In a past post I described how to install a FreeBSD remotely via a tiny UFS based disk image over a linux system. In this post I describe how to do it with a ZFS based disk image.

Invariants

Given a unix based remote system (in this case a linux system) from which you know what kind of hardware it runs on (e.g. PCI IDs) and what the corresponding FreeBSD drivers are.

HOWTO

In the title of this post I wrote “via a tiny disk im­age”. This is true for a suit­able defin­i­tion of tiny.

What we have in the root­server are two ~900 GB hard­disks. They shall be used in a soft­ware mir­ror. The ma­chine has 8 GB of RAM. I do not ex­pect much ker­nel pan­ics (= crash dumps) there, so we do not really need >8 GB of swap (for­get the rule of hav­ing twice as much swap than RAM, with the cur­rent amount of RAM in a ma­chine you are in “trouble” when you need even the same amount of swap than RAM). I de­cided to go with 2 GB of swap.

Pushing/pulling a 900 GB im­age over the net­work to in­stall a sys­tem is not really some­thing I want to do. I am OK to transfer 5 GB (that is 0.5% of the en­tire disk) to get this job done, and this is feas­ible.

First let us define some vari­ables in the shell, this way you just need to change the val­ues in one place and the rest is copy&paste (I use the SVN revision of the source which I use to install the system as the name of the sysutils/beadm compatible boot-dataset in the rootfs, as such I also have the revision number available in a variable):

ROOTFS_SIZE=5G
ROOTFS_NAME=root
FILENAME=rootfs
POOLNAME=mpool
VERSION=r$(cd /usr/src; svnliteversion)
SWAPSIZE=2G

Then change your cur­rent dir­ect­ory to a place where you have enough space for the im­age. There we will cre­ate a con­tainer for the im­age, and make it ready for par­ti­tion­ing:

truncate –s ${ROOTFS_SIZE} ${FILENAME}
mdconfig –a –t vnode –f ${FILENAME}
# if you want to fully allocate
# dd if=/dev/zero of=/dev/md0 bs=1m

Cre­ate the partition table and the rootfs (in a sysutils/beadm compatible way — as I install FreeBSD-current there — and mount it temporary to /temppool):

gpart create –s GPT /dev/md0
gpart add –s 512K –t freebsd-boot –l bootcode0 /dev/md0
gpart add –a 4k –t freebsd-swap –s ${SWAPSIZE} –l swap0 /dev/md0
gpart add –a 1m –t freebsd-zfs –l ${POOLNAME}0 /dev/md0
gpart bootcode –b /boot/pmbr –p /boot/gptzfsboot –i 1 /dev/md0

# if not already the case and you want to have 4k physical sector size of the pool
# syscl vfs.zfs.min_auto_ashift=12
zpool create –o cachefile=/boot/zfs/zpool.cache_temp –o altroot=/temppool –O compress=lz4 –O atime=off –O utf8only=on ${POOLNAME} /dev/gpt/${POOLNAME}0
zfs create –o mountpoint=none ${POOLNAME}/ROOT
zfs create –o mountpoint=/ ${POOLNAME}/ROOT/${VERSION}
zfs create –o mountpoint=/tmp –o exec=on –o setuid=off ${POOLNAME}/tmp
zfs create –o mountpoint=/usr –o canmount=off ${POOLNAME}/usr
zfs create –o mountpoint=/home ${POOLNAME}/home
zfs create –o setuid=off ${POOLNAME}/usr/ports
zfs create ${POOLNAME}/usr/src
zfs create –o mountpoint=/var –o canmount=off ${POOLNAME}/var
zfs create –o exec=off –o setuid=off ${POOLNAME}/var/audit
zfs create –o exec=off –o setuid=off ${POOLNAME}/var/crash
zfs create –o exec=off –o setuid=off ${POOLNAME}/var/log
zfs create –o atime=on ${POOLNAME}/var/mail
zfs create –o setuid=off ${POOLNAME}/var/tmp
zfs create ${POOLNAME}/var/ports
zfs create –o exec=off –o setuid=off –o mountpoint=/shared ${POOLNAME}/shared
zfs create –o exec=off –o setuid=off ${POOLNAME}/shared/distfiles
zfs create –o exec=off –o setuid=off ${POOLNAME}/shared/packages
zfs create –o exec=off –o setuid=off –o compression=lz4 ${POOLNAME}/shared/ccache
zfs create ${POOLNAME}/usr/obj

zpool set bootfs=${POOLNAME}/ROOT/${VERSION} ${POOLNAME}

In­stall FreeBSD (from source):

cd /usr/src
#make buildworld >&! buildworld.log
#make buildkernel –j 8 KERNCONF=GENERIC >&! buildkernel_generic.log

make installworld DESTDIR=/temppool/ >& installworld.log
make distribution DESTDIR=/temppool/ >& distrib.log
make installkernel KERNCONF=GENERIC DESTDIR=/temppool/ >& installkernel.log

Copy the temporary zpool cache created above in the pool-creation part to the image (I have the impression it is not really needed and will work without, but I have not tried this):

cp /boot/zfs/zpool.cache_temp /temppool/boot/
cp /boot/zfs/zpool.cache_temp /temppool/boot/zpool.cache

Add the zfs module to loader.conf:

zfs_load=“yes“
opensolaris_load=“yes”

Now you need to create /temppool/etc/rc.conf (set the de­faultrouter, the IP ad­dress via ifconfig_IF (and do not for­get to use the right IF for it), the host­name, set sshd_enable to yes, zfs_enable=”YES”)  /temppool/boot/loader.conf (zfs_load=”yes”, opensolaris_load=”yes”, vfs.root.mountfrom=”zfs:${POOLNAME}/ROOT/r${VERSION}”)
/temppool/etc/hosts, /temppool/etc/resolv.conf and maybe /temppool/etc/sysctl.conf and /temppool/etc/periodic.conf.

Do not allow password-less root logins in single-user mode on the physical console, create a resolv.conf and an user:

cd /temppool/etc
sed –ie „s:console.*off.:&in:“ ttys
cat >resolv.conf «EOT
search YOURDOMAIN
nameserver 8.8.8.8
EOT

pw –V /temppool/etc groupadd YOURGROUP –g 1001
pw –V /temppool/etc useradd YOURUSER –u 1001 –d /home/YOURUSER –g YOURUSER –G wheel –s /bin/tcsh
pw –V /temppool/etc usermod YOURUSER –h 0
pw –V /temppool/etc usermod root –h 0

zfs create mpool/home/YOURUSER
chown YOURUSER:YOURGROUP /temppool/home/YOURUSER

Now you can make some more modifications to the system if wanted, and then export the pool and detach the image:

zpool export ${POOLNAME}

mdconfig –d –u 0

Depending on the upload speed you can achieve, it is beneficial to compress the image now, e.g. with bzip2. Then transfer the image to the disk of the remote system. In my case I did this via:

ssh –C –o CompressionLevel=9 root@remote_host dd of=/dev/hda bs=1m < /path/to/${FILENAME}

Then reboot/power-cycle the remote system.

Post-install tasks

Now we have a new FreeBSD system which uses only a fraction of the the harddisk and is not resilient against harddisk-failures.

FreeBSD will detect that the disk is bigger than the image we used when creating the GPT label and warn about it (corrupt GPT table). To fix this and to resize the partition for the zpool to use the entire disk we first mirror the zpool to the second disk and resize the partition of the first disk, and when the zpool is in-sync and then we resize the boot disk (attention, you need to change the “-s” part in the following to match your disk size).

First backup the label of the first disk, this makes it more easy to create the label of the second disk:

/sbin/gpart backup ada0 > ada0.gpart

Edit ada0.gpart (give different names for the labels, mainly change the number 0 on the label-name to 1) and then use it to create the partition of the second disk:

gpart restore –Fl ada1 < ada0.gpart
gpart resize –i 3 –a 4k –s 929g ada1
gpart bootcode –b /boot/pmbr –p /boot/gptzfsboot –i 1 ada1
zpool set autoexpand=on mpool

Fix the warning about the GPT label and resize the partition:

gpart recover ada0
gpart resize –i 3 –a 4k –s 929g ada0

Afterwards it should look similar to this:

gpart show –l
=>        40  1953525088  ada0  GPT  (932G)
          40        1024     1  bootcode0  (512K)
        1064     4194304     2  swap0  (2.0G)
     4195368         984        – free –  (492K)
     4196352  1948254208     3  mpool0  (929G)
  1952450560     1074568        – free –  (525M)

=>        40  1953525088  ada1  GPT  (932G)
          40        1024     1  bootcode1  (512K)
        1064     4194304     2  swap1  (2.0G)
     4195368         984        – free –  (492K)
     4196352  1948254208     3  mpool1  (929G)
  1952450560     1074568        – free –  (525M)

Add the second disk to the zpool:

zpool attach mpool gpt/mpool0 gpt/mpool1

When the mirror is in sync (zpool status mpool), we can extend the size of the pool itself:

zpool offline mpool /dev/gpt/mpool0
zpool online mpool /dev/gpt/mpool0

As a last step we can add now an encrypted swap (depending on the importance of the system maybe a gmirror-ed one — not explained here), and specify where to dump (text-dumps) on.

/boot/loader.conf:

dumpdev=”/dev/ada0p2”

/etc/rc.conf:

dumpdev=”/dev/gpt/swap0“
crashinfo_enable=“YES“
ddb_enable=“yes“
encswap_enable=“YES“
geli_swap_flags=”-a hmac/sha256 –l 256 –s 4096 –d”

/etc/fstab:

# Device        Mountpoint      FStype  Options                 Dump    Pass#
/dev/ada1p2.eli none    swap    sw      0       0

Now the system is ready for some applications.

Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Image may be NSFW.
Clik here to view.Share/Save