I recently got a Dedibox from Online.net in order to move my Nextcloud instance over (need that storage!).
Since I’m really interesting in FreeBSD these days, I decided to use it on this new server along with ZFS.
FreeBSD 12.0-RELEASE only came out a few days ago so there is no installation option available on the online.net console yet. However, FreeBSD 11.1-RELEASE is available. The thing, it only allows to use UFS. I don’t like proprietary installers anyway, so I decided that I’d install FreeBSD manually on this box. Plus, it really feels great being in total control of my machine, in the sense that I know how it is set up.
Installing FreeBSD is surprisingly easy as it just involves partitioning the disk and extracting some archives.
First, boot in rescue mode with FreeBSD 11.0:
Login via SSH then escalate as root (sudo su -l)
My installation process was largely inspired from this great tutorial in the FreeBSD forums.
Since I did a lot of trials and fails, I tested a few different setups, which I describe below. Feel free to adapt them to your needs!
The 4 setups are:
- I - Full ZFS (with swap on the zpool)
- II - Full UFS with (encrypted) swap
- III - UFS + (encrypted) swap + encrypted ZFS partition
- IV - ZFS root partition + (encrypted) swap + encrypted ZFS partition with geli
My goal was to use ZFS and store my Nextcloud data on an encrypted ZFS dataset, so my final setup is IV.
Other setups can easily be adapted from the ones above, for instance using UFS on the encrypted partition.
This setup assumes your NIC is igb0 and uses DHCP, and your hard drive is ada0. My current use case is an online.net Dedibox, but it can easily be used for any kind of machine with little changes. It should work for other FreeBSD versions.
I - Full ZFS
In this setup I use a single zpool for the root partition and swap.
1. Partitioning
# Destroy the partitioning scheme if needed
#gpart destroy -F ada0
gpart create -s gpt ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart add -t freebsd-boot -s 128k -l boot ada0
gpart add -t freebsd-zfs -l zroot ada0
2. ZFS setup:
Create a zpool, set it as the boot filesystem and create a 2G dataset for the swap:
zpool create -m / -R /mnt zroot /dev/gpt/zroot
zpool set bootfs=zroot zroot
zfs create -V 2G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
3. System installation
Basically we download and extract archives containing the base system and the kernel into the future root filesystem.
cd /tmp/
fetch http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/base.txz
fetch http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/kernel.txz
cd /mnt
tar xvJf /tmp/base.txz
tar xvJf /tmp/kernel.txz
rm /tmp/*.txz
4 - Basic system configuration:
We need to enable ZFS at boot:
echo 'zfs_load="YES"\
vfs.root.mountfrom="zfs:zroot"' >> boot/loader.conf
Basic setup of rc.conf:
echo 'zfs_enable="YES"\
hostname="fdb"\
ifconfig_igb0="DHCP"\
sshd_enable="YES"' > etc/rc.conf
5 - Users setup
In order to connect via SSH afterwards, you can set a root password and a user in a chroot:
chroot /mnt
passwd
adduser
II - Full UFS with (encrypted) swap
In this setup we’ll use UFS with a separate swap partition, which we can auto-encrypt easily.
1. Partitioning
# Destroy the partitioning scheme if needed
#gpart destroy -F ada0
gpart create -s gpt da0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da0
# The system partition will use the rest of the space
gpart add -t freebsd-boot -s 128k -l boot ada0
gpart add -t freebsd-swap -s 2g -l swap ada0
gpart add -t freebsd-ufs -l system ada0
# Create a UFS filesystem and mout it
newfs /dev/gpt/system
mount /dev/gpt/system /mnt
2. System installation
Basically we download and extract archives containing the base system and the kernel into the future root filesystem.
cd /tmp/
fetch http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/base.txz
fetch http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/kernel.txz
cd /mnt
tar xvJf /tmp/base.txz
tar xvJf /tmp/kernel.txz
rm /tmp/*.txz
3. Basic system configuration
fstab setup:
echo "/dev/gpt/system / ufs rw,noatime 1 1\
/dev/gpt/swap.eli none swap sw 0 0" > etc/fstab
Basic rc.conf setup:
echo 'hostname="fdb"\
ifconfig_igb0="DHCP"\
sshd_enable="YES"' > etc/rc.conf
The swap encryption is automatically encrypted with a random key (at each boot) using geli when adding .eli to the name of the device. See https://www.freebsd.org/doc/handbook/swap-encrypting.html.
4. Users setup
In order to connect via SSH afterwards, you can set a root password and a user in a chroot:
chroot /mnt
passwd
adduser
III - UFS + encrypted swap + encrypted ZFS partition
This setup is useful if you want to keep your root partition on UFS but want to store sensitive data on an encrypted ZFS pool. ZFS native encryption is in the works but not available yet so we’ll use geli and create a zpool on top of it.
1. Partitioning
# Destroy the partitioning scheme if needed
#gpart destroy -F ada0
gpart create -s gpt da0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da0
# 50 GB for the system, 2G for swap and the rest for the zpool
gpart add -t freebsd-boot -s 128k -l boot ada0
gpart add -t freebsd-ufs -s 50g -l system ada0
gpart add -t freebsd-swap -s 2g -l swap ada0
gpart add -t freebsd-zfs -l zdata ada0
# Create a UFS filesystem and mount it
newfs /dev/gpt/system
mount /dev/gpt/system /mnt
2. System installation
Basically we download and extract archives containing the base system and the kernel into the future root filesystem.
cd /tmp/
fetch http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/base.txz
fetch http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/kernel.txz
cd /mnt
tar xvJf /tmp/base.txz
tar xvJf /tmp/kernel.txz
rm /tmp/*.txz
3. Basic system configuration
fstab setup:
echo "/dev/gpt/system / ufs rw,noatime 1 1\
/dev/gpt/swap.eli none swap sw 0 0" > etc/fstab
The swap encryption is automatically encrypted with a random key (at each boot) using geli when adding .eli to the name of the device. See https://www.freebsd.org/doc/handbook/swap-encrypting.html.
We don’t need to specify anything for the encrypted zpool since we’ll mount it manually as it requires a password.
Basic rc.conf setup:
echo 'hostname="fdb"\
ifconfig_igb0="DHCP"\
sshd_enable="YES"' > etc/rc.conf
4. Users setup
In order to connect via SSH afterwards, you can set a root password and a user in a chroot:
chroot /mnt
passwd
adduser
5. Encrypted ZFS pool setup
After a reboot, let’s create and attach our encrypted device using geli(8).
geli init -e AES-XTS -l 256 /dev/gpt/zdata
geli attach /dev/gpt/zdata
You’ll be asked for a password at each step.
Then, let’s create a zpool on it:
zpool create -m /data zdata /dev/gpt/zdata.eli
You can check that everything went well with geli status and zpool status.
After a reboot, you can attach the device:
geli attach /dev/gpt/zdata
Then let ZFS do the rest for you:
zfs mount -a
The zpool will be automatically mounted.
IV - ZFS root partition + encrypted swap + encrypted ZFS partition with geli
This is the setup I’m currently using.
1. Partitioning
# Destroy the partitioning scheme if needed
#gpart destroy -F ada0
gpart create -s gpt da0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da0
# 50 GB for the system, 2G for swap and the rest for zdata
gpart add -t freebsd-boot -s 128k -l boot ada0
gpart add -t freebsd-zfs -s 50g -l system ada0
gpart add -t freebsd-swap -s 2g -l swap ada0
gpart add -t freebsd-zfs -l zdata ada0
2. ZFS setup
zpool create -m / -R /mnt zroot /dev/gpt/zroot
zpool set bootfs=zroot zroot
3. System installation
Basically we download and extract archives containing the base system and the kernel into the future root filesystem.
cd /tmp/
fetch http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/base.txz
fetch http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/kernel.txz
cd /mnt
tar xvJf /tmp/base.txz
tar xvJf /tmp/kernel.txz
rm /tmp/*.txz
4. Basic system configuration
fstab setup:
echo "/dev/gpt/system / ufs rw,noatime 1 1\
/dev/gpt/swap.eli none swap sw 0 0" > etc/fstab
The swap encryption is automatically encrypted with a random key (at each boot) using geli when adding .eli to the name of the device. See https://www.freebsd.org/doc/handbook/swap-encrypting.html.
We need to enable ZFS at boot time:
echo 'zfs_load="YES"\
vfs.root.mountfrom="zfs:zroot"' >> boot/loader.conf
Basic rc.conf setup:
echo 'zfs_enable="YES"\
hostname="fdb"\
ifconfig_igb0="DHCP"\
sshd_enable="YES"' > etc/rc.conf
5. Users setup
In order to connect via SSH afterwards, you can set a root password and a user in a chroot:
chroot /mnt
passwd
adduser
6. Encrypted ZFS pool setup
After a reboot, let’s create and attach our encrypted device using geli(8).
geli init -e AES-XTS -l 256 /dev/gpt/zdata
geli attach /dev/gpt/zdata
You’ll be asked for a password at each step.
Then, let’s create a zpool on it:
zpool create -m /data zdata /dev/gpt/zdata.eli
You can check that everything went well with geli status and zpool status.
After a reboot, you can attach the device:
geli attach /dev/gpt/zdata
Then let ZFS do the rest for you:
zfs mount -a
The zpool will be automatically mounted.
Enable AES-NI
Since we’re using disk encryption, we want to make it as fast as possible.
A way do that is to enable the accelerated AES instructions.
See aesni(5). Add this line to your /boot/loader.conf:
aesni_load="YES"
That’s it. I hope it will be useful to other people, at leat it will be for me when I’ll have to reinstall the server. 🙂