Moving to ZFS with NixOS

My journey on migrating from EXT4 to ZFS with my primary NixOS install. 02 Dec 2020

After using EXT4 as my primary filesystem on my workstation for years I decided to try to something else. After looking for alternatives I landed on giving ZFS a try.

Why ZFS

ZFS (Zettabyte filesystem) is a filesystem with built-in volume management and originated from the Solaris operating system. It has many features1 including but not limited to: Copy-on-write model, snapshots, encryption, software RAID and multi level caching.

Personally my biggest reason for choosing ZFS its maturity and proven track record.

Structure

In ZFS the highest element in the hierarchy is zpools. A zpool is one or multiple vdevs (disks) which make out a pool of datasets, it allows for configuring parameters like redundancy and more.

A ZFS dataset is a volume in a dataset and does not have a fixed size by default. It will continue to expand in the pool until it hits the pool limit or a set quota.

ZFS snapshots can be taken on either the pool or dataset level and replicated to other ZFS powered machines.

Partitioning

After quickly backing up all user data I wanted to keep, I decided to make two zpools for my system. One for my NVMe SSD and one for my spinning disks that can have some redundancy.

Pools

To create a zpool there is the zpool create command. You can put the ZFS vdev on either a partition or entire disk (preferred).

NOTE: -O xattr=sa -O acltype=posixacl can be added when creating pool to make POSIX ACLs work and store attributions in inodes. This might increase performance and fix things that would otherwise break without POSIX ACLs2.

zpool create -O mountpoint=none -R /mnt rpool [disk/diskpartition]
zpool create -O mountpoint=none -R /mnt tank mirror [disk/diskpartition] [2nd disk/diskpartition]

I decided to namespace my datasets with system/ and user/ by creating empty datasets with no mountpoint.

zfs create -o mountpoint=none rpool/system
zfs create -o mountpoint=none rpool/user
zfs create -o mountpoint=none tank/user

Datasets

Now it is time to create the datasets that will hold all my data. I decided to use lz4 compression on most of my datasets for now. Legacy mountpoint means that I will have to manually mount all the datasets myself.

zfs create -o mountpoint=legacy rpool/system/root
zfs create -o mountpoint=legacy -o compression=lz4 rpool/system/nix
zfs create -o mountpoint=legacy -o compression=lz4 rpool/user/home

zfs create -o mountpoint=legacy -o compression=lz4 tank/user/archive
zfs create -o mountpoint=legacy -o compression=lz4 tank/user/games
zfs create -o mountpoint=legacy tank/user/media
zfs create -o mountpoint=legacy tank/user/vms

In the end my ZFS dataset list looked like this:

$ zfs list -o name
NAME
rpool
rpool/system
rpool/system/nix
rpool/system/root
rpool/user
rpool/user/home
tank
tank/user
tank/user/archive
tank/user/games
tank/user/media
tank/user/vms

Mounting

The last step is to mount all the datasets in my NixOS config which is just including:

fileSystems."/" =
  {
    device = "rpool/system/root";
    fsType = "zfs";
  };

fileSystems."/nix" =
  {
    device = "rpool/system/nix";
    fsType = "zfs";
  };

fileSystems."/home" =
  {
    device = "rpool/user/home";
    fsType = "zfs";
  };

fileSystems."/mnt/archive" =
  {
    device = "tank/user/archive";
    fsType = "zfs";
  };

...

Verdict

So far everything has been running stable and I have been impressed of how little performance regression I noticed. I have only a noticed seconds difference in compile time on fairly large codebases. Some larger applications starts up a bit slower than it used to but not in an annoying way.

Also, the compression ratio has been impressive, almost saving half of the space for /nix.

NAME               PROPERTY       VALUE  SOURCE
rpool/system/nix   compressratio  1.84x  -
rpool/user/home    compressratio  1.29x  -

The next step is looking into snapshots, replication and performance tweaking. Also looking into automatically creating datasets for users instead of sharing the /home dataset. But automating this one is fairly tricky.

In the end I believe I have found a superior filesystem that I can use for the next few years until a better alternative shows up.


Footnotes

  1. For the complete list, check out the Wikipedia article.

  2. systemd