Use of ZFS with PMS is entirely optional. However, it does have some pedigree in keeping data safe and provides features for remote replication that prove useful when backing up. This article will cover the basics of ZFS for use with PMS.
Use of ZFS with PMS is entirely optional.
See combining ZFS with MergerFS for full information on using ZFS alongside other filesystems thus avoiding "the hidden cost of ZFS"1.
What is ZFS?¶
ZFS merges the traditional volume management and filesystem layers, and it uses a copy-on-write transactional mechanism—both of these mean the system is very structurally different than conventional filesystems and RAID arrays.
Jim Salter wrote a ZFS 101 piece for Ars Technica. It goes into excruciating detail but is a fantastic primer if you're curious to learn more.
Simply put, ZFS is the gold standard for storing data. You must pay attention when creating a ZFS vdev due to the lack of flexibility on offer1. This is due to change with the eventual inclusion of RAIDZ expansion but at the time of writing this feature is not available.
Why use ZFS for PMS?¶
PMS is used to store all sorts of data ranging from easily replacable files to absolutely irreplaceable data such as photos, documents and drone footage. These files were created over a lifetime and they should outlast me. ZFS has checksumming2 3 to detect data corruption during reads, automatic corruption repair and replication corruption protection.
ZFS will do a lot to try and protect my data as long as I keep the hard drives healthy.
Container appdata is a fantastic use case for ZFS in a PMS system. ZFS datasets provide a simple way to version the configuration and persistent data for containers. Create a dataset per container and before making any drastic changes take a snapshot. If you screw something up, revert the snapshot.
What about the license?¶
It is true, the ZFS and Linux licensing schemes are famously incompatible.
And honestly, there is no way I can merge any of the ZFS efforts until I get an official letter from Oracle that is signed by their main legal counsel or preferably by Larry Ellison himself that says that yes, it's ok to do so and treat the end result as GPL'd.
Other people think it can be ok to merge ZFS code into the kernel and that the module interface makes it ok, and that's their decision. But considering Oracle's litigious nature, and the questions over licensing, there's no way I can feel safe in ever doing so.
And I'm not at all interested in some "ZFS shim layer" thing either that some people seem to think would isolate the two projects. That adds no value to our side, and given Oracle's interface copyright suits (see Java), I don't think it's any real licensing win either.
Don't use ZFS. It's that simple. It was always more of a buzzword than anything else, I feel, and the licensing issues just make it a non-starter for me.
- Linus Torvalds 5
Despite this, Ubuntu has been shipping ZFS baked in since 2016 and the world has not ended. Efforts by Canonical to improve ZFS support are on-going as they recently added experimental support for ZFS on root. My experience with ZFS on Ubuntu has been solid - it just works.
The fact remains that ZFS is technically an out of tree module and either requires running a distro like Ubuntu willing to incur the legal wrath of Oracle or for the user to use DKMS4. DKMS is unfortunately not super reliable in my experience and I avoid it wherever possible.
Ultimately whether or not this "risk" fits within your personal values is up to you.
What about BTRFS?¶
Fedora recently adopted6 BTRFS as the default so it must be ready for primetime right?
The Btrfs community has users that have been using it for most of the past decade at scale. It's been the default on openSUSE (and SUSE Linux Enterprise) since 2014, and Facebook has been using it for all their OS and data volumes, in their data centers, for almost as long. Btrfs is a mature, well-understood, and battle-tested file system, used on both desktop/container and server/cloud use-cases. We do have developers of the Btrfs filesystem maintaining and supporting the code in Fedora, one is a Change owner, so issues that are pinned to Btrfs can be addressed quickly.
Using BTRFS would be easier than ZFS simply because it is shipped as part of the Linux kernel.
Perhaps one day PMS will switch out to BTRFS but that day is a ways off yet - ZFS is here to stay for now if for no other reason than I have replicated multiple TBs of data reliably for a year+ with no data loss whatsoever. In otherwords, I'm locked in.
Creating a storage pool¶
I originally wrote an article on this topic for the LinuxServer.io blog here.
This article assumes you are using Ubuntu. We'll be creating a mirrored pair of drives for use with ZFS here. For reasons why you should use mirrors see Jim's blog.
- Begin by installing the userspace tools for ZFS:
- Create a mirrored pair using the drive
- Make sure to set
ashiftcorrectly as below
- Make sure to set
/dev/sdX for these commands, always use a unique identifier that will transpose between systems or reboots without changing such as
/dev/disk/by-id/. See the OpenZFS docs for more information.
- Once created verify the new pool with
1 2 3 4 5 6 7 8 9 10 11 12 13
- Create some datasets - do not store any data in the root of the pool
1 2 3 4 5 6 7
- Take a snapshot of the new dataset with
1 2 3 4
- View snapshots with
zfs list -t snapshot
1 2 3 4 5 6 7
- Finally ensure the dataset mountpoint is set as you require
Jim Salter's blog at jrs-s.net7 has a number of excellent posts about ZFS including some about tuning in addition to his Ars Technica article 8.
You must make sure you set
ashift correctly. Disks often lie about their sector size and if you ignore this setting it can drastically degrade performance. Most large drives have 4k sectors so an
ashift=12 is usually fine. Some Samsung SSD have 8k sectors where
ashift=13 would be required.
Ashift is per-vdev and immutable once set. It cannot be set at any level below the vdev.— Jim Salter (@jrssnet) May 1, 2019
Scrubs are an important safety check to prevent bitrot. Ubuntu automatically schedules these for you.
New blog post about my new adventures with ZFS.https://t.co/gmGESknfL1— Alex Kretzschmar (@IronicBadger) May 14, 2019
Thanks @allanjude and @jrssnet for the drive to actually, finally do this.
View the status of a pools scrubs with
1 2 3 4 5 6 7 8 9 10 11 12 13
One of the most compelling reasons to use ZFS is replication. We get more into the weeds of backups in Day Two -> Backups but this section will focus on the ZFS specific
zfs send/receive functionality using Sanoid and Syncoid.
Replication requires a remote system also running ZFS. I built a small system at a parent's house under the stairs to perform this function.
rsync.net is expensive but provides this functionality 'in the cloud'.