Don's use ZSYS

[fubar-1]

This is more a statement / opinion / warning than a bug, I just want to leave an artifact here to help others understand the situation around Ubuntu's ZSYS subsystem and a little rant on zfs-on-linux in general.

First, I want to say I really LIKE the ideas behind zsys. From a high-level perspective, it's a major missing component for what's needed to make linux more resilient to software failures like failed system updates, leveraging the power of zfs snapshots while not unduly burdening the user with management tasks. Further, the clear thought and good engineering work that went into zsys is impressive.

However, I've decided to remove zsys and search for another or roll my own solution for a number of reasons:

1. Apparent abandonment by Canonical. No updates beyond bare compatibility with Ubuntu 22.04.
2. Lack of support, here or elsewhere, either by Canonical or the community. It wasn't even installed by default on a new 22.04 system.
3. Bugs. Bug USERDATA datasets removed #218, etc really concern me. Putting a system at risk of unbootable self-destruciton should be considered a CRITICAL bug and should either be fixed or the package should be removed. And I can't even find guidance for how to safely remove system states with confidence of not encountering USERDATA datasets removed #218 using zsys gc or user-error attempting a home-rolled solution.

As a zfs-on-linux early-adopter/enthusiast who's experienced a boot failure first-hand need to say for the record that ZoL system recovery is ugly at best. About a year ago I experienced a failure (due to grub customizer combined with hubris from a false sense of security based on the promise of zfs & zsys). Complexity of the implementation combined with lack of support by the zfs-on-linux community "what? documentation? we just figure it out. conflicts with selinux? ah, well..." led me, after about a week of lost time struggling to recover the system, to just reinstall from scratch and recover my data filesets from a backup. It was a painful but educational experience.

Finally I just want to say I regret having to say this. ZFS and ZSYS are both great technologies. They're ideal solutions to big problems and really move the state of the art forward. Unfortunately, their implementations are both 90% ideal but 10% incomplete. And when it comes to system/data integrity that last 10% is to critical to avoid risk. Sadly it appears that @canonical has abandoned (or at a minimum depreciated) all the the hard work they put into supporting zfs. It's missed the Ubuntu 22.04 LTS bandwagon. Hopefully they or some other distro will take up the effort and complete that last 10%. I switched to Ubuntu for it's zfs support and I'll strongly consider switching to another distro if/when it better supports zfs-on-linux.

@didrocks is an excellent & dedicated developer, I hope he's given the time to get zsys over the finish line.

(Feel free to close this thread if it's felt to not be helpful.)

[Lockszmith-GH]

Just adding my point-of-view.

I have been following this project avidly for a while, and I agree with most of the points you make - I'm not sure about your statement of 90/10% regarding ZFS - I think it is rather complete, but I do agree ZSYS needs more attention and the job isn't fully finished and the 90/10 assessment is probably true.

Let's remember that ZFS is mature, and should be much more widespread - even with its high RAM requirements (I mean BTRFS still has issues, Bcachefs is still in its infancy - so there no better CoW solutions for Linux right now)

You correctly stated, this is NOT due to neglect on the developer part @didrocks (which BTW still rocks! - pun intended, none the less, still true) - and canonical is abandoning the headway they made with making Linux adoption of ZFS easy to consume.

As for the title 'Don't use ZSYS' - from my testing, I think the statement should be 'Only install ZSYS with 20.04 LTS (focal fossa)'.
My own experience shows that the 20.04 LTS installer is solid, and upgrading afterwards works very well and provides all the necessary bug fixes for a stable-home installation.

Should this be used in a business-critical setting - right now the answer would be a resounding NO! - too risky.

[emanuc]

Let's remember that ZFS is mature, and should be much more widespread - even with its high RAM requirements (I mean BTRFS still has issues, Bcachefs is still in its infancy - so there no better CoW solutions for Linux right now)

Excuse me, I disagree here.
Opensuse has been using it for many years without problems, and enables automatic snapshots with Snapper and Boot Snapshot.
Fedora from version 33 uses it by default, without problems.
Valve with Steamos 3.0, on Steam Deck he uses it as roots FS
There are problems on areas where a desktop user is not interested, but these problems are in progress to be solved:

• Extent tree v2 btrfs/btrfs-todo#25
• https://lore.kernel.org/linux-btrfs/YXGyq+buM79A1S0L@relinquished.localdomain/T/
• https://lore.kernel.org/linux-btrfs/cover.1670422590.git.johannes.thumshirn@wdc.com/
• https://patchwork.kernel.org/project/linux-btrfs/patch/beae638a06ad08d16a4eaf799b0aaf01fc1d5573.1652612110.git.wqu@suse.com/

Ubuntu could use Btrfs as the root filesystem..
My two cents. Sorry for the noise, good job to the developers of Ubuntu.

[Venomtek]

I have been using zsys on ubuntu since they offered it in EXPERIMENTAL form. I use it on my virt server and on my urbackup server. I have at least a few hundred datasets on my backup server.

The only problem I've encountered so far is the bpool full issue, which I wrote a script for.
#155 (comment)
There is also a zsys conf file fix if I recall, that lets gc behave a little better.

Not sure what everyone else is doing. I suppose if a person doesn't understand the underlying zfs system then they may have issues, If you are unwilling to fix your own problems may I suggest you don't use EXPERIMENTAL scenarios.

I have had data loss on btrfs and will never use it for anything, having never lost a thing on zfs yet in the 10 years I've been using it. I don't understand why the conversation always ends with "use btrfs", especially in a ZFS forum. These people are brainwashed or something.

I also don't understand why people are whining and crying about an experimental program

I have had much usefulness of this program even with it's quirks, and saves me time with my own scripting.
Thanks to everyone who had a part in it.

[tilgovi]

Zsys has saved me so many times I have lost count. I might have switched to another distro by now were it not for zsys.

[fpga-guy]

I installed Ubuntu 22.04 on a gaming system I inherited from my son and decided to go with ZFS out of interest in having an encrypted filesystem. I had no idea it was "experimental" when the installer gave me the choice. Given recent issues with grub interacting poorly with ZFS (the 10_linux_zfs script is bonkers) I'd say the experience hasn't been pleasant, but I really don't have any other reference points.

Rsync.net uses it to host their cloud service in an industrial, at-scale, environment. That sounds like pretty solid technology to me.

However, if Canonical isn't fully 100% supporting ZFS going forward then I'm going to be upset.

[46cv8]

Does anyone know the state of zsys in ubuntu 24.04? It seems like it isn't installed by default, but will it continue to work if upgraded from 20.04 / 22.04? I don't want to upgrade my system until I know whether zsys continues to function ok. That or I want to know how to purge zsys from my system. I can't find any instructions on how to properly remove zsys without risking being unable to boot. The follow reddit post and gist was the closest I found to uninstall instructions. https://old.reddit.com/r/Ubuntu/comments/um7h6i/zfs_on_root_with_2204_without_zsys/ https://gist.github.com/kimono-koans/7641ee7bc7ce91f520b324bb08d699c8

[Venomtek]

Does anyone know the state of zsys in ubuntu 24.04? It seems like it isn't installed by default, but will it continue to work if upgraded from 20.04 / 22.04? I don't want to upgrade my system until I know whether zsys continues to function ok. That or I want to know how to purge zsys from my system. I can't find any instructions on how to properly remove zsys without risking being unable to boot. The follow reddit post and gist was the closest I found to uninstall instructions. https://old.reddit.com/r/Ubuntu/comments/um7h6i/zfs_on_root_with_2204_without_zsys/ https://gist.github.com/kimono-koans/7641ee7bc7ce91f520b324bb08d699c8

If you don't know how to use ZSYS then I don't suggest you install it. It has nothing to do with your booting ZFS on root, it's just a auto snapshot program. You can uninstall ZSYS then use ZFS utils to remove the snapshots. There are other auto-snapshot tools available on github that are actively being maintained. I suggest you look there. Also they claim to be using the latest kernel from now on, yet openzfs does not always keep up with the latest kernel so who knows the future of ZFS on root for Ubuntu. I've since moved on myself and now use Arch, switching between latest and LTS kernels depending on what state openzfs is in. I got tired of their released upgrades borking my entire system even with ZSYS snapshots. I've also tossed GRUB away and instead use ZFSBootMenu which is way better than GRUB. If you use ZFS on root for any distro, stay away from GRUB.