zfs-2.2.0 building native-deb-utils target fails early on #15404

thisisnotmyrealname · 2023-10-14T13:18:56Z

I don't file many bug reports. I did my best to fill out the template and provide useful information. I have no fixes for the issue so far. Thank you.

There was a lot more dh_missing errors, however I hit the max character limit in a bug report, which is fair.

System information

zfs-2.1.12-1
zfs-kmod-2.1.12-1
6.2.0-32-generic

Describe the problem you're observing

building native-deb or native-deb-utils fails.
I thought this would work per: https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html#build-options
Support for native Debian packaging will be available starting from openzfs-2.2 release.

Describe how to reproduce the problem

extract fresh 2.2.0 source tree

./configure --enable-systemd
make native-deb
 cp -r contrib/debian debian; chmod +x debian/rules;
 cp contrib/debian/control debian/control; \
 dpkg-buildpackage -b -rfakeroot -us -uc;
 cp: cannot stat 'contrib/debian/control': No such file or directory
 dpkg-buildpackage: error: cannot open file debian/changelog: No such file or directory
 make: *** [Makefile:14317: native-deb-utils] Error 255

If I copy over contrib/debian/control* and changelog from git version 2.2.0 into the 2.2.0 release directory, it builds a bit longer, but then seems to fail when it gets to

make[2]: Entering directory '/home/user/test/zfs-2.2.0'
dh_missing --fail-missing
dh_missing: warning: etc/default/zfs exists in debian/tmp but is not installed to anywhere
dh_missing: warning: etc/init.d/zfs-import exists in debian/tmp but is not installed to anywhere
dh_missing: warning: lib/udev/rules.d/90-zfs.rules exists in debian/tmp but is not installed to anywhere
dh_missing: warning: lib/udev/vdev_id exists in debian/tmp but is not installed to anywhere
dh_missing: warning: lib/udev/zvol_id exists in debian/tmp but is not installed to anywhere

dh_missing: warning: usr/src/zfs-2.2.0/module/zstd/lib/decompress/zstd_decompress_block.c exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/module/zstd/lib/decompress/zstd_decompress_block.h exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/module/zstd/lib/decompress/zstd_decompress_internal.h exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/module/zstd/lib/zstd.h exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/module/zstd/zfs_zstd.c exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/module/zstd/zstd-in.c exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/module/zstd/zstd_sparc.c exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/scripts/dkms.postbuild exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/scripts/enum-extract.pl exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/zfs.release.in exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/src/zfs-2.2.0/zfs_config.h.in exists in debian/tmp but is not installed to anywhere
        The following debhelper tools have reported what they installed (with files per package)
         * dh_install: openzfs-libnvpair3 (0), openzfs-libpam-zfs (0), openzfs-libuutil3 (0), openzfs-libzfs-dev (0), openzfs-libzfs4 (0), openzfs-libzfsbootenv1 (0), openzfs-libzpool5 (0), openzfs-python3-pyzfs (0), openzfs-pyzfs-doc (0), openzfs-zfs-dkms (0), openzfs-zfs-dracut (0), openzfs-zfs-initramfs (0), openzfs-zfs-test (0), openzfs-zfs-zed (0), openzfs-zfsutils (0)
         * dh_installdocs: openzfs-libnvpair3 (0), openzfs-libpam-zfs (0), openzfs-libuutil3 (0), openzfs-libzfs-dev (0), openzfs-libzfs4 (0), openzfs-libzfsbootenv1 (0), openzfs-libzpool5 (0), openzfs-python3-pyzfs (0), openzfs-pyzfs-doc (0), openzfs-zfs-dkms (0), openzfs-zfs-dracut (0), openzfs-zfs-initramfs (0), openzfs-zfs-test (0), openzfs-zfs-zed (0), openzfs-zfsutils (0)
        If the missing files are installed by another tool, please file a bug against it.
        When filing the report, if the tool is not part of debhelper itself, please reference the
        "Logging helpers and dh_missing" section from the "PROGRAMMING" guide for debhelper (10.6.3+).
          (in the debhelper package: /usr/share/doc/debhelper/PROGRAMMING.gz)
        Be sure to test with dpkg-buildpackage -A/-B as the results may vary when only a subset is built
        If the omission is intentional or no other helper can take care of this consider adding the
        paths to debian/not-installed.

        Remember to be careful with paths containing "x86_64-linux-gnu", where you might need to
        use a wildcard or (assuming compat 13+) e.g. ${DEB_HOST_MULTIARCH} in debian/not-installed
        to ensure it works on all architectures (see #961104).
dh_missing: error: missing files, aborting
make[2]: *** [debian/rules:160: override_dh_missing] Error 255
make[2]: Leaving directory '/home/user/test/zfs-2.2.0'
make[1]: *** [debian/rules:28: binary] Error 2
make[1]: Leaving directory '/home/user/test/zfs-2.2.0'
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make: *** [Makefile:14317: native-deb-utils] Error 2

Also
make native-deb-kmod fails with

/bin/sh: 1: cannot open debian/control.modules.in: No such file
make[1]: *** [debian/rules:185: override_dh_prep-deb-files] Error 2
make: *** [Makefile:14320: native-deb-kmod] Error 2

If I copy over that file from the 2.2.0 tagged git source, I get

sh scripts/make_gitrev.sh; \
fakeroot debian/rules override_dh_binary-modules;
make[1]: Entering directory '/home/user/test/zfs-2.2.0'
for templ in ; do \
        sed -e 's/##KVERS##/6.2.0-32-generic/g ; s/#KVERS#/6.2.0-32-generic/g ; s/_KVERS_/6.2.0-32-generic/g ; s/##KDREV##//g ; s/#KDREV#//g ; s/_KDREV_//g ; s/_ARCH_/amd64/' \
        < $templ > `echo $templ | sed -e 's/_KVERS_/6.2.0-32-generic/g ; s/_ARCH_/amd64/g ; s/\.in$//'` ; \
done
sed -e 's/##KVERS##/6.2.0-32-generic/g ; s/#KVERS#/6.2.0-32-generic/g ; s/_KVERS_/6.2.0-32-generic/g ; s/##KDREV##//g ; s/#KDREV#//g ; s/_KDREV_//g ; s/_ARCH_/amd64/g' \
< debian/control.modules.in > debian/control
dh override_dh_configure_modules --with autoreconf,dkms,python3,sphinxdoc
dh_testdir
dh_testroot
dh_prep
/usr/bin/make -j32 -C /home/user/test/zfs-2.2.0/module modules
make[2]: Entering directory '/home/user/test/zfs-2.2.0/module'
mkdir -p os/linux/spl/
mkdir -p avl/ icp/ icp/algs/aes/ icp/algs/blake3/ icp/algs/edonr/ icp/algs/modes/ icp/algs/sha2/ icp/algs/skein/ icp/api/ icp/asm-aarch64/blake3/ icp/asm-aarch64/sha2/ icp/asm-arm/sha2/ icp/asm-ppc64/blake3/ icp/asm-ppc64/sha2/ icp/asm-x86_64/aes/ icp/asm-x86_64/blake3/ icp/asm-x86_64/modes/ icp/asm-x86_64/sha2/ icp/core/ icp/io/ icp/spi/ lua/ lua/setjmp/ nvpair/ os/linux/zfs/ unicode/ zcommon/ zfs/ zstd/ zstd/lib/common/ zstd/lib/compress/ zstd/lib/decompress/
/usr/bin/make -C   \
          \
        M="$PWD"  CONFIG_ZFS=m modules
make[3]: Entering directory '/home/user/test/zfs-2.2.0/module'
make[3]: *** M=/home/user/test/zfs-2.2.0/module: No such file or directory.  Stop.
make[3]: Leaving directory '/home/user/test/zfs-2.2.0/module'
make[2]: *** [Makefile:56: modules-Linux] Error 2
make[2]: Leaving directory '/home/user/test/zfs-2.2.0/module'
make[1]: *** [debian/rules:203: override_dh_binary-modules] Error 2
make[1]: Leaving directory '/home/user/test/zfs-2.2.0'
make: *** [Makefile:14320: native-deb-kmod] Error 2

The text was updated successfully, but these errors were encountered:

blind-oracle · 2023-10-15T18:31:45Z

If I copy over that file from the 2.2.0 tagged git source, I get

Since you already checked out zfs-2.2-release branch - just build from there. It works.

# ./autogen.sh
# ./configure
# make native-deb

The release tar is broken, yeah.

usaleem-ix · 2023-10-16T08:22:18Z

It looks like the zfs-2.2.0.tar.gz on the release page only contains rules.in under contrib/debian, other contents are missing. However, the Source code (zip) and Source code (tar.gz) do seem to have all the contents.

I was able to successfully build the Debian packages with Source code (zip) on Debian 12, as mentioned above:

# sh autogen.sh
# ./configure
# make native-deb

mwpastore · 2023-10-16T18:02:46Z

release tar is missing files
git clone instructions should probably also recommend checking out a specific tag (e.g. zfs-2.2-release)
two build dependencies are not listed: libcurl4-openssl-dev libpam0g-dev
the instructions specify to make native-deb-utils but @blind-oracle is giving make native-deb, what's the difference and which one is correct?
I can't install the openzfs-zfs-dkms package on ubuntu server 22.04.3 because it conflicts with the zfs-dkms provided by the kernel:

$ sudo apt install ./openzfs-zfs-zed_2.2.0-0_amd64.deb ./openzfs-zfsutils_2.2.0-0_amd64.deb ./openzfs-zfs-dkms_2.2.0-0_all.deb ./openzfs-libzpool5_2.2.0-0_amd64.deb ./openzfs-libzfs4_2.2.0-0_amd64.deb ./openzfs-libnvpair3_2.2.0-0_amd64.deb ./openzfs-libuutil3_2.2.0-0_amd64.deb 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'openzfs-zfs-zed' instead of './openzfs-zfs-zed_2.2.0-0_amd64.deb'
Note, selecting 'openzfs-zfsutils' instead of './openzfs-zfsutils_2.2.0-0_amd64.deb'
Note, selecting 'openzfs-zfs-dkms' instead of './openzfs-zfs-dkms_2.2.0-0_all.deb'
Note, selecting 'openzfs-libzpool5' instead of './openzfs-libzpool5_2.2.0-0_amd64.deb'
Note, selecting 'openzfs-libzfs4' instead of './openzfs-libzfs4_2.2.0-0_amd64.deb'
Note, selecting 'openzfs-libnvpair3' instead of './openzfs-libnvpair3_2.2.0-0_amd64.deb'
Note, selecting 'openzfs-libuutil3' instead of './openzfs-libuutil3_2.2.0-0_amd64.deb'
The following packages were automatically installed and are no longer required:
  amd64-microcode libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 libusbmuxd6 linux-headers-generic-hwe-22.04 thermald upower usbmuxd
Use 'sudo apt autoremove' to remove them.
Suggested packages:
  debhelper nfs-kernel-server openzfs-zfs-initramfs | openzfs-zfs-dracut
The following packages will be REMOVED:
  linux-generic linux-generic-hwe-20.04 linux-generic-hwe-22.04 linux-image-5.15.0-86-generic linux-image-6.2.0-33-generic linux-image-6.2.0-34-generic linux-image-generic linux-image-generic-hwe-22.04
  linux-modules-5.15.0-86-generic linux-modules-extra-5.15.0-86-generic
The following NEW packages will be installed:
  openzfs-libnvpair3 openzfs-libuutil3 openzfs-libzfs4 openzfs-libzpool5 openzfs-zfs-dkms openzfs-zfs-zed openzfs-zfsutils
0 upgraded, 7 newly installed, 10 to remove and 0 not upgraded.
Need to get 0 B/4,851 kB of archives.
After this operation, 480 MB disk space will be freed.

blind-oracle · 2023-10-16T18:33:54Z

@mwpastore Yes, these native deb packages are in some weird state, probably gonna have to postpone upgrade to 2.2 until those conflicts are fixed...

usaleem-ix · 2023-10-17T06:43:59Z

There are three targets for native packaging, make native-deb, make native-deb-utils and make native-deb-kmod:

make native-deb-utils: Builds the userland utilities and the DKMS package. If kernel modules are to be installed via DKMS package, this target should be sufficient.

make native-deb-kmod: Builds the kernel modules and package the binaries.

make native-deb: Runs native-deb-utils and native-deb-kmod. All userland packages, DKMS package and kernel module package should be built.

There two ways to build the Debian packages. RPM converted packages from Alien can be built by using make deb, and native Debian packages, targets for which are mentioned above.

The conflicts were added intentionally with RPM converted Debian packages and the downstream Debian packages. This was done to make sure, the local native Debian packages cannot co-install with other Debian packages maintained by downstream distributions and RPM converted packages.

To install native Debian packages, please remove the previously installed packages and then install the native Debian packages. Alternatively, RPM converted Debian packages can be installed to upgrade the previously installed ones.

blind-oracle · 2023-10-17T07:08:59Z

To install native Debian packages, please remove the previously installed packages and then install the native Debian packages.

Sadly it does not work like this, at least in Ubuntu - zfs-dkms is provided by the Linux kernel packages and apt refuses to install openzfs-zfs-dkms. Removing kernel, is, obviously a bad idea... :)

Which is actually strange - why kernel should provide DKMS which is purposely created for out-of-kernel modules.

# dpkg -i openzfs-zfs-dkms_2.2.0-0_all.deb 
dpkg: regarding openzfs-zfs-dkms_2.2.0-0_all.deb containing openzfs-zfs-dkms:
 openzfs-zfs-dkms conflicts with zfs-dkms
  linux-image-6.2.0-34-generic provides zfs-dkms and is present and installed.
  linux-image-6.2.0-33-generic provides zfs-dkms and is present and installed.

dpkg: error processing archive openzfs-zfs-dkms_2.2.0-0_all.deb (--install):
 conflicting packages - not installing openzfs-zfs-dkms
Errors were encountered while processing:
 openzfs-zfs-dkms_2.2.0-0_all.deb

AllKind · 2023-10-17T07:41:08Z

@blind-oracle
would apt purge -s zfs-dkms suggest to remove the kernel?

blind-oracle · 2023-10-17T08:06:52Z

@AllKind No it does not. But when you try to install openzfs-zfs-dkms then it would:

# apt install ./openzfs-zfs-dkms_2.2.0-0_all.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'openzfs-zfs-dkms' instead of './openzfs-zfs-dkms_2.2.0-0_all.deb'
The following packages were automatically installed and are no longer required:
  amd64-microcode intel-microcode iucode-tool libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 libusbmuxd6 linux-headers-generic-hwe-22.04 thermald upower usbmuxd
Use 'apt autoremove' to remove them.
Recommended packages:
  openzfs-zfs-zed openzfs-zfsutils
The following packages will be REMOVED:
  linux-generic-hwe-22.04 linux-image-6.2.0-33-generic linux-image-6.2.0-34-generic linux-image-generic-hwe-22.04
The following NEW packages will be installed:
  openzfs-zfs-dkms
0 upgraded, 1 newly installed, 4 to remove and 0 not upgraded.
Need to get 0 B/2557 kB of archives.
After this operation, 7845 kB disk space will be freed.

AllKind · 2023-10-17T08:23:44Z

So it's like usaleem-ix said. You have to remove the old package first.
Not saying I have any idea why apt wants to uninstall the kernel in the first place. Which sounds like something is wrong.

blind-oracle · 2023-10-17T08:26:04Z

So it's like usaleem-ix said. You have to remove the old package first.

No, no old packages are installed, I've purged all 2.1.13 stuff. It just wants to remove the kernel, for sure it's wrong.

AllKind · 2023-10-17T09:12:53Z

dpkg -i openzfs-zfs-dkms_2.2.0-0_all.deb
dpkg: regarding openzfs-zfs-dkms_2.2.0-0_all.deb containing openzfs-zfs-dkms:
openzfs-zfs-dkms conflicts with zfs-dkms

It just looked like you still have zfs-dkms installed.

blind-oracle · 2023-10-17T09:28:53Z

@AllKind zfs-dkms is some virtual thing that the kernel provides, it just happen to have the same name as zfs-2.1.x package and somehow they don't conflict and can be co-installed. But not in 2.2.x

Can be seen here in the Provides section: https://ubuntu.pkgs.org/22.04/ubuntu-updates-main-arm64/linux-image-unsigned-6.2.0-33-generic_6.2.0-33.33~22.04.1_arm64.deb.html

AllKind · 2023-10-17T10:01:49Z

@blind-oracle
I don't think that's true.
Provides does not mean the package must be virtual. It can also be a real package name.

searching for all virtual packages and grepping for zfs:

aptitude search "~v" |grep zfs
v zfs-modules -
v zfsutils -

apt show -a zfs-dkms will show it's a real package and the install status.

rincebrain · 2023-10-17T11:00:38Z

Yeah, the conflicts with the provides is probably not going to work. Hm.

I'm not sure what the right way to resolve this is, tbh - Ubuntu's kernel is probably correct to say it provides zfs-dkms, in some sense. Really, it'd be nice if we could have a provides: zfs-modules and then zfs-dkms also provides that and we just have the Conflicts on zfs-dkms, not zfs-modules, but, I don't think that's necessarily feasible to get everyone to do Right Now.

The simplest way might just be to drop the conflicts for zfs-dkms for now and try to get zfs-dkms and Ubuntu's kernel tree to have a different provides so that we can conflict with zfs-dkms the package later, not the virtual thing it's providing. But I'm not an expert in best practices on this.

blind-oracle · 2023-10-17T11:29:02Z

@AllKind I don't get what you mean. I don't have a real zfs-dkms package installed, but there's still a conflict that makes apt remove the kernel.

@rincebrain Well, I would be happy with how it was working in 2.1.x and before (with alien + rpm conversion). Probably it still works now, but having native debs are probably nicer. There were no conflicts, everything worked like a charm.

Update:
I guess the reason is that 2.2 packages have this in the deb definition. So apt tries to remove anything that provides those.

 Replaces: spl-dkms, zfs-dkms

rincebrain · 2023-10-17T11:57:37Z

I hadn't realized it now forces you to build the native deb packages either way in 2.2. How...well thought out, in literally no way.

blind-oracle · 2023-10-17T12:06:07Z

@rincebrain maybe the old way works (thru alien), I haven't yet tried. But I wanted to have the native ones if they were announced...

AllKind · 2023-10-17T12:19:34Z

@blind-oracle
I was just trying to say, that it's not a virtual package. But I think I just misunderstood, what you were saying with "virtual thing".

I didn't know, that Provides could produce a Conflict. That seems to be the problem.

@rincebrain
What about naming the package names the old way. Without the openzfs- prefix. For me it sounds redundant (and kinda silly). And then just remove zfs-dkms from the Replaces definition?
Wouldn't that make things work like they used to, without the need to change Ubuntus definitions?

AllKind · 2023-10-17T12:24:24Z

@rincebrain maybe the old way works (thru alien), I haven't yet tried. But I wanted to have the native ones if they were announced...

And then you could easily run into #15253 which is likely the same as #15328.
That with the broken tar ball of the 2.2.0 release... Quite a mess installing ZFS right now :-O

rincebrain · 2023-10-17T12:26:56Z

The entire point of the prefix was to ensure you couldn't end up with a mix of the two package sets or have to worry about the versioning causing weird hard to debug problems.

You could just drop the Replaces, but then you could wind up with both module packages installed.

You could, I imagine, try having something janky like a post-install step that checks if zfs-dkms is virtual and if not, removes it, if you still wanted to try enforcing that.

AllKind · 2023-10-17T12:36:15Z

Hm, are you saying apt does not recognize version 2.2 is higher than 2.1?
And even then, isn't that just postponing the same thing just with different names?

rincebrain · 2023-10-17T12:55:18Z

I'm saying that this prefix avoids you winding up with Ubuntu's patched libzfs5-2.2.0 and OpenZFS's vanilla userland, for example.

An additional remark is that, if Ubuntu 24.04, for example, shipped 2.2.0, and you wanted to install OpenZFS's 2.2.0, and hypothetically the package names were the same, apt treats 2.2.0 as less than 2.2.0ubuntu1 for version numbering purposes, I believe, so it would always want to upgrade to the stock packages unless you put an explicit version freeze in.

blind-oracle · 2023-10-17T15:03:30Z

I can confirm that after removing Conflicts: zfs-dkms (Replaces does not play a role) from contrib/debian/control the dkms deb installs fine, though it shows some strange empty dialog window with a single OK button and some warnings in console then:

debconf: Unknown template field '_description', in stanza #1 of /var/lib/dpkg/info/openzfs-zfs-dkms.templates
debconf: Unknown template field '_description', in stanza #2 of /var/lib/dpkg/info/openzfs-zfs-dkms.templates
debconf: Unknown template field '_description', in stanza #3 of /var/lib/dpkg/info/openzfs-zfs-dkms.templates

mwpastore · 2023-10-17T16:51:57Z

I am perplexed as to why the Ubuntu kernel image package provides zfs-dkms and spl-dkms. I can understand why it provides zfs-modules and spl-modules—and it makes sense that those would conflict with modules installed by another package. But it kind of defeats the point of a DKMS package to say that the kernel image package provides it.

Also, why is the provides set on the package containing vmlinux (linux-image-6.2.0-34-generic) and not the metapackage (linux-generic-hwe-22.04) or modules package (linux-modules-6.2.0-34-generic)? Even the headers package (linux-headers-6.2.0-34-generic) would make more sense (to me, but I am naive).

Anyway, I'm sure the good folks over at Ubuntu have sound reasons for having done this—probably to prevent someone shooting themselves in the foot—but it's quite frustrating.

I suppose these new native debs created by OpenZFS have the Conflicts directive in the control file to ensure it doesn't get installed alongside an older zfs-dkms package (i.e. one not provided by the Ubuntu kernel). I wonder if there's a way to write the Conflicts directive so that it excludes such packages without conflicting with the Ubuntu kernel?

EDIT: One other note. I edited the control file in the openzfs-zfs-dkms package I built and removed the Conflicts directive, per @blind-oracle, and it installed fine. However, it didn't automatically uninstall the openzfs-zfs-modules package I installed yesterday. Surely these two things should be mutually exclusive? Maybe the Conflicts was being used for this too?

EchterAgo · 2023-10-17T17:44:10Z

I can confirm that after removing Conflicts: zfs-dkms (Replaces does not play a role) from contrib/debian/control the dkms deb installs fine, though it shows some strange empty dialog window with a single OK button and some warnings in console then:
debconf: Unknown template field '_description', in stanza #1 of /var/lib/dpkg/info/openzfs-zfs-dkms.templates
debconf: Unknown template field '_description', in stanza #2 of /var/lib/dpkg/info/openzfs-zfs-dkms.templates
debconf: Unknown template field '_description', in stanza #3 of /var/lib/dpkg/info/openzfs-zfs-dkms.templates

I can confirm the same, EchterAgo@8553e34

blind-oracle · 2023-10-18T09:50:53Z

@mwpastore Yeah I agree that the kernel should not provide anything DKMS related since it defeats DKMS's purpose. Maybe we should file a bug to Ubuntu, but it will take a long time to be acted upon for sure. So I guess we need to decide smth on OpenZFS side for now...

blind-oracle · 2023-11-22T08:00:13Z

2.2.1 tar.gz still fails with same

cp: cannot stat 'contrib/debian/control': No such file or directory

~~If using git with 2.2.1 tag then it fails later with some cryptic~~
Disregard that, after re-cloning the repo it built fine.

AllKind · 2023-11-22T13:32:58Z

@tonyhutter
Could you take a look? Why are the 2.2.x release tarballs broken?
Is it something that is automated by github, or???

luckylinux · 2023-11-27T12:24:05Z

@AllKind, @tonyhutter, @blind-oracle: See my analysis #15586 (comment)

It seems that one tarball works and the other one doesn't. Depends which one URL you take.

blind-oracle · 2023-11-27T13:01:31Z

@luckylinux well that's obvious since using the 1st link you just get the GIT revision which is .tar.gz-ed by github, similar to git clone + git checkout, which works.

And the "normal" link (from release page) leads to some .tar.gz that was somehow prepared manually/by script/by some github workflow - this one doesn't work for whatever reason.

luckylinux · 2023-11-27T13:10:23Z

@blind-oracle maybe it was obvious for you which tarball to take, for me it wasn't. And something tells me that I'm not the only one, because I wasn't the only one with that error.

Anyway, I just wanted to share my experience, hoping it could help somebody in similar situation.

blind-oracle · 2023-11-27T13:22:57Z

Ideally there should be some github action set up here which build DEB and RPM packages for most recent Ubuntu/Debian/Fedora and published them along with the .tar.gz so that we don't need to do it manually.

FL140 · 2023-11-27T14:48:48Z

Ideally there should be some github action set up here which build DEB and RPM packages for most recent Ubuntu/Debian/Fedora and published them along with the .tar.gz so that we don't need to do it manually.

That is exactly the way it should be.
For now I will provide a script for compiling on Ubuntu in #15586 (The initial version is already there. Update with information from here later today.)

gdevenyi · 2023-12-01T02:18:58Z

#10333

FL140 · 2023-12-01T03:40:37Z

#10333

Well, reading through that one, it is obvious, that the current approach is NOT always the best. While I am also very conservative on upgrading ZFS, I got hit fully by the last two weeks events. And Ubuntu has still no bugfix release version 2.2.1 available for mantic (which doesn't fix it fully as we know now, but helps), IMHO this is nuts. So at least there should be an option to install a hotfix on those systems possible being hit, (which is in case of the events of the last two weeks every system running ZFS for a long time, with agreeable difference in probability, BUT we didn't know that two weeks ago when everything boild up and a hotfix was required to "stop the bleeding".

This hotfix still has not made it in Ubuntu Mantic which IMHO is a shame! So I am right now building packages for Ubuntu 2.2.2 after reading through a LOT of threads here over the last two weeks. So while for sure it is not the fault of the ZFS developers that Ubuntu is ignoring this issue it would help massively in the case of a bug like the one we had now to go here on GitHub and get a hotfix deb package from the release page. This would greatly help!

So finally I was just able to build all the deb packages for Ubuntu 23.10 and they installed with just a minor hickup. So if my system boots (fingers crossed) and nothing blows up I will post the script for others.

It would be great if the ZFS developers could have a look over it then (I had to fix a bit from the documentation available to make it working.) and integrate something similar in the build scripts.

The filesystem driver used is such an important part of the OS that downloading it directly from the developers would give us users much more confidence than having to relay on private PPAs of enthusiastic but unknown (and so not well trusted) unofficial folks like me on GitHub.

I would propose packges for Debian stable, Ubuntu LTS and Ubuntu current, THX!

luckylinux · 2023-12-01T05:09:59Z

Thanks for the update. I didn't know ZFS 2.2.2 was just released.

But @FL140, are you able to boot "normally" using Ubuntu Mantic 23.10 ?

I was and still am hit by
#15512
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2043253

For me the workaround was to type root password at the prompt, then

zfs mount -a
systemctl restart networking
systemctl restart ssh
init 3

And

/etc/init.d/lightdm restart

Ubuntu until 23.04 was working fine in that regards.

Concerning the potential silent corruption issue on ZFS 2.2.0 which should be fixed on ZFS 2.2.1 I see

zpool get all rpool | grep bclone
rpool  bcloneused                     0                              -
rpool  bclonesaved                    0                              -
rpool  bcloneratio                    1.00x                          -

But on the thread it is said that a value of "0" is not a sufficient condition to indicate that NO corruption is taking place.

FL140 · 2023-12-01T06:12:07Z

Thanks for the update. I didn't know ZFS 2.2.2 was just released.

Well me too, it just got released while I build the package from the tonyhutter 2.2.2 branch.
I am currently building new packages from the official release, but this should produce the same packages.

As for your boot error, no I don't run into that one! BUT I will have to say, that I am using dracut and not initramfs, so probably this has an impact regarding the initramfs. I had to switch because GRUB, how to put it nicely? "Had problems" with after Upgrading to 23.10 with ZFS on LUKS.

You can find my build script here:
#15586 (comment)
(I just noted you already ran it. Switch the source to the official branch.)

But be warned! I ran into a bad pool crash while testing some block cloned data, see here:
#15602

luckylinux · 2023-12-01T06:13:55Z

I didn't use your script, I used mine ...

But this builds 2.2.2 as expected

# Use tar (working)
wget https://github.com/openzfs/zfs/archive/refs/tags/zfs-$version.tar.gz -O zfs-$version.tar.gz
mkdir -p zfs-$version
tar xvf zfs-$version.tar.gz -C zfs-$version --strip-components 1

While the git command wanted to build 2.2.99 for whatever reason

# Use git and clone zfs-$version tag
git clone https://github.com/openzfs/zfs.git --depth 1 --tag zfs-$version

luckylinux · 2023-12-01T06:17:10Z

@FL140 But are you using bclone ?
#15602 (comment) states that defaults should be zfs_bclone_enabled=0 and zfs_dmu_offset_next_sync=1

FL140 · 2023-12-01T06:39:12Z

@luckylinux

The thing is that blone was active between 2.2.0 and 2.2.1 producing ~750MB of cloned data IIRC. I wanted to check which files are affected by bclone and the investigate if there is a problem with those files.

Yes 2.2.2 runs here with zfs_bclone_enabled=0 and zfs_dmu_offset_next_sync=1 which I verified just prior executing the bclone information script, which led to the panic.

luckylinux · 2023-12-01T06:42:43Z

So you mean it's kind of ... "The damage is already done" ?

I tried the script but in my case all I get is

./bclonecheck.sh rpool
zdb: can't open 'rpool': No such file or directory
BRT empty. There are no bcloned files in the pool.

./bclonecheck.sh -C rpool
zdb: can't open 'rpool': No such file or directory
BRT empty. There are no bcloned files in the pool.

Temporary directory: /tmp/tmp.izPYmD6TDj

Probably related to needing a "special" zdb (according to https://github.com/0x5c/zfs-bclonecheck), but I don't know how/where to get one.

rincebrain · 2023-12-01T06:44:24Z

git zdb knows how to print the BRT. I don't think 2.2.x got a backport of that yet.

The top of the link to the script repo points this out and has a link to the PR for it.

0x5c · 2023-12-01T06:46:29Z

git zdb knows how to print the BRT. I don't think 2.2.x got a backport of that yet.

The top of the link to the script repo points this out and has a link to the PR for it.

It got merged in #15602

FL140 · 2023-12-01T06:49:22Z

So you mean it's kind of ... "The damage is already done" ?

exactly, but this has to be investigated

I tried the script but in my case all I get is
./bclonecheck.sh rpool
zdb: can't open 'rpool': No such file or directory
BRT empty. There are no bcloned files in the pool.
./bclonecheck.sh -C rpool
zdb: can't open 'rpool': No such file or directory
BRT empty. There are no bcloned files in the pool.

Temporary directory: /tmp/tmp.izPYmD6TDj
Probably related to needing a "special" zdb (according to https://github.com/0x5c/zfs-bclonecheck), but I don't know how/where to get one.

Hmmm, that is interesting as the script started here without a problem and it worked until the panic. The two BRT related commits are in 2.2.2. I think those include all we need.

luckylinux · 2023-12-01T07:04:42Z

So you mean it's kind of ... "The damage is already done" ?

exactly, but this has to be investigated
I tried the script but in my case all I get is
./bclonecheck.sh rpool
zdb: can't open 'rpool': No such file or directory
BRT empty. There are no bcloned files in the pool.
./bclonecheck.sh -C rpool
zdb: can't open 'rpool': No such file or directory
BRT empty. There are no bcloned files in the pool.

Temporary directory: /tmp/tmp.izPYmD6TDj
Probably related to needing a "special" zdb (according to https://github.com/0x5c/zfs-bclonecheck), but I don't know how/where to get one.
Hmmm, that is interesting as the script started here without a problem and it worked until the panic. The two BRT related commits are in 2.2.2. I think those include all we need.

I installed zfs 2.2.2 but I didn't reboot thud the kernel module is still zfs 2.2.0

rincebrain · 2023-12-01T07:05:46Z

Running kernel modules don't matter to zdb.

luckylinux · 2023-12-01T07:15:46Z

Even more weird that the check script doesn't work for me then, since zfs 2.2.2 has already been installed

0x5c · 2023-12-01T07:27:15Z

zdb: can't open 'rpool': No such file or directory

Are you able to run zdb on the pool at all? like zdb -C <pool> (just prints the pool configuration)

luckylinux · 2023-12-01T07:58:32Z

zdb: can't open 'rpool': No such file or directory

Are you able to run zdb on the pool at all? like zdb -C <pool> (just prints the pool configuration)

Nope. It's weird.

root@UbuntuWorkstation02:/usr/src/zfs-bclonecheck# zdb -C rpool
zdb: can't open 'rpool': No such file or directory

But

root@UbuntuWorkstation02:/usr/src/zfs-bclonecheck# zfs list
NAME                                        USED  AVAIL  REFER  MOUNTPOINT
rpool                                       429G  20.9G    96K  /
rpool/ROOT                                  304G  20.9G    96K  none
rpool/ROOT/ubuntu                           304G  20.9G  16.1G  /
rpool/ROOT/ubuntu/srv                        96K  20.9G    96K  /srv
rpool/ROOT/ubuntu/tmp                      87.4M  20.9G  5.03M  /tmp
rpool/ROOT/ubuntu/usr                      1.17G  20.9G    96K  /usr
rpool/ROOT/ubuntu/usr/local                1.17G  20.9G  1.17G  /usr/local
rpool/ROOT/ubuntu/var                       284G  20.9G    96K  /var
rpool/ROOT/ubuntu/var/games                  96K  20.9G    96K  /var/games
rpool/ROOT/ubuntu/var/lib                   281G  20.9G   278G  /var/lib
rpool/ROOT/ubuntu/var/lib/AccountsService   216K  20.9G   152K  /var/lib/AccountsService
rpool/ROOT/ubuntu/var/lib/NetworkManager    116K  20.9G   116K  /var/lib/NetworkManager
rpool/ROOT/ubuntu/var/lib/apt               585M  20.9G   503M  /var/lib/apt
rpool/ROOT/ubuntu/var/lib/dpkg              145M  20.9G   102M  /var/lib/dpkg
rpool/ROOT/ubuntu/var/log                  2.79G  20.9G  1.19G  /var/log
rpool/ROOT/ubuntu/var/mail                   96K  20.9G    96K  /var/mail
rpool/ROOT/ubuntu/var/spool                9.68M  20.9G  4.23M  /var/spool
rpool/ROOT/ubuntu/var/www                    96K  20.9G    96K  /var/www
rpool/USERDATA                              124G  20.9G    96K  none
rpool/USERDATA/podman                      1.06G  20.9G  1.06G  /home/podman
rpool/USERDATA/root                        17.5M  20.9G  16.8M  /root
rpool/USERDATA/stefano                      123G  20.9G   119G  /home/stefano

And

dpkg -s openzfs-zfsutils
Package: openzfs-zfsutils
Status: install ok installed
Priority: optional
Section: contrib/admin
Installed-Size: 1731
Maintainer: ZFS on Linux specific mailing list <zfs-discuss@list.zfsonlinux.org>
Architecture: amd64
Source: openzfs-linux
Version: 2.2.2-1
Replaces: spl (<< 0.7.9-2), spl-dkms, zfsutils-linux
Provides: openzfsutils
Depends: openzfs-libnvpair3 (= 2.2.2-1), openzfs-libuutil3 (= 2.2.2-1), openzfs-libzfs4 (= 2.2.2-1), openzfs-libzpool5 (= 2.2.2-1), python3, libblkid1 (>= 2.16), libc6 (>= 2.38), libssl3 (>= 3.0.0), libudev1 (>= 183), libuuid1 (>= 2.16)
Pre-Depends: init-system-helpers (>= 1.54~)
Recommends: lsb-base, openzfs-zfs-modules | openzfs-zfs-dkms, openzfs-zfs-zed
Suggests: nfs-kernel-server, samba-common-bin (>= 3.0.23), openzfs-zfs-initramfs | openzfs-zfs-dracut
Breaks: openrc, openzfs-zfs-dkms (>> 2.2.2-1...), openzfs-zfs-dkms (<< 2.2.2-1), spl (<< 0.7.9-2), spl-dkms (<< 0.8.0~rc1)
Conflicts: zfs, zfs-fuse, zfsutils-linux
Conffiles:
 /etc/default/zfs c331d379511e423297a088c6060b3db4
 /etc/init.d/zfs-import f7e910e8d6044ba3fb6edba754764711
 /etc/init.d/zfs-load-key 06ed525d980ec7cf986bad47e69cd338
 /etc/init.d/zfs-mount 7cde24d35cf6a5605ecaca2b2da271d2
 /etc/init.d/zfs-share 94b89d78b9c960b705428d5009e36bbc
 /etc/zfs/zfs-functions 53915c6a2375869a2106b0acdaa957d4
Description: command-line tools to manage OpenZFS filesystems
 OpenZFS is a storage platform that encompasses the functionality of
 traditional filesystems and volume managers. It supports data checksums,
 compression, encryption, snapshots, and more.
 .
 This package provides the zfs and zpool commands to create and administer
 OpenZFS filesystems.
Homepage: https://openzfs.org/

Running zdb without arguments yields

root@UbuntuWorkstation02:/usr/src/zfs-bclonecheck# zdb
cannot open '/etc/zfs/zpool.cache': No such file or directory

But the cache is usually in /etc/zfs/zfs-list.cache/rpool.

And

root@UbuntuWorkstation02:/usr/src/zfs-bclonecheck# zpool get cachefile
NAME   PROPERTY   VALUE      SOURCE
rpool  cachefile  -          default

As shown in openzfs#15404#issuecomment-1765002181, Ubuntu kernel has 'Provides: zfs-dkms', which will cause uninstall of the kernel, when attempting to install openzfs-zfs-dkms. As a workaround remove the 'Conflicts: zfs-dkms' definition from the debian control file. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Mart Frauenlob <AllKind@fastest.cc> Closes openzfs#15503

behlendorf · 2024-10-04T21:18:25Z

The recently tagged 2.3.0-rc1 release included a couple of fixed. It'd be very helpful if anyone who had issues could try again so we can work through any potential removing issues. We'll backport these for 2.2.7.

thisisnotmyrealname added the Type: Defect Incorrect behavior (e.g. crash, hang) label Oct 14, 2023

rincebrain mentioned this issue Oct 15, 2023

2.1 libzpool5 deb relies on older libzfs4 #12322

Closed

gmelikov mentioned this issue Oct 17, 2023

Add missing dependencies to Debian custom packages instructions openzfs/openzfs-docs#458

Merged

AllKind mentioned this issue Oct 19, 2023

Fix dkms installation of deb packages created with Alien. #15415

Merged

13 tasks

FL140 mentioned this issue Nov 27, 2023

Missing contrib/debian/control when building deb dkms package on Ubuntu 23.10 #15586

Open

AllKind mentioned this issue Nov 29, 2023

zfs-2.2.2 patchset #15602

Merged

13 tasks

rincebrain mentioned this issue Sep 7, 2024

make native-deb should be removed, since it's never worked in a 2.2 release tarball #16513

Closed

nabijaczleweli mentioned this issue Oct 2, 2024

Kill contrib/debian #16595

Closed

usaleem-ix mentioned this issue Oct 4, 2024

Fix packaging issues for native Debian packages #16604

Closed

13 tasks

zfs-2.2.0 building native-deb-utils target fails early on #15404

zfs-2.2.0 building native-deb-utils target fails early on #15404

Comments

thisisnotmyrealname commented Oct 14, 2023 • edited Loading

System information

Describe the problem you're observing

Describe how to reproduce the problem

blind-oracle commented Oct 15, 2023

usaleem-ix commented Oct 16, 2023

mwpastore commented Oct 16, 2023

blind-oracle commented Oct 16, 2023

usaleem-ix commented Oct 17, 2023

blind-oracle commented Oct 17, 2023 • edited Loading

AllKind commented Oct 17, 2023

blind-oracle commented Oct 17, 2023

AllKind commented Oct 17, 2023

blind-oracle commented Oct 17, 2023 • edited Loading

AllKind commented Oct 17, 2023 • edited Loading

blind-oracle commented Oct 17, 2023

AllKind commented Oct 17, 2023

rincebrain commented Oct 17, 2023

blind-oracle commented Oct 17, 2023 • edited Loading

rincebrain commented Oct 17, 2023

blind-oracle commented Oct 17, 2023

AllKind commented Oct 17, 2023

AllKind commented Oct 17, 2023

rincebrain commented Oct 17, 2023

AllKind commented Oct 17, 2023

rincebrain commented Oct 17, 2023 • edited Loading

blind-oracle commented Oct 17, 2023

mwpastore commented Oct 17, 2023 • edited Loading

EchterAgo commented Oct 17, 2023

blind-oracle commented Oct 18, 2023

blind-oracle commented Nov 22, 2023 • edited Loading

AllKind commented Nov 22, 2023

luckylinux commented Nov 27, 2023 • edited Loading

blind-oracle commented Nov 27, 2023

luckylinux commented Nov 27, 2023

blind-oracle commented Nov 27, 2023

FL140 commented Nov 27, 2023 • edited Loading

gdevenyi commented Dec 1, 2023

FL140 commented Dec 1, 2023 • edited Loading

luckylinux commented Dec 1, 2023 • edited Loading

FL140 commented Dec 1, 2023

luckylinux commented Dec 1, 2023

luckylinux commented Dec 1, 2023

FL140 commented Dec 1, 2023

luckylinux commented Dec 1, 2023

rincebrain commented Dec 1, 2023 • edited Loading

0x5c commented Dec 1, 2023

FL140 commented Dec 1, 2023 • edited Loading

luckylinux commented Dec 1, 2023

rincebrain commented Dec 1, 2023

luckylinux commented Dec 1, 2023

0x5c commented Dec 1, 2023

luckylinux commented Dec 1, 2023

behlendorf commented Oct 4, 2024

thisisnotmyrealname commented Oct 14, 2023 •

edited

Loading

blind-oracle commented Oct 17, 2023 •

edited

Loading

blind-oracle commented Oct 17, 2023 •

edited

Loading

AllKind commented Oct 17, 2023 •

edited

Loading

blind-oracle commented Oct 17, 2023 •

edited

Loading

rincebrain commented Oct 17, 2023 •

edited

Loading

mwpastore commented Oct 17, 2023 •

edited

Loading

blind-oracle commented Nov 22, 2023 •

edited

Loading

luckylinux commented Nov 27, 2023 •

edited

Loading

FL140 commented Nov 27, 2023 •

edited

Loading

FL140 commented Dec 1, 2023 •

edited

Loading

luckylinux commented Dec 1, 2023 •

edited

Loading

rincebrain commented Dec 1, 2023 •

edited

Loading

FL140 commented Dec 1, 2023 •

edited

Loading