zfs-2.2.0 - after upgrade from 2.1.12 zfs fails to honor the mountpoint unless zoned is set to on

[RzTen1]

System information

Type	Version/Name
Distribution Name	Slackware
Distribution Version	15.0
Kernel Version	6.5.7
Architecture	x86_64
OpenZFS Version	2.2.0

Describe the problem you're observing

After an upgrade to zfs 2.2.0 from 2.1.12 it was observed that none of the zfs filesystems automounted. zfs mount -a mounted nothing. Trying to mount a specific filesystem, ie: zfs mount backup, resulted in 'permission denied'.

Trying to change the mountpoint resulted in the following message:
cannot set property for 'backup': 'mountpoint' cannot be set while dataset 'zoned' property is set

turning zoned on, default was off, results in in the mount working again and mountpoint allowed to be set.

turning it back off causes it to fail:
root@darkstar:/home/zfs-2.2.0# zfs set zoned=off backup
root@darkstar:/home/zfs-2.2.0# zfs get mountpoint backup
NAME PROPERTY VALUE SOURCE
backup mountpoint /home/backup local
root@darkstar:/home/zfs-2.2.0# zfs set mountpoint=/home/backup backup
cannot set property for 'backup': 'mountpoint' cannot be set while dataset 'zoned' property is set
root@darkstar:/home/zfs-2.2.0# zfs set zoned=on backup
root@darkstar:/home/zfs-2.2.0# zfs set mountpoint=/home/backup backup
root@darkstar:/home/zfs-2.2.0# zfs mount backup
root@darkstar:/home/zfs-2.2.0#

Describe how to reproduce the problem

Create filesystem on zfs 2.1.12. set the mountpoint to something other than legacy. Upgrade to zfs 2.2.0.

Workaround

setting the mountpoint to legacy works if zoned is off. While it's set to legacy mount -t zfs backup /home/backup works without error.

Include any warning/errors/backtraces from the system logs

Nothing is logged during any of these errors. As near as I can tell the global ID has somehow changed and no longer matches the ID stored in the filesystem, but I have no idea how to modify these IDs. The documentation suggests that zoned shouldn't do anything on linux, but I'd still prefer it to be off since it's the default.

[rincebrain]

I would suspect you're running the old module version and new userland and there's some inconsistent behavior around interpreting the properties struct returned, somehow, but zfs version output would make it clear if I'm correct or not.

[RzTen1]

:~# zfs version
zfs-2.2.0-1
zfs-kmod-2.2.0-1

I've also tried exporting and re-importing the pools in case the ID is updated on import but there was no change.

[deliciouslytyped]

I believe I my be running into a related issue.
Linux namespaces are complicated...
This is more of an experience report because I don't quite understand what is supposed to be happening.

I created a file backed zpool in an unshare, and I was able to import it in the unshare.
This resulted it being visible on the host but not in the unshare (and AFAICT I can't actually do anything with it in the unshare at this point, though it still feels a bit weird that I can get the pool to show up in zpool list on the "host" like this).

After this I believe I created a dataset on the host, and zoned it to the unshare's user namespace.

After fooling around with mounting and unmounting on both sides, I somehow managed to get into a state where I couldn't mount the dataset in the unshare, even though it was unmounted on the host. (mount | grep zfs was empty).

Export/import didnt seem to help, neither did share/unshare.
The mount point of the dataset ended up owned by nobody:nogroup which are the namespace "overflow" / unmapped / something like that id's.

I wasn't able to change any properties on the dataset either.

The documentation for allow says:

Delegations are supported under Linux with the exception of mount, unmount, mountpoint, canmount, rename, and share. These permissions cannot be delegated because the Linux mount(8) command restricts modifications of the global namespace to the root user

Wouldn't this be the whole point of user / mount namespaces though?

After changing zoned=on on the host (didn't try in the unshare) I was able to change the mountpoint property to legacy in the unshare and mount to some other directory. Until I did this, I could not, even though I used zfs allow to add the capabilities listed as unsupported above. (Which is expected I guess.)

Using nsenter to enter the mount namespace, ls shows that the directory that was automounted into is owned by root:root.

[RzTen1]

It looks like the above entry, #15419, does explain my issue. My build doesn't have CONFIG_USER_NS set:

grep CONFIG_USER_NS 6.5.7.config
# CONFIG_USER_NS is not set

[RzTen1]

Since 15419 describes this issue better and matches my case I'm going to close this as a dup. @deliciouslytyped I think you should resubmit your issue as it's own thing as it looks like your build has namespaces enabled.