Cross-platform compatible xattr user namespace

[ghost]

Motivation and Context

ZFS on Linux originally implemented xattr namespaces in a way that is incompatible with other operating systems. On illumos, xattrs do not have namespaces. Every xattr name is visible. FreeBSD has two universally defined namespaces: EXTATTR_NAMESPACE_USER and EXTATTR_NAMESPACE_SYSTEM. The system namespace is used for protected FreeBSD-specific attributes such as MAC labels and pNFS state. These attributes have the namespace string "freebsd:system:" prefixed to the name in the encoding scheme used by ZFS. The user namespace is used for general purpose user attributes and obeys normal access control mechanisms. These attributes have no namespace string prefixed, so xattrs written on illumos are accessible in the user namespace on FreeBSD, and xattrs written to the user namespace on FreeBSD are accessible by the same name on illumos.

Linux has several xattr namespaces. On Linux, ZFS encodes the namespace in the xattr name for every namespace, including the user namespace. As a consequence, an xattr in the user namespace with the name "foo" is stored by ZFS with the name "user.foo" and therefore appears on FreeBSD and illumos to have the name "user.foo" rather than "foo". Conversely, none of the xattrs written on FreeBSD or illumos are accessible on Linux unless the name
happens to be prefixed with one of the Linux xattr namespaces, in which case the namespace is stripped from the name. This makes xattrs entirely incompatible between Linux and other platforms.

We want to make the encoding of user namespace xattrs compatible across platforms. A crucial requirement of this compatibility is for xattrs from existing pools from FreeBSD and illumos to be accessible by the same names in the user namespace on Linux. It is also necessary that existing pools with xattrs written by Linux retain access to those xattrs by the same names on Linux. Making user namespace xattrs from Linux accessible by the correct names on other platforms is important. The handling of other namespaces is not required to be consistent.

Description

Add a fallback mechanism for listing and getting xattrs to treat xattrs as being in the user namespace if they do not match a known prefix.

When setting user namespace xattrs, do not prefix the namespace to the name. If the xattr is already present with the namespace prefix, remove it so only the non-prefixed version persists. This ensures other platforms will be able to read the xattr with the correct name.

Do not allow setting or getting xattrs with a name that is prefixed with one of the namespace names used by ZFS on supported platforms.

Make xattr namespace compatibility dependent on a new feature. New pools will use the compatible namespace encoding by default, and existing pools will continue using the old Linux-specific encoding until the feature is enabled.

Allow choosing between cross-platform compatability and legacy Linux compatibility with a per-dataset property. This facilitates replication between hosts with different compatibility needs.

Add xattr_fallback property to skip fallbacks.

This property currently defaults to "on" so we do not miss xattrs in datasets from pools for which the encoding cannot be known or may even be a mix of both. It can be turned "off" for a performance boost when it is known only the configured xattr_compat format is relevant.

Future work will investigate how we can change the default to "off" and automatically force it to "on" when we cannot know whether skipping the fallbacks is safe.

TODO:

• If xattrs with the user namespace prefix are already present, they
  will not be automatically fixed. Any existing xattrs must be
  manually rewritten on Linux for the name to be correct on FreeBSD.
  This also means that files may have a mix of incompatible and
  compatible names. Other platforms could strip the Linux user
  namespace prefix from xattr names so they are presented correctly.

• The newly written xattrs will no longer be visible on previous
  versions of ZFS on Linux. This behavior needs to be made optional
  with a feature flag and possibly a per-dataset property.

• There is no attempt to handle xattr names that clash with a namespace
  prefix. If you write an xattr named "user.foo" to the user namespace
  on FreeBSD, the "user." prefix will be stripped on Linux. This was
  partially the case already, except now the stripped name will also
  replace the prefixed name when updating the xattr. Likewise, setting
  an xattr to the user namespace using a name with the prefix of
  another namespace may cause the xattr to be manipulated in the other
  namespace. This is potentially a security issue. Such names must be
  forbidden.

• New tests should be added when the functionality is complete.

• Documentation will be needed.

• Performance optimizations should be investigated.

How Has This Been Tested?

So far just existing ZTS tests and manually checking xattr naming/visibility on snaps sent between FreeBSD and Linux.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[ghost]

Ideally I'd like to be able to switch between the old and the new behavior with a per-dataset property. My intent with this would be to retain compatibility with older ZFS on Linux versions for replicated filesystems. Would this be possible? I'm not sure if the pool feature would prevent replication with older ZFS versions if it is not active on the specific dataset, or if the property would end up breaking replication because the other pool doesn't know about it.

[ghost]

I've not bothered storing the ABI change for the sake of this RFC, it would just clutter up the diff.

[ghost]

• Rebased
• Add feature@xattr_compat
• WIP xattr_compat property (needs tests, docs)

[ghost]

I've noticed that FreeBSD does not actually handle xattr=sa, it behaves as if it were xattr=on. I have gone ahead and implemented xattr=sa for FreeBSD but it is not included in this RFC as it does not directly relate to the proposed change to user namespace handling on Linux. It will be something I include in my test matrix.

Update: xattr=sa was merged in #11997.

[ghost]

• Rebased
• WIP xattr_compat for FreeBSD

[ghost]

• Rebased

[lundman]

We noticed that our xattr tests would fail, especially when going from xattr=sa to xattr=on, or v-v. This inspired us to take Linux xattr code and re-apply it; openzfsonosx/openzfs@a1c2e6c

The upshot of this is that macOS xattrs sources (albeit a different filename) is nearly identical to Linux (but better, as we take uio instead of that tmp struct! :) )
https://github.com/openzfsonosx/openzfs/blob/development/module/os/macos/zfs/zfs_vnops_osx_xattr.c

But, as for the xattr name-space, the macro XATTR_USER_PREFIX is still in there but not used. The name-space code can also be moved over, depending on what the consensus is?

[ghost]

• Rebased (with Use SET_ERROR for more errors in FreeBSD vnops #12356)
• Squashed
• Added xattr_fallback boolean property to enable skipping fallback lookups as an optimization
• Added xattr_compat and xattr_fallback properties to general fs prop lists in tests
• Added xattr_compat feature to general feature list in tests

[ghost]

• Rebased so I can grab the ABI files from the CI

[ghost]

• Rebased

[ghost]

• Rebased
• Restored zpool-features.7 changes lost in rebase after man pages moved
• Add dependency on SPA_FEATURE_ENABLED_TXG
• Activate feature on FreeBSD when writing an xattr with xattr_compat=linux

[behlendorf]

@freqlabs can you rebase this one last time, that should get us a clean build on all the builders. @ahrens @lundman @problame if you could give this a final review I'd like to move forward with integrating this PR.

[ghost]

• Rebased

[ghost]

• Rebase
• Make xattr_compat=linux the default on Linux for now

[ghost]

• Rebase
• Make xattr_compat a tunable, zfs_xattr_compat

[behlendorf]

@freqlabs thanks for updating this so quickly. You'll want to make sure to rebase after e2909fa to sort out the GitHub actions CI failure. Sorry about that.

[ghost]

• Incorporated feedback

[ghost]

• Test native format first then toggle back and forth
• Try reading native format first on Linux

[ahrens]

I don't have time to review the code thoroughly, but I discussed the design with @freqlabs and @behlendorf and I think it's great. The explanation in the tunables manpage is super helpful!