FreeBSD: Implement xattr=sa

FreeBSD historically has not cared about the xattr property; it was
always treated as xattr=on.  With xattr=on, xattrs are stored as files
in a hidden xattr directory.  With xattr=sa, xattrs are stored as
system attributes and get cached in nvlists during xattr operations.
This makes SA xattrs simpler and more efficient to manipulate.  FreeBSD
needs to implement the SA xattr operations for feature parity with
Linux and to ensure that SA xattrs are accessible when migrated or
replicated from Linux.

Following the example set by Linux, refactor our existing extattr vnops
to split off the parts handling dir style xattrs, and add the
corresponding SA handling parts.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes openzfs#11997

man/man8/zfsprops.8

@@ -1773,7 +1773,7 @@ The default value is
 This property is not used on Linux.
 .It Sy xattr Ns = Ns Sy on Ns | Ns Sy off Ns | Ns Sy sa
 Controls whether extended attributes are enabled for this file system. Two
-styles of extended attributes are supported either directory based or system
+styles of extended attributes are supported: either directory based or system
 attribute based.
 .Pp
 The default value of
@@ -1785,7 +1785,7 @@ can be set on a file. Although under Linux the
 and
 .Xr setxattr 2
 system calls limit the maximum size to 64K. This is the most compatible
-style of extended attribute and is supported by all OpenZFS implementations.
+style of extended attribute and is supported by all ZFS implementations.
 .Pp
 System attribute based xattrs can be enabled by setting the value to
 .Sy sa .
@@ -1798,6 +1798,9 @@ based xattr. System attribute based extended attributes are not accessible
 on platforms which do not support the
 .Sy xattr=sa
 feature.
+OpenZFS supports
+.Sy xattr=sa
+on both FreeBSD and Linux.
 .Pp
 The use of system attribute based xattrs is strongly encouraged for users of
 SELinux or POSIX ACLs. Both of these features heavily rely on extended

module/os/freebsd/zfs/zfs_vfsops.c

@@ -831,6 +831,10 @@ zfsvfs_init(zfsvfs_t *zfsvfs, objset_t *os)
 		    &sa_obj);
 		if (error != 0)
 			return (error);
+
+		error = zfs_get_zplprop(os, ZFS_PROP_XATTR, &val);
+		if (error == 0 && val == ZFS_XATTR_SA)
+			zfsvfs->z_xattr_sa = B_TRUE;
 	}
 
 	error = sa_setup(os, sa_obj, zfs_attr_table, ZPL_END,