Lift userspace definitions out of zfs_context.h

[robn]

Motivation and Context

As part of my ongoing project to make userspace into a "first class" platform, and to make OpenZFS easier to port to other platforms, I am working to make headers consistent across all platforms. I would like to get to a point where zfs_context.h is the same for all platforms and compile modes, and so provides a complete list of "necessary to implement" for any platform.

The first part of that is to lift all the userspace-specific stuff out of zfs_context.h into separated headers in lib/libspl/include. That's this PR.

This PR deliberately stops at emptying zfs_context.h of definitions. The next PR I open on this project will collapse it into a single list of headers that works across all platforms, but that will require movement across all platform headers, not just userspace, and I don't want to complicate review on this further.

Description

There's a lot of commits here, but they are basically just picking a specific concept or object from the userspace part zfs_context.h and moving it out to a specific header under lib/libspl/include. For most, that header does not already exist. For some, it does exist but wasn't used, or was in conflict, and has been updated appropriately.

After that, there's a long tail of misc stuff, which has been either moved somewhere more appropriate or deleted.

I expect that much of this could be collapsed into a smaller set of commits, but for the moment I'm keeping them separate for review.

How Has This Been Tested?

This should all be entirely mechanical, so anything that compiles should work.

The final series compiles on Linux and FreeBSD, and ZTS passes.

It should compile clean at each individual commit, but I have moved things around a little while assembling the final series, so something may be stacked wrong. I'll go and check that now, but its also pretty low-key.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

Very nice! Aside from a a couple small nits this is a welcome bit of long overdue refactoring.

For next steps I think it would make the most sense to move the remaining user space implementations out of libzpool and into libspl. It looks like lib/libzpool/kernel.c and lib/libzpool/taskq.c should cover almost everything.

Do you want to tackle that in this PR or another? I'm fine with pulling in this whole stack of commits, having it broken up made it much easier to review.

[robn]

Actually, responding to both your comments, I realised what happened here: zfs_context_os.h should not be in libspl - its for the "zfs module", not the SPL! That's why it doesn't make sense!

Both those functions are very libzpool-specific, so they can either say in zfs_context_os.h when I move it away from libspl, or they can go to a libzpool.h, along with kernel_init() etc - things the program needs to drive libzpool, not things that the module internals need to operate.

I'll give it a try. I think there's at least enough overlap with this PR that it could go here, but I'll see how it shakes out - if its ick enough, maybe the next PR is better. More soon!

[robn]

@behlendorf ok, here we go. This actually doubles the size of the PR, but I'm not sure it cleanly splits into two. Let me know what you think. If I don't split, I should rewrite the OP at least.

The whole thing here is ensuring the the implementations of things live in the same place as their headers. So there's a bunch of stuff moved over to libspl from libzpool, libspl gains a dedicated libspl_init() and libspl_fini() to initialise physmem, the rng, etc, while libzpool gains a libzpool.h with kernel_init(), handle_tunable_option(), etc.

It's a lot, but I think it lands in a good place.

[behlendorf]

This is looking good, and it definitely feels like we're moving in the right direction.

Where I'd really like to end up is only including the sys/zfs_context.h headers when building the zfs kmod, libzpool, or low-level commands like zdb, zhack, ztest which can muck around with a pools internals. We're getting closer, but it seems like libzfs ends up including zfs_ioctl.hfor some shared user/kernel structures. That's fine, but it unfortunately drags in a bunch of other zfs headers we don't need. So there's some refactoring needed there stiil.

I took the liberty of pushing this PR a bit further with the above goal in mind. What do think of these proposed additions from https://github.com/behlendorf/zfs/tree/userspace-context-lift-part-2.

1. Unify the new lib/libzpool/include headers with the existing include/sys/ / include/os/<OS>/zfs/sys headers. This is already where all of the other core zfs headers live, several of which are already user/kernel aware, so we might as well keep them in one place.

2. Pulled all of the SET_ERROR / DTRACE_PROBE usage and bogus headers out of the spl layer to fix this layering violation. If we're going to wire those up to the internal zfs debug infrastructure they should live in zfs_debug.h. Luckily they were lightly used in the spl layer so I don't think we're losing much.

3. Stopped adding -I$(top_srcdir)/include/os/@ac_system_l@/zfs to the default user include paths. I've added LIBZPOOL_CPPFLAGS to make it easy to add this path were we still need it today. My goal is the gradually clean up the headers so we can drop it from most of the Makefiles. Adding a similar LIBSPL_CPPFLAGS and giving it the same treatment seems like a reasonable thing to do. Although I haven't tried it yet.

4. Stop adding -I$(top_srcdir)/module/icp/include to the default user include paths. Again, only set it when it's needed.

It might be a good idea to go even farther and move the headers in to include/spl/ and include/zfs/ directories. Something to mull over a bit more. Anyway, that's enough for now.

[robn]

Where I'd really like to end up is only including the sys/zfs_context.h headers when building the zfs kmod, libzpool, or low-level commands like zdb, zhack, ztest which can muck around with a pools internals.

Agreed,

We're getting closer, but it seems like libzfs ends up including zfs_ioctl.hfor some shared user/kernel structures. That's fine, but it unfortunately drags in a bunch of other zfs headers we don't need. So there's some refactoring needed there stiil.

Yeah, I was holding off on zfs_ioctl.h and fs/zfs.h for a while until I had a better sense of what the split between tools, libs and kernel should be. They're sorta related to converting remaining ioctls to nvlists, and to what a control port might look like for userspace, etc. Not that we can't move things again later, but it felt like a lot more churn than I wanted for now. Maybe nbd though, I can bang on it.

I took the liberty of pushing this PR a bit further with the above goal in mind. What do think of these proposed additions from https://github.com/behlendorf/zfs/tree/userspace-context-lift-part-2.

1. Unify the new lib/libzpool/include headers with the existing include/sys/ / include/os/<OS>/zfs/sys headers. This is already where all of the other core zfs headers live, several of which are already user/kernel aware, so we might as well keep them in one place.

I think this is going in the opposite direction to what I was aiming for. I want to remove #ifdef _KERNEL etc from all "shared" code, if possible. To my mind, the "userspace port" should just be what we imagine os/posix is (and I'm still thinking about calling it that!), and libzpool is kinda just the "loader" for that core. So _KERNEL should be meaningless, and anything "OS-specific" should be in separate headers.

This is not perfectly drawn though, and maybe you have a different vision for it?

2. Pulled all of the SET_ERROR / DTRACE_PROBE usage and bogus headers out of the spl layer to fix this layering violation. If we're going to wire those up to the internal zfs debug infrastructure they should live in zfs_debug.h. Luckily they were lightly used in the spl layer so I don't think we're losing much.

Yeah, that works for me for now. In a couple of work/prototype branches (most notably a new unified tunables infrastructure) I concluded that the SPL has to be considered as an extension of the platform, so anything we want to do with eg probes will have to be done directly using platform-specific idioms. So more to do, but we're in agreement here.

3. Stopped adding -I$(top_srcdir)/include/os/@ac_system_l@/zfs to the default user include paths. I've added LIBZPOOL_CPPFLAGS to make it easy to add this path were we still need it today. My goal is the gradually clean up the headers so we can drop it from most of the Makefiles. Adding a similar LIBSPL_CPPFLAGS and giving it the same treatment seems like a reasonable thing to do. Although I haven't tried it yet.

Yeah, that's the right thing to do.

4. Stop adding -I$(top_srcdir)/module/icp/include to the default user include paths. Again, only set it when it's needed.

Yep, for sure!

(incidentally; making libicp optional in userspace and allowing OpenSSL etc to be used is another thing I intent to do, and have a working prototype for already).

It might be a good idea to go even farther and move the headers in to include/spl/ and include/zfs/ directories. Something to mull over a bit more.

Aha this is exactly where I want to take it, but I found that the split has to happen first to even make it tractable.

[behlendorf]

Yeah, I was holding off on zfs_ioctl.h and fs/zfs.h for a while until I had a better sense of what the split between tools, libs and kernel should be.

Yup, that makes sense. It's going to take a little work to tease this all apart.

I think this is going in the opposite direction to what I was aiming for. I want to remove #ifdef _KERNEL etc from all "shared" code, if possible. To my mind, the "userspace port" should just be what we imagine os/posix is (and I'm still thinking about calling it that!), and libzpool is kinda just the "loader" for that core.

Agreed. Adding a os/posix is exactly where I'd like to end up. I'm on board with the name too! I was mainly suggesting this change as a stopgap along the way. I'm fine with having these headers live under lib/libzpool/include/sys/ until we're ready to move them as long as we avoid breaking the build along the way.

Yeah, that works for me for now. In a couple of work/prototype branches (most notably a new unified tunables infrastructure) I concluded that the SPL has to be considered as an extension of the platform, so anything we want to do with eg probes will have to be done directly using platform-specific idioms. So more to do, but we're in agreement here.

Exactly. It'd be nice to eventually move this down in the SPL layer and tie it in more closely with the platform. But as it currently stands we're not there, so we might as well be honest about it in the layering.

Great, it sounds like we're on the same page about all of this. Regard this PR, if you can sort out the remaining build failures I'm comfortable with merging it. It's probably better to do that sooner rather than latter to avoid this getting stale. Then follow up with the next round of refactoring.

[robn]

@behlendorf great to hear we're on the same page!

Last push is rebased to master, includes four of your five commits, with some small adjustments to address a few more missed things from the FreeBSD build. If the builders approve, I think we're good!

[robn]

I'm working on the ABI stuff now. Worst case, I just dump it in as-is, but I notice that we're now exposing a lot of libspl symbols through libuutil, which doesn't actually need any of it. So I'm just going through the static linkage now, seeing if there's a more sensible layering we can do now so everything depends on only the things it uses, and doesn't reexport. We'll see!

As for the others, I need to go back and do all the compile checks again against older kernels, make sure that different sets of defines are including the right things in the right places.

Bit painful, but I'm really happy about the number of quirks and hacks this work has shaken out :)

[robn]

Last push is a reorg of the lib makefiles and dependencies. I want to see what checkabi makes of it. If included, it won't be in exactly that form, so don't look too closely yet.

[robn]

@behlendorf last push is a rebase, and adding the ABI files as-is. It should be no big deal for this dev cycle at least, as we're not removing any symbols, only adding them (all of libspl is exposed through libuutil, libzfs and libzfs_core). So it shouldn't be breaking anything.

I've got a reorg coming to try to fix up symbol visibility in the libs we produce, but its kind of a nightmare and very invasive. I tried to do it on top of master first, but it's just as invasive and also different because the line between libzpool and libspl is so confused. So I'd rather do it after, if you don't mind.

(FWIW, #17909 is a bit of support for that work, and you can see what I'm working on in robn@0a08eb3, though it's not yet where I want it.)

[robn]

Build failures are:

2025-11-07T03:29:40.6817474Z make[4]: Entering directory '/tmp/zfs-build-zfs-h8XkkuO0/BUILD/zfs-2.4.99'
2025-11-07T03:29:40.7436222Z   CC       module/zfs/libicp_la-zfs_impl.lo
2025-11-07T03:29:41.3194922Z In file included from ./include/sys/zfs_context.h:131,
2025-11-07T03:29:41.3195491Z                  from ./include/sys/zio.h:39,
2025-11-07T03:29:41.3195971Z                  from ./include/sys/zio_checksum.h:32,
2025-11-07T03:29:41.3196427Z                  from module/zfs/zfs_impl.c:27:
2025-11-07T03:29:41.3197203Z ./include/os/linux/zfs/sys/zfs_context_os.h:27:10: fatal error: linux/dcache_compat.h: No such file or directory
2025-11-07T03:29:41.3197980Z    27 | #include <linux/dcache_compat.h>
2025-11-07T03:29:41.3198368Z       |          ^~~~~~~~~~~~~~~~~~~~~~~
2025-11-07T03:29:41.3198728Z compilation terminated.
2025-11-07T03:29:41.3241600Z make[4]: *** [Makefile:9391: module/zfs/libicp_la-zfs_impl.lo] Error 1

Locally, it's ok for me:

$ env CCACHE_RECACHE=1 make module/zfs/libicp_la-zfs_impl.lo V=1
/bin/bash ./libtool  --tag=CC --silent  --mode=compile gcc -DHAVE_CONFIG_H -I.  -include ./zfs_config.h -I./include -I./include -I./lib/libspl/include -I./lib/libspl/include/os/linux -I./lib/libzpool/include -D_GNU_SOURCE -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DLIBEXECDIR=\"/usr/local/libexec\" -DZFSEXECDIR=\"/usr/local/libexec/zfs\" -DRUNSTATEDIR=\"/usr/local/var/run\" -DSBINDIR=\"/usr/local/sbin\" -DSYSCONFDIR=\"/usr/local/etc\" -DPKGDATADIR=\"/usr/local/share/zfs\" -DDEBUG -UNDEBUG  -DTEXT_DOMAIN=\"zfs-linux-user\"   -DDEBUG -UNDEBUG -DZFS_DEBUG -I./include/os/linux/zfs -I./module/icp/include  -std=gnu99 -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes -Wwrite-strings -Wno-sign-compare -Wno-missing-field-initializers -fno-strict-aliasing -fno-omit-frame-pointer -Wimplicit-fallthrough -Werror -g -fno-inline -fno-ipa-sra    -Wno-format-zero-length -Wno-format-truncation  -Wframe-larger-than=4096 -no-suppress -g -O2 -MT module/zfs/libicp_la-zfs_impl.lo -MD -MP -MF module/zfs/.deps/libicp_la-zfs_impl.Tpo -c -o module/zfs/libicp_la-zfs_impl.lo `test -f 'module/zfs/zfs_impl.c' || echo './'`module/zfs/zfs_impl.c
mv -f module/zfs/.deps/libicp_la-zfs_impl.Tpo module/zfs/.deps/libicp_la-zfs_impl.Plo

My guess; the Redhat-flavoured compiler or autotools is putting -I./include/os/linux/zfs before -I./lib/libzpool/include. If I hack it that way myself, I get the same break:

$ /bin/bash ./libtool  --tag=CC --silent  --mode=compile gcc -DHAVE_CONFIG_H -I.  -include ./zfs_config.h -I./include -I./include -I./lib/libspl/include -I./lib/libspl/include/os/linux -I./include/os/linux/zfs -I./lib/libzpool/include -D_GNU_SOURCE -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DLIBEXECDIR=\"/usr/local/libexec\" -DZFSEXECDIR=\"/usr/local/libexec/zfs\" -DRUNSTATEDIR=\"/usr/local/var/run\"-DSBINDIR=\"/usr/local/sbin\" -DSYSCONFDIR=\"/usr/local/etc\" -DPKGDATADIR=\"/usr/local/share/zfs\" -DDEBUG -UNDEBUG  -DTEXT_DOMAIN=\"zfs-linux-user\"   -DDEBUG -UNDEBUG -DZFS_DEBUG -I./include/os/linux/zfs -I./module/icp/include  -std=gnu99 -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes -Wwrite-strings -Wno-sign-compare -Wno-missing-field-initializers -fno-strict-aliasing -fno-omit-frame-pointer -Wimplicit-fallthrough -Werror -g -fno-inline -fno-ipa-sra    -Wno-format-zero-length -Wno-format-truncation  -Wframe-larger-than=4096 -no-suppress -g -O2 -MT module/zfs/libicp_la-zfs_impl.lo -MD -MP -MF module/zfs/.deps/libicp_la-zfs_impl.Tpo -c -o module/zfs/libicp_la-zfs_impl.lo `test -f 'module/zfs/zfs_impl.c' || echo './'`module/zfs/zfs_impl.c
In file included from ./include/sys/zfs_context.h:131,
                 from ./include/sys/zio.h:39,
                 from ./include/sys/zio_checksum.h:32,
                 from module/zfs/zfs_impl.c:27:
./include/os/linux/zfs/sys/zfs_context_os.h:27:10: fatal error: linux/dcache_compat.h: No such file or directory
   27 | #include <linux/dcache_compat.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Probably the real issue is icp depending on zfs at all, but it is still a little shaky either way I think. I'll have a swing at it soon; maybe we'll just lift the headers we need to so we don't need the extra OS include in LIBZPOOL_CPPFLAGS.

[robn]

Lets see how that goes. Maybe the only missing piece was zfs_bootenv_os.h.

[robn]

And one more. This does fully decouple libicp from libpzool, which was the next sticking point that popped up.

(Still don't get why its only happening on Redhat flavours though).

[robn]

Aha, not "Redhat flavours"! Not exactly. On those, we build pkg-utils, which generates a distribution tarball and builds from that. I'd neglected to wire up lib/libzpool/include/Makefile.am, so those headers weren't available during build.

Here we go then! 🤞

[robn]

@behlendorf that might be it!

[behlendorf]

Yup, I think we're we're there for this round of changes!

I'd just like to get #17911 merged first to finalize the libzfs7 and libzpool7 ABI for the 2.4 release branch. Then we can get this PR merged and I'm hoping work on cleaning up these library interfaces and headers.

It may be aspirational, but at least for libzfs.so I'd love to get to the point where we don't have to bump the major version bump for each release. That seems very doable with some minor refactoring of the headers to not expose more than we need too. As for libzpool that's harder, but perhaps we could still provide a minimal useful stable interface. I'm not convinced we really want to limit ourselves in that way, but it's something to discuss.

[behlendorf]

Merged! Thanks for iterating with me on this Rob. As part of the merge I refreshed to top-commit to resolve the ABI file conflicts.

[robn]

@behlendorf thank you, this was fun! On to the next one!