Make ZFS optional as a shared library

[wkozaczuk]

ZFS is one of the three filesystems that OSv implements and had been ported from FreeBSD. ZFS is one of the best file systems for persistent mutable data and what ones needs to run mysql or elasticsearch on OSv for example. Although a lot of effort has gone into integrating ZFS into OSv, its sheer code base volume makes OSv kernel bigger than it needs to be and there are plethora of stateless use cases like redis, memcached, miscroservices, etc where ZFS is not needed and instead we can use ROFS now. ZFS-less kernel is expected to be at least 1MB smaller, which should make OSv images smaller, use less memory and boot slightly faster. So it would be nice to make it optional as a shared library.

To that purpose I have created a branch where I simply commented ZFS-related code from 7 files (mostly in Makefile). There is already libzfs.so artifact that we could reuse and make target of all ZFS code.

There is obviously a question how OSv would exactly initialize ZFS. First of if ZFS optional, I am not sure, we would want to support pivoting to ZFZ (make ZFS partition root). I am not even sure, if it is possible. In that case OSv would either boot with ROFS or RAMFS and then optionally mount ZFS if found in fstab at /data or something. Obviously libzfs.so and any other related ZFS tools would have to be present on ROFS or RAMFS (provided we still support BOOTFS).

With this approach we would not have to duplicate ZFS tools (mkfs.so, etc) anymore in kernel (bootfs) and in ZFS partition and only load what is needed in memory. Given this approach would make OSv always depend on ROFS ((except ramfs images) it would be nice to make ROFS compress-able and possibly implement some memory utilization improvements.

[wkozaczuk]

Please note this is related to this closed issue - #195 - and one can find many interesting thoughts there.

[ClosingBracket]

Hi there! My name is Tony and I am a student studying computer science at The University of Texas at Austin. My partner and I are looking for a medium difficulty level issue to work on as part of our virtualization class. Could you tell us more about what will exactly need to be done, and whether you think we would be able to tackle this issue in the timespan of a semester?

[nyh]

Hi Tony! @wkozaczuk explained above what this issue is about, and #195 gave some context too, but perhaps not in a way approachable for beginners - so if you'd like to discuss the bigger picture - what is OSv, why OSv included ZFS, and there an interest to make OSv buildable without ZFS - or take ZFS out to become a dynamically-loadable library - I'll be happy to participate in such discussion on the OSv mailing list, osv-dev@googlegroups.com.
As for the scope of this project - I think it's relatively simple, not something you could do in a couple of hours, but also shouldn't take an entire semester. Obviously just learning what OSv is, how to use it, how to build it, and how to change anything in it, will already take some time. But after you do that and manage to do the first improvement to OSv (such as this issue), you may also decide it's fun and continue to do more improvements to it. You will see that seeing your improvements come to life is addictive :-)

[wkozaczuk]

Hi! The commit 4ffb0fa might give you an idea of how NFS was made a loadable library (please note /usr/lib/fs directory). ZFS is different in the sense, that OSv is a "bootable" (root) filesystem (like RAMFS or ROFS) and we might want to retain that capability of ZFS (but maybe not) as it takes the longest to boot from. So if you want to make ZFS a library then its .so files would have to be loaded from another root filesystem (probably ROFS makes the most sense). For starters, as a 1st step, I would suggest adding a conditional build flag like ZFS_ENABLE that would allow compiling ZFS code out (see one of my branches). Then we can try to make that a library - ideally one that does not require C++ standard library.

Lastly, I would like to say that this effort would be part of a broader "librarization"/modularization effort that I am trying to describe here - https://groups.google.com/g/osv-dev/c/BHwN6Stm3n4/m/Wp4smtklAAAJ and https://groups.google.com/g/osv-dev/c/JsJfnhTX_EY/m/yiVXFlWEAgAJ.

[ClosingBracket]

Thank you both @wkozaczuk and @nyh for the information! I believe we are ready to begin work on this pull request. I saw that you mentioned a branch that has the commented zfs code out, and you mention that the first step would most likely to be create a conditional build flag and wrap the ZFS code accordingly. Do you want us to work on the branch compile_zfs_out2 or compile_zfs_out branch?

[wkozaczuk]

@ClosingBracket, both branches are pretty old (2 years) and lots of things have changed since. Logically some of the changed places in the code still apply so you can still learn a lot from it.

Now, this is a fairly new very experimental branch where I played with many things to minimize kernel size, compiling ZFS being one of it. The ZFS-related commit is this - wkozaczuk@b088aaf. Please note that even since this commit many things have changed.

The most tricky will be core/pagecache.cc where you need to untangle ZFS part based on what you see in my commit. Also rather you commenting out things in the main makefile you will need to add some sort of if logic to it. Ideally one should be able to pass an option to ./scripts/build command to build a kernel with ZFS out (the default should be on).

Please note OSv build system uses configuration *.mk files used by main makefile:

ls conf/*
conf/aarch64.mk  conf/base.mk  conf/debug.mk  conf/release.mk  conf/x64.mk

It would be nice to follow the same pattern and mechanism for compiling ZFS on/out. See another commit - 8ef3841 - on how this could be done. There are similar other build options already used.

[ClosingBracket]

Could you elaborate as to how we could particularly work to make the ZFS_ENABLE flag a conditional build flag?

From our current understanding, it is possible to make some sort of a toggleable build flag by adding another parameter to ./scripts/build similar to --append-manifest or --create-disk. From there, we interpret to see whether that flag was present and if so pass it into the makefile with the appropriate 'vars[FLAG_NAME]="true". Then, finally, we do what we need in the makefile by checking for some sort of ifeq($(...), true) to include the appropriate parameters and providing an appropriate preprocessor define? Is this a correct understanding?

[wkozaczuk]

@ClosingBracket are you still working on this issue?