Building custom variant using twoliter and kits?

[mikn]

Hi!

I have been keeping an eye on Bottlerocket for a couple of years now for use as a secure base, and with the recent introduction of kits, it seems very doable to build our own out-of-tree variants.
There are a few things that remain relatively unclear from browsing the repository and trying to build our own however;

1. The settings-* packages in the bottlerocket repository are very implicit in the actual result and how they are consumed in the build process. This makes it relatively difficult for me to assess how to both, a) what I actually need to implement in my own Twoliter.toml repo, and b) figure out how to override or include what I consider a "sane base-set" from the bottlerocket repository.
2. There are still a few (what seems like) critical sources in the bottlerocket sources/ folder, such as the api server, schnauzer, constants, migration-helpers, etc, which seems to be needed to be copied over verbatim as they are indirect dependencies from especially the settings-migrations/ package. What are the plans there? I tried a bit slimming down the dependencies, but building the metal-dev variant in my own tree without the settings-* packages (and sources) failed, and skipping the migrations also failed with it being seemingly required by twoliter itself.

For now I copied over the three packages (settings-defaults, settings-migrations and settings-plugins) verbatim, including the entire sources/ folder into my Twoliter workspace and I got something that seems like a working image, is this the right path given the current state of things or did I miss something major?

I understand if you do not want to provide support for OOTB with twoliter yet, but Bottlerocket is one of few (if not the only) distro that is closely compatible with the security posture we are looking to maintain, so we will probably keep tilting at the windmills either way. :)

[mikn]

Follow me on the journey of trying to package etcd!

My first attempt was to just naively follow how fedora packages it - I found the spec file through downloading the src.rpm file from rpmfind, but alas they seem to just woefully download the go dependencies over the network in the build step. Something Bottlerocket prevents rather forcefully in their rpmbuild step here: https://github.com/bottlerocket-os/twoliter/blob/77bf67e44c472159e8ff9356ebeba688a2eca0ce/twoliter/embedded/build.Dockerfile#L116

I started trying to understand how external package dependencies are assembled (such as for Go software) - but it seems that the external go projects you have (containerd and kubernetes) have vendored their dependencies, either in your own tar archive that you manage in your cache or by good practice.

After looking a bit more I also found host-ctr which has its sources in the sources/ folder of bottlerocket-core-kit and I found docker-go and the fetch-vendored Makefile.toml target which gave me the idea of putting in a dummy.go file and the go.mod from etcd itself to fetch the dependencies, this... sort of worked? But since the go toolchain is too clever, it doesn't actually fetch the dependencies if they are not included by a code file.

The issue seems to be a combination of the assumption of the Bottlerocket packaging system, that firstly, there is only one go.mod file in the root of the archive/source folder, and the go module layout that etcd has chosen (pre-workspace), where the target I want to build lives in etcd/server/ (the go.mod file) but, there are go.mod files in many places for the different binaries and libraries that etcd produces, and they all create a web of dependencies.

I have done a few experiments, such as adding each go.mod file in their own folder in my sources/ folder and including them under source-groups = [...], then copying the resulting vendor folder over to each location. Sadly none of my experiments actually worked.

Next stop was to just copy in the entire repository of etcd into the sources folder - this did not work either, as for some reason that I do not quite understand, I get this error:

[cargo-make] INFO - Running Task: fetch-vendored
pattern ./...: directory prefix . does not contain main module or its selected dependencies
Error while executing command, exit code: 1
Error: Command was unsuccessful, exit code 105

I tried running the same command on my local toolchain (go mod list -mod=readonly ./...) and it works just fine.

Now I am at an impasse of tinkering with the default build of twoliter however and I am not sure what to try next wrt getting etcd to build. I think I may disable the no networking during the build phase? As that is really the only thing I can think of that would resolve this problem on the short term.

Perhaps there is some valuable information here for a maintainer to extract to an actual action that would result in etcd being packaged without significantly worsening the security posture of Bottlerocket.

[bcressey]

The issue seems to be a combination of the assumption of the Bottlerocket packaging system, that firstly, there is only one go.mod file in the root of the archive/source folder, and the go module layout that etcd has chosen (pre-workspace), where the target I want to build lives in etcd/server/ (the go.mod file) but, there are go.mod files in many places for the different binaries and libraries that etcd produces, and they all create a web of dependencies.

You may find the soci-snapshotter package useful, both the metadata for its multiple bundle-modules passes over the same file, and the spec for the unpackaging dance.

I've opened an issue in Twoliter to better support projects with multiple go.mod files: bottlerocket-os/twoliter#418.

[mikn]

Oh, yes of course it was that late in the alphabet :D
Thank you, that was really unlocking.

[bcressey]

For now I copied over the three packages (settings-defaults, settings-migrations and settings-plugins) verbatim, including the entire sources/ folder into my Twoliter workspace and I got something that seems like a working image, is this the right path given the current state of things or did I miss something major?

That's the right path. Note that settings-migrations can be made into basically a no-op which removes a bunch of the other crates (schnauzer, etc) and makes it less of a rote copy/paste exercise overall.

settings-plugins defines the plugin (cdylib loaded at runtime by apiserver) that backs the settings API, so you can reuse (copy) one of the existing plugins unless or until you need to add settings of your own. settings-defaults is related to that - it's where you set default values for settings, along with various metadata on which services need to restart when those settings change, how to restart them, and which config files need to be rendered.

The long-term plan is still to replace the above with settings extensions which will move OOTBs the rest of the way toward a model where they are composed primarily through packages you define locally or install via kit dependencies. Settings extensions will be just another package you install, and the API server will find them and use them automatically.

I understand if you do not want to provide support for OOTB with twoliter yet, but Bottlerocket is one of few (if not the only) distro that is closely compatible with the security posture we are looking to maintain, so we will probably keep tilting at the windmills either way. :)

I've shared a project with you that might be a better reference point for getting started with an OOTB, since it dispenses with a lot of the legacy concerns.

Eventually there will be something like a twoliter init command that creates a project skeleton for you. It's not so much that we don't want to support OOTBs yet, it's more that the ramp-up experience is very rough and we'd like to finish the settings extensions work first.

[mikn]

Ah yes, okay - that makes sense.
I do need some settings for what I am building.

[mikn]

I am not sure which one I should mark as an answer, as they were both super helpful! I got etcd building with soci-snapshotter strategy.