General musings when working with the build toolchain of Bottlerocket

[mikn]

I have worked quite a bit over the weekend with your toolchain and whilst it is pretty impressive, it is also quite obvious it is still young.
I wrote these things down during my testing, mostly for my own sake, but I thought I should share, again as with previous discussions in the hopes that it may be useful to someone to do something with. :)

Thoughts:

• you need to run twoliter fetch before you can run twoliter build ... otherwise you get a very unintuitive message about external-kit-metadata.json missing.
• the archives downloaded for the packages end up in the respective packages dir - perhaps this is an RPM-ism, but it is a bit strange when there is also a build/ folder.
• the dependency on the "release" package is... not obvious, you get errors about /usr/lib/licenses and /usr/modules folders missing during the final stages of the build if you do not include it.
• Do you have any recommendations wrt a shared crypto library? Very impressively, you have managed to build the entire OS without one!
• You cannot currently create your own SDK whilst depending on bottlerocket's kits, even if it is just adding packages to the one that your kits depend on because of this check: https://github.com/bottlerocket-os/twoliter/blob/77bf67e44c472159e8ff9356ebeba688a2eca0ce/twoliter/src/project/lock/mod.rs#L310 (I need dracut in the image for kata-containers)

I will keep adding things to this discussion as I keep working on it.

[bcressey]

Thanks for the feedback! Your first three points are spot on. The third one especially - packages that are required for the build to succeed - is something I've been chipping away at intermittently over the last year, and need to get back to.

For the shared crypto library, I suggested trying AWS-LC in the other thread, so hopefully that ends up working out.

For the SDK, it is possible to redefine the source registry or image name of the SDK using a Twoliter.override file, e.g.:

[bottlerocket.bottlerocket-sdk]
name = "thar-be-a-beta-sdk"
registry = "public.ecr.aws/c2b5m1s5"

The version (tag) must match what the kits were built with, but otherwise it's pretty easy to replace an SDK with a slightly different one without having to rebuild the world each time.

(I'm also open to adding more tools to the SDK.)

[mikn]

Ah, that is very useful! Thanks!
I may take a look at trying to make a libcrypto out of AWS-LC. We are, ironically, using BoringSSL in other places, so I would've maybe preferred to try with that.

[mikn]

Another interesting snag is that if you decide to start up a new variant that is not prefixed by the variant prefixes you have established (aws, vmware, metal), there is no early-boot-config-providers being built or installed, which very unintuitively breaks loading user-data.toml when it is being placed into the /var/lib/bottlerocket/user-data.toml (i.e it just doesn't get loaded).
I did some code walking to understand how the early-boot-config packages actually gets inserted as dependencies, but I am no expert in RPM spec files nor your build system, so I must admit that I did not actually figure out how the requirement chain works here.
I however added the "early-boot-config-local" package on my variant list and it worked, so there is that. :D

[mikn]

So I read up a bit in Twoliter, specifically here: https://github.com/bottlerocket-os/twoliter/blob/1a7935b61d2aafa08d4894e2a57989a78867f282/tools/bottlerocket-variant/src/lib.rs#L52 and realised that the variant name is actually very meaningful across the build environment.
We will change our naming strategy to make sure that we stick with the metal-runtime-version-something-or-other-naming strategy.

[mikn]

Have you considered adding source RPMs to the builder? That would allow downstream packagers to just patch core packages with additional features instead of having to do a full fork. :)

Not sure where exactly they would be available, but the workflow would be something like:

# Define the source SRPM
Source0: wellknown/path/to/original-package-1.0-1.src.rpm

%prep
# Extract the source SRPM
rpm2cpio %{SOURCE0} | cpio -idmv

# Setup original sources
%setup -q -n original-package-%{version}

# Add your modifications (including renaming the build and install macros)
patch -p1 < %{PATCH0}

%include original-package-%{version}/original-package.spec

%build
%orig_build

# Add your custom build steps
make custom-target

Not sure how else to implement it 🤔

[cbgbt]

We've discussed this before a bit -- the two big concerns here were:

• Security patching surface area expansion: As an example, we know exactly which kernel modules we have enabled and this can make it much easier to reasonably assert "CVE X affects/does not affect Bottlerocket." Supporting source-redistribution re-introduces a lot of "dark" code that currently doesn't fit into that equation.
• Source-level compatibility guarantees -- Compatibility guarantees at the source level are much harder to reason about than at the binary level. Perhaps this would be "expected" by folks who do re-packaging, but it's something we'd need to consider if we were to approach source re-distribution in this way.

[mikn]

I think this is mostly an unavoidable problem in the monolithic source trees, such as systemd and the kernel itself.
I now ended up in a situation where I had to copy the entire release/ package also now, because we have other disk mount units that we need than what you had built into the release/ package.
We also ended up copying the entire kubernetes package to build more of kubernetes (we probably could've created a separate package with only the extra pieces we would like to build in this specific case however).
So, the packages we are maintaining ourselves from the core kit is now:

1. systemd
2. kubernetes
3. release

I understand that this is not a simple project decision. But I can speak as an upstream implementer and I would think anyone who does source distribution patching in their own variants would understand that you do not maintain CVE workflows for the (potentially) additional binaries that upstreams activate through patching build scripts.
I am however not super happy with the idea of having to carry literal patches and arcane spec file mappings (as I previously suggested) to activate a few extra binaries either.

I will try to suggest upstream updates as much as possible to these core packages, and as I did with the systemd package, try to package them as optional to allow anyone to activate these paths. But it does unavoidably leave you with more CVE surface.

[cbgbt]

Thanks very much for the feedback, I'll engage others on the Bottlerocket team to discuss this topic. Taking changes into core-kit is probably the best way to make progress in the meantime, and we really appreciate your efforts there!

[mikn]

So, I have this value:
bash-5.2# apiclient get settings.etcd.initial-cluster

{
  "settings": {
    "etcd": {
      "initial-cluster": "ct-78560717.local.molnett.host=https://192.168.37.10:2380"
    }
  }
}

That when rendered looks like this:

ETCD_INITIAL_CLUSTER='ct-78560717.local.molnett.host=https://192.168.37.10:2380'

I am rendering it like this:

{{#if settings.etcd.initial-cluster}}
ETCD_INITIAL_CLUSTER='{{settings.etcd.initial-cluster}}'
{{else}}

I googled the handlebars syntax and it turns out you need to do triple {{{ }}} to avoid escaping.