Versioned CoreOS Linux container image to build kernel modules / eBPF programs

[alban]

I would like to be able to build kernel modules or eBPF programs that depends on kernel headers for specific versions of CoreOS Linux without having to instantiate a CoreOS Linux VM.

Capturing #589 (comment):

That would be nice if there was an official container image for each official version of CoreOS Linux that would include the kernel headers of the correct version and some compilers (I need gcc, llvm and clang). That would allow to compile kernel modules for a specific CoreOS Linux version using a free CI service such as CircleCI or SemaphoreCI, without having to deploy a CoreOS Linux VM in the CI pipeline (which is usually not possible in free CI services).

I would use that for building eBPF/kprobes programs that depend on a specific kernel headers version. My workflow would be something like this script in CircleCI:

curl https://coreos.com/releases/releases.json | parse_versions.sh > versions
for VERSION in $(cat versions) ; do
  docker pull quay.io/coreos/coreos-kernel-headers:${VERSION}
  docker run -v $PWD:/src \
        quay.io/coreos/coreos-kernel-headers:${VERSION} \
            gcc -o /src/foo-${VERSION} /src/foo.c
done

Then it would generate one compiled object for each CoreOS Linux version and I could publish all of them.

Since coreos/docs#915, there is documentation for Building custom kernel modules but this cannot be done in a container.

[crawford]

It should just be a matter of not binding /lib/modules. That's only done so the modules can be installed.

[alban]

Thanks for looking at this!

The file "coreos_developer_container.bin.bz2" is 3.1G after decompression. If I download several CoreOS versions to build a against several versions of the kernel headers, that might be too big for free CI tools like Semaphore/Circle. I'm wondering if a smaller tarball with the kernel sources & headers + another generic distribution container with llvm/clang would be better for my use case. But I have not checked what are the disk quotas there, maybe it just works.

Also, Semaphore/Circle supports running Docker containers, but I don't know about using systemd-nspawn as suggested by the current doc.

[alban]

We are experimenting with this code (work in progress):
https://github.com/kinvolk/tcptracer-bpf/pull/9/files#diff-500835e5ce5f997360abcb7f7faf4d49R16

This downloads the coreos image, mounts it somewhere just to copy the kernel headers. But the compilation is done in a debian container. In this way, no need to install clang/llvm in the coreos image, and no need to use systemd-nspawn.

/cc @schu @iaguis

[lorenz]

I'd like to shift the focus of this bug towards providing official CoreOS developer container images since these are by far the best way to build for CoreOS. The systemd-nspawn images are bad for automation and most people that have published kernel module builds for CoreOS build their own docker images from the official ones, which is duplicated effort.

I have a process documented (and internally automated) for my torcx-zfs project and @BugRoger has his own automated build. I could probably find more. Any chance CoreOS could provide official Docker images?

Size-wise (some concerns were noted two comments up) I can see that 3GB is not ideal, but honestly since it will only run for short periods of time as part of a CI process that is totally worth the simplicity of using the original build toolchain.

Ref coreos/torcx#93
/cc @lucab @bgilbert

[bgilbert]

Thank you for reporting this issue. Unfortunately, we don't think we'll end up addressing it in Container Linux.

As we recently announced, we're working on a successor to Container Linux, and we expect most major development to occur there instead. Meanwhile, Container Linux won't see many new features, but will still be fully maintained into 2020. Stay tuned for more details about that.