ROADMAP: Build slim and full Falco container images

[mfdii]

The current Falco images are quite large (720 MB) and carry lots of build tools that we want to remove from the current image. The build tools are required to compile kernel modules dynamically when the Falco container starts.

Under the new design we wish to remove the build tools and create two image categories:

• Init container image for Kernel Module/eBPF probe delivery
  • Container for building probe dynamically (current model)
  • Container for pulling module via HTTPS (falcosecurity/httploader)
  • Container for building & packaging custom kernel modules
  • Container(s) shipping prebuilt modules
• Falco container image containing the minimum required software
  • Minimal Image falcosecurity/falco-minimal - only required executables and libraries (~19.5mb)
  • Slim Image falcosecurity/falco-slim - Distribution based image (~224mb)

Other tasks before this issue is complete:

• Update kubernetes manifests to support new model
• Update helm chart (external PR)
• Automated build process for new container images
• Testing of images in deployment scenarios
  • GKE
  • Response Engine
  • IKS
  • Minikube

[JPLachance]

Greetings,

I discussed this with @mstemm and I would like to write down the outcome:

• Create a Kubernetes initContainer that runs the image that builds the kernel module
• Mount the built kernel module in the light Falco container

Separating build tools from the Falco agent will also remove vulnerabilities that those build tools adds in the current falcosecurity/falco Docker image. Because of those build tools, snyk test --docker falcosecurity/falco:latest currently finds 3 "High" vulnerabilities that we can get rid off :)

If you need some hands to develop this, poke me!

[mfdii]

@JPLachance Not sure if you have time to work on this, but your help would be appreciated.

I'd also like to create a mutli-stage Dockerfile that builds Falco in one stage and then installs it in the second stage. Not sure how much work that is but it would make it easier to test developer builds in minikube, etc.

[Issif]

@mfdii building the kernel module in one stage and use it in finale one is exactly what I'm doing in my Dockerfile for building a falco image which is compatible with Docker for Windows.

See : https://github.com/Issif/falco-docker4windows/blob/4.9.125/Dockerfile

The finale image is as big as official, but it can help for start.

[fntlnz]

I agree with @JPLachance - the idea seems the one we would likely need to implement.

He just wrote me on slack he is not sure whether to use Alpine or Ubuntu as base images.

Wdyt @mstemm @mfdii @leodido ?

Would we have problem with Musl for example? Let's just collect some opinions and make a decision on that so that we can go on and do this

[leodido]

I tend to prefer Alpine because it's smaller and usually faster.
Nevertheless the main point here is about having to use a port of glibc instead of musl libc.

[JPLachance]

Hello!

I don’t think a multi-stage Dockerfile is the solution. Building the kernel module needs to be done at boot of the Falco container when the kernel is unsupported (no builds provided). The multi-stage Dockerfile won’t help because building the kernel module would happen at the build of the container image 🙂

About Alpine VS Ubuntu 18.04, I think it will depend on where is Falco happier. Does Alpine offer all the libraries and packages Falco needs to run without crashing?

I'm sure we all agree we prefer something stable over a super slim image 😄

[Issif]

@JPLachance You're right.

I've bypassed the problem in Docker for Windows by this way : https://github.com/Issif/falco-docker4windows/blob/4.9.125/Dockerfile

I agree it's not native, but it works.

[JPLachance]

@Issif, it works to build one Falco container image for one specific version of the kernel. If we want to take that approach, we need to build one container image per kernel version we want to support. I'm sure we can do a script that generates and publish a ton of container images but is there where we want to go? 🙂

We could build two container images. One with all the required build tools, another one that is slim and only contains and run Falco. Falco users on systems for which the kernel module is pre-built will have the option to use the slim image. Other Falco users on systems for which we don't have a pre-built kernel module should then have three options:

• Use the eBPF probe (if supported by their system)
• Use the large image, build the kernel module on boot
• Use an init container if they are in Kubernetes, make the init container build the kernel module, mount it in the main Falco container through a volume (could be an emptyDir).

Sounds good?

[mstemm]

Yes, I agree with the approach of having an alternate set of container images. We take this approach with our commercial agent, where you can either install a single container that builds the kernel module and contains the agent, or run two containers--one builds the kernel module, and the other only runs the agent.

Alpine would be a good choice for a small image, but we'll need to rebuild falco as alpine linux uses a different libc and it's awkward to get glibc-linked binaries to run cleanly in alpine linux.

[michiels]

Just chipping in here from a possible new Falco/Sysdig user:

We are currently aiming at using Falco as IDS for our Kubernetes clusters on DigitalOcean. As the DigitalOcean K8s VM images do not include kernel headers by default, it is not possible for us to use the "on demand" probe build method inside your current container image.

Pending that you might start building the probe module for the DigitalOcean Kubernetes image based on Debian Stretch: We are now going to follow the strategy to base our own Docker image off your Docker image and include a pre-compiled probe module in that Docker image. So it would be very helpful to get the tiniest base image from your side so that we can simply drop in our precompiled via a very simple Dockerfile of our own.

It will also help if you keep your Docker base image clear of vulnerabilities, as your current Docker image reports various applicable CVEs when checking with Snyk.io:

[danbeaulieu]

+1 for a slim image.

There is a trend of moving away from Alpine linux for CNCF project images. While Alpine is small, motivation seems to be partly stability (alpine has shown issues with golang programs in the past) and partly "standardizing" which helps with supply chain auditing.

See this kep for an example: https://github.com/kubernetes/enhancements/blob/master/keps/sig-release/20190316-rebase-images-to-distroless.md

[antoniobeyah]

I would prefer to have an image that includes everything it needs and no modifications at boot time. We are moving towards requiring a readonly file system and no egress for everything and our major offenders preventing us from achieving this right now are our monitoring and security tools. We would prefer the deploys to hard fail if the image didn't work (ex: wasn't built properly w/ proper module) vs trying to build on demand. The fewer moving parts the better.

[JPLachance]

Hello!

A couple of weeks ago we talked about reworking the Falco container image to make it work like the commercial Sysdig Secure image: create an init container image that builds the kernel probe, loads it, then create a slim Falco image that uses the kernel probe and unloads it when exiting.

Where are we in this project? 😄

The list of known vulnerabilities in the Falco container image is only getting longer and longer. It's quite painful to analyze all those and make sure we are not vulnerable.

Thanks for the update!

[mfdii]

Yup. As @leodido and @fntlnz are finishing getting onboard with the project, we are beginning to do some release planning that includes this. Thanks for the patience.

[mfdii]

@JPLachance please look at #776 and add any comments. I'll get some images published using these docker files for testing.