Migrate Falco image from debian to distroless

[CashWilliams]

Motivation

To reduce attack surface and patching maintenance, we should use the most minimal base image for the Falco image as possible.

Currently debian:11-slim is used, however Google's distroless image would be more lightweight.

See https://github.com/GoogleContainerTools/distroless/blob/main/base/README.md

Feature

Migrate to distroless as the base image.

Alternatives

TBD

Additional context

A quick test showed that at least some libraries were missing from distroless. There are methods to add these libraries, but I'm not sure what all would need to be included.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[leogr]

/remove-lifecycle stale

any news?

[leogr]

cc @LucaGuerra

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[alacuku]

/remove-lifecycle stale

[LucaGuerra]

I think we can use this issue to discuss which image we want and why, and I really agree with the sentiment of using a light image.

The hard requirements:

• Needs to have core system libraries that are compatible with the binaries we ship for Falco. We try to build Falco as statically as possible but in order to run plugins at least a little bit of compatibility with glibc is necessary.
• Needs to work on both ARM and x86_64

Recently @dlorenc suggested to try the Wolfi image that he develops with his team at Chainguard. I think it's a good idea to try it out, also considering that he'll be able to answer our questions :)

[dlorenc]

Yes! Let's help out here! cc @amouat who should be able to jump in from our side when he gets back from vacation.

[hpvd]

Since traditional, this approach is somehow strenuous to implement and associated with restrictions...
there are 2 tools that seems to make it pretty easy and straight forward
Haven't tested it on my own, but it looks promising...

Maybe this is useful:

**good overview on distroless containers **
https://dev.to/dansiviter/distroless-alpine-ci8
and new approach:
https://blog.chainguard.dev/minimal-container-images-towards-a-more-secure-future/
=> see last paragraph for how it may work

the tools:
source to abk:
https://github.com/chainguard-dev/melange

abk to oci:
https://github.com/chainguard-dev/apko

to debug distroless containers:
official: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/
detail flow: https://iximiuz.com/en/posts/kubernetes-ephemeral-containers/

[MukulKolpe]

Hey, I'm new in the Falco community. Once all the requirements along with the base image are discussed and finalized, I would be really interested to work on this one.

[amouat]

Hi, I'm more than happy to help out here!

Is this the main Dockerfile you're looking to port?

falco/docker/no-driver/Dockerfile

Line 23 in 0b6e243

FROM debian:11-slim

I can have a look at porting that file as a start. Is there a simple test I can run to confirm things are working as expected?

I see there's a lot of builder stuff happening under the docker folder. I'm happy to take a look at anything else, but it would be good to have a chat first about your goals for the build and what you'd like to see.

[LucaGuerra]

Hey @amouat , thanks for looking into this!

So, as you can see, there are two images for Falco one is the no-driver and the other is the self-contained Falco / driver-loader images. The two are very different and this issue is mostly related to the no-driver image because it's so much easier to handle as you correctly identified.

I can have a look at porting that file as a start. Is there a simple test I can run to confirm things are working as expected?

The simplest thing you can do is running the main falco executable and verifying it runs. i.e. just run the falco-no-driver image and see that Falco start, possibly complaining about the missing kmod.

However, if you have BTF support in your kernel (e.g. Ubuntu 22.04) you can very easily run the entire thing from the lightweight image through the (experimental) modern eBPF probe:

docker run --rm -i -t \
           --privileged \
           -v /var/run/docker.sock:/host/var/run/docker.sock \
           -v /proc:/host/proc:ro \
           falcosecurity/falco-no-driver:latest falco --modern-bpf

This should start Falco with default rules. To check that it is working try spawning a new ubuntu container and running apt-get update from there, one of the default rules should inform you of that.

[amouat]

Great, thanks @LucaGuerra! I'll have a look this week.

@MukulKolpe I'll update progress in this thread so you're welcome to follow along & help test etc.

[MukulKolpe]

Sounds great, @amouat.

[amouat]

Apologies for the delay, I got thrown by KubeCon and a couple of other things.

@LucaGuerra the current Dockerfile uses the precompiled release binaries. I have got that working with our images, but there are potential issues, so I thought I'd take a look at compiling from the source release as well.

[amouat]

@LucaGuerra @MukulKolpe please take a look at the PR. It's still a WIP but I thought it would be worth getting some feedback.

[amouat]

I've updated the PR. I think it's pretty good to go now. We added some extra libs to one of our base images (glibc-dynamic) to support this and similar scenarios.

[LucaGuerra]

Thank you so much @amouat ! There is a ton of work for 0.35 already underway so I don't think it will fit this release, but even if it doesn't I would be happy to merge it afterwards after a bit of testing because it looks like a massive improvement to me!

[leogr]

Thank you so much @amouat ! There is a ton of work for 0.35 already underway so I don't think it will fit this release, but even if it doesn't I would be happy to merge it afterwards after a bit of testing because it looks like a massive improvement to me!

@LucaGuerra
I agree

@falcosecurity/core-maintainers
I believe this should be a top priority in our roadmap for Falco 0.36

@amouat
Btw, thank you for your amazing work! 🤩

[leogr]

/milestone 0.36.0

[amouat]

No problem, good luck with the release!

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[leogr]

/remove-lifecycle stale
cc @LucaGuerra

[amouat]

BTW I just rebased so the tests pass again.

[Andreagit97]

This should solve the issue #2768
I will close this, if there are other issues please re-open it :)
Thank you all!

[LucaGuerra]

To better explain what we did to @CashWilliams and others that may be following this issue:

We have decided not to fully migrate the no-driver image due to potential breaking changes, but we have introduced an experimental falco-distroless image which is based on Wolfi for us all to use.

This is now built in master and tagged and signed releases will be available starting with the upcoming Falco 0.36.0, planned for this month. If we don't see issues compared to the regular no-driver image we can discuss full migration.

[CashWilliams]

Thanks for the update. We will test moving to https://hub.docker.com/r/falcosecurity/falco-distroless and provide feedback.