chore: add 4.0.0+driver support for kernel-5.10.0-0.bpo.15 x86_64 #1131

atmosx · 2023-05-11T10:32:44Z

This PR adds ebpf object file and kernel module driver support for debian kernel v5.10.0-0.bpo.15, architecture x86_64. Supported falco driver versions 2.0,3.0 and 4.0.

Upgrading to falco v0.34.1 in DigitalOcean Kubernetes managed service (DOKS) v1.22.13-do.0 requires 5.10.0-0.bpo.15-amd64 driver support for v4 driver version.

Error encountered during driver setup:

================ Cleaning phase ================

* Looking for a falco module locally (kernel 5.10.0-0.bpo.15-amd64)
* Filename 'falco_debian_5.10.0-0.bpo.15-amd64_1.ko' is composed of:
 - driver name: falco
 - target identifier: debian
 - kernel release: 5.10.0-0.bpo.15-amd64
 - kernel version: 1
* Trying to download a prebuilt falco module from https://download.falco.org/driver/4.0.0%2Bdriver/x86_64/falco_debian_5.10.0-0.bpo.15-amd64_1.ko
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco module
install: /usr/lib/gcc/x86_64-linux-gnu/5/
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"

Creating symlink /var/lib/dkms/falco/4.0.0+driver/source ->
                 /usr/src/falco-4.0.0+driver

DKMS: add completed.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc)
install: /usr/lib/gcc/x86_64-linux-gnu/5/
* Trying to dkms install falco module with GCC /usr/bin/gcc-5
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc-5)
install: /usr/lib/gcc/x86_64-linux-gnu/6/
* Trying to dkms install falco module with GCC /usr/bin/gcc-6
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc-6)
install: /usr/lib/gcc/x86_64-linux-gnu/8/
* Trying to dkms install falco module with GCC /usr/bin/gcc-8
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc-8)
* Trying to load a system falco module, if present
Consider compiling your own falco driver and loading it or getting in touch with the Falco community

Command used generate driver config:

make generate -e TARGET_DISTRO=debian -e TARGET_KERNEL=5.10.0-0.bpo.15-amd64_1 -e TARGET_ARCH=x86_64

poiana · 2023-05-11T10:32:53Z

Hi @atmosx. Thanks for your PR.

I'm waiting for a falcosecurity member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

poiana · 2023-05-11T10:33:45Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: atmosx
Once this PR has been reviewed and has the lgtm label, please assign zuc for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

FedeDP · 2023-05-11T13:14:35Z

Hi! I think this might be the same issue as falcosecurity/falco#2374.
Can you share your uname -a output?

@atmosx

FedeDP · 2023-05-11T13:15:19Z

Btw if that's the case, Falco 0.35 will ship the fix (falcosecurity/falco#2377) and i already provided a Falco-driver-loader 0.34.1 with the fix in my own docker repo to test: falcosecurity/falco#2377 (comment)

FedeDP · 2023-05-11T13:15:24Z

/hold

This PR adds ebpf object file and kernel module driver support for debian kernel v5.10.0-0.bpo.15, architecture `x86_64`. Supported falco driver versions 2.0,3.0 and 4.0. Upgrading to falco v0.34.1 in DigitalOcean Kubernetes managed service (DOKS) v1.22.13-do.0 requires `5.10.0-0.bpo.15-amd64` driver support for v4 driver version. Error encountered during driver setup: ``` ================ Cleaning phase ================ * Looking for a falco module locally (kernel 5.10.0-0.bpo.15-amd64) * Filename 'falco_debian_5.10.0-0.bpo.15-amd64_1.ko' is composed of: - driver name: falco - target identifier: debian - kernel release: 5.10.0-0.bpo.15-amd64 - kernel version: 1 * Trying to download a prebuilt falco module from https://download.falco.org/driver/4.0.0%2Bdriver/x86_64/falco_debian_5.10.0-0.bpo.15-amd64_1.ko curl: (22) The requested URL returned error: 404 Unable to find a prebuilt falco module install: /usr/lib/gcc/x86_64-linux-gnu/5/ * Trying to dkms install falco module with GCC /usr/bin/gcc DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" Creating symlink /var/lib/dkms/falco/4.0.0+driver/source -> /usr/src/falco-4.0.0+driver DKMS: add completed. * Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc) install: /usr/lib/gcc/x86_64-linux-gnu/5/ * Trying to dkms install falco module with GCC /usr/bin/gcc-5 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc-5) install: /usr/lib/gcc/x86_64-linux-gnu/6/ * Trying to dkms install falco module with GCC /usr/bin/gcc-6 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc-6) install: /usr/lib/gcc/x86_64-linux-gnu/8/ * Trying to dkms install falco module with GCC /usr/bin/gcc-8 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc-8) * Trying to load a system falco module, if present Consider compiling your own falco driver and loading it or getting in touch with the Falco community ``` Command used generate driver config: ``` make generate -e TARGET_DISTRO=debian -e TARGET_KERNEL=5.10.0-0.bpo.15-amd64_1 -e TARGET_ARCH=x86_64 ``` Signed-off-by: Panagiotis Atmatzidis <atma@convalesco.org>

atmosx · 2023-05-12T11:40:34Z

Hi! I think this might be the same issue as falcosecurity/falco#2374. Can you share your uname -a output?

@atmosx

Hello!

Thanks for the comment. Here is the output:

root@http-fxwfa:~# uname -a
Linux http-fxwfa 5.10.0-0.bpo.15-amd64 #1 SMP Debian 5.10.120-1~bpo10+1 (2022-06-13) x86_64 GNU/Linux

Hmm, looks like it's a related issue 🤔

UPDATE: testing your docker image, I'm having the same exact error:

> k logs falco-rhvpn -f -n falco -c falco-driver-loader

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.34.1, driver version=4.0.0+driver, arch=x86_64, kernel release=5.10.0-0.bpo.15-amd64, kernel version=1
* Running falco-driver-loader with: driver=module, compile=yes, download=yes

================ Cleaning phase ================

* 1. Check if kernel module 'falco' is still loaded:
- OK! There is no 'falco' module loaded.

* 2. Check all versions of kernel module 'falco' in dkms:
- OK! There are no 'falco' module versions in dkms.

[SUCCESS] Cleaning phase correctly terminated.

================ Cleaning phase ================

* Looking for a falco module locally (kernel 5.10.0-0.bpo.15-amd64)
* Filename 'falco_debian_5.10.0-0.bpo.15-amd64_1.ko' is composed of:
 - driver name: falco
 - target identifier: debian
 - kernel release: 5.10.0-0.bpo.15-amd64
 - kernel version: 1
* Trying to download a prebuilt falco module from https://download.falco.org/driver/4.0.0%2Bdriver/x86_64/falco_debian_5.10.0-0.bpo.15-amd64_1.ko
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco module
install: /usr/lib/gcc/x86_64-linux-gnu/5/
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"

Creating symlink /var/lib/dkms/falco/4.0.0+driver/source ->
                 /usr/src/falco-4.0.0+driver

DKMS: add completed.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc)
install: /usr/lib/gcc/x86_64-linux-gnu/5/
* Trying to dkms install falco module with GCC /usr/bin/gcc-5
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc-5)
install: /usr/lib/gcc/x86_64-linux-gnu/6/
* Trying to dkms install falco module with GCC /usr/bin/gcc-6
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc-6)
install: /usr/lib/gcc/x86_64-linux-gnu/8/
* Trying to dkms install falco module with GCC /usr/bin/gcc-8
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/4.0.0+driver/build/make.log (with GCC /usr/bin/gcc-8)
* Trying to load a system falco module, if present
Consider compiling your own falco driver and loading it or getting in touch with the Falco community

FedeDP · 2023-05-12T11:42:47Z

Yay! And we should have correct driver built: https://download.falco.org/driver/4.0.0%2Bdriver/x86_64/falco_debian_5.10.120-1-amd64_1.ko

Can you test using my falco-driver-loader image (linked in the comment i shared above?)
fededp/falco-driver-loader:0.34.1_fixed
Thank you!

atmosx · 2023-05-12T11:49:04Z

5.10.0-0.bpo.15-amd64

You're too fast :-P - see the "update".

FedeDP · 2023-05-12T11:55:40Z

Weird, are you sure you are actually running my falco-driver-loader image?
I just tried the exact same bash code being executed in the patched falco-driver-loader, and its output is 5.10.120-1-amd64 that matches exactly the driver we are hosting on download.falco.org previously linked!

Eg (see https://github.com/falcosecurity/falco/pull/2377/files):

#!/bin/bash

OS_ID="debian"
KERNEL_RELEASE="5.10.0-0.bpo.15-amd64"
KERNEL_V="5.10.120-1~bpo10+1"
TARGET_ID=$(echo "${OS_ID}" | tr '[:upper:]' '[:lower:]')
ARCH_extra=""
if [[ $KERNEL_RELEASE =~ -(amd64|arm64) ]];
then
    ARCH_extra="-${BASH_REMATCH[1]}"
fi
if [[ $KERNEL_V =~ ([0-9]+\.[0-9]+\.[0-9]+\-[0-9]+) ]];
then
    KERNEL_RELEASE="${BASH_REMATCH[1]}${ARCH_extra}"
fi

echo "${KERNEL_RELEASE}"

atmosx · 2023-05-12T12:09:46Z

Weird, are you sure you are actually running my falco-driver-loader image? I just tried the exact same bash code being executed in the patched falco-driver-loader, and its output is 5.10.120-1-amd64 that matches exactly the driver we are hosting on download.falco.org previously linked!

Eg (see https://github.com/falcosecurity/falco/pull/2377/files):
#!/bin/bash

OS_ID="debian"
KERNEL_RELEASE="5.10.0-0.bpo.15-amd64"
KERNEL_V="5.10.120-1~bpo10+1"
TARGET_ID=$(echo "${OS_ID}" | tr '[:upper:]' '[:lower:]')
ARCH_extra=""
if [[ $KERNEL_RELEASE =~ -(amd64|arm64) ]];
then
    ARCH_extra="-${BASH_REMATCH[1]}"
fi
if [[ $KERNEL_V =~ ([0-9]+\.[0-9]+\.[0-9]+\-[0-9]+) ]];
then
    KERNEL_RELEASE="${BASH_REMATCH[1]}${ARCH_extra}"
fi

echo "${KERNEL_RELEASE}"

I made a mistake earlier. Your image works for me as expected, the init container loads the module:

 k logs falco-s2m4h -n falco -c falco-driver-loader
* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.34.1, driver version=4.0.0+driver, arch=x86_64, kernel release=5.10.0-0.bpo.15-amd64, kernel version=1
* Running falco-driver-loader with: driver=module, compile=yes, download=yes

================ Cleaning phase ================

* 1. Check if kernel module 'falco' is still loaded:
- Kernel module 'falco' is still loaded.
- Trying to unload it with 'rmmod falco'...
- OK! Unloading 'falco' module succeeded.

* 2. Check all versions of kernel module 'falco' in dkms:
- OK! There are no 'falco' module versions in dkms.

[SUCCESS] Cleaning phase correctly terminated.

================ Cleaning phase ================

* Looking for a falco module locally (kernel 5.10.120-1-amd64)
* Filename 'falco_debian_5.10.120-1-amd64_1.ko' is composed of:
 - driver name: falco
 - target identifier: debian
 - kernel release: 5.10.120-1-amd64
 - kernel version: 1
* Trying to download a prebuilt falco module from https://download.falco.org/driver/4.0.0%2Bdriver/x86_64/falco_debian_5.10.120-1-amd64_1.ko
* Download succeeded
* Success: falco module found and inserted

However running falco fails 🤔 :

> k logs falco-s2m4h -n falco -c falco

* Setting up /usr/src links from host
Unknown driver: /usr/bin/falco

FedeDP · 2023-05-12T12:22:19Z

Unknown driver: /usr/bin/falco

😆
How are you running/deploying Falco?
The only reference i see is: https://github.com/falcosecurity/falco/blob/master/scripts/falco-driver-loader#L740

atmosx · 2023-05-12T12:40:49Z

Unknown driver: /usr/bin/falco

😆 How are you running/deploying Falco? The only reference i see is: https://github.com/falcosecurity/falco/blob/master/scripts/falco-driver-loader#L740

I'm producing a YAML definition from the helm chart. This is a configuration issue, probably some options missing there, we can consider this problem solved :-)

My understanding is that to proceed right now I have two choices:

Use the linked module
Use your container

The fix will ship with falco v0.35 ?

FedeDP · 2023-05-12T13:20:20Z

Use the linked module

Well that's not really an option unless you want to manually load it on all your nodes and then skip the falco-driver-loader stage :D

Use your container

Yep, i know it's not ideal. In any case, please remember to switch back to falcosecurity one once Falco 0.35 is released :)

The fix will ship with falco v0.35 ?

Yes!

You can also try the modern bpf probe driver instead! It does not require any external artifact since it leverages modern CO-RE approach!
See https://github.com/falcosecurity/charts/tree/master/falco#deploying-falco-in-kubernetes.

EDIT: a note about modern-bpf; it is experimental as of Falco 0.34; but will be official since Falco 0.35!

FedeDP · 2023-05-12T13:23:32Z

we can consider this problem solved

Yay! I am glad we were able to test my patch once more; i don't know when and why debian started doing that kernel release ABI kernel API thing, but hopefully all issues will be solved by Falco 0.35!

Feel free to close this one!

poiana added dco-signoff: no needs-ok-to-test labels May 11, 2023

poiana added the size/S label May 11, 2023

poiana requested review from jonahjon and zuc May 11, 2023 10:33

atmosx force-pushed the master branch from 9ba76a0 to bbf8b8a Compare May 11, 2023 11:01

poiana added dco-signoff: yes and removed dco-signoff: no labels May 11, 2023

poiana added the do-not-merge/hold label May 11, 2023

atmosx force-pushed the master branch from bbf8b8a to f3e8303 Compare May 11, 2023 13:27

atmosx closed this May 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add 4.0.0+driver support for kernel-5.10.0-0.bpo.15 x86_64 #1131

chore: add 4.0.0+driver support for kernel-5.10.0-0.bpo.15 x86_64 #1131

atmosx commented May 11, 2023

poiana commented May 11, 2023

poiana commented May 11, 2023

FedeDP commented May 11, 2023 •

edited

Loading

FedeDP commented May 11, 2023

FedeDP commented May 11, 2023

atmosx commented May 12, 2023 •

edited

Loading

FedeDP commented May 12, 2023 •

edited

Loading

atmosx commented May 12, 2023

FedeDP commented May 12, 2023 •

edited

Loading

atmosx commented May 12, 2023 •

edited

Loading

FedeDP commented May 12, 2023

atmosx commented May 12, 2023

FedeDP commented May 12, 2023 •

edited

Loading

FedeDP commented May 12, 2023 •

edited

Loading

chore: add 4.0.0+driver support for kernel-5.10.0-0.bpo.15 x86_64 #1131

chore: add 4.0.0+driver support for kernel-5.10.0-0.bpo.15 x86_64 #1131

Conversation

atmosx commented May 11, 2023

poiana commented May 11, 2023

poiana commented May 11, 2023

FedeDP commented May 11, 2023 • edited Loading

FedeDP commented May 11, 2023

FedeDP commented May 11, 2023

atmosx commented May 12, 2023 • edited Loading

FedeDP commented May 12, 2023 • edited Loading

atmosx commented May 12, 2023

FedeDP commented May 12, 2023 • edited Loading

atmosx commented May 12, 2023 • edited Loading

FedeDP commented May 12, 2023

atmosx commented May 12, 2023

FedeDP commented May 12, 2023 • edited Loading

FedeDP commented May 12, 2023 • edited Loading

FedeDP commented May 11, 2023 •

edited

Loading

atmosx commented May 12, 2023 •

edited

Loading

FedeDP commented May 12, 2023 •

edited

Loading

FedeDP commented May 12, 2023 •

edited

Loading

atmosx commented May 12, 2023 •

edited

Loading

FedeDP commented May 12, 2023 •

edited

Loading

FedeDP commented May 12, 2023 •

edited

Loading