get ebPF to work on Linux Kernel 6.2

[hhgsplk]

What happened?

Description

otel eBPF Kernel Driver is not getting compiled on Ubuntu 22.04.03 LTS with the latest Kernel Version

Steps to Reproduce

install Ubuntu 22.04.03 LTS and uprade to the the latest versions with apt-get dist-upgrade
try systemctl start kernel-collector

Expected Result

Actual Result via journalctl -u kernel-collector

Okt 06 19:40:37 snmpv8 bash[163510]: 2023-10-06 19:40:37.596040+02:00 info [p:163510 t:163510] Starting Kernel Collector version 0.10.0 (release)
Okt 06 19:40:37 snmpv8 bash[163510]: 2023-10-06 19:40:37.596049+02:00 info [p:163510 t:163510] Kernel Collector agent ID is FAIDYY4TNW86LK6IOU4HUHOE8WFS2USC710F
Okt 06 19:40:37 snmpv8 bash[163510]: 2023-10-06 19:40:37.596052+02:00 info [p:163510 t:163510] Running on:
Okt 06 19:40:37 snmpv8 bash[163510]: sysname: Linux
Okt 06 19:40:37 snmpv8 bash[163510]: nodename: snmpv8
Okt 06 19:40:37 snmpv8 bash[163510]: release: 6.2.0-34-generic
Okt 06 19:40:37 snmpv8 bash[163510]: version: #34~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 7 13:12:03 UTC 2
Okt 06 19:40:37 snmpv8 bash[163510]: machine: x86_64
Okt 06 19:40:37 snmpv8 bash[163510]: 2023-10-06 19:40:37.596071+02:00 info [p:163510 t:163510] HTTP Metrics: Enabled
Okt 06 19:40:37 snmpv8 bash[163510]: 2023-10-06 19:40:37.596072+02:00 info [p:163510 t:163510] Socket stats interval in seconds: 10
Okt 06 19:40:37 snmpv8 bash[163510]: 2023-10-06 19:40:37.596073+02:00 info [p:163510 t:163510] Userland TCP: Disabled
Okt 06 19:40:38 snmpv8 bash[163510]: 2023-10-06 19:40:38.607000+02:00 info [p:163510 t:163510] Kernel Collector version 0.10.0 (release) started on host snmpv8
Okt 06 19:40:38 snmpv8 bash[163510]: 2023-10-06 19:40:38.607303+02:00 info [p:163510 t:163510] No "labels" were specified.
Okt 06 19:40:51 snmpv8 bash[163510]: 2023-10-06 19:40:51.304489+02:00 info [p:163510 t:163510] connecting to 127.0.0.1:8000 (binary)...
Okt 06 19:40:55 snmpv8 bash[163510]: In file included from ../../../src/collector/kernel/bpf_src/render_bpf.c:39:
Okt 06 19:40:55 snmpv8 bash[163510]: In file included from include/net/tcp.h:35:
Okt 06 19:40:55 snmpv8 bash[163510]: In file included from include/net/sock_reuseport.h:5:
Okt 06 19:40:55 snmpv8 bash[163510]: In file included from include/linux/filter.h:9:
Okt 06 19:40:55 snmpv8 bash[163510]: include/linux/bpf.h:315:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_rb_root'
Okt 06 19:40:55 snmpv8 bash[163510]: return sizeof(struct bpf_rb_root);
Okt 06 19:40:55 snmpv8 bash[163510]: ^ ~~~~~~~~~~~~~~~~~~~~
Okt 06 19:40:55 snmpv8 bash[163510]: include/linux/bpf.h:315:24: note: forward declaration of 'struct bpf_rb_root'
Okt 06 19:40:55 snmpv8 bash[163510]: return sizeof(struct bpf_rb_root);
Okt 06 19:40:55 snmpv8 bash[163510]: ^
Okt 06 19:40:55 snmpv8 bash[163510]: include/linux/bpf.h:317:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_rb_node'
Okt 06 19:40:55 snmpv8 bash[163510]: return sizeof(struct bpf_rb_node);
Okt 06 19:40:55 snmpv8 bash[163510]: ^ ~~~~~~~~~~~~~~~~~~~~
Okt 06 19:40:55 snmpv8 bash[163510]: include/linux/bpf.h:317:24: note: forward declaration of 'struct bpf_rb_node'
Okt 06 19:40:55 snmpv8 bash[163510]: return sizeof(struct bpf_rb_node);
Okt 06 19:40:55 snmpv8 bash[163510]: ^
Okt 06 19:40:55 snmpv8 bash[163510]: include/linux/bpf.h:339:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_rb_root'
Okt 06 19:40:55 snmpv8 bash[163510]: return alignof(struct bpf_rb_root);
Okt 06 19:40:55 snmpv8 bash[163510]: ^ ~~~~~~~~~~~~~~~~~~~~
Okt 06 19:40:55 snmpv8 bash[163510]: include/linux/bpf.h:339:29: note: forward declaration of 'struct bpf_rb_root'
Okt 06 19:40:55 snmpv8 bash[163510]: return alignof(struct bpf_rb_root);
Okt 06 19:40:55 snmpv8 bash[163510]: ^
Okt 06 19:40:55 snmpv8 bash[163510]: include/linux/bpf.h:341:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_rb_node'
Okt 06 19:40:55 snmpv8 bash[163510]: return alignof(struct bpf_rb_node);
Okt 06 19:40:55 snmpv8 bash[163510]: ^ ~~~~~~~~~~~~~~~~~~~~
Okt 06 19:40:55 snmpv8 bash[163510]: include/linux/bpf.h:341:29: note: forward declaration of 'struct bpf_rb_node'
Okt 06 19:40:55 snmpv8 bash[163510]: return alignof(struct bpf_rb_node);
Okt 06 19:40:55 snmpv8 bash[163510]: ^
Okt 06 19:40:56 snmpv8 bash[163510]: 4 errors generated.
Okt 06 19:40:56 snmpv8 bash[163510]: 2023-10-06 19:40:56.341642+02:00 error [p:163510 t:163510] Cannot initialize BPF program, res=-1
Okt 06 19:40:56 snmpv8 bash[163510]: Failed to compile eBPF code for the Linux distro 'unknown' running kernel version 6.2.0-34-generic.
Okt 06 19:40:56 snmpv8 bash[163510]: troubleshoot item bpf_compilation_failed (os=Linux,flavor=unknown,headers_src=unknown,kernel=6.2.0-34-generic): ProbeHandler couldn't load BPFModule: Success
Okt 06 19:40:56 snmpv8 bash[163510]: This usually means that kernel headers weren't installed correctly.
Okt 06 19:40:56 snmpv8 bash[163510]: Please reach out to support and include this log in its entirety so we can diagnose and fix
Okt 06 19:40:56 snmpv8 bash[163510]: the problem.
Okt 06 19:40:56 snmpv8 bash[163510]: In the meantime, please install kernel headers manually on each host before running
Okt 06 19:40:56 snmpv8 bash[163510]: the Kernel Collector.
Okt 06 19:40:56 snmpv8 bash[163510]: To manually install kernel headers, follow the instructions below:
Okt 06 19:40:56 snmpv8 bash[163510]: - for Debian/Ubuntu based distros, run:
Okt 06 19:40:56 snmpv8 bash[163510]: sudo apt-get install --yes "linux-headers-uname -r"
Okt 06 19:40:56 snmpv8 bash[163510]: - for RedHat based distros like CentOS and Amazon Linux, run:
Okt 06 19:40:56 snmpv8 bash[163510]: sudo yum install -y "kernel-devel-uname -r"
Okt 06 19:40:56 snmpv8 bash[163510]: 2023-10-06 19:40:56.341808+02:00 error [p:163510 t:163510] Exception during BPFHandler initialization, closing connection: ProbeHandler couldn't load BPFModule: Success
Okt 06 19:40:56 snmpv8 bash[163510]: 2023-10-06 19:40:56.341823+02:00 error [p:163510 t:163510] troubleshoot item bpf_compilation_failed (os=Linux,flavor=unknown,headers_src=unknown,kernel=6.2.0-34-generic): ProbeHandler couldn't load BPFModule: Success

Btw: kernel headers are installed:
hgehrts@snmpv8:$ sudo apt-get install --yes "linux-headers-uname -r"
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
linux-headers-6.2.0-34-generic is already the newest version (6.2.0-34.3422.04.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

eBPF Collector version

0.10.0

Environment information

Environment

OS: Ubuntu 22.04.03 LTS , latest dist-upgrade with Kernel 6.2
Linux snmpv8 6.2.0-34-generic #34~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 7 13:12:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

eBPF Collector configuration

intake:
  host: 127.0.0.1
  port: 8000

Log output

see above

Additional context

No response

[hagen-p]

This affect the collector in k8s as well if the nodes are Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

The kernel collector will go in a CrashLoopback loop

[KnightChaser]

Still no updates at this issue? Not related to the OpenTelemetry, but had a similar issue that eBPF doesn't work in the Linux Kernel 6. :(

[yonch]

@jakub-racek-swi was this something you were hoping to tackle?