Socket tracer unable to start on 6.10 and later kernels #2035
Comments
|
After supplying one-off built kernel headers from #2036, this community user's ARM64 and x86 PEMs are still seeing BPF compilation issues. I ran one of the trace bpf tests in qemu with a 6.11.1 kernel and the new headers, and I'm able to reproduce the same error message that they have. observability__vizier-pem-dv96n__pem.log |
ddelnano
changed the title
|
I was able to track down the problem and upgrading bcc fixes the issue. BCC has certain "virtual" files it includes behind the scenes. The I need to double check that my rebasing of bcc's updated changes is correct, and update our fork (pixie-io/bcc) first, but I should be able to open a PR for this soon. |
…er kernels (#2041) Summary: Upgrade bcc and libbpf to fix BPF program compilation on 6.10 and later kernels Bcc provides some "[virtual](https://github.com/iovisor/bcc/blob/cb1ba20f4800f556dc940682ba7016c50bd0a3ac/src/cc/exported_files.cc#L28-L48)" includes to BPF programs. The `compat/linux/virtual_bpf.h` file in particular needs to be kept in sync with libbpf and matches the [header guard](https://github.com/iovisor/bcc/blob/cb1ba20f4800f556dc940682ba7016c50bd0a3ac/src/cc/compat/linux/virtual_bpf.h#L9) of the `include/uapi/linux/bpf.h` file. This means that while our linux headers were updated, our older bcc install was inserting an older copy of the `uapi/linux/bpf.h` file -- one that didn't contain the `bpf_wq` declaration. ``` include/linux/bpf.h:348:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_wq' return sizeof(struct bpf_wq); ^ ~~~~~~~~~~~~~~~ include/linux/bpf.h:348:24: note: forward declaration of 'struct bpf_wq' return sizeof(struct bpf_wq); ^ include/linux/bpf.h:377:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_wq' return __alignof__(struct bpf_wq); ^ ~~~~~~~~~~~~~~~ include/linux/bpf.h:377:29: note: forward declaration of 'struct bpf_wq' return __alignof__(struct bpf_wq); ``` Note: while this fixes the 6.10 compilation issue, our 6.10 qemu build fails without disabling [this logic](https://github.com/pixie-io/pixie/blob/3c41d554215528e688328aef94192e696db617dc/src/stirling/source_connectors/socket_tracer/socket_trace_connector.cc#L464-L472). 6.10 kernels added BPF token support. This changes the BPF permission model slightly and causes the BPF instruction limit to be dependent on the permissions of the BPF syscall caller ([linux source](https://elixir.bootlin.com/linux/v6.11.1/source/kernel/bpf/syscall.c#L2757)). This new BPF token logic coupled with our qemu setup, causes our 6.10 build to fallback to the 4096 instruction limit. I'll be addressing this in #2040 and #2042. Those issues shouldn't block this change since that loop limit code can be bypasses at runtime with our current cli flags. Relevant Issues: Closes #2035 Type of change: /kind bugfix Test Plan: Built 6.10 and 6.11 kernels and the associated linux headers from #2036 and verified that a local qemu build passes - [x] Verify `#ci:bpf-build-all-kernels` build passes Changelog Message: Upgraded bcc and libbpf to support kernels 6.10 and later --------- Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
Describe the bug
From working with someone in the community, I received a report that their OpenSUSE MicroOS instances were failing to start the socket tracer. Their PEMs fail to compile multiple BPF programs (as seen below). Their instances are running a 6.11 kernel while our latest kernel headers are 6.1.x.
I'm in the process of verifying that newer kernel headers resolves their problems, and if that's the case, our linux header kernels should be updated through 6.11.
Logs
pixie_logs_20241001120707.zip
The relevant logs from that PEM are the following:
App information (please complete the following information):
The text was updated successfully, but these errors were encountered: