Tracee doesn't work on CentOS 8

[NeonSludge]

Tracee doesn't seem to be compatible with the stock kernel on CentOS 8.

Steps to reproduce:

1. Get a machine with CentOS 8.2 or 8.3, install latest updates and use the stock kernel.
2. Install prerequisites for Tracee.
3. Download a binary release of Tracee or build it from source.
4. Run it with debugging enabled.
5. Observe a failure to build the BPF program that Tracee uses.

Here is the output of an attempt to run Tracee 0.3.0 on CentOS 8.3.2011 (Linux 4.18.0-240.1.1.el8_3.x86_64, clang 10.0.1). I've tried building Tracee from source (master) but it doesn't make any difference.

At the same time, BCC and bpftrace (latest versions built from source) work perfectly and do not produce any errors on the same machine.

[yanivagman]

Hi @NeonSludge ,
Thanks for reporting this issue!

It seems that redhat backported a commit from a more recent kernel to their 4.18 kernel.
In the mainline linux sources, task_struct had "pids" field until version 4.18, which was removed in kernel 4.19, where we use the thread_pid field instead.
https://elixir.bootlin.com/linux/v4.18.20/source/include/linux/sched.h#L778
https://elixir.bootlin.com/linux/v4.19.162/source/include/linux/sched.h#L782

We will need to figure out in which kernel version that changed for redhat (and thus for centos as well).
In the meanwhile, if you are compiling from source, you can try to workaround this issue by modifying the tracee.bpf.c code.
In lines 270 and 284 there is an #if that checks for the kernel version used.
You can just change the number 19 to 18 in these lines to look like:
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 18, 0)

Please let me know if that worked for you

[itaysk]

related: draios/sysdig#1650

[NeonSludge]

Please let me know if that worked for you

Yep, Tracee works with these changes in tracee.bpf.c:

diff --git a/tracee/tracee.bpf.c b/tracee/tracee.bpf.c
index 4685cb5..20ac93c 100644
--- a/tracee/tracee.bpf.c
+++ b/tracee/tracee.bpf.c
@@ -270,7 +270,7 @@ static __always_inline u32 get_task_ns_pid(struct task_struct *task)
 {
     unsigned int level = READ_KERN(READ_KERN(READ_KERN(task->nsproxy)->pid_ns_for_children)->level);

-#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0)
+#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 18, 0)
     // kernel 4.14-4.18:
     return READ_KERN(READ_KERN(task->pids[PIDTYPE_PID].pid)->numbers[level].nr);
 #else
@@ -284,7 +284,7 @@ static __always_inline u32 get_task_ns_tgid(struct task_struct *task)
     unsigned int level = READ_KERN(READ_KERN(READ_KERN(task->nsproxy)->pid_ns_for_children)->level);
     struct task_struct *group_leader = READ_KERN(task->group_leader);

-#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0)
+#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 18, 0)
     // kernel 4.14-4.18:
     return READ_KERN(READ_KERN(group_leader->pids[PIDTYPE_PID].pid)->numbers[level].nr);
 #else
@@ -298,7 +298,7 @@ static __always_inline u32 get_task_ns_ppid(struct task_struct *task)
     struct task_struct *real_parent = READ_KERN(task->real_parent);
     unsigned int level = READ_KERN(READ_KERN(READ_KERN(real_parent->nsproxy)->pid_ns_for_children)->level);

-#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0)
+#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 18, 0)
     // kernel 4.14-4.18:
     return READ_KERN(READ_KERN(real_parent->pids[PIDTYPE_PID].pid)->numbers[level].nr);
 #else

[yanivagman]

Thanks @NeonSludge

I just opened a PR to fix this issue, and I'll appreciate if you can test it on your machine and verify if it works:
#351

[NeonSludge]

Thanks @NeonSludge

I just opened a PR to fix this issue, and I'll appreciate if you can test it on your machine and verify if it works:
#351

I've compiled a version of Tracee from that PR on a CentOS 8.3 machine with the stock RedHat kernel and it works fine.

[NeonSludge]

Thank you for the quick fix! 👏