tcplife not working on Ubuntu 19.10

[mwuertinger]

Running tcplife-bpfcc on Ubuntu 19.10 results in the following error:

# tcplife-bpfcc
In file included from /virtual/main.c:4:
In file included from include/linux/tcp.h:19:
In file included from include/net/sock.h:46:
In file included from include/linux/netdevice.h:42:
In file included from include/net/netprio_cgroup.h:11:
In file included from include/linux/cgroup.h:28:
In file included from include/linux/cgroup-defs.h:22:
In file included from include/linux/bpf-cgroup.h:5:
include/linux/bpf.h:115:3: error: variable has incomplete type 'struct bpf_spin_lock'
                (struct bpf_spin_lock){};
                ^~~~~~~~~~~~~~~~~~~~~~~~
[...]
49 warnings and 12 errors generated.
Traceback (most recent call last):
  File "/usr/sbin/tcplife-bpfcc", line 486, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 320, in __init__
    raise Exception("Failed to compile BPF text")
Exception: Failed to compile BPF text

This can be reproduced on a freshly installed Ubuntu 19.10 by installing the bpfcc-tools package version 0.8.0-4. Running other TCP related tools results in similar error messages.

$ uname -a
Linux pc-540 5.3.0-24-generic #26-Ubuntu SMP Thu Nov 14 01:33:18 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[bodgergely]

Can you show the output of clang -v as well?

[mwuertinger]

$ /usr/lib/llvm-7/bin/clang -v
clang version 7.0.1-9build1 (tags/RELEASE_701/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-7/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64

[yonghong-song]

The reason is kernel internal headers are more advanced then bcc internal uapi bpf.h headers. You have 5.3 kernel and its kernel-devel headers trying to access uapi bpf.h header which is provided by bcc a little bit old.

The solution is to use newer bcc versions.

[mwuertinger]

Hmm, I just installed the packages that are included in Ubuntu. Is this maybe a bug in the Ubuntu packages or am I doing something wrong here?

[yonghong-song]

It just the package is not up-to-date. Not sure how to push ubuntu to update packages. cc @drzaeus77

[mwuertinger]

Turns out this is a known Ubuntu issue for quite a while:
https://bugs.launchpad.net/ubuntu/+source/bpfcc/+bug/1848137

Since the packages are currently partially broken for Ubuntu 19.10 I'm wondering whether a warning should be added to INSTALL.md?

[dailyherold]

Marked the launchpad bug as affecting me, encourage others to do same if you're running 19.10.

Given the packaging issue, I attempted to build from source (HEAD of master) following documentation. However, regardless of running (e.g. tcplife) via built tools, or via snap package, or via quick start docker image, I get the same error:

❯ sudo /usr/share/bcc/tools/tcplife
could not open bpf map: whoami, error: Operation not permitted
Traceback (most recent call last):
  File "/usr/share/bcc/tools/tcplife", line 456, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 349, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

Some system information:

❯ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=19.10
DISTRIB_CODENAME=eoan
DISTRIB_DESCRIPTION="Ubuntu 19.10"
❯ uname -r
5.3.0-29-generic
❯ clang -v
clang version 9.0.0-2 (tags/RELEASE_900/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/9
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64

Also checked out v0.12.0 tag and tried to rebuild but received some make errors during Linking C executable bps step:

Scanning dependencies of target bps
[ 37%] Building C object introspection/CMakeFiles/bps.dir/bps.c.o
[ 38%] Linking C executable bps
/usr/bin/ld: ../src/cc/libbcc_bpf.a(libbpf.c.o): undefined reference to symbol 'gzgets'
/usr/bin/ld: /lib/x86_64-linux-gnu/libz.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [introspection/CMakeFiles/bps.dir/build.make:85: introspection/bps] Error 1
make[1]: *** [CMakeFiles/Makefile2:726: introspection/CMakeFiles/bps.dir/all] Error 2
make: *** [Makefile:141: all] Error 2

Anything else I can provide to help out? I can open a separate issue but figured I'd start here since this is a fresh 19.10 install.

Updates

• dmesg showed that apparmor might be interfering however stopping the service didn't remedy.

[yonghong-song]

Not sure why gzgets symbol is missing. But if you want to continue, you may tentatively remove introspection from toplevel CMakefile to see how far you can go.

[mwuertinger]

@dailyherold Could you please also post the output of dmesg? I recently had a similar issue and it was caused by kernel lockdown.

[dailyherold]

@mwuertinger looks like dmesg errors only came up when trying to use the snap package. Building off of 0a7da74 (to keep things consistent with last night) I didn't see any movement while watching dmesg.

@yonghong-song moving to v0.12.0 to repeat last night's test, I think I removed introspection properly:

--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -98,7 +98,6 @@ set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wall")
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall ${CXX_ISYSTEM_DIRS}")

 add_subdirectory(src)
-add_subdirectory(introspection)
 if(ENABLE_CLANG_JIT)
 add_subdirectory(examples)
 if(ENABLE_MAN)

Was able to compile without issue, but running still results in same error:

❯ sudo /usr/share/bcc/tools/tcplife
could not open bpf map: whoami, error: Operation not permitted
Traceback (most recent call last):
  File "/usr/share/bcc/tools/tcplife", line 456, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 347, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

Also pulled latest from master to see if any difference, but unfortunately not and still same error.

I did build from source (HEAD of master) on my 18.04 machine and was able to run without issue. So has to be something odd with package/deps/libs combo in 19.10. Let me know how I can be of more assistance in debugging!

[dailyherold]

@mwuertinger, yup you were right. Turns out it was lockdown mode. When I first read your post I was visualizing the apparmor dmesg events I was getting with the snap package versions. And since I wasn't getting any dmesg events when running my compiled binaries, I kept trying other stuff. I also didn't I know what lockdown mode was until...

I found bpftrace/bpftrace#853, and after disabling, my issues were resolved. Both bcc and bpftrace now working for me without problem. Subscribed to both #2565 and bpftrace/bpftrace#925 because I think these may start affecting all people who have a fresh 19.10 install. And with the BPF Performance Tools book being recent, I bet we'll see more people confused.

[McSim85]

seems like relevant to #4758