nix build generate static bpftrace,but still depend on other dynamic libraries when running

[baiwfg2]

Hi, I follow this tutorial : https://github.com/bpftrace/bpftrace/blob/master/docs/nix.md to build static bpftrace within docker container successfully.

$ ls -l
llrwxrwxrwx 1 builduser builduser     69 Dec  2 19:41 result -> /nix/store/8qxcrm5zx7ck53q4m05imkbv4377yam5-bpftrace-aarch64.AppImage
$file /nix/store/8qxcrm5zx7ck53q4m05imkbv4377yam5-bpftrace-aarch64.AppImage
/nix/store/8qxcrm5zx7ck53q4m05imkbv4377yam5-bpftrace-aarch64.AppImage: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped

I can run it inside the container. But I find that it still depend on many dynamic libraries such as :

$strace ./result --info |& egrep 'lzma|std'
openat(AT_FDCWD, "/nix/store/rhg9am19c4441prvkjwx04fzxq6w481n-bcc-0.30.0/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ipawp7l6xqybrhc745d7man4rn6xabd8-elfutils-0.191/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/2sgnmxla3sqqbflkzrf3828040ya0inm-libbpf-1.4.2/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/7jdg58lh5m9kijmyp2yxlfsfk2y4zj58-binutils-2.41-lib/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/mga994giknx03d5bl9vxj2nx7mpj13f6-clang-18.1.7-lib/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/0s8bwfldgxv9fsbr776kjwvd59dc058p-libpcap-1.10.4/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/krv3q032j5iwsb58j36isp88gsjqnz4b-lldb-18.1.7-lib/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/67zz1fxgn27y2726blaghi26vl6gd1ry-zlib-1.3.1/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/f1l61vql8qrhc2vhcsczpv8fzzgmnp74-llvm-18.1.7-lib/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/27fg1mkiymj2b344j80kygsbxfcdl5qi-glibc-2.39-52/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/66qm2gvcb3img7faxli0jv10dahg61qa-gcc-13.2.0-lib/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/nix/store/hi9f8lm3g3hy52drhxzjq4knbwwrlyji-bzip2-1.0.8/lib/libzstd.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/67zz1fxgn27y2726blaghi26vl6gd1ry-zlib-1.3.1/lib/libzstd.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/wv2wlkg19xis5rnpr1dhazd308w6dv0p-zstd-1.5.6/lib/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/nix/store/hi9f8lm3g3hy52drhxzjq4knbwwrlyji-bzip2-1.0.8/lib/liblzma.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/67zz1fxgn27y2726blaghi26vl6gd1ry-zlib-1.3.1/lib/liblzma.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/wv2wlkg19xis5rnpr1dhazd308w6dv0p-zstd-1.5.6/lib/liblzma.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/k0wiplr20nqdrk158vl4wzmy4cfx27w5-xz-5.4.6/lib/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/nix/store/wv2wlkg19xis5rnpr1dhazd308w6dv0p-zstd-1.5.6/lib/libssl.so.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/wv2wlkg19xis5rnpr1dhazd308w6dv0p-zstd-1.5.6/lib/libcrypto.so.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/wv2wlkg19xis5rnpr1dhazd308w6dv0p-zstd-1.5.6/lib/libbrotlidec.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

That makes it unusable when I move it outside the container, since there're no lzma.so, no higher version stdc++.so and so on.

I wonder whether it's static or not. I thought static binary can run anywhere.

[danobi]

Please try building from master. There was a bug with the appimages that was fixed but not released yet.

[baiwfg2]

Right, I try master and problem in NixOS/nixpkgs#329246 disappear. I met with other issue:

$./bpf.master/result -v tmp-pkg/a.bt
BTF: failed to find BTF data for vmlinux, errno 3
BTF: failed to find BTF data
BTF: failed to find BTF data for vmlinux, errno 3
BTF: failed to find BTF data
AST node count: 34
Attaching 3 probes...
ERROR: Error loading BPF program for uprobe__var_jenkins_home_tmp_pkg_lib_libgmpersistence_so_5_1_SeDatabaseInit_1.
Kernel error log:
back-edge from insn 1108 to 1088

WARNING: Kernel log seems to be trimmed. This may be due to buffer not being big enough, try increasing the BPFTRACE_LOG_SIZE environment variable beyond the current value of 1000000 bytes

ERROR: Loading BPF object(s) failed.

My kernel version is v4.19.36, I try to figure it out.

However, the problem in this thread still occur. I can't run it outside the container independently

[danobi]

Can you share the contents of your a.bt script? 4.19 kernel is quite old - we are dropping support for it soon. But it should probably work right now. In a few months we may start deleting support as upstream bpftrace supports up to the oldest LTS kernel in serivce. 4.19 goes out of service this month.

[baiwfg2]

It's just setting uprobe:

uprobe:/var/jenkins_home/tmp-pkg/lib/libgmpersistence.so.5.1:SeDatabaseInit
{
    printf("stacktrace:%s\n", ustack);
    printf("arg1: %p\n", arg1);
}

What I find is that the newest x86_64 appImage you built also depends many so file, we can see that from lsof -p <bpftrace.AppImage pid>. Where does those so come from ? Well, it does some weird mounting !

root@vm:/mnt# df
Filesystem        1K-blocks      Used Available Use% Mounted on
tmpfs                813244      1640    811604   1% /run
overlay           104805380  75670512  29134868  73% /mnt/docker-data/overlay2/3997f026bfd4deb2bce4bc9c487941da81df924a2ffc2684d35f7479cd3d41eb/merged
bpftrace.AppImage    219136    219136         0 100% /tmp/.mount_bpftraDmpaoI
tmpfs               4066212         0   4066212   0% /tmp/.mount_bpftraDmpaoI/mountroot
root@vm:/mnt# find /tmp/.mount_bpftraDmpaoI/ -name '*.so' | grep lzma  (lzma just for example)
/tmp/.mount_bpftraDmpaoI/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/lib-dynload/_lzma.cpython-311-x86_64-linux-gnu.so
/tmp/.mount_bpftraDmpaoI/nix/store/lphbn1va4i43fj7f3m9xskf9y86khzf3-xz-5.4.6/lib/liblzma.so

root@vm:/mnt# lsof -p 4067816  | grep 'lzma'
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
lsof: WARNING: can't stat() fuse.portal file system /run/user/1000/doc
      Output information may be incomplete.
bpftrace 4067816 root  mem       REG   0,64          3067811175 /tmp/.mount_bpftraDmpaoI/mountroot/nix/store/lphbn1va4i43fj7f3m9xskf9y86khzf3-xz-5.4.6/lib/liblzma.so.5.4.6 (stat: No such file or directory)

Why does a simple bpftrace statement like ./bpftrace.AppImage -e 'k:vfs_read { @++ }' result in this kind of mount ? I don't like this behavior , probably caused by nix build ! Very strange static binary !

[danobi]

Why does a simple bpftrace statement like ./bpftrace.AppImage -e 'k:vfs_read { @++ }' result in this kind of mount ? I don't like this behavior , probably caused by nix build ! Very strange static binary !

It's b/c it's an appimage. Appimage looks and behaves like a static binary. But under the hood it mounts a squashfs image with all of the application's dependencies. This allows us to ship a static-like build without actually doing static linking. Static linking when this many dependencies and LLVM involved is nearly impossible to maintain.

[danobi]

It's just setting uprobe:

uprobe:/var/jenkins_home/tmp-pkg/lib/libgmpersistence.so.5.1:SeDatabaseInit
{
printf("stacktrace:%s\n", ustack);
printf("arg1: %p\n", arg1);
}

Hmm, that's really odd. My guess is the combination of latest LLVM + really old kernel is causing some sort of bad interaction. If you can try on a newer kernel that would be a good data point.