Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make ebpf exporter to support CORE #130

Merged
merged 36 commits into from
Oct 18, 2022

Conversation

wenlxie
Copy link
Contributor

@wenlxie wenlxie commented Dec 28, 2021

This PR is for ebpf exporter to support CORE.
This can make ebpf exporter to get CORE benefits and tolerate kernel changes.

I'd believe that there are more things need to do with this so please share your suggestions. Thanks.

@wenlxie
Copy link
Contributor Author

wenlxie commented Dec 28, 2021

@bobrik Can you help to take a look? thanks.

@bobrik
Copy link
Contributor

bobrik commented Dec 28, 2021

There are lots of changes here and many of them are unrelated:

  • Removed comments that still apply (starting with Dockerfile)
  • Inconsistent indentation in C files
  • Either missing or multiple newlines at the end of files
  • Broken Go formatting, even on the lines you didn't really change
  • Error strings starting with capital letters (only log lines should)
  • Functions like tableValues moved around and generating large diff with no logical changes

Could you take care of this first to make it easier to look at the actual changes?

There are SPDX headers for LGPL, I'll need to ask legal people whether we can accept it.

I would also like to see some documentation in the README on how to use this.

@wenlxie
Copy link
Contributor Author

wenlxie commented Dec 29, 2021

@bobrik Thanks for the suggestion. I will take care of them.
After move to CORE mode, BCC related will be removed, so I raised this PR for your to review first. Move from BCC to CORE is a right direction, but need your agreement with this.
Thanks again.

@wenlxie wenlxie force-pushed the upstream.master.core branch 4 times, most recently from ad9fcd3 to 2347e3c Compare December 29, 2021 07:49
@wenlxie
Copy link
Contributor Author

wenlxie commented Dec 29, 2021

How to run ebpf_exporter with CORE mode?

  1. Run make container generate the image directly, the default image is ebpf-exporter-build:lastest
  2. Run the image ebpf-exporter-build:lastest with cmd like:

docker run -d --net=host --privileged -v /sys/kernel/debug:/sys/kernel/debug ebpf-exporter:latest

@bobrik

Copy link
Contributor

@bobrik bobrik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, I was not able to build this either with the Dockerfile or in a Debian Bullseye VM with libbpf from backports. Happy to have another look when I can build and run it myself.

Dockerfile Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
examples/CORE/shrinklat.yaml Outdated Show resolved Hide resolved
exporter/exporter.go Show resolved Hide resolved
exporter/exporter.go Outdated Show resolved Hide resolved
exporter/exporter.go Show resolved Hide resolved
exporter/exporter.go Show resolved Hide resolved
@wenlxie wenlxie force-pushed the upstream.master.core branch 3 times, most recently from 8c0e953 to d66bcf3 Compare January 4, 2022 12:02
@wenlxie
Copy link
Contributor Author

wenlxie commented Jan 4, 2022

@bobrik I addressed some comments today but not finished all of these, will continue to do this tomorrow.
Thanks . The image (#130 (comment) ) runs well on my workstation, you can have a try when you have time.

Dockerfile Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
examples/CORE/biolatency.bpf.c Outdated Show resolved Hide resolved
examples/CORE/config.yaml Outdated Show resolved Hide resolved
exporter/exporter.go Show resolved Hide resolved
go.mod Outdated Show resolved Hide resolved
@rob-scheepens
Copy link

rob-scheepens commented Feb 14, 2022

Due to not wanting to install Docker on all machines I intend to run ebpf_exporter on I tried to build on Bullseye vm as well. When running "./scripts/build-deb.sh release" I got:

dpkg-checkbuilddeps: error: Unmet build dependencies: python-netaddr

When trying to install python-netaddr:

root@rob-bullseye:~/bcc# apt-get install -y python-netaddr
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package python-netaddr is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  python3-netaddr

E: Package 'python-netaddr' has no installation candidate

Installing python3-netaddr works fine on Bullseye, but that doesn't address the dependency issue which comes from ./debian/control:

root@rob-bullseye:~/bcc# cat debian/control | grep netaddr
    python (>= 2.7), python-netaddr, python-pyroute2 | python3-pyroute2, luajit,

I tried changing to python3-netaddr, but still get the same error.

@bobrik
Copy link
Contributor

bobrik commented Feb 15, 2022

@rob-scheepens, I think bcc build script checks out HEAD, so you need to commit your change to apply it.

Not sure how it's related to CO-RE.

@rob-scheepens
Copy link

rob-scheepens commented Feb 16, 2022

@bobrik : Using the Dockerfile from https://github.com/wenlxie/ebpf_exporter/blob/upstream.master.core/Dockerfile I attempted to convert this to commands that can be scripted and run on a Debian Bullseye vm. All commands for now assume the user is root.

export PATH="/usr/lib/go-1.17/bin:$PATH"
export DEBIAN_FRONTEND=noninteractive
export LD_LIBRARY_PATH=/lib64

echo 'deb http://deb.debian.org/debian bullseye-backports main contrib non-free
deb-src http://deb.debian.org/debian bullseye-backports main contrib non-free' >> /etc/apt/sources.list.d/backports.list

sed -i "s/bullseye main/bullseye main non-free/" /etc/apt/sources.list

apt-get update

apt-get --no-install-recommends install -y sudo devscripts build-essential fakeroot pbuilder aptitude git openssh-client ca-certificates git pkg-config libelf-dev libz-dev software-properties-common clang gnupg python3-netaddr pkg-config libz-dev software-properties-common bpftool python2 python3 dh-python libclang-9-dev libclang-common-9-dev libclang-cpp9 libclang1-9 libllvm9 llvm-9 llvm-9-dev llvm-9-runtime llvm-9-tools netperf python3-pyroute2 clang-format bison flex libfl-dev libedit-dev arping iperf ethtool python3 python3-doc python3-tk python3-venv python3.9-venv python3.9-doc cmake luajit libluajit-5.1-dev debhelper
apt-get install -y -t bullseye-backports golang-1.17 libbpfcc-dev libbpfcc bpfcc-tools

git clone --branch=v0.22.0 --depth=1 https://github.com/iovisor/bcc.git /root/bcc && \
git -C /root/bcc submodule update --init --recursive

git clone https://github.com/libbpf/libbpf.git /root/libbpf && \
cd /root/libbpf && git checkout 030ff87857090ae5c9d74859042d05bfb3b613a2 && cd src && mkdir build root && BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=root make install

At this point a modification to the debian/control file is needed, see rob-scheepens/bcc@6335995 and my previous commit on that file, changing python2 dependencies as these do not work on Bullseye anymore. Script to make the changes and commit them (since the build-deb.sh script checks out HEAD):

cd /root/bcc/
git config --global user.email "user@[email.com](http://email.com/)"
git config --global [user.name](http://user.name/) "nutanix"
sed -i "s/python (>= 2.7)/python3/" debian/control
sed -i "s/python-netaddr/python3-netaddr/" debian/control
git add debian/control
git commit -m "python3 changes."
./scripts/build-deb.sh release

git clone https://github.com/wenlxie/ebpf_exporter.git /root/ebpf_exporter
cd /root/ebpf_exporter
git checkout upstream.master.core

This gets me up to the point where I run into the issue described in aquasecurity/libbpfgo#1, where "go install" failed. Closer inspection learned that the -I include path in CFLAGS did not exist. Changing it to /root/libbpf/src/ made things progress more, up to the point where it fails on vmlinux.h missing:

    -X github.com/prometheus/common/version.Version=$(git describe) \
    -X github.com/prometheus/common/version.Branch=$(git rev-parse --abbrev-ref HEAD) \
    -X github.com/prometheus/common/version.Revision=$(git rev-parse --short HEAD) \
    -X github.com/prometheus/common/version.BuildUser=docker@$(hostname) \
    -X github.com/prometheus/common/version.BuildDate=$(date --iso-8601=seconds) \
    " ./cmd/ebpf_exporter
cd /root/ebpf_exporter/examples/CORE && make all
github.com/aquasecurity/libbpfgo
# github.com/aquasecurity/libbpfgo
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1153:37: could not determine kind of name for C.BPF_TC_CUSTOM
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1151:37: could not determine kind of name for C.BPF_TC_EGRESS
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1159:26: could not determine kind of name for C.BPF_TC_F_REPLACE
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1150:37: could not determine kind of name for C.BPF_TC_INGRESS
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1256:9: could not determine kind of name for C.bpf_tc_attach
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1267:9: could not determine kind of name for C.bpf_tc_detach
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1237:9: could not determine kind of name for C.bpf_tc_hook_create
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1246:9: could not determine kind of name for C.bpf_tc_hook_destroy
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1278:9: could not determine kind of name for C.bpf_tc_query
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1202:12: could not determine kind of name for C.sizeof_struct_bpf_tc_hook
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1179:12: could not determine kind of name for C.sizeof_struct_bpf_tc_opts
rm -f *.o
rm -rf config
clang -g -O2 -I"/tmp/libbpf-tools/.output" -I "/tmp/libbpf-tools/x86" -c   -target bpf -D__TARGET_ARCH_x86 biolatency.bpf.c -o biolatency.bpf.o
biolatency.bpf.c:2:10: fatal error: 'vmlinux.h' file not found
#include <vmlinux.h>
         ^~~~~~~~~~~
1 error generated.
make: *** [Makefile:16: biolatency.bpf.o] Error 1

Using bpftool I generated vmlinux.h and put it in /tmp/libbpf-tools/ (is that correct?). Then I still got the C.BPF* errors as seen above:

root@rob-bullseye:~# cd /root/ebpf_exporter && CGO_CFLAGS="-I /root/libbpf/src/" CGO_LDFLAGS="-lbpf" GOPATH="" GOPROXY="off" GOFLAGS="-mod=vendor" go install -v -ldflags=" \
    -X github.com/prometheus/common/version.Version=$(git describe) \
    -X github.com/prometheus/common/version.Branch=$(git rev-parse --abbrev-ref HEAD) \
    -X github.com/prometheus/common/version.Revision=$(git rev-parse --short HEAD) \
    -X github.com/prometheus/common/version.BuildUser=docker@$(hostname) \
    -X github.com/prometheus/common/version.BuildDate=$(date --iso-8601=seconds) \
    " ./cmd/ebpf_exporter
github.com/aquasecurity/libbpfgo
# github.com/aquasecurity/libbpfgo
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1153:37: could not determine kind of name for C.BPF_TC_CUSTOM
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1151:37: could not determine kind of name for C.BPF_TC_EGRESS
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1159:26: could not determine kind of name for C.BPF_TC_F_REPLACE
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1150:37: could not determine kind of name for C.BPF_TC_INGRESS
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1256:9: could not determine kind of name for C.bpf_tc_attach
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1267:9: could not determine kind of name for C.bpf_tc_detach
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1237:9: could not determine kind of name for C.bpf_tc_hook_create
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1246:9: could not determine kind of name for C.bpf_tc_hook_destroy
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1278:9: could not determine kind of name for C.bpf_tc_query
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1202:12: could not determine kind of name for C.sizeof_struct_bpf_tc_hook
vendor/github.com/aquasecurity/libbpfgo/libbpfgo.go:1179:12: could not determine kind of name for C.sizeof_struct_bpf_tc_opts

Digging further... updating libbpf-dev from bullseye-backports got rid of the above warnings, but still ebpf_exporter is not built:

root@rob-bullseye:~# cd /root/ebpf_exporter && CGO_CFLAGS="-I /root/libbpf/src/" CGO_LDFLAGS="-lbpf" GOPATH="" GOPROXY="off" GOFLAGS="-mod=vendor" go install -v -ldflags=" \
    -X github.com/prometheus/common/version.Version=$(git describe) \
    -X github.com/prometheus/common/version.Branch=$(git rev-parse --abbrev-ref HEAD) \
    -X github.com/prometheus/common/version.Revision=$(git rev-parse --short HEAD) \
    -X github.com/prometheus/common/version.BuildUser=docker@$(hostname) \
    -X github.com/prometheus/common/version.BuildDate=$(date --iso-8601=seconds) \
    " ./cmd/ebpf_exporter
github.com/aquasecurity/libbpfgo
github.com/cloudflare/ebpf_exporter/exporter
github.com/cloudflare/ebpf_exporter/cmd/ebpf_exporter

Checking under vendor shows there is no cloudflare (yet):

root@rob-bullseye:~/ebpf_exporter/vendor/github.com# ls -l
total 32
drwxr-xr-x 4 root root 4096 Feb 16 08:54 alecthomas
drwxr-xr-x 3 root root 4096 Feb 16 08:54 aquasecurity
drwxr-xr-x 3 root root 4096 Feb 16 08:54 beorn7
drwxr-xr-x 3 root root 4096 Feb 16 08:54 cespare
drwxr-xr-x 3 root root 4096 Feb 16 08:54 golang
drwxr-xr-x 3 root root 4096 Feb 16 08:54 iovisor
drwxr-xr-x 3 root root 4096 Feb 16 08:54 matttproud
drwxr-xr-x 6 root root 4096 Feb 16 08:54 prometheus

Is that the reason the build is not complete?

Turns out ebpf_exporter binary is under /root/go/bin. :) Running it with config.yaml gives me this error:

root@rob-bullseye:~# ebpf_exporter --config.file=/root/ebpf_exporter/examples/CORE/config.yaml
2022/02/16 10:07:10 Error attaching exporter: stat /root/ebpf_exporter/examples/CORE/biolatency.bpf.o: no such file or directory

The above error was resolved by running "make all" in examples/CORE. Then, when running, I got this error:

root@rob-bullseye:~# ebpf_exporter --config.file=/root/ebpf_exporter/examples/CORE/config.yaml
libbpf: load bpf program failed: Invalid argument
libbpf: -- BEGIN DUMP LOG ---
libbpf:
Unrecognized arg#0 type PTR
; hkey.numa_node = ctx->nid;
0: (61) r2 = *(u32 *)(r1 +8)
; hkey.numa_node = ctx->nid;
1: (63) *(u32 *)(r10 -8) = r2
; hkey.numa_zone = ctx->classzone_idx;
2: (85) call unknown#195896080
invalid func unknown#195896080
processed 3 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

libbpf: -- END LOG --
libbpf: failed to load program 'tracepoint__compaction__mm_compaction_kcompactd_wake'
libbpf: failed to load object '/root/ebpf_exporter/examples/CORE/kcompactd.bpf.o'
2022/02/16 10:45:08 Error attaching exporter: Error to load program:kcompactd object  failed to load BPF object

This seems to be an issue with the config.yaml, when using biolatency.yaml it works:

root@rob-bullseye:~# ebpf_exporter --config.file=/root/ebpf_exporter/examples/CORE/biolatency.yaml
2022/02/16 10:54:08 Starting with 1 programs found in the config
2022/02/16 10:54:08 Listening on :9435
^C

@rob-scheepens
Copy link

@bobrik : for now, this is the script I use to get it working on Bullseye:

export PATH="/usr/lib/go-1.17/bin:$PATH"
export DEBIAN_FRONTEND=noninteractive
export LD_LIBRARY_PATH=/lib64

# APT
echo 'deb http://deb.debian.org/debian bullseye-backports main contrib non-free
deb-src http://deb.debian.org/debian bullseye-backports main contrib non-free' >> /etc/apt/sources.list.d/backports.list

sed -i "s/bullseye main/bullseye main non-free/" /etc/apt/sources.list

apt-get update && \

apt-get --no-install-recommends install -y sudo devscripts build-essential fakeroot pbuilder aptitude git openssh-client ca-certificates git pkg-config libelf-dev libz-dev software-properties-common clang gnupg python3-netaddr pkg-config libz-dev software-properties-common bpftool python2 python3 dh-python libclang-9-dev libclang-common-9-dev libclang-cpp9 libclang1-9 libllvm9 llvm-9 llvm-9-dev llvm-9-runtime llvm-9-tools netperf python3-pyroute2 clang-format bison flex libfl-dev libedit-dev arping iperf ethtool python3-doc python3-tk python3-venv python3.9-venv python3.9-doc cmake luajit libluajit-5.1-dev debhelper

apt-get install -y -t bullseye-backports golang-1.17 libbpfcc-dev libbpfcc bpfcc-tools libbpf-tools libbpf-dev

# BCC
git clone --branch=v0.22.0 --depth=1 https://github.com/iovisor/bcc.git /root/bcc && \
git -C /root/bcc submodule update --init --recursive

cd /root/bcc/
git config --global user.email "user@email.com"
git config --global user.name "nutanix"
sed -i "s/python (>= 2.7)/python3/" debian/control
sed -i "s/python-netaddr/python3-netaddr/" debian/control
git add debian/control
git commit -m "python3 changes."

cd /root/bcc && \
./scripts/build-deb.sh release

cp /root/bcc/libbcc_*.deb /tmp/libbcc.deb
cp -r /root/bcc/libbpf-tools  /tmp/libbpf-tools
dpkg -i /tmp/libbcc.deb

# libbpf
git clone --branch=v0.7.0 --depth=1 https://github.com/libbpf/libbpf.git && \
cd libbpf && cd src && mkdir build root && BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=root make install

cp -r /root/libbpf/ /tmp/libbpf
cd /tmp/libbpf/src && make install

# ebpf_exporter
git clone https://github.com/wenlxie/ebpf_exporter.git /root/ebpf_exporter
cd /root/ebpf_exporter
git checkout upstream.master.core
cd /root/ebpf_exporter && CGO_CFLAGS="-I /usr/include/bpf/" CGO_LDFLAGS="-lbpf" GOPATH="" GOPROXY="off" GOFLAGS="-mod=vendor" go install -v -ldflags=" \
    -X github.com/prometheus/common/version.Version=$(git describe) \
    -X github.com/prometheus/common/version.Branch=$(git rev-parse --abbrev-ref HEAD) \
    -X github.com/prometheus/common/version.Revision=$(git rev-parse --short HEAD) \
    -X github.com/prometheus/common/version.BuildUser=docker@$(hostname) \
    -X github.com/prometheus/common/version.BuildDate=$(date --iso-8601=seconds) \
    " ./cmd/ebpf_exporter
cd /root/ebpf_exporter/examples/CORE && make all
cp -r /root/ebpf_exporter/examples /root/examples
cp /root/go/bin/ebpf_exporter /usr/local/bin/ebpf_exporter
cd

# cleanup
find /tmp -name "*.git*"|xargs rm -Rf
find /root/examples -name "*.git*"|xargs rm -Rf

# RUN
ebpf_exporter --config.file=/root/examples/CORE/biolatency.yaml

cc: @wenlxie

@wenlxie
Copy link
Contributor Author

wenlxie commented Feb 22, 2022

I update the code:

  1. Remove bcc dependency
  2. Add tag support back
  3. Make lib to be static linked
  4. benchmark folder removed because of 1, will add it back

@bobrik @rob-scheepens
Now ebpf_exporter compiled with lib static linked , it works well in my env.

root@4bfcb177e93f:~/go/bin# ldd ebpf_exporter
	not a dynamic executable

Appreciate that if you can help to have a try of this.

@rob-scheepens
Copy link

@wenlxie : using your latest upstream.master.core I checked again and the script above still works.

When you say "Remove bcc dependency", does that mean that there is no need to have bcc installed/built anymore, and that I can remove it from the script? When I tested that, things failed.

@wenlxie
Copy link
Contributor Author

wenlxie commented Mar 8, 2022

When you say "Remove bcc dependency", does that mean that there is no need to have bcc installed/built anymore, and that I can remove it from the script? When I tested that, things failed.

It means remove the bcc codes from the code/vendor

But it still need the header file in bcc to compile the bpf binary.

@rob-scheepens
Copy link

Unfortunately, I was not able to build this either with the Dockerfile or in a Debian Bullseye VM with libbpf from backports. Happy to have another look when I can build and run it myself.

@bobrik : can you give it another try using the script I provided for Bullseye?

cc: @wenlxie

@bobrik
Copy link
Contributor

bobrik commented Oct 14, 2022

I was able to build this locally with some modifications. My plan is to clean this up and see how we can get this merged.

Apologies for the delay.

@bobrik
Copy link
Contributor

bobrik commented Oct 16, 2022

aquasecurity/libbpfgo#258 looks like it could be a good replacement for kaddr (or any argument) passing.

Not sure if we should wait for it though.

@bobrik bobrik changed the base branch from v2 to master October 18, 2022 04:01
@bobrik bobrik merged commit d467240 into cloudflare:master Oct 18, 2022
@bobrik
Copy link
Contributor

bobrik commented Oct 18, 2022

Many thanks @wenlxie! I'll try to roll it out internally to iron any kinks before tagging a proper v2.0.0 release.

@bobrik
Copy link
Contributor

bobrik commented Oct 31, 2022

It's out now: https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants