Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Linux内核如何支持xdp？？ #4

Open
birdofprey opened this issue Oct 8, 2021 · 17 comments
Open

Linux内核如何支持xdp？？ #4

birdofprey opened this issue Oct 8, 2021 · 17 comments

Comments

@birdofprey
Copy link

其实一直挺关注BPF的，就是不知道Gnu/Linux内核如何支持xdp.有没有Debian Gnu/Linux下的详细讲解
@birdofprey
Copy link
Author

Debian Gnu/Linux内核默认支持xdp、ebpf、mptcp、wireguard估计还有很长时间啊

@ihciah
Copy link
Owner

ihciah commented Oct 9, 2021 •
edited
Loading

XDP (eXpress Data Path) is an eBPF based high performance data path merged in the Linux kernel since version 4.8.

ref: https://en.wikipedia.org/wiki/Express_Data_Path

@ihciah
Copy link
Owner

ihciah commented Oct 9, 2021

Debian从9开始就支持了。

ref: https://en.wikipedia.org/wiki/Debian_version_history#Release_table

@birdofprey
Copy link
Author

我的意思是如何开启？
OS:Debian Gnu/Linux 11
kernel:Linux localhost 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux
具体步逐有么有相关的文章？

@ihciah
Copy link
Owner

ihciah commented Oct 9, 2021

不需要手动开启，默认就能用的(当然要是自己编译的内核另说)

@birdofprey
Copy link
Author

ip link set dev eth0
address allmulticast broadcast dynamic multicast netns trailers up
alias arp down mtu name promisc txqueuelen
没有xdp选项

ip link set dev wlp2s0 xdp obj ./clean-dns.elf

段错误
我想我需要一个完整的过程demo,便于实现然后理解

@birdofprey
Copy link
Author

Oct 9 14:49:21 localhost kernel: [ 5677.121935] ip[12873]: segfault at 10 ip 00007ff3a0e82582 sp 00007ffc4814e560 error 4 in libbpf.so.0.3.0[7ff3a0e6c000+23000]
Oct 9 14:49:21 localhost kernel: [ 5677.121941] Code: 49 63 47 60 48 8d 58 01 48 69 c0 d8 00 00 00 48 89 5c 24 18 48 89 44 24 50 49 8b 9f 00 01 00 00 48 b8 ab aa aa aa aa aa aa aa <48> 8b 73 10 48 f7 e6 48 c1 ea 04 48 89 54 24 10 48 83 fe 17 0f 86

@birdofprey birdofprey mentioned this issue Oct 9, 2021
Open
@birdofprey
Copy link
Author

ip -force link set dev wlan0 xdp object clean-dns.elf ver
libbpf: loading clean-dns.elf
libbpf: elf: section(1) prog, size 536, link 0, flags 6, type=1
段错误

@ihciah
Copy link
Owner

ihciah commented Oct 9, 2021

我找了个debian10（Linux debian 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux）试了下Release里的elf，没复现问题。
如果是load上去就挂，那可能是内核的兼容性问题。
你可以试下搭建一个rust开发环境，然后使用依次运行 For Developer 章节的 Build 和 Run 部分。

@ihciah ihciah mentioned this issue Oct 9, 2021
Open
@birdofprey
Copy link
Author

birdofprey commented Oct 10, 2021 •
edited
Loading

我找了个debian10（Linux debian 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux）试了下Release里的elf，没复现问题。 如果是load上去就挂，那可能是内核的兼容性问题。 你可以试下搭建一个rust开发环境，然后使用依次运行 For Developer 章节的 Build 和 Run 部分。

谢谢你的回复，编译会报如下错误：

cargo bpf build clean-dns

error: no such subcommand: bpf

    Did you mean `doc`?

找了互联网所有的帖子，发现没有和这个类似的。不知道具体是那里的问题。祝你开心.
另外在Debian Gnu/Linux 11 中执行
apt-get install -t bullseye-proposed-updates iproute2
会安装最新的iproute2 (5.14.0-1~bpo11+1)，该版本包含了xdp选项

@ihciah
Copy link
Owner

ihciah commented Oct 10, 2021

我找了个debian10（Linux debian 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux）试了下Release里的elf，没复现问题。 如果是load上去就挂，那可能是内核的兼容性问题。 你可以试下搭建一个rust开发环境，然后使用依次运行 For Developer 章节的 Build 和 Run 部分。

谢谢你的回复，编译会报如下错误：

cargo bpf build clean-dns

error: no such subcommand: bpf

    Did you mean `doc`?

找了互联网所有的帖子，发现没有和这个类似的。不知道具体是那里的问题。祝你开心. 另外在Debian Gnu/Linux 11 中执行 apt-get install -t bullseye-proposed-updates iproute2 会安装最新的iproute2 (5.14.0-1~bpo11+1)，该版本包含了xdp选项

cargo install cargo-bpf

@pymongo
Copy link

pymongo commented Oct 11, 2021

manjaro(跟你 arch 差不多的) 5.13 的 kernel 编译时依然 SIGSEGV

确认 5.13 支持 CONFIG_XDP_SOCKETS

[w@ww temp]$ zcat /proc/config.gz | grep CONFIG_XDP_SOCKETS
CONFIG_XDP_SOCKETS=y
CONFIG_XDP_SOCKETS_DIAG=m
[w@ww temp]$ cat /boot/config-$(uname -r) | grep CONFIG_XDP_SOCKETS
cat: /boot/config-5.13.19-2-MANJARO: No such file or directory
[w@ww temp]$ zcat /proc/config.gz | head
#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 5.13.19-2 Kernel Configuration
#
CONFIG_CC_VERSION_TEXT="gcc (GCC) 11.1.0"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=110100
CONFIG_CLANG_VERSION=0
CONFIG_AS_IS_GNU=y
CONFIG_AS_VERSION=23601

SIGSEGV 的 backtrace

(lldb) run bpf build
There is a running process, kill it and restart?: [Y/n] y
Process 2103336 exited with status = 9 (0x00000009) 
Process 2103417 launched: '/home/w/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo' (x86_64)
Process 2103417 stopped
* thread #1, name = 'cargo-bpf', stop reason = exec
    frame #0: 0x00007ffff7fce090 ld-2.33.so`_start
ld-2.33.so`_start:
    0x7ffff7fce090 <+0>: movq   %rsp, %rdi
    0x7ffff7fce093 <+3>: callq  0x7ffff7fcee20            ; _dl_start

ld-2.33.so`_dl_start_user:
    0x7ffff7fce098 <+0>: movq   %rax, %r12
    0x7ffff7fce09b <+3>: movl   0x2dc17(%rip), %eax       ; _dl_skip_args
(lldb) settings set target.process.stop-on-exec false
(lldb) run bpf build
There is a running process, kill it and restart?: [Y/n] y
Process 2103417 exited with status = 9 (0x00000009) 
Process 2103601 launched: '/home/w/.cargo/bin/cargo-bpf' (x86_64)
Process 2103601 stopped and restarted: thread 1 received signal: SIGCHLD
   Compiling clean-dns-bpf v0.1.0 (/home/w/repos/clone_repos/clean-dns-bpf)
warning: due to multiple output types requested, the explicitly specified output file name will be adapted for each output type

warning: ignoring --out-dir flag due to -o flag

warning: `clean-dns-bpf` (bin "clean-dns") generated 2 warnings
    Finished release [optimized] target(s) in 0.35s
Process 2103601 stopped and restarted: thread 1 received signal: SIGCHLD
Process 2103601 stopped
* thread #1, name = 'cargo-bpf', stop reason = signal SIGSEGV: invalid address (fault address: 0x1a80)
    frame #0: 0x0000555559cdce42 cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::lazyLoadOneMetadata(unsigned int, (anonymous namespace)::(anonymous namespace)::PlaceholderQueue&) + 162
cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::lazyLoadOneMetadata:
->  0x555559cdce42 <+162>: movq   (%rax,%rdx,8), %rdx
    0x555559cdce46 <+166>: movl   $0x0, 0x120(%rbp)
    0x555559cdce50 <+176>: movq   %rdx, %rax
    0x555559cdce53 <+179>: shrq   $0x3, %rax
(lldb) bt
* thread #1, name = 'cargo-bpf', stop reason = signal SIGSEGV: invalid address (fault address: 0x1a80)
  * frame #0: 0x0000555559cdce42 cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::lazyLoadOneMetadata(unsigned int, (anonymous namespace)::(anonymous namespace)::PlaceholderQueue&) + 162
    frame #1: 0x0000555559cdd321 cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::resolveForwardRefsAndPlaceholders((anonymous namespace)::(anonymous namespace)::PlaceholderQueue&) + 657
    frame #2: 0x0000555559ce046c cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::parseMetadataAttachment(llvm::Function&, llvm::SmallVectorImpl<llvm::Instruction*> const&) + 1068
    frame #3: 0x0000555559ce0701 cargo-bpf`llvm::MetadataLoader::parseMetadataAttachment(llvm::Function&, llvm::SmallVectorImpl<llvm::Instruction*> const&) + 33
    frame #4: 0x0000555559cc553a cargo-bpf`(anonymous namespace)::BitcodeReader::parseFunctionBody(llvm::Function*) + 15642
    frame #5: 0x0000555559cc989e cargo-bpf`(anonymous namespace)::BitcodeReader::materialize(llvm::GlobalValue*) + 574
    frame #6: 0x0000555559ccf726 cargo-bpf`(anonymous namespace)::BitcodeReader::materializeModule() + 166
    frame #7: 0x0000555559e0a1bb cargo-bpf`llvm::Module::materializeAll() + 59
    frame #8: 0x0000555559ccf3c9 cargo-bpf`llvm::BitcodeModule::getModuleImpl(llvm::LLVMContext&, bool, bool, bool, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>) + 3929
    frame #9: 0x0000555559ccf56b cargo-bpf`llvm::parseBitcodeFile(llvm::MemoryBufferRef, llvm::LLVMContext&, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>) + 267
    frame #10: 0x0000555559390ca7 cargo-bpf`llvm::parseIR(llvm::MemoryBufferRef, llvm::SMDiagnostic&, llvm::LLVMContext&, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>) + 439
    frame #11: 0x0000555559391ccd cargo-bpf`LLVMParseIRInContext + 333
    frame #12: 0x00005555576ccd3b cargo-bpf`cargo_bpf_lib::llvm::compile::h408fb3dc4f4ed60b + 331
    frame #13: 0x00005555576d6d86 cargo-bpf`cargo_bpf_lib::build::build::hd9adab697c50d350 + 3398
    frame #14: 0x00005555576d7d51 cargo-bpf`cargo_bpf_lib::build::cmd_build::h74398175950cebbd + 145
    frame #15: 0x00005555576b526f cargo-bpf`cargo_bpf::main::h75f4e3ecde46d0b3 + 4559
    frame #16: 0x00005555576b8943 cargo-bpf`std::sys_common::backtrace::__rust_begin_short_backtrace::hb975087f78918ca4 + 3
    frame #17: 0x00005555576b61a9 cargo-bpf`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h3842132363a6eb04 (.llvm.16335996392534242537) + 9
    frame #18: 0x000055555a02164a cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::ha9408abe89f69dc4 at function.rs:259:13
    frame #19: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::do_call::h5b0cc9e9102acb65 at panicking.rs:401
    frame #20: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::hddc7f8229138b3ba at panicking.rs:365
    frame #21: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panic::catch_unwind::hfa401ff8bab2986e at panic.rs:434
    frame #22: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::h8163422320d11405 at rt.rs:45
    frame #23: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::do_call::hc742cc7bb4f0fb20 at panicking.rs:401
    frame #24: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::ha37d8d2dd1acf7d3 at panicking.rs:365
    frame #25: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panic::catch_unwind::h8a5381d5ecf437bc at panic.rs:434
    frame #26: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 at rt.rs:45
    frame #27: 0x00005555576b59c2 cargo-bpf`main + 34
    frame #28: 0x00007ffff79aeb25 libc.so.6`__libc_start_main + 213
    frame #29: 0x00005555576b2f5e cargo-bpf`_start + 46

@pymongo
Copy link

pymongo commented Oct 11, 2021

manjaro(跟你 arch 差不多的) 5.13 的 kernel 编译时依然 SIGSEGV

确认 5.13 支持 CONFIG_XDP_SOCKETS

[w@ww temp]$ zcat /proc/config.gz | grep CONFIG_XDP_SOCKETS
CONFIG_XDP_SOCKETS=y
CONFIG_XDP_SOCKETS_DIAG=m
[w@ww temp]$ cat /boot/config-$(uname -r) | grep CONFIG_XDP_SOCKETS
cat: /boot/config-5.13.19-2-MANJARO: No such file or directory
[w@ww temp]$ zcat /proc/config.gz | head
#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 5.13.19-2 Kernel Configuration
#
CONFIG_CC_VERSION_TEXT="gcc (GCC) 11.1.0"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=110100
CONFIG_CLANG_VERSION=0
CONFIG_AS_IS_GNU=y
CONFIG_AS_VERSION=23601

SIGSEGV 的 backtrace

(lldb) run bpf build
There is a running process, kill it and restart?: [Y/n] y
Process 2103336 exited with status = 9 (0x00000009) 
Process 2103417 launched: '/home/w/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo' (x86_64)
Process 2103417 stopped
* thread #1, name = 'cargo-bpf', stop reason = exec
    frame #0: 0x00007ffff7fce090 ld-2.33.so`_start
ld-2.33.so`_start:
    0x7ffff7fce090 <+0>: movq   %rsp, %rdi
    0x7ffff7fce093 <+3>: callq  0x7ffff7fcee20            ; _dl_start

ld-2.33.so`_dl_start_user:
    0x7ffff7fce098 <+0>: movq   %rax, %r12
    0x7ffff7fce09b <+3>: movl   0x2dc17(%rip), %eax       ; _dl_skip_args
(lldb) settings set target.process.stop-on-exec false
(lldb) run bpf build
There is a running process, kill it and restart?: [Y/n] y
Process 2103417 exited with status = 9 (0x00000009) 
Process 2103601 launched: '/home/w/.cargo/bin/cargo-bpf' (x86_64)
Process 2103601 stopped and restarted: thread 1 received signal: SIGCHLD
   Compiling clean-dns-bpf v0.1.0 (/home/w/repos/clone_repos/clean-dns-bpf)
warning: due to multiple output types requested, the explicitly specified output file name will be adapted for each output type

warning: ignoring --out-dir flag due to -o flag

warning: `clean-dns-bpf` (bin "clean-dns") generated 2 warnings
    Finished release [optimized] target(s) in 0.35s
Process 2103601 stopped and restarted: thread 1 received signal: SIGCHLD
Process 2103601 stopped
* thread #1, name = 'cargo-bpf', stop reason = signal SIGSEGV: invalid address (fault address: 0x1a80)
    frame #0: 0x0000555559cdce42 cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::lazyLoadOneMetadata(unsigned int, (anonymous namespace)::(anonymous namespace)::PlaceholderQueue&) + 162
cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::lazyLoadOneMetadata:
->  0x555559cdce42 <+162>: movq   (%rax,%rdx,8), %rdx
    0x555559cdce46 <+166>: movl   $0x0, 0x120(%rbp)
    0x555559cdce50 <+176>: movq   %rdx, %rax
    0x555559cdce53 <+179>: shrq   $0x3, %rax
(lldb) bt
* thread #1, name = 'cargo-bpf', stop reason = signal SIGSEGV: invalid address (fault address: 0x1a80)
  * frame #0: 0x0000555559cdce42 cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::lazyLoadOneMetadata(unsigned int, (anonymous namespace)::(anonymous namespace)::PlaceholderQueue&) + 162
    frame #1: 0x0000555559cdd321 cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::resolveForwardRefsAndPlaceholders((anonymous namespace)::(anonymous namespace)::PlaceholderQueue&) + 657
    frame #2: 0x0000555559ce046c cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::parseMetadataAttachment(llvm::Function&, llvm::SmallVectorImpl<llvm::Instruction*> const&) + 1068
    frame #3: 0x0000555559ce0701 cargo-bpf`llvm::MetadataLoader::parseMetadataAttachment(llvm::Function&, llvm::SmallVectorImpl<llvm::Instruction*> const&) + 33
    frame #4: 0x0000555559cc553a cargo-bpf`(anonymous namespace)::BitcodeReader::parseFunctionBody(llvm::Function*) + 15642
    frame #5: 0x0000555559cc989e cargo-bpf`(anonymous namespace)::BitcodeReader::materialize(llvm::GlobalValue*) + 574
    frame #6: 0x0000555559ccf726 cargo-bpf`(anonymous namespace)::BitcodeReader::materializeModule() + 166
    frame #7: 0x0000555559e0a1bb cargo-bpf`llvm::Module::materializeAll() + 59
    frame #8: 0x0000555559ccf3c9 cargo-bpf`llvm::BitcodeModule::getModuleImpl(llvm::LLVMContext&, bool, bool, bool, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>) + 3929
    frame #9: 0x0000555559ccf56b cargo-bpf`llvm::parseBitcodeFile(llvm::MemoryBufferRef, llvm::LLVMContext&, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>) + 267
    frame #10: 0x0000555559390ca7 cargo-bpf`llvm::parseIR(llvm::MemoryBufferRef, llvm::SMDiagnostic&, llvm::LLVMContext&, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>) + 439
    frame #11: 0x0000555559391ccd cargo-bpf`LLVMParseIRInContext + 333
    frame #12: 0x00005555576ccd3b cargo-bpf`cargo_bpf_lib::llvm::compile::h408fb3dc4f4ed60b + 331
    frame #13: 0x00005555576d6d86 cargo-bpf`cargo_bpf_lib::build::build::hd9adab697c50d350 + 3398
    frame #14: 0x00005555576d7d51 cargo-bpf`cargo_bpf_lib::build::cmd_build::h74398175950cebbd + 145
    frame #15: 0x00005555576b526f cargo-bpf`cargo_bpf::main::h75f4e3ecde46d0b3 + 4559
    frame #16: 0x00005555576b8943 cargo-bpf`std::sys_common::backtrace::__rust_begin_short_backtrace::hb975087f78918ca4 + 3
    frame #17: 0x00005555576b61a9 cargo-bpf`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h3842132363a6eb04 (.llvm.16335996392534242537) + 9
    frame #18: 0x000055555a02164a cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::ha9408abe89f69dc4 at function.rs:259:13
    frame #19: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::do_call::h5b0cc9e9102acb65 at panicking.rs:401
    frame #20: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::hddc7f8229138b3ba at panicking.rs:365
    frame #21: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panic::catch_unwind::hfa401ff8bab2986e at panic.rs:434
    frame #22: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::h8163422320d11405 at rt.rs:45
    frame #23: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::do_call::hc742cc7bb4f0fb20 at panicking.rs:401
    frame #24: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::ha37d8d2dd1acf7d3 at panicking.rs:365
    frame #25: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panic::catch_unwind::h8a5381d5ecf437bc at panic.rs:434
    frame #26: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 at rt.rs:45
    frame #27: 0x00005555576b59c2 cargo-bpf`main + 34
    frame #28: 0x00007ffff79aeb25 libc.so.6`__libc_start_main + 213
    frame #29: 0x00005555576b2f5e cargo-bpf`_start + 46

确认了下直接装载 github release 的 elf 二进制文件是没啥问题的配上 8.8.8.8 的 DNS server，所以不是我 kernel 的问题

@birdofprey
Copy link
Author

我找了个debian10（Linux debian 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux）试了下Release里的elf，没复现问题。 如果是load上去就挂，那可能是内核的兼容性问题。 你可以试下搭建一个rust开发环境，然后使用依次运行 For Developer 章节的 Build 和 Run 部分。

谢谢你的回复，编译会报如下错误：

cargo bpf build clean-dns

error: no such subcommand: bpf

    Did you mean `doc`?

找了互联网所有的帖子，发现没有和这个类似的。不知道具体是那里的问题。祝你开心. 另外在Debian Gnu/Linux 11 中执行 apt-get install -t bullseye-proposed-updates iproute2 会安装最新的iproute2 (5.14.0-1~bpo11+1)，该版本包含了xdp选项

cargo install cargo-bpf

error: No suitable version of LLVM was found system-wide or pointed
to by LLVM_SYS_120_PREFIX.

   Consider using `llvmenv` to compile an appropriate copy of LLVM, and
   refer to the llvm-sys documentation for more information.

   llvm-sys: https://crates.io/crates/llvm-sys
   llvmenv: https://crates.io/crates/llvmenv

--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/llvm-sys-120.2.1/src/lib.rs:486:1
|
486 | / std::compile_error!(concat!(
487 | | "No suitable version of LLVM was found system-wide or pointed
488 | | to by LLVM_SYS_",
489 | | env!("CARGO_PKG_VERSION_MAJOR"),
... |
496 | | llvmenv: https://crates.io/crates/llvmenv"
497 | | ));
| |___^

error: aborting due to previous error

error: could not compile llvm-sys.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...

@ihciah
Copy link
Owner

ihciah commented Oct 11, 2021

manjaro(跟你 arch 差不多的) 5.13 的 kernel 编译时依然 SIGSEGV

确认 5.13 支持 CONFIG_XDP_SOCKETS

[w@ww temp]$ zcat /proc/config.gz | grep CONFIG_XDP_SOCKETS
CONFIG_XDP_SOCKETS=y
CONFIG_XDP_SOCKETS_DIAG=m
[w@ww temp]$ cat /boot/config-$(uname -r) | grep CONFIG_XDP_SOCKETS
cat: /boot/config-5.13.19-2-MANJARO: No such file or directory
[w@ww temp]$ zcat /proc/config.gz | head
#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 5.13.19-2 Kernel Configuration
#
CONFIG_CC_VERSION_TEXT="gcc (GCC) 11.1.0"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=110100
CONFIG_CLANG_VERSION=0
CONFIG_AS_IS_GNU=y
CONFIG_AS_VERSION=23601

SIGSEGV 的 backtrace

(lldb) run bpf build
There is a running process, kill it and restart?: [Y/n] y
Process 2103336 exited with status = 9 (0x00000009) 
Process 2103417 launched: '/home/w/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo' (x86_64)
Process 2103417 stopped
* thread #1, name = 'cargo-bpf', stop reason = exec
    frame #0: 0x00007ffff7fce090 ld-2.33.so`_start
ld-2.33.so`_start:
    0x7ffff7fce090 <+0>: movq   %rsp, %rdi
    0x7ffff7fce093 <+3>: callq  0x7ffff7fcee20            ; _dl_start

ld-2.33.so`_dl_start_user:
    0x7ffff7fce098 <+0>: movq   %rax, %r12
    0x7ffff7fce09b <+3>: movl   0x2dc17(%rip), %eax       ; _dl_skip_args
(lldb) settings set target.process.stop-on-exec false
(lldb) run bpf build
There is a running process, kill it and restart?: [Y/n] y
Process 2103417 exited with status = 9 (0x00000009) 
Process 2103601 launched: '/home/w/.cargo/bin/cargo-bpf' (x86_64)
Process 2103601 stopped and restarted: thread 1 received signal: SIGCHLD
   Compiling clean-dns-bpf v0.1.0 (/home/w/repos/clone_repos/clean-dns-bpf)
warning: due to multiple output types requested, the explicitly specified output file name will be adapted for each output type

warning: ignoring --out-dir flag due to -o flag

warning: `clean-dns-bpf` (bin "clean-dns") generated 2 warnings
    Finished release [optimized] target(s) in 0.35s
Process 2103601 stopped and restarted: thread 1 received signal: SIGCHLD
Process 2103601 stopped
* thread #1, name = 'cargo-bpf', stop reason = signal SIGSEGV: invalid address (fault address: 0x1a80)
    frame #0: 0x0000555559cdce42 cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::lazyLoadOneMetadata(unsigned int, (anonymous namespace)::(anonymous namespace)::PlaceholderQueue&) + 162
cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::lazyLoadOneMetadata:
->  0x555559cdce42 <+162>: movq   (%rax,%rdx,8), %rdx
    0x555559cdce46 <+166>: movl   $0x0, 0x120(%rbp)
    0x555559cdce50 <+176>: movq   %rdx, %rax
    0x555559cdce53 <+179>: shrq   $0x3, %rax
(lldb) bt
* thread #1, name = 'cargo-bpf', stop reason = signal SIGSEGV: invalid address (fault address: 0x1a80)
  * frame #0: 0x0000555559cdce42 cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::lazyLoadOneMetadata(unsigned int, (anonymous namespace)::(anonymous namespace)::PlaceholderQueue&) + 162
    frame #1: 0x0000555559cdd321 cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::resolveForwardRefsAndPlaceholders((anonymous namespace)::(anonymous namespace)::PlaceholderQueue&) + 657
    frame #2: 0x0000555559ce046c cargo-bpf`llvm::MetadataLoader::MetadataLoaderImpl::parseMetadataAttachment(llvm::Function&, llvm::SmallVectorImpl<llvm::Instruction*> const&) + 1068
    frame #3: 0x0000555559ce0701 cargo-bpf`llvm::MetadataLoader::parseMetadataAttachment(llvm::Function&, llvm::SmallVectorImpl<llvm::Instruction*> const&) + 33
    frame #4: 0x0000555559cc553a cargo-bpf`(anonymous namespace)::BitcodeReader::parseFunctionBody(llvm::Function*) + 15642
    frame #5: 0x0000555559cc989e cargo-bpf`(anonymous namespace)::BitcodeReader::materialize(llvm::GlobalValue*) + 574
    frame #6: 0x0000555559ccf726 cargo-bpf`(anonymous namespace)::BitcodeReader::materializeModule() + 166
    frame #7: 0x0000555559e0a1bb cargo-bpf`llvm::Module::materializeAll() + 59
    frame #8: 0x0000555559ccf3c9 cargo-bpf`llvm::BitcodeModule::getModuleImpl(llvm::LLVMContext&, bool, bool, bool, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>) + 3929
    frame #9: 0x0000555559ccf56b cargo-bpf`llvm::parseBitcodeFile(llvm::MemoryBufferRef, llvm::LLVMContext&, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>) + 267
    frame #10: 0x0000555559390ca7 cargo-bpf`llvm::parseIR(llvm::MemoryBufferRef, llvm::SMDiagnostic&, llvm::LLVMContext&, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>) + 439
    frame #11: 0x0000555559391ccd cargo-bpf`LLVMParseIRInContext + 333
    frame #12: 0x00005555576ccd3b cargo-bpf`cargo_bpf_lib::llvm::compile::h408fb3dc4f4ed60b + 331
    frame #13: 0x00005555576d6d86 cargo-bpf`cargo_bpf_lib::build::build::hd9adab697c50d350 + 3398
    frame #14: 0x00005555576d7d51 cargo-bpf`cargo_bpf_lib::build::cmd_build::h74398175950cebbd + 145
    frame #15: 0x00005555576b526f cargo-bpf`cargo_bpf::main::h75f4e3ecde46d0b3 + 4559
    frame #16: 0x00005555576b8943 cargo-bpf`std::sys_common::backtrace::__rust_begin_short_backtrace::hb975087f78918ca4 + 3
    frame #17: 0x00005555576b61a9 cargo-bpf`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h3842132363a6eb04 (.llvm.16335996392534242537) + 9
    frame #18: 0x000055555a02164a cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::ha9408abe89f69dc4 at function.rs:259:13
    frame #19: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::do_call::h5b0cc9e9102acb65 at panicking.rs:401
    frame #20: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::hddc7f8229138b3ba at panicking.rs:365
    frame #21: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panic::catch_unwind::hfa401ff8bab2986e at panic.rs:434
    frame #22: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::h8163422320d11405 at rt.rs:45
    frame #23: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::do_call::hc742cc7bb4f0fb20 at panicking.rs:401
    frame #24: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panicking::try::ha37d8d2dd1acf7d3 at panicking.rs:365
    frame #25: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 [inlined] std::panic::catch_unwind::h8a5381d5ecf437bc at panic.rs:434
    frame #26: 0x000055555a021643 cargo-bpf`std::rt::lang_start_internal::h7e2cee8c90d4a4d3 at rt.rs:45
    frame #27: 0x00005555576b59c2 cargo-bpf`main + 34
    frame #28: 0x00007ffff79aeb25 libc.so.6`__libc_start_main + 213
    frame #29: 0x00005555576b2f5e cargo-bpf`_start + 46

用stable。nightly下会崩我也遇到了，可以看cargo-bpf repo的issue。

@ihciah
Copy link
Owner

ihciah commented Oct 29, 2021

在另一个使用ebpf的实践中，我也发现不同kernel版本似乎出现了一定的不兼容。原因没有具体定位到，感觉可能是部分内核的BTF支持问题or bpf helper签名问题。
可能确实需要较高版本的kernel。

@time-river time-river mentioned this issue Dec 4, 2021
Merged
@21ki
Copy link

21ki commented Aug 29, 2022 •
edited
Loading

Ubuntu 22.04
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ihciah @birdofprey @21ki @pymongo