Add support for writing maps into .maps ELF section in aya-bpf

[v-thakkar]

At the moment generating BTF debug info by default is not enabled. Once LLVM fixes are in and this is merged in bpf-linker, we'll need to teach aya-bpf to support writing maps into .maps section.

Adding this as an issue, so that I don't forget to tackle it once the above 2 things are done. cc @vadorovsky

[vadorovsky]

We have some updates and a bit of progress here.

Discord thread for anyone interested

LLVM fix

@dave-tucker submitted this LLVM fix https://reviews.llvm.org/D134533 - the only thing missing to get it merged is writing some LLVM IR test for that, which Dave is working on.

Building LLVM with this patch and then building bpf-linker with it makes it possible to build eBPF programs with debug_info and BTF. 🎉

Actual BTF map support in Aya

I crated this repo where I'm trying to make a small aya-bpf program with a BTF map:

https://github.com/aya-rs/aya-btf-maps
https://github.com/aya-rs/aya-btf-maps/blob/main/fork-ebpf/src/main.rs

It's using libbpf 1.0 in the userspace and Aya in the eBPF crate.

It's not fully working yet, libbpf complains about the type field:

libbpf: map 'PID_MAP': attr 'type': expected PTR, got int.

It most likely means that there is something behind __type and __u32 macros here which I missed. UPDATE (5.10.2022): this is solved now

These are examples of BTF map definitions preprocessed by clang:

struct {
 int (*type)[BPF_MAP_TYPE_ARRAY];
 typeof(u32) *key;
 typeof(u32) *value;
 int (*max_entries)[256];
} array1

struct {
 int (*type)[BPF_MAP_TYPE_SK_STORAGE];
 int (*map_flags)[BPF_F_NO_PREALLOC];
 typeof(int) *key;
 typeof(int) *value;
} sk_stg_map

I managed to build such a struct with this macro:

https://github.com/vadorovsky/aya-btf-maps/blob/0f952634bf962e78bca54b6f19f9baacf0c868ae/aya-btf-macros/src/expand.rs#L55-L73

libbpf is fine with how the .maps section looks like, but the kernel doesn't like the BTF included in our program:

Error: failed to load bpf object: -22

We need to compare .debug_info produced by clang and Rust and hunt the differences. We will probably need to solve them with some hacks in bpf-linker first, then come up with some proper fixes for LLVM or rustc or kernel, wherever the problem happens.

Loading the same program with Aya (in userspace) works.

Please note that this example is just defining a map as a struct directly in the eBPF program crate. And we use bindings to BPF helpers directly. Once we get it working, we still need to figure out how we design a proper API for BTF maps in Aya.

If anyone wants to try our or/and help, I added instructions to README how to prepare patched LLVM, bpf-linker, build the project and what's done there.

[vadorovsky]

Currently we are doing a following fixup in Aya (userspace, when loading programs):

aya/aya/src/obj/btf/btf.rs

Lines 393 to 401 in 66b4f79

	// Fixup PTR for Rust
	// LLVM emits names for Rust pointer types, which the kernel doesn't like
	// While I figure out if this needs fixing in the Kernel or LLVM, we'll
	// do a fixup here
	BtfType::Ptr(ptr) => {
	let mut fixed_ty = ptr.clone();
	fixed_ty.name_offset = 0;
	types.types[i] = BtfType::Ptr(fixed_ty)
	}

PTR looks like that in BTF (produced by clang):

[1] PTR '(anon)' type_id=3
[...]
[5] PTR '(anon)' type_id=6
[...]
[7] PTR '(anon)' type_id=8
[...]
[11] PTR '(anon)' type_id=0

PTRs in Rust BTF have names corresponding to the type of a pointer:

[1] PTR '*const [i32; 1]' type_id=3
[...]
[5] PTR '*const u32' type_id=6
[...]
[7] PTR '*const [i32; 1024]' type_id=8
[...]
[9] PTR '*const [i32; 0]' type_id=10
[...]
[13] PTR '*mut core::ffi::c_void' type_id=14
[...]
[17] PTR '*mut u8' type_id=18

DWARF (produced by clang):

0x000000ca:   DW_TAG_pointer_type
                DW_AT_type      (0x000000cf "int[1]")
[...]
0x000000df:   DW_TAG_pointer_type
                DW_AT_type      (0x000000e4 "unsigned int")
[...]
0x00000102:   DW_TAG_pointer_type
                DW_AT_type      (0x00000107 "long (const char *, __u32, ...)")

(notice the lack of DW_AT_name field)

We need to move it to bpf-linker. And all the other differences in debug_info between eBPF objects commpiled with C and Rust - also have to be handled in bpf-linker. At least for now (we might consider proper fixes in rustc/LLVM/kernel when we know everything and get stuff working).

Probably the best place for fixup would be somewhere around here:

https://github.com/aya-rs/bpf-linker/blob/0666b6424216ad27651f14b0f433f6200872aff8/src/llvm/mod.rs#L213-L220

On this stage, we have already one big LLVM module after linking, we just need to find a way to find the debug info of that module and be able to iterate over it.

Quotes from Discord:

you probably need to create an iterator (see iter.rs) based on this https://docs.rs/llvm-sys/latest/llvm_sys/core/fn.LLVMGetFirstNamedMetadata.html
filter all the debug metadata
recursively visit all the metadata
build some kind of id => metadata mapping
have some kind of sentinel for metadata id that has been referenced but you haven't found yet
then after you finish scanning, scan the whole id => metadata mapping
look for sentinels that still exists (things for which you have missing definitions)
and create stubs
or you can probably even generate the stubs as you go
and then replace them if you encounter their actual definition
or I mean, I don't remember what that particular crash was about
maybe we can get away by generating stubs just for that
the long term solution is what I'm doing with the #[btf_export] thing
(which now that I looked at the code is easier than I thought)

There was also an idea of modifying DI by writing an LLVM pass, but

I don't think we need a custom pass?
there's already "passes" in bpf-linker
that don't use the actual llvm pass interface
you need to implement that interface if you want it to be called by the pass manage
but we don't want that
https://github.com/aya-rs/bpf-linker/blob/main/src/llvm/mod.rs#L21
internalize is something we do to massage linkage
essentially, bpf-linker gets all the input modules, merges in one big module, that big module contains everything you need
including DI
the fix would consist in visiting all the DI, and fixing missing definitions

[davibe]

I need this for a project I'm working on, so I've started implementing doing the fixups in bpf-linker