WIP: CI for bash completion #2
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
qmonnet
force-pushed
the
wip/ci-bash-completion
branch
30 times, most recently
from
6639ad3
to
3a0bbf9
Compare
LLVM automatically places __arena variables into ".arena.1" ELF section.
In order to use such global variables bpf program must include definition
of arena map in ".maps" section, like:
struct {
__uint(type, BPF_MAP_TYPE_ARENA);
__uint(map_flags, BPF_F_MMAPABLE);
__uint(max_entries, 1000); /* number of pages */
__ulong(map_extra, 2ull << 44); /* start of mmap() region */
} arena SEC(".maps");
libbpf recognizes both uses of arena and creates single `struct bpf_map *`
instance in libbpf APIs.
".arena.1" ELF section data is used as initial data image, which is exposed
through skeleton and bpf_map__initial_value() to the user, if they need to tune
it before the load phase. During load phase, this initial image is copied over
into mmap()'ed region corresponding to arena, and discarded.
Few small checks here and there had to be added to make sure this
approach works with bpf_map__initial_value(), mostly due to hard-coded
assumption that map->mmaped is set up with mmap() syscall and should be
munmap()'ed. For arena, .arena.1 can be (much) smaller than maximum
arena size, so this smaller data size has to be tracked separately.
Given it is enforced that there is only one arena for entire bpf_object
instance, we just keep it in a separate field. This can be generalized
if necessary later.
All global variables from ".arena.1" section are accessible from user space
via skel->arena->name_of_var.
For bss/data/rodata the skeleton/libbpf perform the following sequence:
1. addr = mmap(MAP_ANONYMOUS)
2. user space optionally modifies global vars
3. map_fd = bpf_create_map()
4. bpf_update_map_elem(map_fd, addr) // to store values into the kernel
5. mmap(addr, MAP_FIXED, map_fd)
after step 5 user spaces see the values it wrote at step 2 at the same addresses
arena doesn't support update_map_elem. Hence skeleton/libbpf do:
1. addr = malloc(sizeof SEC ".arena.1")
2. user space optionally modifies global vars
3. map_fd = bpf_create_map(MAP_TYPE_ARENA)
4. real_addr = mmap(map->map_extra, MAP_SHARED | MAP_FIXED, map_fd)
5. memcpy(real_addr, addr) // this will fault-in and allocate pages
At the end look and feel of global data vs __arena global data is the same from
bpf prog pov.
Another complication is:
struct {
__uint(type, BPF_MAP_TYPE_ARENA);
} arena SEC(".maps");
int __arena foo;
int bar;
ptr1 = &foo; // relocation against ".arena.1" section
ptr2 = &arena; // relocation against ".maps" section
ptr3 = &bar; // relocation against ".bss" section
Fo the kernel ptr1 and ptr2 has point to the same arena's map_fd
while ptr3 points to a different global array's map_fd.
For the verifier:
ptr1->type == unknown_scalar
ptr2->type == const_ptr_to_map
ptr3->type == ptr_to_map_value
After verification, from JIT pov all 3 ptr-s are normal ld_imm64 insns.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Quentin Monnet <quentin@isovalent.com>
Link: https://lore.kernel.org/bpf/20240308010812.89848-11-alexei.starovoitov@gmail.com
According to a report, skeletons fail to assign shadow pointers when being compiled with C++ programs. Unlike C doing implicit casting for void pointers, C++ requires an explicit casting. To support C++, we do explicit casting for each shadow pointer. Also add struct_ops_module.skel.h to test_cpp to validate C++ compilation as part of BPF selftests. Signed-off-by: Kui-Feng Lee <thinker.li@gmail.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Yonghong Song <yonghong.song@linux.dev> Acked-by: Quentin Monnet <quentin@isovalent.com> Link: https://lore.kernel.org/bpf/20240312013726.1780720-1-thinker.li@gmail.com
Current 'bpftool link' command does not show pids, e.g.,
$ tools/build/bpftool/bpftool link
...
4: tracing prog 23
prog_type lsm attach_type lsm_mac
target_obj_id 1 target_btf_id 31320
Hack the following change to enable normal libbpf debug output,
--- a/tools/bpf/bpftool/pids.c
+++ b/tools/bpf/bpftool/pids.c
@@ -121,9 +121,9 @@ int build_obj_refs_table(struct hashmap **map, enum bpf_obj_type type)
/* we don't want output polluted with libbpf errors if bpf_iter is not
* supported
*/
- default_print = libbpf_set_print(libbpf_print_none);
+ /* default_print = libbpf_set_print(libbpf_print_none); */
err = pid_iter_bpf__load(skel);
- libbpf_set_print(default_print);
+ /* libbpf_set_print(default_print); */
Rerun the above bpftool command:
$ tools/build/bpftool/bpftool link
libbpf: prog 'iter': BPF program load failed: Permission denied
libbpf: prog 'iter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; struct task_struct *task = ctx->task; @ pid_iter.bpf.c:69
0: (79) r6 = *(u64 *)(r1 +8) ; R1=ctx() R6_w=ptr_or_null_task_struct(id=1)
; struct file *file = ctx->file; @ pid_iter.bpf.c:68
...
; struct bpf_link *link = (struct bpf_link *) file->private_data; @ pid_iter.bpf.c:103
80: (79) r3 = *(u64 *)(r8 +432) ; R3_w=scalar() R8=ptr_file()
; if (link->type == bpf_core_enum_value(enum bpf_link_type___local, @ pid_iter.bpf.c:105
81: (61) r1 = *(u32 *)(r3 +12)
R3 invalid mem access 'scalar'
processed 39 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 2
-- END PROG LOAD LOG --
libbpf: prog 'iter': failed to load: -13
...
The 'file->private_data' returns a 'void' type and this caused subsequent 'link->type'
(insn libbpf#81) failed in verification.
To fix the issue, restore the previous BPF_CORE_READ so old kernels can also work.
With this patch, the 'bpftool link' runs successfully with 'pids'.
$ tools/build/bpftool/bpftool link
...
4: tracing prog 23
prog_type lsm attach_type lsm_mac
target_obj_id 1 target_btf_id 31320
pids systemd(1)
Fixes: 44ba7b30e84f ("bpftool: Use a local copy of BPF_LINK_TYPE_PERF_EVENT in pid_iter.bpf.c")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Tested-by: Quentin Monnet <quentin@isovalent.com>
Reviewed-by: Quentin Monnet <quentin@isovalent.com>
Link: https://lore.kernel.org/bpf/20240312023249.3776718-1-yonghong.song@linux.dev
Wire up BPF cookie for raw tracepoint programs (both BTF and non-BTF aware variants). This brings them up to part w.r.t. BPF cookie usage with classic tracepoint and fentry/fexit programs. Acked-by: Stanislav Fomichev <sdf@google.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Message-ID: <20240319233852.1977493-4-andrii@kernel.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
When trying to load the pid_iter BPF program used to iterate over the PIDs of the processes holding file descriptors to BPF links, we would unconditionally silence libbpf in order to keep the output clean if the kernel does not support iterators and loading fails. Although this is the desirable behaviour in most cases, this may hide bugs in the pid_iter program that prevent it from loading, and it makes it hard to debug such load failures, even in "debug" mode. Instead, it makes more sense to print libbpf's logs when we pass the -d|--debug flag to bpftool, so that users get the logs to investigate failures without having to edit bpftool's source code. Signed-off-by: Quentin Monnet <qmo@kernel.org> Message-ID: <20240320012241.42991-1-qmo@kernel.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Commit d510296d331a ("bpftool: Use syscall/loader program in "prog load"
and "gen skeleton" command.") added new files to the list of objects to
compile in order to build the bootstrap version of bpftool. As far as I
can tell, these objects are unnecessary and were added by mistake; maybe
a draft version intended to add support for loading loader programs from
the bootstrap version. Anyway, we can remove these object files from the
list to make the bootstrap bpftool binary a tad smaller and faster to
build.
Fixes: d510296d331a ("bpftool: Use syscall/loader program in "prog load" and "gen skeleton" command.")
Signed-off-by: Quentin Monnet <qmo@kernel.org>
Message-ID: <20240320013457.44808-1-qmo@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Bpftool's Makefile uses $(HOST_CFLAGS) to build the bootstrap version of bpftool, in order to pick the flags for the host (where we run the bootstrap version) and not for the target system (where we plan to run the full bpftool binary). But we pass too much information through this variable. In particular, we set HOST_CFLAGS by copying most of the $(CFLAGS); but we do this after the feature detection for bpftool, which means that $(CFLAGS), hence $(HOST_CFLAGS), contain all macro definitions for using the different optional features. For example, -DHAVE_LLVM_SUPPORT may be passed to the $(HOST_CFLAGS), even though the LLVM disassembler is not used in the bootstrap version, and the related library may even be missing for the host architecture. A similar thing happens with the $(LDFLAGS), that we use unchanged for linking the bootstrap version even though they may contains flags to link against additional libraries. To address the $(HOST_CFLAGS) issue, we move the definition of $(HOST_CFLAGS) earlier in the Makefile, before the $(CFLAGS) update resulting from the feature probing - none of which being relevant to the bootstrap version. To clean up the $(LDFLAGS) for the bootstrap version, we introduce a dedicated $(HOST_LDFLAGS) variable that we base on $(LDFLAGS), before the feature probing as well. On my setup, the following macro and libraries are removed from the compiler invocation to build bpftool after this patch: -DUSE_LIBCAP -DHAVE_LLVM_SUPPORT -I/usr/lib/llvm-17/include -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -lLLVM-17 -L/usr/lib/llvm-17/lib Another advantage of cleaning up these flags is that displaying available features with "bpftool version" becomes more accurate for the bootstrap bpftool, and no longer reflects the features detected (and available only) for the final binary. Cc: Jean-Philippe Brucker <jean-philippe@linaro.org> Signed-off-by: Quentin Monnet <qmo@kernel.org> Acked-by: Jiri Olsa <jolsa@kernel.org> Message-ID: <20240320014103.45641-1-qmo@kernel.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
There is a difference between kernel uapi bpf.h and tools uapi bpf.h. There is no functionality difference, but let us sync properly to make it easy for later bpf.h update. Signed-off-by: Yonghong Song <yonghong.song@linux.dev> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20240325033842.1693553-1-yonghong.song@linux.dev
The selftests use to tell LLVM about special pointers. For LLVM there is nothing "arena" about them. They are simply pointers in a different address space. Hence LLVM diff llvm/llvm-project#85161 renamed: . macro __BPF_FEATURE_ARENA_CAST -> __BPF_FEATURE_ADDR_SPACE_CAST . global variables in __attribute__((address_space(N))) are now placed in section named ".addr_space.N" instead of ".arena.N". Adjust libbpf, bpftool, and selftests to match LLVM. Signed-off-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Stanislav Fomichev <sdf@google.com> Link: https://lore.kernel.org/bpf/20240315021834.62988-3-alexei.starovoitov@gmail.com
Syncing latest bpftool commits from kernel repository. Baseline bpf-next commit: e63985ecd22681c7f5975f2e8637187a326b6791 Checkpoint bpf-next commit: 14bb1e8c8d4ad5d9d2febb7d19c70a3cf536e1e5 Baseline bpf commit: 2487007aa3b9fafbd2cb14068f49791ce1d7ede5 Checkpoint bpf commit: 443574b033876c85a35de4c65c14f7fe092222b2 Alexei Starovoitov (4): bpf: Introduce bpf_arena. bpf: Disasm support for addr_space_cast instruction. bpftool: Recognize arena map type libbpf, selftests/bpf: Adjust libbpf, bpftool, selftests to match LLVM Andrii Nakryiko (3): bpftool: rename is_internal_mmapable_map into is_mmapable_map libbpf: Recognize __arena global variables. bpf: support BPF cookie in raw tracepoint (raw_tp, tp_btf) programs Kui-Feng Lee (1): bpftool: Cast pointers for shadow types explicitly. Quentin Monnet (3): bpftool: Enable libbpf logs when loading pid_iter in debug mode bpftool: Remove unnecessary source files from bootstrap version bpftool: Clean up HOST_CFLAGS, HOST_LDFLAGS for bootstrap bpftool Yonghong Song (2): bpftool: Fix missing pids during link show bpf: Sync uapi bpf.h to tools directory docs/bpftool-map.rst | 2 +- include/uapi/linux/bpf.h | 20 ++++++++++++++++++-- src/Makefile | 14 ++++++-------- src/gen.c | 34 ++++++++++++++++++++++++---------- src/kernel/bpf/disasm.c | 10 ++++++++++ src/map.c | 2 +- src/pids.c | 19 ++++++++++++------- src/skeleton/pid_iter.bpf.c | 4 ++-- 8 files changed, 74 insertions(+), 31 deletions(-) Signed-off-by: Quentin Monnet <qmo@kernel.org>
A recent patch has touched some portions of bpftool's Makefile that differ between kernel's and mirror's sources. Let's update the diff with the expected differences accordingly, to smoothen future sync ups. Signed-off-by: Quentin Monnet <qmo@kernel.org>
Pull latest libbpf from mirror. Libbpf version: 1.4.0 Libbpf commit: 6d3595d215b014d3eddb88038d686e1c20781534 Signed-off-by: Quentin Monnet <qmo@kernel.org>
Extend the bpf_fib_lookup() helper by making it to utilize mark if the BPF_FIB_LOOKUP_MARK flag is set. In order to pass the mark the four bytes of struct bpf_fib_lookup are used, shared with the output-only smac/dmac fields. Signed-off-by: Anton Protopopov <aspsk@isovalent.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Reviewed-by: David Ahern <dsahern@kernel.org> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20240326101742.17421-2-aspsk@isovalent.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
BPF verifier emits "unknown func" message when given BPF program type does not support BPF helper. This message may be confusing for users, as important context that helper is unknown only to current program type is not provided. This patch changes message to "program of this type cannot use helper " and aligns dependent code in libbpf and tests. Any suggestions on improving/changing this message are welcome. Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com> Acked-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Quentin Monnet <qmo@kernel.org> Link: https://lore.kernel.org/r/20240325152210.377548-1-yatsenko@meta.com Signed-off-by: Alexei Starovoitov <ast@kernel.org>
The rST manual pages for bpftool would use a mix of tabs and spaces for indentation. While this is the norm in C code, this is rather unusual for rST documents, and over time we've seen many contributors use a wrong level of indentation for documentation update. Let's fix bpftool's indentation in docs once and for all: - Let's use spaces, that are more common in rST files. - Remove one level of indentation for the synopsis, the command description, and the "see also" section. As a result, all sections start with the same indentation level in the generated man page. - Rewrap the paragraphs after the changes. There is no content change in this patch, only indentation and rewrapping changes. The wrapping in the generated source files for the manual pages is changed, but the pages displayed with "man" remain the same, apart from the adjusted indentation level on relevant sections. [ Quentin: rebased on bpf-next, removed indent level for command description and options, updated synopsis, command summary, and "see also" sections. ] Signed-off-by: Rameez Rehman <rameezrehman408@hotmail.com> Signed-off-by: Quentin Monnet <qmo@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20240331200346.29118-2-qmo@kernel.org
As it turns out, the terms in definition lists in the rST file are
already rendered with bold-ish formatting when generating the man pages;
all double-star sequences we have in the commands for the command
description are unnecessary, and can be removed to make the
documentation easier to read.
The rST files were automatically processed with:
sed -i '/DESCRIPTION/,/OPTIONS/ { /^\*/ s/\*\*//g }' b*.rst
Signed-off-by: Rameez Rehman <rameezrehman408@hotmail.com>
Signed-off-by: Quentin Monnet <qmo@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20240331200346.29118-3-qmo@kernel.org
Improve the formatting of the attach flags for cgroup programs in the
relevant man page, and fix typos ("can be on of", "an userspace inet
socket") when introducing that list. Also fix a couple of other trivial
issues in docs.
[ Quentin: Fixed trival issues in bpftool-gen.rst and bpftool-iter.rst ]
Signed-off-by: Rameez Rehman <rameezrehman408@hotmail.com>
Signed-off-by: Quentin Monnet <qmo@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20240331200346.29118-4-qmo@kernel.org
In a few places in the bpf uapi headers, EOPNOTSUPP is missing a "P" in the doc comments. This adds the missing "P". Signed-off-by: David Lechner <dlechner@baylibre.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20240329152900.398260-2-dlechner@baylibre.com
When generated BPF skeleton header is included in C++ code base, some
compiler setups will emit warning about using language extensions due to
typeof() usage, resulting in something like:
error: extension used [-Werror,-Wlanguage-extension-token]
obj->struct_ops.empty_tcp_ca = (typeof(obj->struct_ops.empty_tcp_ca))
^
It looks like __typeof__() is a preferred way to do typeof() with better
C++ compatibility behavior, so switch to that. With __typeof__() we get
no such warning.
Fixes: c2a0257c1edf ("bpftool: Cast pointers for shadow types explicitly.")
Fixes: 00389c58ffe9 ("bpftool: Add support for subskeletons")
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Kui-Feng Lee <thinker.li@gmail.com>
Acked-by: Quentin Monnet <qmo@kernel.org>
Acked-by: John Fastabend <john.fastabend@gmail.com>
Link: https://lore.kernel.org/bpf/20240401170713.2081368-1-andrii@kernel.org
Syncing latest bpftool commits from kernel repository. Baseline bpf-next commit: 14bb1e8c8d4ad5d9d2febb7d19c70a3cf536e1e5 Checkpoint bpf-next commit: 2a24e2485722b0e12e17a2bd473bd15c9e420bdb Baseline bpf commit: 443574b033876c85a35de4c65c14f7fe092222b2 Checkpoint bpf commit: 443574b033876c85a35de4c65c14f7fe092222b2 Andrii Nakryiko (1): bpftool: Use __typeof__() instead of typeof() in BPF skeleton Anton Protopopov (1): bpf: Add support for passing mark with bpf_fib_lookup David Lechner (1): bpf: Fix typo in uapi doc comments Mykyta Yatsenko (1): bpf: improve error message for unsupported helper Rameez Rehman (3): bpftool: Use simpler indentation in source rST for documentation bpftool: Remove useless emphasis on command description in man pages bpftool: Clean-up typos, punctuation, list formatting in docs docs/Makefile | 6 +- docs/bpftool-btf.rst | 116 +++++----- docs/bpftool-cgroup.rst | 193 ++++++++-------- docs/bpftool-feature.rst | 115 +++++----- docs/bpftool-gen.rst | 338 +++++++++++++--------------- docs/bpftool-iter.rst | 52 ++--- docs/bpftool-link.rst | 73 +++--- docs/bpftool-map.rst | 232 ++++++++++---------- docs/bpftool-net.rst | 98 ++++----- docs/bpftool-perf.rst | 34 +-- docs/bpftool-prog.rst | 426 +++++++++++++++++------------------- docs/bpftool-struct_ops.rst | 71 +++--- docs/bpftool.rst | 60 ++--- docs/common_options.rst | 26 +-- include/uapi/linux/bpf.h | 24 +- src/feature.c | 3 +- src/gen.c | 4 +- 17 files changed, 900 insertions(+), 971 deletions(-) Signed-off-by: Quentin Monnet <qmo@kernel.org>
If a value is assigned to the variable DESTDIR when compiling bpftool, this variable will be passed to libbpf's Makefile. In this case, libbpf's header files will be installed in an unexpected directory, causing bpftool to be unable to find the header files. Signed-off-by: jinzhiguang <jinzhiguang@kylinos.cn>
Signed-off-by: Quentin Monnet <qmo@kernel.org>
Signed-off-by: Quentin Monnet <qmo@kernel.org>
Won't work because programs are loaded only for tests with links - we need to load it before, e.g. as class fixture? Signed-off-by: Quentin Monnet <qmo@kernel.org>
No need for a copy, let's go faster Signed-off-by: Quentin Monnet <qmo@kernel.org>
Signed-off-by: Quentin Monnet <qmo@kernel.org>
qmonnet
force-pushed
the
wip/ci-bash-completion
branch
from
0198d83
to
128c423
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.