bpf, arm64: JIT support for private stack #9339
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
puranjaymohan
force-pushed
the
arm64_private_stack
branch
from
35cf8d9 to
c6aee9b
Compare
In the ARM64 BPF JIT when prog->aux->exception_boundary is set for a BPF program, find_used_callee_regs() is not called because for a program acting as exception boundary, all callee saved registers are saved. find_used_callee_regs() sets `ctx->fp_used = true;` when it sees FP being used in any of the instructions. For programs acting as exception boundary, ctx->fp_used remains false even if frame pointer is used by the program and therefore, FP is not set-up for such programs in the prologue. This can cause the kernel to crash due to a pagefault. Fix it by setting ctx->fp_used = true for exception boundary programs as fp is always saved in such programs. Fixes: 5d4fa9e ("bpf, arm64: Avoid blindly saving/restoring all callee-saved registers") Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
The private stack is allocated in bpf_int_jit_compile() with 16-byte
alignment. It includes additional guard regions to detect stack
overflows and underflows at runtime.
Memory layout:
+------------------------------------------------------+
| |
| 16 bytes padding (overflow guard - stack top) |
| [ detects writes beyond top of stack ] |
BPF FP ->+------------------------------------------------------+
| |
| BPF private stack (sized by verifier) |
| [ 16-byte aligned ] |
| |
BPF PRIV SP ->+------------------------------------------------------+
| |
| 16 bytes padding (underflow guard - stack bottom) |
| [ detects accesses before start of stack ] |
| |
+------------------------------------------------------+
On detection of an overflow or underflow, the kernel emits messages
like:
BPF private stack overflow/underflow detected for prog <prog_name>
After commit bd737fc ("bpf, arm64: Get rid of fpb"), Jited BPF
programs use the stack in two ways:
1. Via the BPF frame pointer (top of stack), using negative offsets.
2. Via the stack pointer (bottom of stack), using positive offsets in
LDR/STR instructions.
When a private stack is used, ARM64 callee-saved register x27 replaces
the stack pointer. The BPF frame pointer usage remains unchanged; but it
now points to the top of the private stack.
Relevant tests:
kernel-patches#415/1 struct_ops_private_stack/private_stack:OK
kernel-patches#415/2 struct_ops_private_stack/private_stack_fail:OK
kernel-patches#415/3 struct_ops_private_stack/private_stack_recur:OK
kernel-patches#415 struct_ops_private_stack:OK
kernel-patches#549/1 verifier_private_stack/Private stack, single prog:OK
kernel-patches#549/2 verifier_private_stack/Private stack, subtree > MAX_BPF_STACK:OK
kernel-patches#549/3 verifier_private_stack/No private stack:OK
kernel-patches#549/4 verifier_private_stack/Private stack, callback:OK
kernel-patches#549/5 verifier_private_stack/Private stack, exception in main prog:OK
kernel-patches#549/6 verifier_private_stack/Private stack, exception in subprog:OK
kernel-patches#549/7 verifier_private_stack/Private stack, async callback, not nested:OK
kernel-patches#549/8 verifier_private_stack/Private stack, async callback, potential nesting:OK
kernel-patches#549 verifier_private_stack:OK
Summary: 2/11 PASSED, 0 SKIPPED, 0 FAILED
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
puranjaymohan
force-pushed
the
arm64_private_stack
branch
from
c6aee9b to
fd7affb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Private stack is allocated in bpf_int_jit_compile() with an alignment of 16. Private stack allocation size includes the stack size determined by verifier and additional space to protect stack overflow and underflow. See below an illustration:
---> memory address increasing
[16 bytes to protect overflow] [normal stack] [16 bytes to protect underflow]
If overflow/underflow is detected, kernel messages will be emited in dmesg like
BPF private stack overflow/underflow detected for prog Fx
BPF Private stack overflow/underflow detected for prog bpf_prog_a41699c234a1567a_subprog1x
After commit bd737fc ("bpf, arm64: Get rid of fpb"), Jited BPF programs use the stack in two ways:
When a private stack is used, ARM64 callee-saved register x27 replaces the stack pointer. The frame pointer usage remains unchanged; but it now points to the top of the private stack.