Skip to content

Commit

Permalink
ARC: Add eBPF JIT support
Browse files Browse the repository at this point in the history
This will add eBPF JIT support to the 32-bit ARCv2 processors. The
implementation is qualified by running the BPF tests on a Synopsys HSDK
board with "ARC HS38 v2.1c at 500 MHz" as the 4-core CPU.

Deployment and structure
------------------------
The related codes are added to "arch/arc/net":

- bpf_jit.h       -- The interface that a back-end translator must provide
- bpf_jit_core.c  -- Knows how to handle the input eBPF byte stream
- bpf_jit_arcv2.c -- The back-end code that knows the translation logic

The bpf_int_jit_compile() at the end of bpf_jit_core.c is the entrance
to the whole process. Normally, the translation is done in one pass,
namely the "normal pass". In case some relocations are not known during
this pass, some data (arc_jit_data) is allocated for the next pass to
come. This possible next (and last) pass is called the "extra pass".

1. Normal pass       # The necessary pass
     1a. Dry run       # Get the whole JIT length, epilogue offset, etc.
     1b. Emit phase    # Allocate memory and start emitting instructions
2. Extra pass        # Only needed if there are relocations to be fixed
     2a. Patch relocations

Support status
------------------------
This JIT compiler does NOT yet provide support for:

- Tail calls
- Atomic operations
- 64-bit division/remainder
- BPF_PROBE_MEM* (exception table)

The result of "test_bpf" test suite on an HSDK board is:

hsdk-lnx# insmod test_bpf.ko test_suite=test_bpf

  test_bpf: Summary: 863 PASSED, 186 FAILED, [851/851 JIT'ed]

All the failing test cases are due to the ones that were not JIT'ed.
Categorically, they can be represented as:

  .-----------.------------.-------------.
  | test type |   opcodes  | # of cases  |
  |-----------+------------+-------------|
  | atomic    | 0xC3, 0xDB |         149 |
  | div64     | 0x37, 0x3F |          22 |
  | mod64     | 0x97, 0x9F |          15 |
  `-----------^------------+-------------|
                           | (total) 186 |
                           `-------------'

Setup: build config
-------------------
The following configs must be set to have a working JIT test:

  CONFIG_BPF_JIT=y
  CONFIG_BPF_JIT_ALWAYS_ON=y
  CONFIG_TEST_BPF=m

The following options are not necessary for the tests module,
but are good to have:

  CONFIG_DEBUG_INFO=y             # prerequisite for below
  CONFIG_DEBUG_INFO_BTF=y         # so bpftool can generate vmlinux.h

  CONFIG_FTRACE=y                 #
  CONFIG_BPF_SYSCALL=y            # all these options lead to
  CONFIG_KPROBE_EVENTS=y          # having CONFIG_BPF_EVENTS=y
  CONFIG_PERF_EVENTS=y            #

Some BPF programs provide data through /sys/kernel/debug:
  CONFIG_DEBUG_FS=y
arc# mount -t debugfs debugfs /sys/kernel/debug

Setup: elfutils
---------------
The libdw.{so,a} library that is used by pahole for processing
the final binary must come from elfutils 0.189 or newer. The
support for ARCv2 [1] has been added since that version.

[1]
https://sourceware.org/git/?p=elfutils.git;a=commit;h=de3d46b3e7

Setup: pahole
-------------
The line below in linux/scripts/Makefile.btf must be commented out:

pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats

Or else, the build will fail:

$ make V=1
  ...
  BTF     .btf.vmlinux.bin.o
pahole -J --btf_gen_floats                    \
       -j --lang_exclude=rust                 \
       --skip_encoding_btf_inconsistent_proto \
       --btf_gen_optimized .tmp_vmlinux.btf
Complex, interval and imaginary float types are not supported
Encountered error while encoding BTF.
  ...
  BTFIDS  vmlinux
./tools/bpf/resolve_btfids/resolve_btfids vmlinux
libbpf: failed to find '.BTF' ELF section in vmlinux
FAILED: load BTF from vmlinux: No data available

This is due to the fact that the ARC toolchains generate
"complex float" DIE entries in libgcc and at the moment, pahole
can't handle such entries.

Running the tests
-----------------
host$ scp /bld/linux/lib/test_bpf.ko arc:
arc # sysctl net.core.bpf_jit_enable=1
arc # insmod test_bpf.ko test_suite=test_bpf
      ...
      test_bpf: #1048 Staggered jumps: JMP32_JSLE_X jited:1 697811 PASS
      test_bpf: Summary: 863 PASSED, 186 FAILED, [851/851 JIT'ed]

Acknowledgments
---------------
- Claudiu Zissulescu for his unwavering support
- Yuriy Kolerov for testing and troubleshooting
- Vladimir Isaev for the pahole workaround
- Sergey Matyukevich for paving the road by adding the interpretor support

Signed-off-by: Shahab Vahedi <shahab@synopsys.com>
  • Loading branch information
Shahab Vahedi authored and Kernel Patches Daemon committed Feb 13, 2024
1 parent e9e616a commit 742909a
Show file tree
Hide file tree
Showing 8 changed files with 4,598 additions and 2 deletions.
1 change: 1 addition & 0 deletions Documentation/admin-guide/sysctl/net.rst
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ two flavors of JITs, the newer eBPF JIT currently supported on:
- riscv64
- riscv32
- loongarch64
- arc

And the older cBPF JIT supported on the following archs:

Expand Down
4 changes: 2 additions & 2 deletions Documentation/networking/filter.rst
Original file line number Diff line number Diff line change
Expand Up @@ -513,7 +513,7 @@ JIT compiler
------------

The Linux kernel has a built-in BPF JIT compiler for x86_64, SPARC,
PowerPC, ARM, ARM64, MIPS, RISC-V and s390 and can be enabled through
PowerPC, ARM, ARM64, MIPS, RISC-V, s390, and ARC and can be enabled through
CONFIG_BPF_JIT. The JIT compiler is transparently invoked for each
attached filter from user space or for internal kernel users if it has
been previously enabled by root::
Expand Down Expand Up @@ -650,7 +650,7 @@ before a conversion to the new layout is being done behind the scenes!

Currently, the classic BPF format is being used for JITing on most
32-bit architectures, whereas x86-64, aarch64, s390x, powerpc64,
sparc64, arm32, riscv64, riscv32, loongarch64 perform JIT compilation
sparc64, arm32, riscv64, riscv32, loongarch64, arc perform JIT compilation
from eBPF instruction set.

Testing
Expand Down
1 change: 1 addition & 0 deletions arch/arc/Kbuild
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
obj-y += kernel/
obj-y += mm/
obj-y += net/

# for cleaning
subdir- += boot
1 change: 1 addition & 0 deletions arch/arc/Kconfig
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ config ARC
select PCI_SYSCALL if PCI
select HAVE_ARCH_JUMP_LABEL if ISA_ARCV2 && !CPU_ENDIAN_BE32
select TRACE_IRQFLAGS_SUPPORT
select HAVE_EBPF_JIT if ISA_ARCV2

config LOCKDEP_SUPPORT
def_bool y
Expand Down
6 changes: 6 additions & 0 deletions arch/arc/net/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# SPDX-License-Identifier: GPL-2.0-only

ifeq ($(CONFIG_ISA_ARCV2),y)
obj-$(CONFIG_BPF_JIT) += bpf_jit_core.o
obj-$(CONFIG_BPF_JIT) += bpf_jit_arcv2.o
endif
161 changes: 161 additions & 0 deletions arch/arc/net/bpf_jit.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* The interface that a back-end should provide to bpf_jit_core.c.
*
* Copyright (c) 2024 Synopsys Inc.
* Author: Shahab Vahedi <shahab@synopsys.com>
*/

#ifndef _ARC_BPF_JIT_H
#define _ARC_BPF_JIT_H

#include <linux/bpf.h>
#include <linux/filter.h>

/* Print debug info and assert. */
//#define ARC_BPF_JIT_DEBUG

/* Determine the address type of the target. */
#ifdef CONFIG_ISA_ARCV2
#define ARC_ADDR u32
#endif

/*
* For the translation of some BPF instructions, a temporary register
* might be needed for some interim data.
*/
#define JIT_REG_TMP MAX_BPF_JIT_REG

/************* Globals that have effects on code generation ***********/
/*
* If "emit" is true, the instructions are actually generated. Else, the
* generation part will be skipped and only the length of instruction is
* returned by the responsible functions.
*/
extern bool emit;

/* An indicator if zero-extend must be done for the 32-bit operations. */
extern bool zext_thyself;

/************** Functions that the back-end must provide **************/
/* Extension for 32-bit operations. */
extern inline u8 zext(u8 *buf, u8 rd);
/***** Moves *****/
extern u8 mov_r32(u8 *buf, u8 rd, u8 rs, u8 sign_ext);
extern u8 mov_r32_i32(u8 *buf, u8 reg, s32 imm);
extern u8 mov_r64(u8 *buf, u8 rd, u8 rs, u8 sign_ext);
extern u8 mov_r64_i32(u8 *buf, u8 reg, s32 imm);
extern u8 mov_r64_i64(u8 *buf, u8 reg, u32 lo, u32 hi);
/***** Loads and stores *****/
extern u8 load_r(u8 *buf, u8 rd, u8 rs, s16 off, u8 size, bool sign_ext);
extern u8 store_r(u8 *buf, u8 rd, u8 rs, s16 off, u8 size);
extern u8 store_i(u8 *buf, s32 imm, u8 rd, s16 off, u8 size);
/***** Addition *****/
extern u8 add_r32(u8 *buf, u8 rd, u8 rs);
extern u8 add_r32_i32(u8 *buf, u8 rd, s32 imm);
extern u8 add_r64(u8 *buf, u8 rd, u8 rs);
extern u8 add_r64_i32(u8 *buf, u8 rd, s32 imm);
/***** Subtraction *****/
extern u8 sub_r32(u8 *buf, u8 rd, u8 rs);
extern u8 sub_r32_i32(u8 *buf, u8 rd, s32 imm);
extern u8 sub_r64(u8 *buf, u8 rd, u8 rs);
extern u8 sub_r64_i32(u8 *buf, u8 rd, s32 imm);
/***** Multiplication *****/
extern u8 mul_r32(u8 *buf, u8 rd, u8 rs);
extern u8 mul_r32_i32(u8 *buf, u8 rd, s32 imm);
extern u8 mul_r64(u8 *buf, u8 rd, u8 rs);
extern u8 mul_r64_i32(u8 *buf, u8 rd, s32 imm);
/***** Division *****/
extern u8 div_r32(u8 *buf, u8 rd, u8 rs, bool sign_ext);
extern u8 div_r32_i32(u8 *buf, u8 rd, s32 imm, bool sign_ext);
/***** Remainder *****/
extern u8 mod_r32(u8 *buf, u8 rd, u8 rs, bool sign_ext);
extern u8 mod_r32_i32(u8 *buf, u8 rd, s32 imm, bool sign_ext);
/***** Bitwise AND *****/
extern u8 and_r32(u8 *buf, u8 rd, u8 rs);
extern u8 and_r32_i32(u8 *buf, u8 rd, s32 imm);
extern u8 and_r64(u8 *buf, u8 rd, u8 rs);
extern u8 and_r64_i32(u8 *buf, u8 rd, s32 imm);
/***** Bitwise OR *****/
extern u8 or_r32(u8 *buf, u8 rd, u8 rs);
extern u8 or_r32_i32(u8 *buf, u8 rd, s32 imm);
extern u8 or_r64(u8 *buf, u8 rd, u8 rs);
extern u8 or_r64_i32(u8 *buf, u8 rd, s32 imm);
/***** Bitwise XOR *****/
extern u8 xor_r32(u8 *buf, u8 rd, u8 rs);
extern u8 xor_r32_i32(u8 *buf, u8 rd, s32 imm);
extern u8 xor_r64(u8 *buf, u8 rd, u8 rs);
extern u8 xor_r64_i32(u8 *buf, u8 rd, s32 imm);
/***** Bitwise Negate *****/
extern u8 neg_r32(u8 *buf, u8 r);
extern u8 neg_r64(u8 *buf, u8 r);
/***** Bitwise left shift *****/
extern u8 lsh_r32(u8 *buf, u8 rd, u8 rs);
extern u8 lsh_r32_i32(u8 *buf, u8 rd, u8 imm);
extern u8 lsh_r64(u8 *buf, u8 rd, u8 rs);
extern u8 lsh_r64_i32(u8 *buf, u8 rd, s32 imm);
/***** Bitwise right shift (logical) *****/
extern u8 rsh_r32(u8 *buf, u8 rd, u8 rs);
extern u8 rsh_r32_i32(u8 *buf, u8 rd, u8 imm);
extern u8 rsh_r64(u8 *buf, u8 rd, u8 rs);
extern u8 rsh_r64_i32(u8 *buf, u8 rd, s32 imm);
/***** Bitwise right shift (arithmetic) *****/
extern u8 arsh_r32(u8 *buf, u8 rd, u8 rs);
extern u8 arsh_r32_i32(u8 *buf, u8 rd, u8 imm);
extern u8 arsh_r64(u8 *buf, u8 rd, u8 rs);
extern u8 arsh_r64_i32(u8 *buf, u8 rd, s32 imm);
/***** Frame related *****/
extern u32 mask_for_used_regs(u8 bpf_reg, bool is_call);
extern u8 arc_prologue(u8 *buf, u32 usage, u16 frame_size);
extern u8 arc_epilogue(u8 *buf, u32 usage, u16 frame_size);
/***** Jumps *****/
/*
* Different sorts of conditions (ARC enum as opposed to BPF_*).
*
* Do not change the order of enums here. ARC_CC_SLE+1 is used
* to determine the number of JCCs.
*/
enum ARC_CC {
ARC_CC_UGT = 0, /* unsigned > */
ARC_CC_UGE, /* unsigned >= */
ARC_CC_ULT, /* unsigned < */
ARC_CC_ULE, /* unsigned <= */
ARC_CC_SGT, /* signed > */
ARC_CC_SGE, /* signed >= */
ARC_CC_SLT, /* signed < */
ARC_CC_SLE, /* signed <= */
ARC_CC_AL, /* always */
ARC_CC_EQ, /* == */
ARC_CC_NE, /* != */
ARC_CC_SET, /* test */
ARC_CC_LAST
};
/*
* A few notes:
*
* - check_jmp_*() are prerequisites before calling the gen_jmp_*().
* They return "true" if the jump is possible and "false" otherwise.
*
* - The notion of "*_off" is to emphasize that these parameters are
* merely offsets in the JIT stream and not absolute addresses. One
* can look at them as addresses if the JIT code would start from
* address 0x0000_0000. Nonetheless, since the buffer address for the
* JIT is on a word-aligned address, this works and actually makes
* things simpler (offsets are in the range of u32 which is more than
* enough).
*/
extern bool check_jmp_32(u32 curr_off, u32 targ_off, u8 cond);
extern bool check_jmp_64(u32 curr_off, u32 targ_off, u8 cond);
extern u8 gen_jmp_32(u8 *buf, u8 rd, u8 rs, u8 cond, u32 c_off, u32 t_off);
extern u8 gen_jmp_64(u8 *buf, u8 rd, u8 rs, u8 cond, u32 c_off, u32 t_off);
/***** Miscellaneous *****/
extern u8 gen_func_call(u8 *buf, ARC_ADDR func_addr, bool external_func);
extern u8 arc_to_bpf_return(u8 *buf);
/*
* Perform byte swaps on "rd" based on the "size". If "force" is
* set to "true", do it unconditionally. Otherwise, consider the
* desired "endian"ness and the host endianness.
*/
extern u8 gen_swap(u8 *buf, u8 rd, u8 size, u8 endian, bool force);

#endif /* _ARC_BPF_JIT_H */
Loading

0 comments on commit 742909a

Please sign in to comment.