| Age | Commit message (Collapse) | Author |
|---|
|
The JITs pass the first stack arguments of a call in registers:
x86-64 r9 argument 6
arm64 x5-x7 arguments 6-8
riscv64 a5-a7 arguments 6-8
'*(u64 *)(r11 - 8) = rX' writes such a register in the caller.
'rX = *(u64 *)(r11 + 8)' reads it in the callee. A call changes these
registers. Thus the verifier accepts no call between the write and the call
of the callee, and no call in the callee before the read.
may_goto is not a call for the verifier. But one time in 65535 iterations
it calls arch_bpf_timed_may_goto(), and this function calls C code. It
saves R0-R5 only.
*(u64 *)(r11 - 8) = 0 ; the verifier: argument 6 is 0
call sub
sub:
1: may_goto 2f ; C code changes r9
goto 1b
2: r2 = *(u64 *)(r11 + 8) ; r2 is a kernel address
r1 = map_value
r1 += r2
*(u64 *)(r1 + 0) = 0
On x86-64 the C code changes r9 when the time of may_goto is up. With the
hpet clocksource a look at the clock changes r9 too. The prog above crashes
the kernel. LLVM does not emit such code: it reads stack arguments at the
entry of a function, and it writes them immediately before the call.
Save and restore these registers too. The cost is 2 insns on x86-64, 4 on
arm64 and 6 on riscv64, one time in 65535 iterations.
Fixes: 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")
Fixes: 235b2fe772f5 ("bpf, arm64: Add JIT support for stack arguments")
Fixes: ba754185f902 ("riscv, bpf: Add BPF stack arguments support for RV64 JIT")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261027190314.1440199-19-alexei.starovoitov@gmail.com
|
|
x86 has no fetching AND/OR/XOR. Thus the JIT emits a loop with cmpxchg.
cmpxchg uses RAX, and RAX holds BPF R0. Thus the JIT saves R0 before the
loop.
prog the address is in the JIT saves R0 in
-------------------------- ------------------ -------------------
without the flag a BPF register BPF_REG_AX
with BPF_F_ARENA_SCALAR BPF_REG_AX the stack
The flag is removed. Remove the second variant.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261027190314.1440199-7-alexei.starovoitov@gmail.com
|
|
Rust has no address spaces. Thus a prog in Rust accesses arena through
scalars. With BPF_F_ARENA_SCALAR the verifier accepts each load and each
store through a scalar as an access to arena.
The verifier rewrites one insn at a time. Thus it must add one insn to each
access. The compiler knows the source of each pointer. It can emit one
addr_space_cast for each pointer, as it does for the __arena pointers of C.
flag: one move for each access cast: one cast for each pointer
ax = (u32)r1 r1 = addr_space_cast(r1, 0, 1)
r2 = *(u64 *)(ax + 0) r2 = *(u64 *)(r1 + 0)
ax = (u32)r1 r3 = *(u64 *)(r1 + 8)
r3 = *(u64 *)(ax + 8)
20 progs of rust-bpf, with the flag and with the casts that a prototype of
an LLVM pass emits:
flag casts
moves or casts 1568 571
insns 14864 13938
jited bytes 72351 69111
No released kernel has the flag. LLVM does not emit the casts for Rust at
this time. Until it does, Rust progs that access arena do not load.
scx_mavd and scx_nitosis set the flag too. They are experimental schedulers
in C in sched-ext/scx, and they use the flag to omit __arena. They must use
__arena again.
This reverts commit 4c651a91bdfc
("bpf: Treat load and store through a number as arena access"). One part
stays: BPF_REG_AX in reg2pt_regs[] of the x86 JIT. The fetching AND/OR/XOR
atomics in arena use it now.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261027190314.1440199-6-alexei.starovoitov@gmail.com
|
|
A backward jump is sized from its target's current-pass address and its
own previous-pass address, so it can grow while other jumps shrink. Two
passes may then give the same image length with jumps still changing,
which ends in "bpf_jit: fatal error", and a jump growing in a padding
pass ends in "unexpected jmp_cond padding".
Shift the jump's previous-pass address by how far the current pass has
moved it, so a backward jump is sized by the insns it jumps over, like
a forward one. Also start every insn at BPF_MAX_INSN_SIZE instead of 64,
so none can grow in the first pass. Then no insn grows from one pass to
the next, and a repeated image length means nothing moved. The 123
limit on imm8 offsets worked around that growth and is dropped. The
ja -1 special case is subsumed, such a jump is now imm8 from the first
pass.
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://patch.msgid.link/20261022-jit_size_fix-v2-1-4bee6921b218@meta.com
|
|
clang inlines emit_unwind_ret_addr() and emit_patch() into do_jit() and
folds 'bpf_unwind_target - X86_PATCH_SIZE' into one relocation. It points
to a nop in the padding in front of the function, and objtool complains:
vmlinux.o: warning: objtool: do_jit+0x1330: relocation to !ENDBR: .text+0xc7fab
It's only used to compute rel32 of a direct call, so nothing is wrong at
run time.
emit_rsb_call() does call depth accounting too and hides func from
the optimizer. Use it instead of open coding.
Fixes: cff94a3d78d8 ("bpf, x86: Support unwind_goto and unwind_exit")
Reported-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Tested-by: Nathan Chancellor <nathan@kernel.org> # build
Link: https://patch.msgid.link/20261021220844.239400-1-alexei.starovoitov@gmail.com
Closes: https://lore.kernel.org/bpf/20261021201456.GA3197963@ax162/
|
|
For the verifier unwind_exit of an exception callback is exit. For JITs
the callback is a subprog, so they emit the lookup of the return address.
The idea was that the kernel calls the callback and the lookup finds
nothing.
But an exception callback has no frame of its own. It returns from
the frame of the main prog that threw, and the kernel isn't always
the caller of that one:
- An extension returns to the caller of the subprog that it replaced.
- After a tail call in a subprog the main prog returns to the caller of
that subprog.
When unwind_goto follows that call, the callback goes to the pad, and
the pad wasn't verified with it:
r6 = 1
call a // can unwind
unwind_goto pad
r6 = 2
call b // cannot unwind
unwind_goto pad // stays, since the pad is not dead code
exit
pad:
r0 = r6 // r6 is 1 for the verifier
An extension without unwind_exit can replace b(). When it throws and its
exception callback ends with unwind_exit, the pad runs with r6 = 2.
Both JITs ask bpf_prog_can_unwind(). Let it say no for an exception
callback.
An async callback has a frame of its own and the kernel is the caller.
Fixes: a2d274771321 ("bpf: Add unwind_goto and unwind_exit insns")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/5f24cbed1ec4a6a0ecd0b834d738c48dc9eb64362be4fa50dfb0eaddfad928b3@mail.kernel.org/
Link: https://patch.msgid.link/20261020231942.2568163-2-alexei.starovoitov@gmail.com
Closes: https://lore.kernel.org/bpf/sashiko-outbox-179498@kernel.org/
|
|
bpf x86
call inner call inner
unwind_goto pad
w0 = w6 mov eax, ebx
r0 = 0 xor eax, eax
unwind_exit push rax
push rsi
mov rdi, [rbp + 8]
call bpf_unwind_target
mov [rbp + 8], rax
pop rsi
pop rax
leave
ret
unwind_goto is an extable entry and no code. unwind_exit looks its return
address up there and returns to the pad. r0 and r2 are what a subprog
returns, so rax and rsi are saved around the lookup.
When there is no entry, the kernel called the subprog as an async or an
exception callback, and unwind_exit is exit.
The caller of the main prog doesn't tell exit from unwind_exit, so there
it's exit. An extension is different. freplace is a jump, so its return
address is the one of the subprog that it replaced.
With private stack r9 is saved around a call, and 'pop r9' has to be done
on the way to the pad too. In such progs unwind_goto is:
push r9
call foo
pop r9 <- exit returns here
jmp 1f
pop r9 <- unwind_exit returns here
jmp pad
1:
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261020181320.1284798-3-alexei.starovoitov@gmail.com
|
|
A JIT sets up the arena base in the prologue of every function of a
program that has an arena, a callee-saved register loaded with a 64-bit
immediate and spilled around the function on x86. That is paid for by
every program with an arena, whether it touches the arena or not, and
now that libbpf binds the arena of an object to every program in it, by
every program of an object that happens to contain an arena.
Decide per function, on its final insns, right before it is JITed.
bpf_prog_has_arena_insns() looks for an insn that works with the base:
an access through an arena pointer, or through a number in a program
loaded with BPF_F_ARENA_SCALAR, has been rewritten to a BPF_PROBE_MEM32
form by then, a cast to the user address space keeps
BPF_ADDR_SPACE_CAST, and a kfunc call with an __arena argument, which
the JIT rebases, has a function model that says so. A load of an
__arena struct member is rewritten with an immediate and needs none;
dereferencing the result does.
Looking at the final insns rather than recording the uses during
verification keeps the answer in step with what the JIT sees. A
constant NULL passed to a nullable __arena kfunc argument takes the
early exit for NULL arguments in the verifier, and the JIT rebases the
call all the same; an insn that is dead code has disappeared by then.
bpf_prog_jit_compile() caches the answer in bpf_prog_aux for a single
function program, jit_subprogs() for each subprogram once it has its
kfunc table and arena. It implies that the program has an arena, as the
scan says no without one. The JITs with arena support pick up the arena
base and user address only when bpf_prog_uses_arena_insns() says so,
and so generate the prologue and epilogue of a program bound to an
arena it never touches as if it had none.
The other users of the arena address of a program stay as they are on
purpose. The struct_ops trampoline converts __arena arguments on entry
and arena_member_fixup() converts loaded __arena members after the load,
both with the base as an immediate. Both have to happen whether or not
the program dereferences the result, it may compare, store or pass it
on, so neither depends on the base register nor on the flag. The arena
fault report reads the user address of the arena the faulting insn
belongs to, and such an insn sets the flag anyway.
check_map_prog_compatibility() checks the user address when the arena
is bound, before any use is known. The rest, in arena.c and sched_ext,
works with the map and not with a program.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019132203.3034698-14-memxor@gmail.com
|
|
The kernel test robot reports for a clang 22 randconfig with UBSAN
and ThinLTO:
vmlinux.o: error: objtool: do_jit(): unexpected end of section .text.do_jit
Fix by masking the counts to their range like rol64() does, which keeps the
shifts defined for the sanitizer and removes the __ubsan_handle_shift_out_of_bounds()
checks. The counts are below 32 and 64 anyway, so the emitted code does not
change. Reproduced with clang 20 and the robot's config without LTO.
Fixes: 02cfd66f3d9a ("bpf, x86: Divide by constants via multiplication with a magic number")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202610171029.DfoBvNXv-lkp@intel.com/
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261018113352.730799-1-daniel@iogearbox.net
|
|
save_args() numbers the on-stack argument slots consecutively, and the
area built for the original function is sized the same way, while the
x86-64 ABI rounds the stack slot up to an even one for an argument
aligned to 16 bytes: an __int128, or an aggregate holding one. When such
an argument lands on an odd stack slot, it and every argument after it
are read from the wrong place, so a bpf program attached to the function
reads a neighbouring eightbyte, and the arguments handed on to the
original function are shifted. For example:
/* s holds an __int128 */
void foo(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, u64 g,
struct s h, u64 i)
x86-64 puts g in the first stack slot, leaves the second empty and puts
h in the third and fourth, while the trampoline reads h from the second
and third.
Take the position of each argument slot from bpf_jit_place_args(), which
knows the x86-64 rules, backfill included. save_args() then reads each
slot from its register or incoming stack slot, restore_regs() puts back
only the register slots, and the area built for the original function
mirrors the incoming one, holes included.
A trampoline on a BPF function is not changed by this: since the
previous patch, the function's model has one u64 per argument slot,
which bpf_jit_place_args() places in slot order, so the trampoline still
takes the arguments where BPF code put them.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261016183105.1134967-1-yonghong.song@linux.dev
|
|
A trampoline on a function of a program that can tail call keeps the
tail call counter pointer by pushing rax after allocating its frame.
When it calls the original function, for fexit, fmod_ret or fsession,
the push sits between the stack arguments and the return address, which
the x86-64 calling convention does not allow. For example, in
run() -> call_stack() -> callee_stack(a, b, c, d, e, f, g, h)
run() -> subprog_tail() -> bpf_tail_call()
callee_stack() is not on the path to the tail call, so the verifier
allows its stack arguments g and h. fexit on it builds, with T the
trampoline's rbp:
T-120 h save_args()
T-128 g save_args(), RSP after sub rsp, 128
T-136 tail_call_cnt_ptr push rax, RSP at the call
T-144 return address call orig_call
and callee_stack() finds the counter pointer where g belongs, at 8(%rsp)
on entry, and g where h belongs.
Keep the pointer in a slot of the frame instead, so that the stack
arguments sit right above the return address again:
T-120 tail_call_cnt_ptr mov [rbp - 120], rax
T-128 (pad to 16)
T-136 h save_args()
T-144 g save_args(), RSP at the call
T-152 return address call orig_call
Fixes: 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261016183044.1132751-1-yonghong.song@linux.dev
|
|
x86-64 has no single instruction for a fetching AND/OR/XOR, so the JIT
lowers them to a CMPXCHG loop. That loop could not be used against arena
memory: it contains two memory accesses, the load of the old value and
the CMPXCHG itself, either of which can fault when the arena page goes
away, while the verifier reserves only one exception table entry per
instruction. bpf_jit_supports_insn() therefore rejected the three opcodes
and such programs failed to load with
BPF_ATOMIC stores into R1 arena is not allowed
Emit the same loop with R12-indexed addressing and give each of the two
accesses its own exception table entry. The verifier still reserves one
per instruction, so the JIT accounts for the extra entry itself by
rescanning the instruction stream in bpf_int_jit_compile() before the
extable is sized, as s390 does in bpf_jit_alloc() for its BPF_XCHG
lowering. The rescan sits after the extra-pass early exit, so a program
is counted exactly once. Drop the bpf_jit_supports_insn() rejection.
A loop without the load would need only one entry, since a CMPXCHG that
loses the comparison reloads RAX, but its first CMPXCHG would then almost
always lose. Every operation would pay for a second locked CMPXCHG, which
makes the fetching insn more expensive than the load and CMPXCHG loop
programs write by hand today. The second entry is paid for once, in the
JIT.
Both entries resume past the whole loop rather than past the faulting
instruction alone, with the fetch destination cleared, so a fault cannot
re-enter the loop, and both are reported as writes since the BPF insn is
a read-modify-write. This repurposes the INSN_LEN field of the fixup as a
resume distance rather than an instruction length; a sequence long enough
to overflow its 8 bits would first have to exceed BPF_MAX_INSN_SIZE, which
do_jit() rejects before the image is used.
The loop needs RAX for CMPXCHG and saves R0 in BPF_REG_AX, which it also
substitutes for R0 when either operand is R0. ex_handler_bpf() may then
have to name r10 as the register holding the arena address or the one to
clear on fault; reg2pt_regs[] already has that entry.
An access through a number, made by a program loaded with
BPF_F_ARENA_SCALAR, keeps its address in BPF_REG_AX itself (see
arena_scalar_access()), which is where the loop would save R0. Save R0 on
the stack in that case, as the div and mod lowering does. An R0 operand
stays in that stack slot: the new value is built from it, RAX ends up
holding the old value that R0 is to return, and the slot is dropped
instead of popped. Both entries resume before that pop or drop, so a
fault leaves the stack balanced.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20261016171319.1623447-2-puranjay@kernel.org
|
|
The cast of an arena pointer to its user address zero extended the low
half, shifted it up, or'ed the user base in, rotated it back down and
replaced the result with zero from a cleared r11 through a cmove for a
NULL pointer: seven instructions with a dependency chain of six. Or the
user base into r11 instead and pick it with a cmovne when the low half
is not zero, five instructions with a chain of three. LLVM emits the
cast for every arena pointer handed back to user space.
Before, for r1 = addr_space_cast(r7, 1, 0) with a user base of
0x7f1200000000:
44 89 ef movl %r13d,%edi
48 c1 e7 20 shlq $0x20,%rdi
48 81 cf 12 7f 00 00 orq $0x7f12,%rdi
48 c1 c7 20 rolq $0x20,%rdi
4d 31 db xorq %r11,%r11
85 ff testl %edi,%edi
49 0f 44 fb cmoveq %r11,%rdi
After:
44 89 ef movl %r13d,%edi
49 bb 00 00 00 00 12 7f 00 00 movabsq $0x7f1200000000,%r11
49 09 fb orq %rdi,%r11
85 ff testl %edi,%edi
49 0f 45 fb cmovneq %r11,%rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-17-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The trampoline prologue open codes the choice between the imm8 and the
imm32 form of sub rsp which emit_sub_rsp() already makes. Use it.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-16-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
LLVM turns a multiplication by a power of two into a shift, but keeps
the other constants as a multiplication since BPF has no lea to lower
them into, so r *= 3 and friends reach the JIT as BPF_MUL with BPF_K
and were emitted as imul with an immediate, a 3 cycle instruction on
Intel and AMD. Emit lea dst, [dst + dst * 2/4/8] for 3, 5 and 9
instead, which is the same size and a single cycle on Intel and on
Zen 3 and later, two cycles on older AMD cores. r13 keeps the imul,
since it needs a displacement byte as base register, which makes the
lea a byte longer and a slow three operand form with 3 cycles of
latency on Intel.
Before/after for r1 *= 3:
48 6b ff 03 imulq $0x3,%rdi,%rdi
48 8d 3c 7f leaq (%rdi,%rdi,2),%rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-15-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
LLVM strength-reduces 32-bit divisions by a constant into a 64-bit
multiplication, but has no high-half multiply for the 64-bit ones, so
these reach the JIT as BPF_DIV/BPF_MOD with BPF_K and were emitted as
div or idiv, which take 20-100 cycles on Intel before Ice Lake, about
15 cycles on Ice Lake and later, and 9-19 cycles on AMD Zen 3-5,
whereas a mul is 3 to 4 cycles on all current cores.
Emit the multiplication with a magic number instead, with the constants
computed per Hacker's Delight, 2nd ed., figures 10-1 and 10-2, which is
also what compilers do for native code: an unsigned division becomes a
mul plus a shift, or plus a sub/shr/add/shr sequence when the magic
number needs 65 bits, which an even divisor avoids by shifting the
dividend by its trailing zeros first, a signed one an imul, an optional
add or sub of the dividend, a sar and the sign bit adjustment. A modulo
multiplies the quotient back and subtracts it from the dividend.
Divisors 0 and +-1 as well as unsigned divisors above 2^63 (negative
imm32) are left to div/idiv as before, and so are the 32-bit divisions.
The magic numbers were validated in user space against the bounds from
Hacker's Delight for every divisor an imm32 can hold, in both
signednesses, and the emitted sequences were executed with dividends
around every multiple boundary of the divisor, at the extremes and at
random.
Before, for r1 /= 1000:
50 pushq %rax
52 pushq %rdx
49 c7 c3 e8 03 00 00 movq $0x3e8,%r11
48 89 f8 movq %rdi,%rax
31 d2 xorl %edx,%edx
49 f7 f3 divq %r11
48 89 c7 movq %rax,%rdi
5a popq %rdx
58 popq %rax
After:
50 pushq %rax
52 pushq %rdx
48 89 f8 movq %rdi,%rax
48 c1 e8 03 shrq $0x3,%rax
49 bb cf f7 53 e3 a5 9b c4 20 movabsq $0x20c49ba5e353f7cf,%r11
49 f7 e3 mulq %r11
48 c1 ea 04 shrq $0x4,%rdx
48 89 d7 movq %rdx,%rdi
5a popq %rdx
58 popq %rax
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-14-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The PROBE_MEM check rejects src_reg + off - VSYSCALL_ADDR up to a limit
of TASK_SIZE_MAX + PAGE_SIZE - VSYSCALL_ADDR, which does not fit into a
32-bit immediate and therefore needed a movabs into r10 and a register
compare. Compare against the next power of two above the limit instead,
which is a shift by fls64(limit) followed by a jne on the result. The
addresses between the limit and that power of two are non-canonical on
every x86-64 layout, since user space ends at TASK_SIZE_MAX and the
kernel half starts far above 2^57, so they faulted and had the register
zeroed by the exception handler before. Now they are zeroed by the check
without taking the fault. The compare and branch stay two uops on Intel
and AMD, while the sequence shrinks from 15 to 6 bytes.
Before, for the compare of a 4-level paging kernel:
49 ba 00 00 a0 00 00 80 00 00 movabsq $0x800000a00000,%r10
4d 39 d3 cmpq %r10,%r11
77 05 ja 0x...
After:
49 c1 eb 30 shrq $0x30,%r11
75 05 jne 0x...
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-13-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The kernel address check in front of a BPF_PROBE_MEM load computed
src_reg + off - VSYSCALL_ADDR with a mov of VSYSCALL_ADDR, a mov of
the base register, an add of the offset in imm32 form and a sub, that
is 20 bytes and 4 uops. VSYSCALL_ADDR is -10 MiB, so together with
the 16-bit instruction offset it fits into a 32-bit displacement and
a single lea does the same in 7 bytes and 1 uop on Intel and AMD.
Before, for the check on r0 = *(u64 *)(r1 + 16):
49 c7 c2 00 00 60 ff movq $0xffffffffff600000,%r10
49 89 fb movq %rdi,%r11
49 81 c3 10 00 00 00 addq $0x10,%r11
4d 29 d3 subq %r10,%r11
After:
4c 8d 9f 10 00 a0 00 leaq 0xa00010(%rdi),%r11
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-12-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Commit 77d8f5d47bfb ("bpf,x64: use shrx/sarx/shlx when available") kept
the legacy shift by cl when the count already is in rcx, since that
encoding is two bytes shorter. However, shl/shr/sar r,cl are 3 uops on
Intel from Haswell through Skylake and still 2 uops on Ice Lake, as a
zero count must leave the flags untouched, whereas shlx/shrx/sarx are a
single uop with a latency of 1. On AMD both forms are a single uop, so
the two extra bytes are the only cost there. Use the BMI2 form whenever
it is available. LLVM keeps the count in whichever register it was
allocated to, r4 included.
Before/after for r1 <<= r4:
48 d3 e7 shlq %cl,%rdi
c4 e2 f1 f7 ff shlxq %rcx,%rdi,%rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-11-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The 16-bit byte swap was emitted as a 16-bit rotate followed by a zero
extending move. bswap of the 32-bit register followed by a right shift
by 16 has the same result, is two bytes shorter, and avoids the operand
size prefix and the 16-bit partial register write. Both sequences are
two single cycle uops on Intel and AMD. LLVM emits be16 for bpf_htons()
and bpf_ntohs() with cpu v3 and bswap16 with cpu v4, both of which
take this path.
Before, for r1 = be16 r1:
66 c1 cf 08 rorw $0x8,%di
0f b7 ff movzwl %di,%edi
After:
0f cf bswap %edi
c1 ef 10 shrl $0x10,%edi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-10-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
An add or sub of 1 or -1 is an inc or dec, which is a byte shorter and a
single uop on all current cores, and the flags they leave untouched are
not read across BPF instructions anyway. An add of 128 is the sub of
-128 and the other way round, which fits into an imm8. An and of a 64-bit
register with a non-negative immediate clears the upper half just like
the 32-bit and does, which drops the REX.W prefix on the lower eight
registers. A shift left by one is the add of the register to itself, the
same size and a single uop that runs on more ports. LLVM emits all of
them, r += 1 for every counter, r &= mask for the masks beyond the
byte and word ones and r <<= 1 for x * 2 and x + x, and does the same
replacements for native code.
Before, for r1 += 1, r1 += 128, r1 &= 0x3ff and r1 <<= 1:
48 83 c7 01 addq $0x1,%rdi
48 81 c7 80 00 00 00 addq $0x80,%rdi
48 81 e7 ff 03 00 00 andq $0x3ff,%rdi
48 d1 e7 shlq %rdi
After:
48 ff c7 incq %rdi
48 83 ef 80 subq $-0x80,%rdi
81 e7 ff 03 00 00 andl $0x3ff,%edi
48 01 ff addq %rdi,%rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-9-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The add, sub, and, or and xor with an immediate or a register, the shifts
by an immediate and neg are open coded in the ALU cases of do_jit(), the
add with an immediate once more in the tail call and KASAN check sequences.
Add emit_alu_imm(), emit_alu_reg(), emit_shift_imm() and emit_neg() for
them and use them there, together with emit_mov_reg(), emit_mov_imm32()
and emit_add_rsp() for the open coded moves and stack adjustment in the
tail call, the division and the trampoline. The emitted code is unchanged.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-8-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
LLVM lowers u8 and u16 truncations to r &= 0xff and r &= 0xffff, and
a bitwise not to r ^= -1, since BPF has neither a zero extending move
from a sub-register nor a not instruction. On x86 these masks need the
imm32 form of and as they do not fit into a sign-extended imm8. Emit
a movzx from the low byte or word instead, which is 3 to 4 bytes and
zero extends into the full register for both the 32 and 64-bit variant,
and a not for the xor with -1. Both are single cycle instructions on
Intel and AMD cores.
Before/after for r1 &= 0xff, r1 &= 0xffff and r1 ^= -1:
48 81 e7 ff 00 00 00 andq $0xff,%rdi
40 0f b6 ff movzbl %dil,%edi
48 81 e7 ff ff 00 00 andq $0xffff,%rdi
0f b7 ff movzwl %di,%edi
48 83 f7 ff xorq $-1,%rdi
48 f7 d7 notq %rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-7-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
A strict compare against 1 or 128 is the non-strict compare against 0
or 127, and a strict compare against -129 or, with a signed condition,
-1 the non-strict compare against -128 or 0. LLVM canonicalizes x <= 0
to x < 1, x >= 0 to x > -1 and x <= 127 to x < 128, so these reach the
JIT as compares against 1, -1 and 128. Adjust the condition and the
immediate, so that the compare against 0 becomes the test without an
immediate already emitted for it and 127 and -128 fit into an imm8.
Before, for if r1 s> -1 goto ... and if r1 < 128 goto ...:
48 83 ff ff cmpq $-0x1,%rdi
7f xx jg ...
48 81 ff 80 00 00 00 cmpq $0x80,%rdi
72 xx jb ...
After:
48 85 ff testq %rdi,%rdi
7d xx jge ...
48 83 ff 7f cmpq $0x7f,%rdi
76 xx jbe ...
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-6-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
test has no sign-extended imm8 form, so a JSET against a constant was
always emitted as a 6 to 7 byte test with imm32. When the mask fits
into the low byte, the result only depends on that byte, so test the
8-bit register instead, which takes 2 to 4 bytes. Reading the low byte
of a register that was written as a whole has no partial register
penalty on Intel or AMD cores. Also use the one byte shorter accumulator
forms of test and cmp with imm32 when the destination is r0, like the
ALU immediates already do.
Before/after for if r1 & 0x40 goto ...:
48 f7 c7 40 00 00 00 testq $0x40,%rdi
40 f6 c7 40 testb $0x40,%dil
Before/after for if r0 == 0x1000 goto ...:
48 81 f8 00 10 00 00 cmpq $0x1000,%rax
48 3d 00 10 00 00 cmpq $0x1000,%rax
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-5-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The prologue and the tail call sequences always used the imm32 forms
of sub/add rsp and a disp32 load of the tail call counter pointer, even
when the rounded stack depth fits into a byte. Route them through the
existing emit_sub_rsp()/emit_add_rsp() and emit_ldx() helpers which pick
the short forms, and derive the ip adjustment for the counter load in
front of a subprog call from the emitted length instead of hardcoding
the 7 bytes. Nothing depends on the size of these instructions: the tail
call entry offset covers only the code in front of the stack adjustment,
and the poke targets are computed from the emitted bytes.
Before/after for a program with 32 bytes of stack:
48 81 ec 20 00 00 00 subq $0x20,%rsp
48 83 ec 20 subq $0x20,%rsp
48 8b 85 d0 ff ff ff movq -0x30(%rbp),%rax
48 8b 45 d0 movq -0x30(%rbp),%rax
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-4-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
A memory operand with a zero offset was encoded with a mod 01 ModR/M and
an explicit zero displacement byte. Use the mod 00 form instead, which
saves a byte for every load, store and atomic with a zero offset, as
LLVM emits them for the first field of a struct and for every plain
pointer dereference. The exception are rbp and r13 as base register,
where mod 00 with r/m 101 selects a RIP-relative disp32 and with SIB
base 101 a disp32 without base, so these keep their displacement byte.
Also let emit_st() reuse emit_insn_suffix() instead of open-coding the
ModR/M encoding.
Before/after for r0 = *(u64 *)(r1 + 0):
48 8b 47 00 movq 0x0(%rdi),%rax
48 8b 07 movq (%rdi),%rax
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-3-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
movzx into a 32-bit register already clears the upper 32 bits, so the
REX.W prefix on the 8 and 16-bit BPF_LDX loads only costs a byte. This
also matches what the arena variants of these loads emit. Likewise,
the 32-bit movsx from an 8-bit register only needs a REX prefix when
the destination is an extended register or the source is sil, dil, bpl
or an extended register. LLVM emits these loads for u8 and u16 fields
and, with cpu v4, the movsx for a cast to s8 or s16.
Before/after for r0 = *(u8 *)(r1 + 0):
48 0f b6 47 00 movzbq 0x0(%rdi),%rax
0f b6 47 00 movzbl 0x0(%rdi),%eax
Before/after for w1 = (s8)w0:
40 0f be f8 movsbl %al,%edi
0f be f8 movsbl %al,%edi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-2-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
There are no address spaces in Rust. LLVM emits plain loads and stores
for arena memory, without cast_kern:
r3 = 0x20 ll // R_BPF_64_64 .bss, libbpf puts it into arena
r2 = *(u64 *)(r3 + 0) // R3 invalid mem access 'scalar'
lock *(u64 *)(r2 + 8) += r1
Add BPF_F_ARENA_SCALAR flag of BPF_PROG_LOAD. When the program is loaded
with it and has an arena treat ldx, stx, st and atomics through a number
as arena access. It's as safe as access through PTR_TO_ARENA: JIT adds
the base of arena to the low 32 bits of the address. The program has
an arena only with CAP_BPF and CAP_PERFMON.
Registers of the program are not changed, since Rust compares and
stores the address after the access. bpf_do_misc_fixups() copies
the low 32 bits into BPF_REG_AX and the access goes through it:
r2 = *(u64 *)(r3 + 0) -> w12 = w3
r2 = *(u64 *)(r12 + 0)
Constant blinding needs BPF_REG_AX for immediates and leaves insns that
use BPF_REG_AX alone. So the immediate of st through a number is not
blinded, the rest of the program is:
*(u64 *)(r3 + 0) = 1 -> w12 = w3
*(u64 *)(r12 + 0) = 1
The same insn may see PTR_TO_ARENA on another path. It works for both.
It's a flag, since a number is also what the verifier makes of a pointer
that went away when the program has CAP_PERFMON: a ringbuf record after
bpf_ringbuf_submit(), a pointer to the packet after bpf_skb_pull_data().
Programs in C that have an arena keep "invalid mem access 'scalar'" for
such bugs.
JIT tells with bpf_jit_supports_arena_scalar() that it takes BPF_REG_AX
as the address of arena access. With other JITs the flag is rejected
with -EOPNOTSUPP. x86 and arm64 do:
- x86: the fault handler finds the register that holds the address
through reg2pt_regs[]. Add BPF_REG_AX, r10 of x86, there.
- arm64: nothing else is needed. The JIT takes any register as the
address and the fault handler reads it by its number. Compile tested
only.
Any number is an address of arena, NULL and small numbers included, like
it is for PTR_TO_ARENA: arena code in C dereferences NULL and relies on
the fault being handled.
Still rejected:
- insn that sees a number on one path and a pointer that is not
PTR_TO_ARENA on another.
- numbers passed to helpers and kfuncs, except __arena arguments.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261002124714.180012-4-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Cross-merge BPF and other fixes after downstream PR.
Conflicts:
arch/arm64/net/bpf_jit_comp.c
arch/x86/net/bpf_jit_comp.c
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
bpf_tramp_image_put() makes sure a trampoline image is not freed while
a task may still be running in it, but nothing similar is done for the
progs called by that image. Detach drops the last prog reference right
away and the prog is freed after grace periods, on the basis that a
task still in the traced function skips the fexit progs once the nop at
ip_after_call is patched to a jump.
That leaves out a task sleeping in a sleepable prog that runs before
the detached one, which no grace period waits for:
CPU 0 CPU 1
in image I, sleeping in prog S
detach P from I's trampoline
-> new image, bpf_tramp_image_put(I)
bpf_prog_put(P), last ref
grace periods, P freed
back from S
__bpf_prog_enter(P)
call P->bpf_func
If S and P are fexit progs the task is already past the patched jump,
and fentry only images don't have one. On x86 this is an int3 in
poisoned bpf_prog_pack memory:
Oops: int3: 0000 [#1] SMP NOPTI
CPU: 18 UID: 0 PID: 94573 Comm: x169 Not tainted 6.18.44 #1 PREEMPT(lazy)
RIP: 0010:0xffffffffc0601d8d
Call Trace:
<TASK>
? bpf_trampoline_6442515411+0x1a4/0x21b
bpf_lsm_bprm_committed_creds+0x5/0x10
security_bprm_committed_creds+0x5f/0x70
begin_new_exec+0x2d6/0x410
...
We hit this in production when progs attached through trampolines got
detached while their hooks were busy, and it was independently found
with a fuzzer and KASAN.
Have the JITs emit a patchable nop in front of each prog call sequence
and record it in the image. When a prog is detached, patch its nop to a
jump over the call sequence. Progs that stay attached keep running for
the tasks that are in the image, and ip_after_call isn't needed anymore.
The task can be in any image that isn't freed yet, not only in the
current one. It sleeps in image I1 that calls S, P and Q, then P is
detached and the trampoline moves to image I2, then Q is detached and
its call is still in I1. So the trampoline keeps a list of its images
until they are freed, and detaching a prog patches its nop in all of
them. Images hold a reference on the trampoline for that long.
On riscv and loongarch a jump of any range takes several instructions,
and a task preempted in the middle of them could resume into half of the
new sequence. The nop is a single instruction there, patched to a near
branch through arch_bpf_trampoline_skip().
With the extra nops, BPF_MAX_TRAMP_LINKS progs no longer fit in a page
on x86 and arm64 (and already didn't on powerpc), so lower the limit
there like s390 does.
Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")
Reported-by: Sechang Lim <rhkrqnwk98@gmail.com>
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260926135605.1217928-3-florent.revest@linux.dev
Closes: https://lore.kernel.org/bpf/20260815071927.147049-1-zirajs7@gmail.com/
|
|
arch_bpf_timed_may_goto() computes the address of count and timestamp
as rbp + offset. When the prog uses private stack the JIT maps r10 to r9
and nothing is reserved under rbp, so bpf_check_timed_may_goto() reads
and writes the timestamp in the kernel stack, where
arch_bpf_timed_may_goto() has just saved r0-r5 or further down.
The following loop in fentry prog with r0-r5 set to zero:
1: may_goto +2
r6 += 1
goto 1b
leaves r3 with ktime in it when the prog has 64 bytes of stack and
ends after 65535 iterations when it has 128.
Add the frame pointer to the offset in the JIT, where rbp vs r9 is
known, and pass the pointer in AX.
arm64 and powerpc JITs keep the address of private stack in BPF_REG_FP
and are not affected.
Fixes: 2fb761823ead ("bpf, x86: Add x86 JIT support for timed may_goto")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260924233130.1213812-2-alexei.starovoitov@gmail.com
|
|
The x86-64 JIT encodes frame sizes as 32-bit immediates in its
prologue, epilogue and tail call sequences, a tail call pops the frame
of the program making it and lands in the target's prologue before the
target allocates its own frame, and private stacks are allocated from
each program's depth, so nothing in it depends on frames staying within
512 bytes. Report bpf_jit_supports_large_stack(), which raises the
budget of JITed programs to MAX_BPF_STACK_JIT: 2 KiB combined over a
call chain, or per frame on a private stack, with no separate limit on
a single frame. Interpreted programs keep 512 bytes.
The worst case kernel stack use of a chain of tail calls grows
accordingly: the callers of a tail call may still leave at most 256
bytes on the stack, so 33 programs can accumulate 8 KiB of dead frames
below the last one, which may now use 2 KiB instead of 512 bytes, for
a little over 10 KiB in total on a 16 KiB kernel stack. The per-cpu
memory behind a private stack grows in proportion to the frames a
program asks for, up to 2 KiB plus guards per frame.
The budget does not depend on the privileges of the loader: an
unprivileged program cannot call other BPF functions, so a tail call
from one leaves no frame behind and its worst case is a single 2 KiB
frame. Programs nested through helpers or attach points, such as a
tracing program entered from a helper of a networking program, are not
accounted against each other before or after this change; each level
of nesting may now add up to 1.5 KiB more. The one nesting a program
can force on itself, bpf_clone_redirect() to its own device, keeps that
program at 512 bytes.
The verifier state of a frame grows with the stack it uses, up to four
times as many stack slots as before; the allocation is on demand, so
only programs using deep frames pay for them.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260924165740.2146806-17-memxor@gmail.com
|
|
Emit indirect call via the register that holds the address of bpf subprog.
ld_imm64 BPF_PSEUDO_FUNC is resolved to func[i]->bpf_func by
jit_subprogs() already. It's the same address that is passed to helpers
as a callback.
Add emit_indirect_call() similar to emit_indirect_jump() used by tail
calls and gotox: use ITS or retpoline thunks when enabled.
Save/restore r9 around the call when private stack is used, like direct
calls do.
The verifier guarantees that callees of callx are not tail call
reachable, so don't pass tail call counter in rax, which may hold
the callee address.
FineIBT expects indirect callers to go through CFI preamble. Not done
yet. Return false from bpf_jit_supports_callx() in this mode, so such
progs are rejected at load time.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260924031042.1690890-8-alexei.starovoitov@gmail.com
|
|
Currently arch_bpf_trampoline_size allocates and frees a temporary
trampoline buffer on every invocation. The buffer is only used as a
scratch space while __arch_prepare_bpf_trampoline() calculates the
required size, and the generated trampoline is discarded.
Allocating a writable scratch page during kernel initialization and
reusing it for all size calculations. This improves tracing_multi
attachment time.
With current code:
# ./test_progs -t tracing_multi_bench_attach -v
...
serial_test_tracing_multi_bench_attach: found 55227 functions
serial_test_tracing_multi_bench_attach: attached in 1.563s
serial_test_tracing_multi_bench_attach: detached in 0.256s
With the fix:
# ./test_progs -t tracing_multi_bench_attach -v
...
serial_test_tracing_multi_bench_attach: found 55235 functions
serial_test_tracing_multi_bench_attach: attached in 0.798s
serial_test_tracing_multi_bench_attach: detached in 0.258s
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260910094353.1256316-1-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
The x86-64 allows 6 register arguments and then stack arguments, but
backfilling the 6th register is possible, as in the following example:
/* s is a 16-byte struct */
void kfunc(u64 a, u64 b, u64 c, u64 d, u64 e, struct big s, u64 f)
BPF puts s in slots 5 and 6, R9 and the first stack slot, and f in slot
7, while x86-64 puts s in the two stack slots and f in R9. Emit the
moves bpf_jit_plan_arg_moves() plans to bridge the two.
Only one argument ever moves down, as an argument frees one register at
most, so carrying that single value in the scratch register past its own
destination is enough.
In addition, the arena argument walk counts eightbytes rather than
parameters, as an argument may take two registers.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/r/20260912195247.989857-1-yonghong.song@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Cross-merge BPF and other fixes after downstream PR.
Conflicts:
kernel/bpf/backtrack.c
include/linux/bpf_verifier.h
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Insert KASAN shadow memory checks before memory load and store
operations in JIT-compiled BPF programs. This helps detect memory safety
bugs such as use-after-free and out-of-bounds accesses at runtime.
The main instructions being targeted are BPF_ST, BPF_STX and BPF_LDX,
but not all of them are being instrumented:
- if the load/store instruction is in fact accessing the program stack,
emit_kasan_check silently skips the instrumentation, as we can already
benefit from guard pages to monitor stack accesses.
- if the load/store instruction is a BPF_PROBE_MEM or a BPF_PROBE_ATOMIC
instruction, we do not instrument it, as the passed address can fault
(hence the custom fault management with BPF_PROBE_XXX instructions),
and so the corresponding kasan check could fault as well.
To support those new instructions insertion, create the
emit_kasan_check() helper that emits KASAN shadow memory checks before
memory accesses in JIT-compiled BPF programs. The implementation relies
on the existing __asan_{load,store}X functions from KASAN subsystem. The
helper:
- saves registers. This includes caller-saved registers, but also
temporary registers, as those were possibly used by the
affected program.
- computes the accessed address and stores it in %rdi
- calls the relevant function, depending on the instruction being a load
or a store, and the size of the access.
- restores registers
The special care needed when inserting this instrumentation comes at the
cost of a non negligeable increase in JITed code size. For example, a
bare
mov 0x0(%si),rbx # Load in rbx content at address stored in rsi
becomes
push %rax
push %rcx
push %rdx
push %rsi
push %rdi
push %r8
push %r9
push %r10
push %r11
mov %rsi,%rdi
call 0xffffffff81da0a60 <__asan_load8>
pop %r11
pop %r10
pop %r9
pop %r8
pop %rdi
pop %rsi
pop %rdx
pop %rcx
pop %rax
mov 0x0(%rsi),rbx
Signed-off-by: Alexis Lothoré (eBPF Foundation) <alexis.lothore@bootlin.com>
Link: https://lore.kernel.org/r/20260903-kasan-v9-4-2407fe99255a@bootlin.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
In order to prepare for KASAN checks insertion before every
memory-related load or store, group all BPF_ST instructions that indeed
access memory in a single helper to allow instrumenting those in one
call, rather than having to instrument all cases individually.
Signed-off-by: Alexis Lothoré (eBPF Foundation) <alexis.lothore@bootlin.com>
Link: https://lore.kernel.org/r/20260903-kasan-v9-3-2407fe99255a@bootlin.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
LLVM 23 returns an __int128, or a struct/union larger than 8 bytes and no
larger than 16 bytes, in the BPF R0:R2 register pair. The previous patch
added the shared helpers describing that convention; wire up the JIT side
so that the second half of the return value actually lands in R2.
Placing the second return half into R2 is possible on any JIT, but it needs
architecture-specific JIT work. Rather than requiring every JIT to
implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair()
capability, defaulting to false in the generic core; an architecture opts
in once its JIT handles the R0:R2 pair, and the remaining ones are left for
future work. Only the x86-64, arm64 and riscv64 JITs opt in so far.
On arm64 and riscv64 the native second return register is already BPF R2
(x1 in bpf2a64[] and a1 in regmap[] respectively), so the upper half needs
no move at all, unlike x86-64's RDX->RSI. The lower half is covered by the
move into BPF R0 that those JITs already emit after every call, from x0
into x8 and from a0 into a5. This has been tested on x86-64 and arm64. The
riscv64 path is expected to work by the same register-mapping reasoning as
arm64 but has not been tested.
bpf_add_kfunc_call() also rejects a kfunc that is marked KF_FASTCALL and
returns more than 8 bytes. The bpf_fastcall contract implemented by
mark_fastcall_pattern_for_call() assumes a call clobbers R0 plus the
registers holding its arguments, so a return in the R0:R2 pair would
clobber an R2 the caller expects the fastcall pattern to preserve. Such
a kfunc is rejected with -EOPNOTSUPP as well.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260819055254.3295610-1-yonghong.song@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge
number of trampolines [1].
The slowdown caused by extra protection changes in execmem_alloc_rw() and
execmem_free().
With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA
are ROX after the allocation. execmem_alloc_rw() temporarily sets them to
W+NX and execmem_free() resets them back to ROX.
The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size()
that only needs a temporary writable buffer in the modules address space.
On x86 executable memory and module data are constrained to the same
address range, so x86::arch_bpf_trampoline_size() can directly use
execmem_alloc(EXECMEM_MODULE_DATA)
Replace the call to bpf_jit_alloc_exec_rw() with a call to
execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and
drop bpf_jit_alloc_exec_rw() helper.
Fixes: 5bf02dbf39fa ("bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable")
Reported-by: Jiri Olsa <olsajiri@gmail.com>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/all/an8r7EODLIL-bZM3@krava
Link: https://lore.kernel.org/bpf/20260818130510.3110054-1-rppt@kernel.org
|
|
The destination of the per-CPU address MOV is encoded in ModRM.reg,
which is extended by REX.R, but the REX prefix is built with
add_1mod(), which sets REX.B. REX.B extends ModRM.rm and SIB.base, and
this instruction addresses memory as disp32 with no base, so the bit
has no effect at all and the high register bit is simply lost.
Every is_ereg() destination therefore resolves to the wrong register,
picking whichever one shares the low three bits:
R5 -> RAX R7 -> RBP R8 -> RSI R9 -> RDI
With BPF_REG_5, whose reg2hex is 0, the emitted
65 49 03 04 25 <off> add %gs:<off>,%rax
adds the per-CPU offset to RAX rather than R8. The destination keeps
the unadjusted address and RAX is clobbered, so the program goes on to
dereference a pointer that was never made per-CPU:
BUG: unable to handle page fault for address: 0000607e386a8894
RIP: bpf_prog_707837aafd2aa9ae_update_percpu_data+0x93/0xc9
Call Trace:
__bpf_prog_test_run_raw_tp+0x2dc/0x7d0
__flush_smp_call_function_queue+0x1e9/0xc80
Kernel panic - not syncing: Fatal exception in interrupt
R5 is the mildest of the four, aliasing a scratch register and faulting
at the store. R7 aliases RBP and would corrupt the frame pointer, R8
and R9 alias the argument registers.
Use add_2mod() so the register goes through REX.R, matching how
add_2reg() places it in ModRM.reg and how emit_priv_frame_ptr()
hardcodes 0x4c for the same instruction with R9. Encodings for the
non-extended registers are unchanged.
Problem showed up when trying to resurrect BPF_GCC CI (selftests built
with BPF_GCC).
This has gone unnoticed because clang reloads the address into R1
before each per-CPU access, so the destination is never an extended
register. GCC keeps several per-CPU addresses live at once, and
test_progs-bpf_gcc panics the kernel in global_percpu_data/init, where
the address of a .percpu variable ends up in R5.
Fixes: 7bdbf7446305 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs")
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260814220254.3797467-2-vineet.gupta@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
populate_extable() encodes "there is no destination register to clear" as
DONT_CLEAR in the DST_REG field of the exception table metadata, and later
ex_handler_bpf() then reuses that very value to derive the direction it
reports the fault with is_write = (reg == DONT_CLEAR). The two coincide
for a plain load or store, but not for a RMW carrying BPF_FETCH. Such an
atomic writes memory, so it has to be reported as a WRITE, and it also reads
the old value into a register, src_reg for BPF_ADD | BPF_FETCH and BPF_XCHG,
r0 for BPF_CMPXCHG, so that register has to be cleared on fault. A single
DONT_CLEAR cannot say both, and the store branch picks it unconditionally:
[...]
} else {
arena_reg = reg2pt_regs[dst_reg];
fixup_reg = DONT_CLEAR;
}
[...]
The reported direction is therefore right, but on a fault over an unmapped
arena page the fetch destination keeps whatever it held before the atomic,
where every other BPF_PROBE_* access delivers 0. Give the metadata its own
ARENA_WRITE bit so that the reported direction no longer depends on whether
there is a register to clear, and fill DST_REG in from bpf_atomic_load_reg().
BPF_{AND,OR,XOR} | BPF_FETCH need no handling here, bpf_jit_supports_insn()
already rejects those in the arena.
Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811131600.506721-3-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
save_args() reads stack-passed arguments relative to rbp assuming two
return addresses sit between the saved rbp and the arguments, which
holds when the trampoline is entered through the fentry call from a
traced function. An indirect trampoline is called through a function
pointer, so only the caller's return address is on the stack and the
arguments start at rbp + 16, not rbp + 24. Every stack-passed argument
of a struct_ops callback with more than six argument slots is read one
slot off.
This has gone unnoticed because no in-tree struct_ops member passes
arguments on the stack. The jmp-entry form already accounts for having
a single return address; treat BPF_TRAMP_F_INDIRECT the same way.
Fixes: 473e3150e30a ("bpf, x86: allow function arguments up to 12 for TRACING")
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Tested-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-12-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Implement the struct_ops arena argument conversion on x86. save_args()
receives the arena base from bpf_tramp_arena_base() and consults the
btf_func_model argument flags as it copies each native argument into the
BPF ctx, routing a marked argument through RAX:
movl %esrc, %eax /* truncate and clear the upper 32 bits */
subl $base_lo, %eax
movq %rax, ctx_slot
A nullable argument tests the full 64-bit kernel pointer first:
movq %rsrc, %rax
testq %rax, %rax
jz 1f
subl $base_lo, %eax
1:
movq %rax, ctx_slot
The 32-bit subtraction is sufficient since (u32)(kaddr - base) ==
(u32)kaddr - (u32)base, and it clears the upper half as the JITs require
of arena pointer registers. Stack-passed arguments already reload
through RAX, so only the subtraction (and the NULL test) is inserted
there.
Keep arena and nullable classification in btf_func_model.
bpf_tramp_arena_base() returns a base only for a single-program
struct_ops indirect trampoline; other trampolines pass zero and perform
no conversion. The size probe reruns the same emission with the same
model and nodes, so the image size matches by construction.
With both the kfunc and struct_ops directions implemented, flip
bpf_jit_supports_arena_args() on for x86.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-8-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
Implement arena argument rebasing for kfunc calls on x86. R12 already
holds kern_vm_start whenever the prog has an arena, so each tagged
argument costs two instructions emitted right before the call:
movl %eN, %eN /* truncate, clear the upper 32 bits */
addq %r12, %rN
A nullable argument tests the truncated value and jumps over the add:
movl %eN, %eN
testl %eN, %eN
jz 1f
addq %r12, %rN
1:
addq carries a REX prefix for every argument register and is always
three bytes, so the jz displacement is constant. The sequence is native
code generated after constant blinding has run on the BPF instruction
stream, so blinding never sees the rebase and needs no special handling.
bpf_jit_supports_arena_args() is not flipped yet; that happens when the
struct_ops trampoline side is in place as well.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-7-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
A load-acquire from an arena pointer is converted to BPF_PROBE_ATOMIC and
gets an exception table entry, but the entry is filled in as if it were a
store, since populate_extable() decides based on instruction class alone
and a load-acquire is of BPF_STX class:
if (BPF_CLASS(insn->code) == BPF_LDX) {
arena_reg = reg2pt_regs[src_reg];
fixup_reg = reg2pt_regs[dst_reg];
} else {
arena_reg = reg2pt_regs[dst_reg];
fixup_reg = DONT_CLEAR;
}
For a load-acquire dst_reg holds the loaded value and src_reg holds the
address, so both assignments in the else branch are wrong. On a fault
over an unmapped arena page ex_handler_bpf() then:
- computes the reported address from the value register instead
of the address register
- reports the access as a WRITE, since it derives the direction
from fixup_reg == DONT_CLEAR
- leaves dst_reg untouched, so the program continues with a stale
value instead of the 0 that BPF_PROBE_* loads deliver
The access itself is emitted correctly, emit_atomic_ld_st_index() uses
src_reg as the address, so this is a broken probe contract and a wrong
diagnostic rather than a memory safety issue.
Use bpf_atomic_is_load_acq() helper so a load-acquire takes the load path.
Fixes: 5341c9a4d833 ("bpf, x86: Support load-acquire and store-release instructions")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260806201047.333389-3-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
btf_distill_func_proto() accepts a function argument up to 16 bytes, so a
128-bit scalar such as __int128 reaches the x86 trampoline with
arg_size == 16. But the current implementation assumes an __int128
argument only needs one register, so the register save area is
under-allocated and save_args() overwrites adjacent stack slots.
Compute the register count from arg_size for all arguments to fix it.
Fixes: a9c5ad31fbdc ("bpf: x86: Support in-register struct arguments in trampoline programs")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260729050204.2586457-1-yonghong.song@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
arch_bpf_trampoline_size() allocates a buffer to get actual size required
for a trampoline.
This buffer must be in the module address space because
__arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to
that buffer.
In preparation for enabling ROX mode for EXECMEM_BPF make sure that the
allocated memory is writable.
Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for
buffer allocation in arch_bpf_trampoline_size().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Currently predictor flush on memory reuse is done for all BPF JIT
allocations, but only cBPF programs can be loaded by an unprivileged user.
eBPF is privileged by default, and flushing predictors for all CPUs on
every eBPF reuse penalizes the common case for no security benefit.
eBPF allocations can be frequent on busy systems, only flush predictors
for cBPF programs. Trampoline and dispatcher allocations also skip the
flush as they are eBPF-only.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
|
