bpf, x86: Fix bpf_throw() in a prog with a private stack

When the main prog has a private stack and the prog calls bpf_throw(), it
returns to the kernel with wrong values in r12, rbx, r13, r14 and r15. The
kernel crashes soon after that, for example:

  BUG: kernel NULL pointer dereference, address: 0000000000000010
  RIP: 0010:__bpf_prog_test_run_raw_tp+0x125/0x380

The main prog of a prog that can throw saves these five registers at its
entry. bpf_throw() walks the stack to the frame of the main prog and
gives the stack pointer of that frame to the exception callback, which
pops the registers from there. In a function with a private stack r9 is
the frame pointer, and the JIT emits 'push r9' in front of every call
and 'pop r9' after it. So the stack pointer that bpf_throw() finds is
8 bytes too low:

  r12
  rbx
  r13
  r14
  r15  <- the callback has to pop from here
  r9   <- rsp of the main prog at the call

r15 gets the address of the private stack and every other register gets
the value of its neighbour.

This happens in tracepoint, raw_tp, perf_event and tracing progs, except
iterators, and in struct_ops progs that ask for a private stack, when
the main prog has more than 48 bytes of stack. The throw can be in the
main prog or in a subprog.

aux->stack_arg_sp_adjust already moves the stack pointer from the
outgoing stack arguments up to the saved registers. Add the 8 bytes of
r9 to it when the main prog has a private stack.

A constant is enough. Calls of helpers, kfuncs and subprogs and callx
all have the push. A prog with tail calls has no private stack, and a
callback cannot throw. Only the frame of the main prog matters: the
frames of subprogs are dropped, with or without a private stack. On
x86-64 a function with stack arguments has no private stack, hence only
one of the two parts is not zero.

The JITed code does not change.

Fixes: 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261031050558.2410721-2-alexei.starovoitov@gmail.com

0 files changed, 0 insertions, 0 deletions