Merge branch 'bpf-unwinding-with-unwind_goto-and-unwind_exit'

Alexei Starovoitov says:

====================
bpf: Unwinding with unwind_goto and unwind_exit

From: Alexei Starovoitov <ast@kernel.org>

1. What is the problem?

A rust panic or a C++ exception has to run Drop impls or destructors of
every frame it passes through. bpf_throw() discards the frames, so the
verifier doesn't let a frame that owns a lock or a reference throw. In rust
every index and every unwrap() can panic.

2. Is it worth solving?

Without it rust-bpf [1] drops all landing pads and a panic is bpf_throw().
A prog that can panic with a guard held doesn't load.

3. Is there a simpler way?

[2] keeps the normal path free of unwind code. llvm emits a table, libbpf
passes it to the kernel, bpf_unwind() walks the stack and rewrites return
addresses, and the verifier models hidden edges from calls to pads. That's
1400 lines in the kernel, with new uapi and two kfuncs, and 440 lines in
libbpf.

Here the edge from a call to its pad is an insn, and so is the other way
to leave a frame [3]:

  ret                            exit
  invoke foo unwind pad          call foo; unwind_goto pad
  resume                         r0 = 0; unwind_exit

Pads are ordinary code. The verifier learns one bit: how the callee left.
The normal path is free of unwind code too: JITs emit nothing for
unwind_goto, and exit is what it was.
200 lines in patch 1 and 100 lines in each JIT.

4. Will the api and the layers last?

Two new insns: one more kind of BPF_JCOND and one more kind of BPF_EXIT.
No kfuncs, no BTF, no changes in libbpf and other loaders.
The source level entry point is _Unwind_RaiseException() of the Itanium
ABI. It's what the runtimes of rust and C++ call on other architectures.
unwind_exit returns r0 like exit does. Today it's 0. That's where an
exception object would go.

5. Is it good taste?

Every frame returns through its own epilogue. The only question is where it
returns to. That's one bit, and it's a bit of control flow. v1 kept it in
a register.

It's not specific to rust. A C++ prog with throw, catch and destructors
compiles, loads and runs. It's a selftest in patch 5.

The llvm side is a new PR [3]. It replaces [4].

v3 -> v4:
- CI stayed red on x86-64. Call depth accounting was not the reason. CI
  builds x86-64 with KASAN, where the JIT wraps a load in pushes, a call
  and pops. The patterns of two __jited tests took that call for the call
  of the subprog. Now they start after the loads.
- When the callee never exits, what follows unwind_goto is seen only
  speculatively. It was removed as dead code, and other code moved to where
  the cpu predicts the return. Now 'goto pc-1' is there (sashiko).
- unwind_exit of an async or an exception callback is exit, like in the
  main prog. It was rejected. A sync callback still cannot unwind.
- arm64: no autia1716 and pacia1716. The address of the pad isn't signed.
  The epilogue authenticates LR as it is, and the offset of the pad is
  added to it afterwards.
- BPF_UNWIND_EXIT is enum instead of #define.
- unwind_attach and unwind_cxx skip when JIT is disabled, and don't define
  ENOTSUPP (bpf-ci).

v2 -> v3:
- bpf_unwind_target() is back in the JITs. In core.c it broke the build on
  MIPS, where struct exception_table_entry has no fixup (sashiko, bpf-ci).
- CI was red on x86-64: two __jited tests didn't expect call depth
  accounting in front of a call (bpf-ci).
- A global subprog that returns 16 bytes couldn't unwind: the verifier
  checks r0 and r2 at unwind_exit, and llvm set r0 only. Fixed in llvm.
  Patch 6 is Kumar's test.
- freplace: unwind_exit of an extension was exit, so the exception was lost
  (Eduard). Now the extension returns to the pad of the caller of what it
  replaced. An extension that can unwind replaces only a subprog that can.
  fexit cannot attach to such an extension.
- unwind_exit makes the cpu mispredict the return. Without CAP_PERFMON
  the verifier looks at what follows unwind_goto as a speculative path
  (bpf-ci).
- unwind_goto was 'call pc+0' in the xlated dump: BPF_JCOND is the same
  opcode as BPF_CALL_ARGS.
- selftests: 'the callee never returns' looks at xlated insns, and 'the way
  the callee left is a part of the state' has BPF_F_TEST_STATE_FREQ.
  Neither tested what it said (bpf-ci). New tests for async and exception
  callbacks, speculation and freplace. Every check that the series adds
  was removed in turn to see that a test fails.
- llvm: a pad doesn't set r0 when it only writes w0. Yonghong's callx case
  from [4] is a lit test.
- Rebased.

v1 -> v2:
- unwind_goto and unwind_exit instead of 'if r3 != 0 goto pad' and
  'r3 = 1; exit' (Kumar).
- JIT support is not an optimization anymore. There are no shape tests, no
  look at jump_target, which was stale (sashiko), no exit that looks at r3
  and no fallback to a branch.
- r3 is an argument again and nothing else. In v1 a subprog that didn't
  write it passed it through, and fentry didn't restore it (bpf-ci).
- Async and exception callbacks cannot unwind (sashiko).
- No BTF decl_tag, which was matched by prefix (bpf-ci). It works without
  BTF.
- x86 private stack is supported.
- Tail call and ld_abs are allowed.
- selftests: no #ifdef by arch, which broke the build on the rest of them.
  The messages are checked. fentry is attached and the target is run.
  Skips say why (bpf-ci).
- clang defines __BPF_FEATURE_UNWIND.

v3: https://lore.kernel.org/bpf/20261020043319.3031996-1-alexei.starovoitov@gmail.com/
v2: https://lore.kernel.org/bpf/20261019051846.3012089-1-alexei.starovoitov@gmail.com/
v1: https://lore.kernel.org/bpf/20261018010246.1643877-1-alexei.starovoitov@gmail.com/

Tested on x86-64 in kvm with KASAN, like CI does: test_progs,
test_progs-no_alu32, test_progs-cpuv4 and test_verifier, all of them in
full. The new tests also with retbleed=stuff. C, C++ and rust progs built
with [3] were run on x86-64 and arm64, with and without bpf_jit_harden=2.
arm64 is qemu, with and without pointer authentication, and a small loader
instead of test_progs. v3 passed test_progs on arm64 in CI.

[1] https://github.com/4ast/rust-bpf
[2] https://lore.kernel.org/bpf/20261017025129.1914859-1-yonghong.song@linux.dev/
[3] https://github.com/llvm/llvm-project/pull/233649
[4] https://github.com/llvm/llvm-project/pull/233140
====================

Link: https://patch.msgid.link/20261020181320.1284798-1-alexei.starovoitov@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

3 files changed, 24 insertions, 0 deletions

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 7750df973494..580f89c450b5 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1897,6 +1897,7 @@ struct bpf_prog_aux {
 	bool changes_pkt_data;
 	bool has_prologue;
 	bool might_sleep;
+	bool can_unwind;
 	bool kprobe_write_ctx;
 	/* has insns that work with the arena base, see bpf_prog_has_arena_insns() */
 	bool uses_arena_insns;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 54481da4ed2d..6606b341351e 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -532,6 +532,8 @@ struct bpf_verifier_state {
 
 	bool speculative;
 	bool in_sleepable;
+	/* The callee left with unwind_exit. Next insn is unwind_goto. */
+	bool unwound;
 
 	/*
 	 * First and last insn idx of this verifier state.
@@ -957,6 +959,7 @@ struct bpf_subprog_info {
 	s16 fastcall_stack_off;
 	bool has_tail_call: 1;
 	bool might_throw: 1;
+	bool can_unwind: 1; /* has unwind_exit */
 	bool tail_call_reachable: 1;
 	bool has_ld_abs: 1;
 	bool is_cb: 1;
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 6f4c9bd024a0..fccd8db33647 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -254,6 +254,25 @@ static inline bool insn_is_mov_percpu_addr(const struct bpf_insn *insn)
 		.off   = 0,					\
 		.imm   = 1 })
 
+static inline bool bpf_is_unwind_goto(const struct bpf_insn *insn)
+{
+	return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_UNWIND_GOTO;
+}
+
+static inline bool bpf_is_unwind_exit(const struct bpf_insn *insn)
+{
+	return insn->code == (BPF_JMP | BPF_EXIT) && insn->src_reg == BPF_UNWIND_EXIT;
+}
+
+/*
+ * The caller of the main prog doesn't tell unwind_exit from exit. An extension
+ * is different: its caller is the caller of the subprog that it replaced.
+ */
+static inline bool bpf_prog_can_unwind(const struct bpf_prog *prog)
+{
+	return bpf_is_subprog(prog) || prog->type == BPF_PROG_TYPE_EXT;
+}
+
 static inline bool insn_is_zext(const struct bpf_insn *insn)
 {
 	return insn->code == (BPF_ALU | BPF_MOV | BPF_X) && insn->imm == 1;
@@ -1257,6 +1276,7 @@ bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
 bool bpf_jit_supports_callx(void);
 bool bpf_jit_supports_kfunc_ret_reg_pair(void);
+bool bpf_jit_supports_unwind(void);
 bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);