bpf: One may_goto time slice per prog invocation

The count and the timestamp of may_goto are in the stack frame. So the
limit of 250 ms is for one call of a subprog, not for one invocation of the
prog, and loops that nest across calls multiply it. These 9 insns cause a
soft lockup:

  main:                 sub:
  1: call sub           1: may_goto 2f
     may_goto 2f           goto 1b
     goto 1b            2: r0 = 0
  2: r0 = 1                exit
     exit

Use one count and one timestamp for all may_goto of a prog, in per-cpu
memory of the prog. The main prog sets them in its first insns. When the
time is up each loop ends at its next may_goto, in any function.

  rX = address of the count
  rX = &(void __percpu *)(rX)
  r12 = *(u64 *)(rX +0)
  if r12 == 0x0 goto target
  r12 -= 1
  if r12 != 0x0 goto pc+2
  r12 = rX
  call arch_bpf_timed_may_goto
  *(u64 *)(rX +0) = r12

rX is a register that is dead at the may_goto. When all ten are live R0 is
saved in the stack.

An async callback can interrupt the prog on the same cpu. So its caller
sets count and timestamp, and puts the old values back at the end. A second
invocation of the prog on the same cpu gives more time to the first one.

  x86-64 (Zen 5), nsec                             before  after
  iteration, empty loop                             0.36    0.44
  iteration, the body takes 4 cycles                0.89    0.89
  call of a subprog and 1 iteration                 1.21    1.16

The 56 progs of scx with may_goto get 1.7% more JITed code and use 25% less
stack.

Fixes: e723608bf428 ("bpf: Add verifier support for timed may_goto")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261031215543.682284-8-alexei.starovoitov@gmail.com

0 files changed, 0 insertions, 0 deletions