bpf: End bpf_loop() when the time of may_goto is up

When the time of a prog is up all loops with may_goto end, but bpf_loop()
goes on. It calls its callback up to 1 << 23 times. The loops in the
callback make no iteration, but the rest of the callback runs, and that can
be another bpf_loop(). Two test progs with bpf_loop(1 << 23, cb) and some
work in cb return after 4 sec and 10 sec. With this patch they return after
250 ms.

bpf_loop() and bpf_for_each_map_elem() end their loop when the callback
returns 1. So return 1 in the first insns of the callback when the time is
up:

  r0 = address of the count
  r0 = &(void __percpu *)(r0)
  r0 = *(u64 *)(r0 +0)
  if r0 != 0x0 goto pc+2
  r0 = 1
  exit

The verifier has seen what follows: the loop of the helper can end after
each call of the callback.

A function that BPF code calls too, or that is the callback of another
helper or kfunc, stays as it is: there the return value has another
meaning. The callback of bpf_user_ringbuf_drain() stays as it is too. The
helper takes a sample out of the buffer for each call, so a call that
does nothing loses the sample. Its loop ends after 128K samples.

This patch does not limit bpf_loop() itself. Only may_goto counts and reads
the clock. So nested bpf_loop() are not stopped when the prog has no
may_goto, or has none in the callbacks: bpf_loop(1 << 20, cb) with
bpf_loop(4096, cb2) in cb takes 5 sec. Nested open coded iterators have the
same problem. A follow up will fix these holes.

A call of such a callback executes 4 more insns. The cost on x86-64 (Zen 5)
in nsec:

                                                   before  after
  callback makes 1 iteration, inlined bpf_loop()    1.16    1.54
  callback makes 1 iteration, bpf_loop()            6.52    6.74
  callback without may_goto, inlined bpf_loop()     1.16    1.16
  callback without may_goto, bpf_loop()             4.95    4.95
  callback without may_goto, nested bpf_loop()      1.13    1.38

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261031215543.682284-9-alexei.starovoitov@gmail.com

0 files changed, 0 insertions, 0 deletions