bpf: tcp: Guard fast-path bpf_tcp_ops_call() under per-socket flag.

bpf_tcp_ops.{parse_hdr,hdr_opt_len} are called for every
incoming / outgoing skb.

bpf_tcp_ops.rtt is called once per RTT, which is every
incoming skb in ping-pong workloads like tcp_rr.

Even attaching NULL callbacks in the fast path hurts performance.

Let's guard them (and write_hdr_opt) with the new per-socket flags.

__bpf_tcp_ops_call() and bpf_tcp_ops_call_flag() are added to check
cgroup_bpf_enabled(CGROUP_TCP_SOCK_OPS) first and avoid accessing
tp->bpf_tcp_ops_flags when no bpf_tcp_ops is attached.

Note that bpf_tcp_ops_hdr_opt_len() has 3 callers and previously
checked the static key both in tcp_established_options() and inside
bpf_tcp_ops_call().  Now the static key is checked once at the
beginning of bpf_tcp_ops_hdr_opt_len(), and the redundant check in
tcp_established_options() is removed.

tcp_bpf_hdr_opt_len() must use __always_inline, not just
inline, otherwise Clang inlines bpf_tcp_ops_hdr_opt_len() to
tcp_bpf_hdr_opt_len() instead.

  $ llvm-objdump -S -D --disassemble=tcp_established_options vmlinux
  ...
  ; 	if (unlikely(BPF_SOCK_OPS_TEST_FLAG(tp,
  ffffffff8251270e: 41 f6 87 20 0c 00 00 40      	testb	$0x40, 0xc20(%r15)
  ffffffff82512716: 75 47                	jne	0xffffffff8251275f <tcp_established_options+0x1ff>
  ; 	asm goto(ARCH_STATIC_BRANCH_ASM("%c0 + %c1", "%l[l_yes]")
  ffffffff82512718: 0f 1f 44 00 00       	nopl	(%rax,%rax)
  ; 	return size;
  ffffffff8251271d: 44 89 e0             	movl	%r12d, %eax
  ffffffff82512720: 5b                   	popq	%rbx
  ffffffff82512721: 41 5c                	popq	%r12
  ffffffff82512723: 41 5e                	popq	%r14
  ffffffff82512725: 41 5f                	popq	%r15
  ffffffff82512727: 5d                   	popq	%rbp
  ffffffff82512728: 2e e9 42 03 43 00    	jmp	0xffffffff82942a70 <__x86_return_thunk>

bpf_skops_write_hdr_opt() is still not free, but it can be
optimised later if needed.

Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Link: https://patch.msgid.link/20261008031604.256498-5-kuniyu@google.com

0 files changed, 0 insertions, 0 deletions