From 4e2385abecb03998922afbfb91caff273fe6cbee Mon Sep 17 00:00:00 2001 From: Kuniyuki Iwashima Date: Thu, 8 Oct 2026 03:15:22 +0000 Subject: bpf: tcp: Add a new per-socket flag and kfunc for bpf_tcp_ops. The legacy SOCK_OPS guards some hooks with a per-socket flag, tp->bpf_sock_ops_cb_flags. In contrast, bpf_tcp_ops was initially designed without per-socket flags so that users can simply define only the callbacks they need. However, it turned out that even attaching a bpf_tcp_ops with a NULL callback incurs measurable overhead in the fast path. [0] We could reuse tp->bpf_sock_ops_cb_flags, but it is u8 and has only 1 bit left for a new opt-in callback, while not all of the 7 existing opt-in hooks are in the fast path and really need a flag guard. Moreover, tp->bpf_sock_ops_cb_flags has two bpf helper functions to update it, both of which require sock_owned_by_me(sk): * bpf_sock_ops_cb_flags_set() * bpf_setsockopt(TCP_BPF_SOCK_OPS_CB_FLAGS) Both helpers only overwrite the field, which leads to reading and modifying the flags in the BPF prog and then writing them back via the helper. This prevents use from the fast path (tc or cgroup_skb) hooks where bh_lock_sock() alone cannot prevent races with process context. Let's add a new u32 field, tp->bpf_tcp_ops_flags, at the end of the tcp_sock_read_txrx cacheline group, along with a new kfunc, bpf_tcp_ops_set_flags(). bpf_tcp_ops_set_flags() takes bitmasks of flags to enable and disable and updates tp->bpf_tcp_ops_flags atomically via try_cmpxchg() without relying on lock_sock(). The kfunc is exposed to bpf_tcp_ops, BPF_PROG_TYPE_CGROUP_SOCKOPT, and BPF_PROG_TYPE_CGROUP_SKB. The first argument is struct tcp_sock * so that bpf_tcp_sock() etc is required for cgroup hooks, but not for bpf_tcp_ops where struct sock * is promoted to struct tcp_sock * automatically. The fast-path callbacks will be guarded by the new flags in a later patch after updating the existing selftest to avoid breaking bisection. Signed-off-by: Kuniyuki Iwashima Signed-off-by: Alexei Starovoitov Reviewed-by: Amery Hung Acked-by: Stanislav Fomichev Link: https://lore.kernel.org/netdev/CAMB2axMwBuz3X4Uwn5uzZUqm91EbiMnbQX832wUFOQ44fOwDnQ@mail.gmail.com/ #[0] Link: https://patch.msgid.link/20261008031604.256498-3-kuniyu@google.com --- include/linux/tcp.h | 7 +++++++ 1 file changed, 7 insertions(+) (limited to 'include/linux') diff --git a/include/linux/tcp.h b/include/linux/tcp.h index 6a8c77719322..24bb751cc65a 100644 --- a/include/linux/tcp.h +++ b/include/linux/tcp.h @@ -233,6 +233,13 @@ struct tcp_sock { is_sack_reneg:1, /* in recovery from loss with SACK reneg? */ is_cwnd_limited:1,/* forward progress limited by snd_cwnd? */ recvmsg_inq : 1;/* Indicate # of bytes in queue upon recvmsg */ +#ifdef CONFIG_BPF + u32 bpf_tcp_ops_flags; +#define BPF_TCP_OPS_TEST_FLAG(TP, ARG) \ + (READ_ONCE((TP)->bpf_tcp_ops_flags) & BPF_TCP_OPS_FLAG_ ## ARG) +#else +#define BPF_TCP_OPS_TEST_FLAG(TP, ARG) (0) +#endif __cacheline_group_end(tcp_sock_read_txrx); /* RX read-mostly hotpath cache lines */ -- cgit v1.2.3