| Age | Commit message (Collapse) | Author |
|---|
|
rhashtable is a generic container: it does not know what a key is, so it
carries the key's shape as data. dsq_hash_params declares key_len,
key_offset and head_offset and nothing else, so lookups fall back to
rhashtable_compare(), which reads both the offset and the length back out
of ht->p at runtime:
memcmp(ptr + ht->p.key_offset, arg->key, ht->p.key_len)
The key is one naturally aligned u64, but neither the offset nor the
length is a compile-time constant there, so the compiler cannot narrow
the call and emits an out-of-line memcmp() for every element walked -
two loads, a call and a length dispatch to compare eight bytes. The
interpretation costs more than the comparison it is interpreting.
find_user_dsq() sits on the __schedule() path, and scx_layered calls
scx_bpf_dsq_nr_queued() once per layer per dispatch decision. On Meta's
fleet the lookup has a significant cost and shows up in fleet wide
profile.
Let's supply an obj_cmpfn. dsq_hash_params is a const object passed by
value into the __always_inline __rhashtable_lookup(), so
params.obj_cmpfn is a compile-time constant, the ternary that selects it
folds away and the callback inlines: the compare becomes a single cmp
against dsq->id. Every consumer of the result only tests it against
zero, so the ordering memcmp() also carries is never observed. No
functional change intended.
An in-kernel A/B over the DSQ ids scx_layered creates, both parameter
sets compiled into the same kernel, makes the lookup 2.9x faster.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
MEM_WRITE without MEM_UNINIT denotes memory that is read as well as
written. Helper argument checking requests only BPF_WRITE for it, which
checks stack initialization but omits read permission checks on map values.
For example, bpf_check_mtu() reads its mtu_len argument before overwriting
it, but the verifier permits that argument to point into a BPF_F_WRONLY_PROG
map. bpf_fib_lookup() and bpf_load_hdr_opt() are affected the same way.
Derive generic memory access from argument flags: read-only for inputs,
write-only for MEM_WRITE | MEM_UNINIT, and read/write for MEM_WRITE alone.
Partial-output helpers such as bpf_snprintf() now carry MEM_UNINIT, so
their write-only map destinations remain valid. Generated kfunc prototypes
already encode the same distinction, which retires the kfunc-specific
access override.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260921023843.411943-11-memxor@gmail.com
|
|
Partial-output helpers such as bpf_snprintf() do not read incoming buffer
contents, but may leave some bytes untouched. Let them accept uninitialized
storage without promising full initialization to callers that cannot read
uninitialized stack memory.
Make this the default for generic MEM_UNINIT buffers, including __uninit
kfunc arguments. Allow invalid stack bytes through the output check but
leave them invalid when uninitialized stack reads are not allowed. Scrub
initialized bytes and scalar spills as possible writes, retaining the
existing restrictions on spilled pointers and special stack objects.
Retain the output annotation for variable-sized arguments and track their
raw-mode eligibility separately. Callers allowed uninitialized stack reads
can continue treating the potentially written range as initialized. For
constant ranges, defer that initialization until all inputs are checked.
Keep prior stack contents live for generic outputs when the caller cannot
read uninitialized stack memory. Such calls do not define the entire range,
so liveness must preserve initialization facts that remain relevant after
the call. Dynptr and iterator constructors still define their storage.
Annotate the snprintf, sysctl name, d_path, snprintf_btf and branch-record
destinations with MEM_UNINIT, and document that generic __uninit kfuncs may
leave bytes untouched. This also changes readback from existing full-writing
helpers: without permission to read uninitialized stack memory, programs
must initialize those bytes themselves before reading them after a call.
Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260921023843.411943-9-memxor@gmail.com
|
|
A single output descriptor cannot retain the initialization ranges of two
__uninit memory arguments. Track raw-mode eligibility and the definite
output size separately for each ABI slot, so a variable-size output leaves
independent constant-size outputs intact.
Use the shared argument-checking path to record outputs for both helpers
and kfuncs. With per-slot tracking, neither needs the single-output
prototype restriction. Initialize every recorded output after checking all
arguments, skipping empty slots before looking up their register state.
Use the resolved BTF parameter when looking up __uninit for stack liveness.
A preceding by-value parameter can consume multiple ABI slots, so its slot
number cannot index the BTF parameter array.
Suggested-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260921023843.411943-7-memxor@gmail.com
|
|
Stack liveness treats __uninit kfunc arguments as writes. Even with
write-only access checks, ordinary clobber handling leaves poisoned stack
bytes poisoned after the call, so the output cannot be read.
Reuse the helper output descriptor for generic kfunc buffers. Record the
output after resolving its memory type, and initialize its stack bytes
only after all arguments have been checked. This prevents an output from
making an aliased, uninitialized input appear valid.
Track the output by its ABI argument slot, since a kfunc pointer may be
passed on the stack or follow a multi-slot by-value argument. Keep the
single-output restriction in prototype validation; multiple-output
tracking is a separate extension.
Document that output buffers must be fully initialized on every return
path, including error returns and padding.
Fixes: 2cb27158adb3 ("bpf: poison dead stack slots")
Reported-by: Tejun Heo <tj@kernel.org>
Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/86d966ec88bbf27d21b2bb4e18c8aa00@kernel.org
Link: https://patch.msgid.link/20260921023843.411943-5-memxor@gmail.com
|
|
Generic kfunc memory arguments are checked for both read and write access,
including __uninit outputs. An output pointing into a write-only map value
is therefore rejected, and a stack output must already hold readable
contents even though the kfunc only writes it.
Mark generic buffers in generated kfunc prototypes with MEM_WRITE, and keep
MEM_UNINIT when a struct pointer resolves to generic memory. Check __uninit
buffers as write-only while ordinary kfunc buffers retain read/write
access. Helper access selection is unchanged. Definite initialization of
stack outputs after the call is addressed separately.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260921023843.411943-4-memxor@gmail.com
|
|
Record raw outputs in check_func_arg() after resolving their memory type,
instead of pre-recording them in check_raw_mode_ok(). Keep prototype
validation separate from per-call output tracking so kfuncs can share the
latter in a following patch.
For bpf_strtol(), bpf_strtoul() and bpf_kallsyms_lookup_name(), a variable-size
input is checked before the arg4 output. Recording outputs as their arguments
are checked prevents that input from clearing the output's raw mode early.
Only clear raw mode when a variable size belongs to the recorded output:
an unrelated later input must also preserve an earlier output descriptor.
For bpf_map_peek_elem() on a bloom filter, type resolution removes MEM_UNINIT
and MEM_WRITE because the value is an input. Recording the resolved type makes
the later bloom-filter-specific raw-mode reset unnecessary. Reuse the same
raw-memory predicate for output recording and prototype validation.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260921023843.411943-3-memxor@gmail.com
|
|
|
|
Convert open-coded if/else with set_bit/clear_bit and their
non-atomic __set_bit/__clear_bit variants to the assign_bit/__assign_bit
API.
Done with Coccinelle semantic patch:
// set_bit -> clear_bit => assign_bit
@@
expression cond, bit, addr;
@@
-if (cond)
- set_bit(bit, addr);
-else
- clear_bit(bit, addr);
+assign_bit(bit, addr, cond);
// clear_bit -> set_bit => assign_bit
@@
expression cond, bit, addr;
@@
-if (cond)
- clear_bit(bit, addr);
-else
- set_bit(bit, addr);
+assign_bit(bit, addr, !cond);
// __set_bit -> __clear_bit => __assign_bit
@@
expression cond, bit, addr;
@@
-if (cond)
- __set_bit(bit, addr);
-else
- __clear_bit(bit, addr);
+__assign_bit(bit, addr, cond);
// __clear_bit -> __set_bit => __assign_bit
@@
expression cond, bit, addr;
@@
-if (cond)
- __clear_bit(bit, addr);
-else
- __set_bit(bit, addr);
+__assign_bit(bit, addr, !cond);
Signed-off-by: Peng Fan <peng.fan@nxp.com>
Acked-by: Michal Koutný <mkoutny@suse.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
power supply
Neil Armstrong <neil.armstrong@linaro.org> says:
Document and add regulator support for the Nexperia NEX10000UB dual
output LCD bias power supply which provides programmable positive
and negative output voltages mainly for display panels applications.
The driver supports setting the output voltage from 4V to 6V in
100mV steps for each output via the I2C programming interface and
supports the enable GPIOs for both outputs.
Product datashet can be found online at [1].
[1] https://assets.nexperia.com/documents/data-sheet/NEX10000UB.pdf
Link: https://patch.msgid.link/20260921-topic-sm8x50-nex10000ub-v2-0-eeeaf9b0c913@linaro.org
|
|
In preparation for converting the kmalloc family of allocators to the
type-aware kmalloc_obj family, we need to make sure that the returned
type from the allocation matches the type of the variable being
assigned. (The kmalloc family returns "void *", which can be implicitly
cast to any pointer type.)
argv holds 5 pointers, but the size was taken from the array type
"char *[5]", which would make the allocation type a pointer to that
array rather than the "char **" being assigned. Allocate 5 entries of
the target's type instead. The resulting allocation size is the same.
Build tested ARCH=x86_64 allmodconfig with GCC 16.2.0:
kernel/module/kmod.o
Assisted-by: LLM coccinelle
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
Linux 7.3-rc4
Merged to pick up an x86 build fix needed for the ad9910 driver
and the IIO fixes recently picked up.
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull timer race fixes from Ingo Molnar:
- Fix timer signal <-> exec() race, to prevent UAF (Thomas Gleixner)
- Clean up POSIX CPU timers right after de_thread(), to prevent UAF
(Hyunwoo Kim)
- Fix POSIX CPU timers race between expiry and timer_settime(),
to prevent UAF (Thomas Gleixner)
* tag 'timers-urgent-2026-09-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
posix-cpu-timers: Prevent freeing a timer which is queued on the expiry list
exec: Cleanup POSIX timers right after de_thread()
signal: Prevent exec() race
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fix from Ingo Molnar:
- Avoid false positive migration warning for proxy donors
(Andrea Righi)
* tag 'sched-urgent-2026-09-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched/core: Avoid false migration warning for proxy donors
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull perf events fixes from Ingo Molnar:
- Fix crash when probing CS CALL instructions (Jinke Han)
- Fix NULL pointer crash during module unload (Vinay Belgaumkar)
* tag 'perf-urgent-2026-09-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
perf: Fix null pointer access in is_include_guest_event()
x86/kprobes: Fix crash when probing CS CALL instructions
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull futex fix from Ingo Molnar:
- Also allocate a default private futex hash on vfork() as well, to
avoid races with (private) futex waiters (Peter Zijlstra)
* tag 'locking-urgent-2026-09-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
futex: Also allocate private hash on vfork()
|
|
bpf_crypto_ctx_create() is a kfunc whose second argument is declared
with the __sz annotation, so the verifier only guarantees that
params__sz bytes of params are valid. The function nevertheless reads
params->reserved[0] and params->reserved[1] (offsets 14 and 15) before
comparing params__sz against the size of struct bpf_crypto_params, so a
BPF program can pass a shorter buffer and have the kernel read past the
region that was validated for it.
Move the size check in front of the reserved field reads.
Fixes: 3e1c6f35409f ("bpf: make common crypto API for TC/XDP programs")
Reported-by: Vega <vega@nebusec.ai>
Signed-off-by: Yuqi Xu <xuyuqiabc@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Ren Wei <weir@nebusec.ai>
Link: https://patch.msgid.link/4f3ab4b03e79017e215521743996555439bf0bb3.1789802413.git.xuyuqiabc@gmail.com
|
|
Kijo analyzed another race in the POSIX CPU timer code:
Commit bf635681c906 converted cpu_timer::firing from a tristate value to a
boolean. This lost the distinction between "not owned by the firing list"
and "still owned, but delivery was canceled". The resulting race is:
expiry handler timer_settime() timer_delete()
-------------- --------------- --------------
collect timer onto
private firing list
firing = true
observes firing = true
firing = false
return TIMER_RETRY
wait for handler
observes firing = false
finish deletion
unhash and free timer
resume list traversal
read freed elist.next
-> UAF
The firing bit is clearly the wrong indicator since that commit.
Check whether the timer is queued on the expiry list or not instead. If it
is queued clear the firing bit to prevent signal delivery as before and
return TIMER_RETRY so the caller unlocks the timer which allows the expiry
code to make progress and remove it from the list.
Fixes: bf635681c906 ("posix-cpu-timers: Cleanup the firing logic")
Reported-by: Kijo Park <red993688@gmail.com>
Debugged-by: Kijo Park <red993688@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Kijo Park <red993688@gmail.com>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Cc: stable@vger.kernel.org
|
|
The stubs that bpf_lsm_proto.c overrides are generated as:
#define LSM_HOOK(RET, DEFAULT, NAME, ...) \
__weak noinline RET bpf_lsm_##NAME(__VA_ARGS__) \
{ \
return DEFAULT; \
}
Mark the overrides noinline as well so they hold up the same contract
as the stubs they replace.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260915150739.284189-6-daniel@iogearbox.net
|
|
Many in-kernel LSMs (SELinux, Smack, IMA) store security labels in extended
attributes. For these LSMs, atomic labeling during inode creation is
critical: if the inode becomes accessible before its xattr is set, it is
briefly unlabeled, which can disrupt LSMs making policy decisions based
on file labels. Existing LSMs solve this by setting xattrs in the
inode_init_security hook, which runs before the inode becomes accessible.
BPF LSM programs currently lack this capability because the hook uses an
output parameter (xattr_count) that BPF programs cannot write to, and
existing kfuncs like bpf_set_dentry_xattr() require a dentry that isn't
available until after the inode is accessible.
Add a bpf_inode_init_xattr() kfunc that takes the hook's own xattrs and
xattr_count arguments, passed through from the program's context, and
claims a slot via lsm_get_xattr_slot() on the program's behalf. The
xattr_count output argument is exposed to inode_init_security programs
as trusted read-only memory, so programs can pass it to the kfunc but
cannot modify the count themselves.
Reserve BPF_LSM_INODE_INIT_XATTRS slots in bpf_lsm_blob_sizes the way
every other xattr-providing LSM does, for the life of the kernel. The
framework keys the collection off the reserved slot count, so a kernel
built with CONFIG_BPF_LSM=y now allocates the xattr array on every inode
creation, whether or not a program sits on the hook.
Give the hook a strong prototype in bpf_lsm_proto.c so that its qstr and
xattrs arguments are marked __nullable. Both can be NULL for some callers.
Without the annotation the verifier otherwise hands the program a trusted
non-NULL pointer which it dereferences. Also, keep the hook out of the
sleepable set. inode_init_security runs inside the transaction creating
the inode, with a journal handle held on ext4 and btrfs and the parent's
i_rwsem down, which is why everything on the path allocates GFP_NOFS.
Co-developed-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: David Windsor <dwindsor@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260915150739.284189-5-daniel@iogearbox.net
|
|
Allows programs to pass a context argument that points to a scalar
output value on to a kfunc that writes the result on the program's
behalf.
A ctx_arg_info entry can now describe a fixed-size PTR_TO_MEM context
argument through a new mem_size field, honored in btf_ctx_access().
Loading the argument yields a PTR_TO_MEM | MEM_RDONLY | PTR_TRUSTED
register of known size, readable but not writable by the program. A
kfunc argument tagged with the new "__ctx_out" suffix accepts only that
register type with a size matching the pointed-to type, so the only
value a program can pass is an output argument from its own context.
The pointer stays read-only to the program because the value is
trusted by whoever invoked the BPF program. In the first use case, the
inode_init_security LSM hook, every LSM receives a shared xattr array
and an int *xattr_count that lsm_get_xattr_slot() increments; a program
that could store a garbage count would push another LSM's write out of
bounds, so only the kfunc itself writes through it.
Suggested-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Co-developed-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: David Windsor <dwindsor@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260915150739.284189-4-daniel@iogearbox.net
|
|
The verifier types a load from a tracing-style context by its offset
alone: btf_ctx_access() looks the offset up in the attach hook's
prototype, and check_mem_access() stamps the resulting type onto the
destination register. Nothing tracks what the slot actually holds.
That is sound only as long as the context is read-only, and for BPF LSM
it is not. lsm_verifier_ops wires ->is_valid_access() straight to
btf_ctx_access(), which never looks at enum bpf_access_type, so a store
into the context passes verification. Every other user of the helper
either goes through bpf_tracing_btf_ctx_access() or open-codes the same
type != BPF_READ test first; BPF LSM is the only one that does neither.
A program can therefore store a scalar of its choosing into a context
slot, read the slot back, and get a register the verifier believes is a
trusted pointer of the hook's argument type. Handing that to a kfunc or
helper which dereferences a trusted argument turns it into an arbitrary
kernel access. Route BPF LSM through bpf_tracing_btf_ctx_access() like
everyone else.
Fixes: fc611f47f218 ("bpf: Introduce BPF_PROG_TYPE_LSM")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260915150739.284189-3-daniel@iogearbox.net
|
|
The cid-form API has an obvious hole. A task's cid mask is only visible
through ops.set_cmask(), which fires on affinity changes and class switches
but not when a task enters a scheduler through fork, sub-sched enable or
re-home, and there is no p->cpus_ptr equivalent to fall back on. Schedulers
work around it by seeding the mask in ops.init_task() from p->cpus_ptr cid
by cid, which is subtly wrong: on sub-sched enable and re-home, an affinity
change between init_task() and enable() is delivered to the sched the task
is still on, and nothing corrects the new sched's copy afterwards.
Fix it by adding struct scx_enable_args to cid-form ops.enable() carrying
the task's cmask, built in the per-cpu scratch under the rq lock as the task
enters the scheduler, and calling set_cmask() with the same mask right after
enable(), ahead of set_weight(). A scheduler can then track affinity in
set_cmask() alone, and scx_qmap drops its init_task() seed. set_cmask() no
longer fires for a cid-form task before it is enabled, and the class-switch
republish in switching_to_scx() is limited to the cpu form.
This changes the cid-form ops.enable() signature, which is fine as the
cid-form API is still considered unreleased. An args struct rather than a
bare cmask argument leaves room for more initial state without another
signature change, and the cmask travels as a plain arena address because BTF
can't type arena struct members yet.
v2: The cmask travels as a u64 arena address, cmask_arena_addr, instead of a
kernel-typed pointer, with the typing limitation and the planned typed alias
documented (Sashiko review).
v3: The initial set_cmask() is delivered before set_weight() so that the
mask is in place when weight-dependent state is derived (Andrea Righi).
Selftest added.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
This patch adds necessary infrastructure to attach a struct_ops
map to a cgroup. The initial need was to support migrating
the legacy BPF_PROG_TYPE_SOCK_OPS to a struct_ops.
Recently, there are other struct_ops use cases that
need to attach struct_ops to a cgroup. For example,
the recent BPF OOM and memcg discussion in LSFMMBPF 2026.
The motivation is to create a consistent expectation
for attaching struct_ops to cgroup instead of each subsystem
creating its own infrastructure. This logic includes
hierarchy expectation, ordering expectation,
attachment API, and rcu gp.
There is already an existing implementation for attaching
multiple bpf progs to a cgroup. There are also tools
built around it for querying. Attaching a struct_ops map
(which is a group of bpf programs) could also adhere to
a similar API and potentially reuse most of the existing
implementation.
A couple of ideas have been tried. One of them
is to use mprog.c. In terms of the amount of changes,
I eventually came to the same conclusion as in
commit 120933984460 ("bpf: Implement mprog API on top of existing cgroup progs").
I then shifted the focus to reusing the current
{update,compute,activate,purge}_effective_progs() which has
the main logic that implements the mprog API.
Since then, I tried to add a 'struct cgroup *cgroup' member
to the existing 'struct bpf_struct_ops_link' and link_create
will create a 'struct bpf_struct_ops_link' object to be stored
in the pl->link. This turns out to have more changes on
both cgroup.c and bpf_struct_ops.c than I like.
This patch directly reuses the 'struct bpf_cgroup_link' which
cgroup.c already understands. Add 'struct bpf_map *map'
to 'struct bpf_cgroup_link'. In the future, as more subsystems
are extended by struct_ops, we may consider to make
'struct bpf_map *map' as a primary citizen of a link
like 'struct bpf_prog *prog' and directly add
'struct bpf_map *map' to the generic 'struct bpf_link'.
The pl->link could be the traditional 'prog' link or the
new 'map' link. The places that need to handle them differently
have already been refactored into the new prog_list_*() added in
the earlier patch. In those new prog_list_*(), this patch will
check "pl->link && pl->link->map", learn that it is a 'map' link
and handle it correctly.
The bpf_prog_array also needs to handle that its item can store
the traditional 'prog' or it can store a struct_ops map.
The places that need to handle them differently have also
been refactored into the new bpf_cgroup_array_*() added
in the earlier patch. The two differences are:
- different sentinel (dummy_bpf_prog in prog vs cfi_stub in struct_ops)
- the array for struct_ops may need to go through different
rcu gp.
The bpf_cgroup_array_*() functions use the cgroup_bpf_attach_type (ie atype)
to distinguish the array is storing prog or storing struct_ops map.
This patch also implements a separate struct bpf_link_ops
"cgroup_struct_ops_link_ops" to have a separate link_ops implementation
that only handles the cgroup's struct_ops link.
Questions:
- Although this patch did not change it, it is not obvious to me how
the replace_effective_progs() and purge_effective_progs() handle
cases when there are existing BPF_F_PREORDER progs attached
in the hlist.
Misc notes:
- CGROUP_TCP_SOCK_OPS is added to the 'enum cgroup_bpf_attach_type'.
The actual implementation of the tcp_bpf_ops (a struct_ops)
will be added in the next patch.
- free_after_mult_rcu_gp is added to 'struct bpf_struct_ops' such that
the bpf_prog_array can have a mix of sleepable and
non-sleepable prog in a struct_ops. This can tell
how the bpf_prog_array should be freed.
- For a struct_ops that supports cgroup attachment, it does not need to
implement its own reg/unreg function. reg/unreg to a cgroup is
done by the common infrastructure added in this patch.
- The cgroup's struct_ops link only supports BPF_F_ALLOW_MULTI.
This is enforced internally in cgroup_bpf_struct_ops_attach.
This should be consistent with the current prog's link
behavior in cgroup_bpf_link_attach.
In the future, we may allow each subsystem to choose differently.
- A cgroup_atype member is added to 'struct bpf_struct_ops'.
When a subsystem struct_ops needs to support cgroup attachment,
it needs to add a value to 'enum cgroup_bpf_attach_type'
and then assign it to the newly added cgroup_atype member
in the bpf_struct_ops.
- During LINK_CREATE in syscall, the patch uses the same
BPF_STRUCT_OPS (in attr->link_create.attach_type).
The bpf_struct_ops_link_create learns the map and
from the map it learns the st_ops. If the st_ops->cgroup_atype
is not 0, it will create a cgroup's link.
- When a subsystem registers a struct_ops that supports cgroup
attachment, the struct_ops infrastructure will also ask the
cgroup infrastructure to remember a few things. This is done
by calling cgroup_bpf_struct_ops_register().
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260917200542.3689605-10-ameryhung@gmail.com
|
|
In the upcoming patch, the array can store a struct_ops map.
The array could have a cfi_stubs acting as a dummy instead of
the dummy_bpf_prog. The array logic will need to skip the cfi_stubs
also in order to support storing struct_ops map in the array.
bpf_cgroup_array_length(), bpf_cgroup_array_copy_to_user(), and
bpf_cgroup_array_delete_safe_at() are added as a preparation work
to allow skipping the cfi_stubs in the upcoming patch. This patch
only skips the dummy_bpf_prog which is the same as the existing behavior.
The current bpf_prog_array_*() callers are changed to call the new
bpf_cgroup_array_*(). This is a no-op change.
Unlike bpf_prog_array_copy_to_user(), bpf_cgroup_array_copy_to_user()
does not need a temporary buffer. The cgroup caller already holds
cgroup_mutex and dereferences the effective array with
rcu_dereference_protected(), so it does not copy to userspace
from an RCU read-side critical section. Details in commit 0911287ce32b.
Another addition is the bpf_cgroup_array_free(). This prepares
the array to have a different rcu gp for the struct_ops use case,
for example, a struct_ops could have mix of sleepable ops and
non-sleepable ops. In this patch, bpf_cgroup_array_free() only
goes through the regular rcu gp. This is a no-op change also.
bpf_prog_dummy() is also added to return the global dummy_bpf_prog.
bpf_cgroup_array_dummy() is added to decide the sentinel based on atype.
It now always returns bpf_prog_dummy(). In the upcoming patch,
it can return a cfi_stubs if the atype belongs to a struct_ops.
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://patch.msgid.link/20260917200542.3689605-9-ameryhung@gmail.com
|
|
Move the LSM trampoline unlink into bpf_cgroup_link_auto_detach().
The purpose is to consolidate the auto_detach cleanup logic.
It prepares for the upcoming struct_ops cgroup attachment patch where
bpf_cgroup_link_auto_detach() will need to handle the struct_ops case
(link->map != NULL).
This is a no-op change.
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://patch.msgid.link/20260917200542.3689605-8-ameryhung@gmail.com
|
|
Add three helpers to abstract operations on a bpf_prog_list entry.
Right now, bpf_prog_array_item is initialized from prog_list_prog(pl),
which returns either pl->prog or pl->link->link.prog. This will not work
when struct_ops is attached to a cgroup because the attachment is backed
by a struct_ops map instead of a BPF prog.
The same applies to __cgroup_bpf_query(). Instead of always copying a
prog id to userspace, struct_ops cgroup attachment will need to copy the
struct_ops map id.
Refactor bpf_prog_array_item initialization into prog_list_init_item()
and prog_list_replace_item(), and refactor id lookup into prog_list_id().
These helpers will be extended to support pl->link->map in a later patch.
This is a no-op change.
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://patch.msgid.link/20260917200542.3689605-7-ameryhung@gmail.com
|
|
prog_list_length() and compute_effective_progs() use !prog_list_prog(pl)
to skip a 'detaching' pl.
When pl->link is not NULL, prog_list_prog(pl) returns
the pl->link->link.prog. This does not work for the upcoming struct_ops
patch where pl->link is not NULL but pl->link->link.prog is NULL,
because a struct_ops map is attached to the cgroup instead of a BPF prog.
To prepare for the upcoming struct_ops patch, this patch
replaces the prog_list_prog() test with the
"!pl->prog && !pl->link". In __cgroup_bpf_detach(),
both pl->prog and pl->link are set to NULL, so testing
"!pl->prog && !pl->link" is the same test to tell
if a pl is being detached. This change should be a no-op.
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260917200542.3689605-6-ameryhung@gmail.com
|
|
effective_prog_pos(), called from replace_effective_prog() and
purge_effective_progs(), tests "!prog_list_prog(pl)" to skip a
'detaching' pl.
When detaching a pl, pl->prog and pl->link are set to NULL in case
the update_effective_progs() failed.
However, replace_effective_prog() is not detaching a pl,
so the case "!prog_list_prog()" will not happen.
In purge_effective_prog(), the pl->prog and pl->link are restored
before calling purge_effective_progs(), so the case "!prog_list_prog()"
will not happen either.
This patch removes them as a prep work for the upcoming work
in attaching struct_ops to cgroup. When attaching a struct_ops
to cgroup, there is a link->map case and the prog_list_prog()
will not consider the link->map. The replace_effective_prog()
and purge_effective_progs() will then incorrectly skip a pl
with struct_ops map attached to it.
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://patch.msgid.link/20260917200542.3689605-5-ameryhung@gmail.com
|
|
Add the helper functions bpf_struct_ops_map_kdata() and
bpf_struct_ops_map_cfi_stubs() in bpf_struct_ops.c. They will be called
from cgroup.c in the upcoming patch to create a struct_ops to cgroup
attachment link.
bpf_struct_ops_valid_to_reg() is also exposed for the upcoming caller
in cgroup.c.
The link update validation is also refactored into a new function
bpf_struct_ops_link_update_check() such that it can be reused by the
caller in cgroup.c in the upcoming patch.
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://patch.msgid.link/20260917200542.3689605-4-ameryhung@gmail.com
|
|
bpf_struct_ops_map_free() currently waits for both a regular RCU grace
period and a tasks RCU grace period for every struct_ops map through
synchronize_rcu_mult(call_rcu, call_rcu_tasks).
A regular RCU grace period is still required for all struct_ops maps
because the struct_ops trampoline ksyms requires a rcu grace period
(take a look at the list_del_rcu in __bpf_ksym_del).
Add a map_free_pre_rcu() callback so the struct_ops map can remove
ksyms before bpf_map_put() wait for the regular rcu grace period.
The tasks RCU grace period is only needed by tcp_congestion_ops.
Add free_after_tasks_rcu_gp only to struct bpf_struct_ops instead
of the bpf_map.
When CONFIG_TASKS_RCU=n, synchronize_rcu_tasks() is the same as
synchronize_rcu(). Since all struct_ops maps now complete a regular RCU
grace period before bpf_struct_ops_map_free() runs, skip the extra
synchronize_rcu_tasks() call in this case.
This cleanup prepares for a later patch that needs to support
free_after_mult_rcu_gp.
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260917200542.3689605-3-ameryhung@gmail.com
|
|
st_link->map is always written under update_mutex. The paths that read
st_link->map with rcu_read_lock() are not in the fast path, so they can
simply take update_mutex instead. Remove the __rcu annotation and replace
all RCU accessors with direct pointer reads under update_mutex. Use
READ_ONCE() in bpf_struct_ops_map_link_poll() which reads the pointer
without holding update_mutex.
It is a simplification change.
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260917200542.3689605-2-ameryhung@gmail.com
|
|
Program-allocated objects can own other local objects through referenced
kptrs. bpf_obj_free_fields() follows those pointers through
__bpf_obj_drop_impl() synchronously, before the object storage is freed
through RCU. A self-referential local kptr type therefore permits arbitrarily
deep object chains, and dropping the head can exhaust the kernel stack.
Long acyclic type chains have the same problem.
btf_check_and_fixup_fields() still assumes referenced kptrs only point to
kernel types and checks ownership through list and rbtree roots only. Its
existing rule is sufficient for graph-only cycles: the target of each graph
edge must contain a node, so every type in a cycle has both a root and a
node. The rule rejects such a type owning another root, breaking every
cycle. It also limits graph-only chains to three types, or two if the first
type contains a node, and conservatively rejects longer acyclic chains.
The missing local-kptr edges, rather than a missed graph-only cycle, are the
bug introduced by support for bpf_kptr_xchg() into local kptrs.
Replace that restriction with one bounded ownership walk covering graph
roots and local referenced kptrs. Run it after all BTF records have been
fixed up, reject cycles and paths deeper than eight record-bearing types,
and cache each type's suffix depth while checking it against the remaining
budget. This also permits the longer acyclic graph-only layouts rejected
by the old rule; update their existing BTF tests accordingly.
Keep the bound independent of MAX_CALL_FRAMES because recursive destruction
can run below a BPF call chain. A plain local pointee without special-field
metadata adds only a final non-recursing drop. Non-owning kptrs and
kernel-BTF kptrs do not recurse through local records and remain outside the
walk. Include local percpu-kptr edges too, although allocation of percpu
objects with special fields is currently forbidden, so that relaxing that
restriction cannot bypass the ownership bound.
btf_check_and_fixup_fields() continues to initialize graph_root.value_rec,
including for separately allocated map records. The ownership relationships
belong to immutable program BTF and only need validation at BTF load time.
Fixes: b0966c724584 ("bpf: Support bpf_kptr_xchg into local kptr")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260914132444.2564218-2-memxor@gmail.com
|
|
regs_exact() compares the register state up to id, followed by the ID
mappings, but does not compare frameno. The PTR_TO_STACK case in regsafe()
checks frameno separately, which is bypassed when exact comparison is
requested. Consequently, infinite-loop detection can treat pointers to
different stack frames as the same pointer and reject a finite loop.
For example, initialize fp-8 to zero in the caller and to one in the
callee, then pass the caller's fp-8 to the callee as r1:
loop:
r0 = *(u64 *)(r1 + 0);
if r0 != 0 goto done;
r1 = r10;
r1 += -8;
goto loop;
done:
exit;
The loop terminates after reading the callee's slot on its second
iteration. At the loop header, however, the only relevant difference is
r1's frameno, so exact comparison incorrectly reports an infinite loop.
The same problem occurs when the pointer is spilled to the stack.
Move frameno into the type-specific metadata union, ahead of id, so the
existing prefix comparison in regs_exact() covers it. Ordinary stack
pointers do not use another union member. Iterator and IRQ stack-slot
states use their dedicated union views and do not need a frame lookup.
This also keeps bpf_reg_state at 80 bytes.
Since frameno now shares storage with other pointer metadata, it is only
meaningful for PTR_TO_STACK registers. Return NULL from bpf_func() for
other register types. process_iter_arg(), get_constant_map_key() and
is_dynptr_reg_valid_init() look up the frame before checking the register
type and would otherwise index frame[] with a byte of the register's map
or BTF pointer. They dereference the frame only after their type check.
Move the states_maybe_looping() boundary from frameno to precise after the
field relocation. Its prefix comparison continues to cover the complete
value state and now includes frameno.
Continue to ignore precise. Precision marks control whether pruning may
ignore scalar ranges; they do not change the represented values, and exact
comparison already compares those ranges unconditionally. Marks can also
change through backtracking while an ancestor state is still being
explored.
Fixes: d5b892fd607a ("bpf: make infinite loop detection in is_state_visited() exact")
Reported-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260919014213.1840880-2-memxor@gmail.com
|
|
The fair scheduling class can request lazy rescheduling, deferring an
in-kernel scheduling boundary until returning to user space or until the
next scheduler tick. sched_ext only exposes immediate preemption,
preventing BPF schedulers from making the same trade-off.
Add SCX_ENQ_PREEMPT_LAZY and SCX_KICK_PREEMPT_LAZY. Both expire the
current sched_ext task slice but request lazy rescheduling. Immediate
preemption, WAIT and plain kicks take precedence when requests are
combined, while a lazy enqueue to a non-local DSQ retains the
head-insertion semantics of SCX_ENQ_PREEMPT.
Add SCX_OPS_LAZY_RESCHED as the default slice-expiry policy for newly
enabled tasks and initialize it before ops.enable(). Add
scx_bpf_task_set_lazy_resched() so the owning scheduler can override the
persistent policy per task from any callback while preserving
sub-scheduler task ownership boundaries. Bypass continues to force
immediate rescheduling.
For an infinite-slice task whose tick is stopped, set the lazy
reschedule request before restoring the scheduler tick dependency. This
lets the dependency update's IPI serve the request on return to user
space and guarantees that the next tick promotes it otherwise. Use
pinned rq locking for remote kick delivery so RQCF_UPDATED reflects
clock updates since lock acquisition.
Accumulate kick requests independently and resolve precedence while
holding the target rq lock. Reject invalid SCX_KICK_IDLE combinations
and unknown kick flags, which were previously silently ignored.
Tested-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
|
|
Since ad67b74d2469 ("printk: hash addresses printed with %p") %p is hashed
by default, which is also what %pK prints under the default kptr_restrict=0.
Converting the remaining %pK users makes it possible to remove its handling
from vsprintf.
Documentation/process/deprecated.rst asks that no new %p uses be added, but
these pointers are what makes the css_set debug output useful, and debug.c
already prints css pointers with %p. Keep the pointer output and use %p.
Under kptr_restrict=1 and 2 the files now show the same hashed values as any
other %p output: unprivileged readers see hashes instead of zeros, and a
CAP_SYSLOG reader under kptr_restrict=1 no longer gets raw addresses.
Booting with hash_pointers=never is the remaining way to see them.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Michal Koutný <mkoutny@suse.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The current RCU CPU stall-warning code can say "INFO: Stall ended before
state dump start" but then panic the system anyway. This commit therefore
avoids panicking the system if the stall has already ended.
However, the sched_ext scheduler will still be ejected, as it might have
been the cause of the almost stall. This can easily be changed if needed.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
devices and drivers
The ability to add and remove devices from a driver through the sysfs
"bind" and "unbind" files was created all those decades ago as a way
that kernel developers can iterate faster, and provide a debugging way
for users to attempt to add a new device to a driver without having to
rebuild their kernel.
This api over the years has been abused and recently come under a major
fuzzing "attack" through tools like syzbot which decided that it would
attempt to just randomly bind any driver to any type of device, causing
loads of unneeded errors and pointless kernel patches to be generated by
unsuspecting new developers.
Handle all of this by adding a new taint flag, TAINT_FORCED_BIND, which
will be set on the driver if the bind/unbind sysfs files are ever
written to. This lets kernel developers "know" that a user is
attempting to do something that is not normal, and as such, if the
kernel breaks they get to keep the shiny pieces laying around on the
floor.
The flag is 'Y' which was unused, and can remembered as the user is
"yeeting" the device being operated on here (thrown with force without
regard for the thing being thrown).
Note, the taint flag gets set _BEFORE_ the bind/unbind callback happens,
as many times crashes/oops/warnings/failures happen within the callback,
and the taint flag needs to be there to show what was being attempted.
If it were to be set after the callback happens, the oops report would
not properly reflect what foolishness was being attempted.
Fuzzing tools like syzbot, that doesn't have hand-crafted rules to keep
the tool from hitting bind/unbind, should be run with panic_on_taint
enabled so that they fall over and don't continue on, thinking that they
actually found a real issue.
Userspace operations that rely on the bind/unbind files to work around
the lack of will to upgrade a kernel image to a newer version with
proper support for new devices, or the lack of will to submit valid
device ids to driver authors, will still work properly, but now the
kernel will be flagged in a way that will show that perhaps those users
should reconsider their behavior and work to have the drivers properly
support these devices in a "native" manner.
Finally, the bind/unbind files can find real use-after-free issues with
some drivers by forcing the process to happen virtually without having
to rely on manual removal processes. Those real bugs should still be
worked on, but by adding this taint flag, developers can more easily
determine bug reports that are actually worth looking at.
Reviewed-by: Johan Hovold <johan@kernel.org>
Tested-by: Johan Hovold <johan@kernel.org>
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260914-bind_taint-v4-3-eadf8a090903@linuxfoundation.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
|
|
When add_taint_module() was added, the documentation for it was copied
directly from add_taint(). But that documentation was a bit wrong
(wrong variable name, and misleading boolean reference.) Fix up both
functions to have the correct documentation for how to use the
lockdep_ok parameter to these functions.
Reported-by: Petr Pavlu <petr.pavlu@suse.com>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260914-bind_taint-v4-2-eadf8a090903@linuxfoundation.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
|
|
Other kernel code might want to call add_taint_module() so pull it out
and make it global. If modules are not enabled, this defaults to a call
to add_taint(), so all is fine.
Reviewed-by: Johan Hovold <johan@kernel.org>
Tested-by: Johan Hovold <johan@kernel.org>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260914-bind_taint-v4-1-eadf8a090903@linuxfoundation.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
|
|
Mark linux_binprm->mm as a trusted-or-null nested pointer so BPF programs
can pass it to kfuncs after a NULL check.
The field is either NULL or points to a live mm_struct whenever BPF can
access a linux_binprm. On successful exec, exec_mmap() installs the new
address space before begin_new_exec() clears bprm->mm. The bprm_mm_init()
error path clears the field before mmdrop(), and free_bprm() clears it
before mmput(), as ensured by an earlier patch in this series.
Update the existing LSM selftest to check bprm->mm for NULL before
dereferencing it, as required for trusted-or-null pointers.
Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Sun Jian <sun.jian.kdev@gmail.com>
Reviewed-by: Matt Bobrowski <matt@bobrowski.net>
Link: https://patch.msgid.link/20260918131757.42802-5-tasos.papagiannnis@gmail.com
|
|
On CONFIG_MMU kernels, when security_bprm_check() runs, the argument and
environment strings for the exec have been copied into bprm->mm. The new
address space is not associated with a task_struct until exec_mmap(), so
existing BPF user memory helpers cannot access it.
Add bpf_copy_from_user_mm() and bpf_copy_from_user_mm_str() kfuncs. Both
take a struct mm_struct pointer directly, allowing callers to access
trusted address spaces that are not associated with a task_struct.
bpf_copy_from_user_mm() has similar semantics to
bpf_copy_from_user_task(). bpf_copy_from_user_mm_str() copies one
NUL-terminated string and returns its size including the NUL terminator.
It accepts BPF_F_PAD_ZEROS to clear unused destination bytes on success.
Refactor the task-based helpers and the new mm-based kfuncs to share
static internal implementations. The task-based interfaces validate their
arguments before acquiring and holding a reference to the task's mm for
the copy. No behavior change is intended for the existing task-based
interfaces.
Register both new kfuncs and mark them KF_SLEEPABLE because accessing a
remote address space can fault.
Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Matt Bobrowski <matt@bobrowski.net>
Link: https://patch.msgid.link/20260918131757.42802-4-tasos.papagiannnis@gmail.com
|
|
Proxy execution can move a blocked donor's scheduling context to the
lock owner's CPU even when the donor is migration-disabled. The donor
does not execute there, and its original execution CPU remains recorded
in wake_cpu.
set_task_cpu() warns unconditionally for migration-disabled tasks, so a
subsequent proxy migration or the wakeup path returning the donor home
triggers a false positive: moving a blocked scheduling context does not
violate the migration-disabled execution context.
For example, creating a mutex owner on CPU1 and a migration-disabled
waiter on CPU0 can trigger the following warning:
proxy_migrate_repro: donor blocking on CPU0 with migration disabled
proxy_migrate_repro: donor moved from CPU0 to CPU1
WARNING: kernel/sched/core.c:3389 at set_task_cpu+0x1d3/0x280
...
Call Trace:
try_to_wake_up+0x43f/0x780
__mutex_unlock_slowpath+0x330/0x540
owner_fn+0x9f/0xc0 [proxy_migrate_repro]
...
proxy_migrate_repro: donor woke on CPU0, task_cpu=0
proxy_migrate_repro: completed
Exclude blocked proxy donors from the warning. The proxy wakeup path
restores an executable placement before clearing the blocked state.
Fixes: b049b81bdff6 ("sched: Handle blocked-waiter migration (and return migration)")
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: John Stultz <jstultz@google.com>
Link: https://patch.msgid.link/20260915184101.2621252-1-arighi@nvidia.com
|
|
A typical module unload occurring event when there is an active perf
connection leads to freeing of the pmu pointer. The call log is something
like:
..
__pmu_detach_event
pmu_detach_event
pmu_detach_events
perf_pmu_unregister
..
__pmu_detach_event() sets event->pmu to null. When the perf connection
finally is closed, the following stack trace is observed:
Oops: general protection fault, kernel NULL pointer dereference
...
RIP: 0010:_free_event+0x3e/0x370
...
Call Trace:
...
perf_event_release_kernel+0x260/0x2d0
perf_release+0x12/0x20
A call to mediated_pmu_unaccount_event() inside _free_event() is the root
cause of this crash. Adding a check inside is_include_guest_event() ensures
we don't accidentally access a null pmu ptr. In addition to this, we will
now call mediated_pmu_unaccount_event() before clearing the pmu ptr so that
nr_include_guest_events counts are maintained correctly.
Fixes: eff95e170275 ("perf: Add APIs to create/release mediated guest vPMUs")
Assisted-by: Claude:Claude-Sonnet-5
Signed-off-by: Vinay Belgaumkar <vinay.belgaumkar@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://patch.msgid.link/20260904181625.1394082-1-vinay.belgaumkar@intel.com
|
|
A nested bpf_for_each_map_elem() callback can unlock a different element
of the same map:
static long inner(void *map, int *key, struct value *v,
struct value **outer_value)
{
bpf_spin_lock(&v->lock);
bpf_spin_unlock(&(*outer_value)->lock);
return 0;
}
static long outer(void *map, int *key, struct value *v, void *ctx)
{
bpf_for_each_map_elem(map, inner, &v, 0);
return 0;
}
Both callback values currently have ID zero and the same map_ptr.
process_spin_lock() compares those two fields, so it accepts the unlock
even though the two callbacks can receive different map elements.
Assign a fresh ID to every callback map value in the for-each,
timer/workqueue, and task-work constructors. Copies of one callback
argument retain its ID, so locking and unlocking through that argument
continues to work. Distinct callbacks also get distinct IDs for
single-element arrays, including inner arrays sharing inner_map_meta.
Preserve map_uid for every inner-map lookup and compare it through
check_ids() during state pruning. This preserves relationships between
maps, keys, and values while allowing equivalent states with different
lookup IDs to match. It avoids field-specific rules for when an inner map
needs an identity.
Move map_uid out of the metadata union and next to the other IDs, so
register comparisons can use the existing memcmp() ranges and remap the
IDs separately. Clear it when resetting a register or converting a map
lookup result to a socket pointer. Shrink frameno to u8, which is enough
for MAX_CALL_FRAMES, to make room without growing bpf_reg_state.
Fixes: d0d78c1df9b1 ("bpf: Allow locking bpf_spin_lock global variables")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260917233222.2542500-9-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
check_subprogs() verifies that each subprogram ends in an exit or an
unconditional jump before in-kernel CO-RE relocations are applied. An
unresolved relocation can then replace that terminal instruction with an
invalid helper call. The resulting fall-through into another subprogram
breaks the CFG invariant used by postorder and stack liveness analysis,
which can write past their per-subprogram arrays.
Apply CO-RE relocations immediately after preparing the program BTF, before
subprogram discovery and validation. Keep func_info and line_info validation
after subprogram discovery because those records depend on the complete
subprogram layout.
Reject an ldimm64 first slot at the end of the instruction stream before
CO-RE can inspect its missing second slot. check_subprogs() previously
rejected this form before relocation processing because it is not a valid
subprogram terminator. Moving CO-RE ahead of check_subprogs() removes that
implicit protection, so perform an explicit check before applying
relocations.
Include core_relo_cnt when deciding whether to prepare program BTF. A load
that supplied only CO-RE relocation metadata previously skipped both BTF
setup and relocation processing.
Fixes: fbd94c7afcf9 ("bpf: Pass a set of bpf_core_relo-s to prog_load command.")
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260917233222.2542500-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
regsafe() maps packet pointer IDs between states and checks that each
current register range is a subset of the corresponding explored
register range. It does not, however, preserve the displacement between
registers that share a packet pointer ID.
This is unsound because packet range is shared by ID. A bounds check on
one class member updates every member, and a later access can consume the
range through another member. Commit 022ac0750883 ("bpf: use reg->var_off
instead of reg->off for pointers") folded the fixed pointer offset into
r64 and removed the old off equality check, so two individually narrower
registers can prune even when their displacement has changed. The
explored path can then license an out-of-bounds packet access on the
pruned path.
Require matching range bases for packet pointers with an ID. Together
with the existing ID mapping, this preserves the displacement between
members of each packet-pointer class without adding per-ID state.
Packet pointers without an ID remain unaffected.
Fixes: 022ac0750883 ("bpf: use reg->var_off instead of reg->off for pointers")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260917233222.2542500-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
After do_check() returns, the verifier runs several instruction rewrite
passes. Some of them patch or remove one instruction at a time. Each
operation moves the remaining instruction and auxiliary-data arrays and
adjusts all branch offsets, making the overall work quadratic in the
program length.
A privileged loader can submit 131072 unconditional jumps by zero followed
by a valid return. Verification finishes quickly, but bpf_opt_remove_nops()
then spends a long time removing each jump separately. Since this
post-verification work neither checks for signals nor reschedules, a pending
SIGKILL cannot terminate the task until the rewrite finishes.
Make bpf_patch_insn_data() and verifier_remove_insns() common cancellation
and rescheduling points. These helpers run from BPF_PROG_LOAD process
context, and bpf_patch_insn_data() can already sleep while reallocating
auxiliary data.
Report interrupted constant blinding as -EINTR and propagate it through
both JIT paths, including kernels that permit interpreter fallback.
Other blinding failures retain the existing fallback behavior.
This does not reduce the quadratic cost of the rewrite passes, but it makes
the work preemptible and allows a killed loader to be torn down promptly.
Fixes: 52875a04f4b2 ("bpf: verifier: remove dead code")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260917233222.2542500-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
fprobe_fgraph_entry() reserves shadow stack space for every fprobe with
an exit handler, but only fills it for those whose entry handler returns
0. fgraph_reserve_data() does not clear the area, so fprobe_return()
parses the unused tail as headers left over from an earlier call, and an
exit handler can run twice or despite its entry handler asking to skip
it.
Write a zero word after the last entry to terminate the walk. A zeroed
slot does not decode to a NULL fprobe on the arches that encode the
header into one unsigned long, since arch_decode_fprobe_header_fp() ORs
in FPROBE_HEADER_MSB_PATTERN, so make read_fprobe_header() return NULL
for a zeroed slot.
Link: https://lore.kernel.org/all/20260917212407.384468-1-devnexen@gmail.com/
Fixes: e0a384434ae1 ("tracing: fprobe: do not zero out unused fgraph_data")
Cc: stable@vger.kernel.org
Suggested-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
