| Age | Commit message (Collapse) | Author |
|---|
|
In non-preemptible kernels, a Tiny SRCU grace period implies an RCU
grace period because any context switch suffices. But in preemptible
kernels, it is possible for a Tiny SRCU grace period to elapse without
a corresponding RCU grace period.
Which was OK until RCU Tasks Trace was re-implemented in terms
of SRCU-fast, which in Tiny SRCU is implemented as SRCU, which is
already fast. And RCU Tasks Trace grace periods are required to imply
RCU grace periods. This commit therefore adds a synchronize_rcu(),
but only in preemptible kernels.
Because preemptible Tiny SRCU is not in mainline, this added call to
synchronize_rcu() will not slow anything down: The comparison would
instead be with TREE SRCU. But if this added call ever becomes a problem,
the Tiny SRCU srcu_struct structure could track whether or not this is for
SRCU-fast, and to add the synchronize_rcu() only in the SRCU-fast case.
However, at the moment, this is seen as unnecessary complexity.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
call_rcu() and call_srcu() are now safe to invoke from NMI, but
rcutorture never does, leaving the deferral path untested.
Add an ->nmi_capable flag to rcu_torture_ops. For flavors that set it,
arm a per-CPU hardware perf counter whose overflow handler submits a
callback via ->call(). The handler acts only when in_nmi(), so only a
genuine NMI exercises the deferral path. One preallocated callback per
CPU is kept in flight, guarded by an atomic, to avoid allocating in NMI.
The counter uses a fixed sample period rather than a frequency: a
frequency-based event sets TICK_DEP_BIT_PERF_EVENTS and would pin the tick
for the whole run on NO_HZ_FULL kernels.
Report the count issued from NMI ("nmi-calls:") and the count invoked
("nmi-cbs:"). rcu_torture_cleanup() disables the counters and then calls
cb_barrier(), which drains every deferred callback, so the two counts must
then match; a mismatch fails the test. This relies on
rcu_barrier()/srcu_barrier() flushing deferred callbacks, as added earlier
in the series.
Set ->nmi_capable on the NMI-safe flavors: rcu, srcu, srcud, and
tasks-tracing (call_srcu() under the hood). Tasks and Tasks Rude are left
alone, as call_rcu_tasks_generic() is not yet NMI-safe.
Enabled by default; the nmi_calls parameter disables it, which helps rule
NMI handling in or out when triaging a failure. Requires
CONFIG_PERF_EVENTS and a hardware PMU: without one nothing is issued from
NMI and the end-of-test check compares zero against zero, so a pass does
not by itself mean the path ran. That is the case under kvm.sh, which
boots qemu with -cpu kvm64 and no vPMU; rcu_torture_nmi_cleanup() says so
on the console.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Vanilla RCU readers, when either non-preemptible or preemptible with
priority-boosting enabled, can handle heavy overload conditions.
Other RCU implementations are more fragile. This commit adds an
rcu_torture_ops structure field named ->rdrs_handle_load that is set
for non-fragile configurations of vanilla RCU, but cleared otherwise.
It also refrains from running fragile readers concurrently with overload
testing, thus avoiding false-positive overload-testing failures.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Give Tiny call_srcu() the same treatment as Tree SRCU. When interrupts
are disabled and the scheduler is up, stage the callback on the
srcu_struct's lockless list for an irq_work to re-issue later. Tiny SRCU
is uniprocessor, so there is no CPU-offline drain. A draining flag drops
a deferring call_srcu() that re-enters mid-drain (unless from an NMI), as
in Tree SRCU; such a drop can strand state the caller tied to the callback,
not just the callback itself.
Interrupts stay off for the whole batch. TINY_SRCU implies !SMP, where
arch_irq_work_has_interrupt() is false, so the drain always waits for the
tick and a batch is whatever one tick's worth of interrupts-disabled
call_srcu()s staged. Unlike the other three flavors srcu_do_enqueue() here
has no debug_rcu_head_queue(), so nothing reports a double call_srcu();
termination of the drain rests on srcu_do_enqueue() clearing ->next, and
the callback list self-links at the tail exactly as a double call_srcu()
made it before.
srcu_barrier() (now out of line) and cleanup_srcu_struct() drain the
deferred list first, so a deferred callback is re-issued onto the callback
list and invoked by the grace-period work that cleanup_srcu_struct()
flushes, rather than stranded on a soon-to-be-freed srcu_struct.
cleanup_srcu_struct() also syncs ->defer_iw, since that irq_work is
embedded in the srcu_struct the caller is about to free.
The draining flag is global rather than per-srcu_struct: a re-entrant
call_srcu(B) inside a drain of A raises B's own ->defer_iw, whose drain can
stage back onto A, so a per-srcu_struct flag would not break the chain.
The cost is that a drain of A also drops a non-NMI call_srcu() to any
other srcu_struct for its duration.
Gated by CONFIG_RCU_DEFER like Tree SRCU, though the srcu_struct members
are unconditional.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
call_srcu() has the same constraint as call_rcu():
srcu_gp_start_if_needed() enqueues under raw_spin_lock_irqsave() and may
walk the srcu_node tree, as do callback invocation and grace-period work,
so a call_srcu() with interrupts already disabled can race an operation in
flight on this CPU. call_rcu_tasks_trace() is call_srcu() under the hood,
so a sleepable BPF program freeing an object can reach this.
Defer as call_rcu() does: stage the callback on the srcu_data's
->defer_cbs, chain that srcu_data onto a per-CPU list, and raise a per-CPU
irq_work that re-issues it straight to the enqueue helper, never back
through __call_srcu(). The irqs-enabled path is unchanged; as for
call_rcu() the gate is bare irqs_disabled(), so callers that merely hold
interrupts off are deferred too and pay one irq_work hop, including
call_rcu_tasks_trace() from the BPF memalloc irq_work.
The irq_work is per-CPU rather than per-srcu_struct and statically
initialized, so deferral never runs check_init_srcu_struct(); it is
IRQ_WORK_INIT_HARD as for call_rcu(). srcu_barrier() flushes it first, and
rcutree_migrate_callbacks() calls srcu_offline_drain() for an outgoing CPU.
cleanup_srcu_struct() drains before its "just leak it" early returns, and
srcu_module_going() before freeing any ->sda, since a staged srcu_data left
chained on a per-CPU list would dangle. Staging is two steps, the callback
onto
->defer_cbs and then the srcu_data onto the per-CPU list, so a flusher can
find the per-CPU list empty while a callback whose call_srcu() has not
returned sits on ->defer_cbs; the staging CPU's own irq_work takes that
one.
The per-CPU srcu_defer ->lock, not any srcu_data's, is held with interrupts
off across the whole nested drain: the chain of srcu_datas staged on that
CPU and, for each, its callbacks, with srcu_do_enqueue() taking that
srcu_data's ->lock and possibly starting a grace period for every one.
That is what serializes the drainers. The bound is as for call_rcu(): what
one interrupts-disabled region could stage, normally a single callback, or
whatever accumulates before the tick where arch_irq_work_has_interrupt() is
false. Both lists are drained newest-first; nothing depends on call_srcu()
ordering.
The drain clears ->next before re-issuing, which bounds a node self-linked
by a double call_srcu(); a longer cycle is not bounded, and a double
call_srcu() stays undefined, as for call_rcu().
A callback deferred past the CPUHP_AP_SMPCFD_DYING irq_work flush leaves
that CPU's srcu_defer ->iw claimed with its self-IPI lost, as for
call_rcu(). srcu_offline_drain() still re-issues the callback, but the
irq_work cannot be un-queued, and here the claim is shared by every
srcu_struct on the CPU.
As in call_rcu(), the re-issue runs with interrupts disabled and can be
re-entered by instrumentation, so a per-CPU flag, set only while that CPU
is inside its own irq_work drain, drops a deferring call_srcu() seen
mid-drain unless it comes from an NMI. Such a drop can strand state the
caller associated with the callback, not just the callback itself.
Staging records only the callback, so a deferred expedited call_srcu()
completes as a normal grace period. Only srcu_expedite_current() can hit
that, and only when invoked with interrupts already disabled.
Gated by CONFIG_RCU_DEFER, though the srcu_data members and the per-CPU
srcu_defer are unconditional. Under CONFIG_PROVE_RCU, warn if the direct
path is reached from an NMI.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Give Tiny call_rcu() the same treatment as Tree RCU. When interrupts are
disabled and the scheduler is up, stage the callback on a lockless list
that an irq_work re-issues later. One global list and irq_work suffice
since Tiny RCU is uniprocessor, and there is no CPU-offline drain.
The re-issue runs with interrupts disabled and can be re-entered by
instrumentation, so a draining flag drops a deferring call_rcu() seen
mid-drain (unless from an NMI), as in Tree RCU. Gated by CONFIG_RCU_DEFER,
though the deferral state is unconditional.
Interrupts stay off for the whole batch, but the re-issue is a tail append
with no locks. TINY_RCU implies !SMP, where arch_irq_work_has_interrupt()
is false, so the drain always waits for the tick and a batch is whatever
one tick's worth of interrupts-disabled call_rcu()s staged. As in Tree
RCU the drain clears ->next before re-issuing, which bounds a node
self-linked by a double call_rcu(): rcu_do_enqueue()'s duplicate path
returns without clearing it. A longer cycle is not bounded; a double
call_rcu() stays undefined.
The idle-task reschedule moves out of the enqueue helper so that a drain
does it once for the batch rather than once per callback, which would
otherwise take the runqueue lock N times with interrupts disabled.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
RCU's per-CPU callback list is only touched with interrupts disabled: the
enqueue runs under local_irq_save() (and the nocb locks when offloaded),
as do callback invocation and grace-period work. A call_rcu() that
arrives with interrupts already disabled, whether from an NMI or from
instrumentation that re-enters RCU, can interrupt one of those and corrupt
the list or deadlock.
Defer instead: stage the callback on a per-CPU llist and raise an irq_work
that re-issues it once interrupts are on, straight to the enqueue so it
cannot defer again. The gate is bare irqs_disabled(), so callers that
merely hold interrupts off are deferred too and pay one irq_work hop.
Skip it while the scheduler is down (RCU_SCHEDULER_INACTIVE): irq_work is
not usable that early, rcu_init() already calls call_rcu(), and the per-CPU
deferral state is not initialised until rcu_init_one() runs later in it.
rcu_barrier() drains every CPU's ->defer_head before it scans the lists,
and rcutree_migrate_callbacks() drains an outgoing CPU's. A drain
re-issues onto the draining CPU, so a barrier moves other CPUs' staged
callbacks onto
its own ->cblist; call_rcu() promises no CPU affinity for invocation.
->defer_lock is held across llist_del_all() and the whole re-issue so the
drainers
serialize: one that finds the list empty can conclude that everything
staged before it is already on a callback list. Interrupts stay off for
the batch. Where the arch has an irq_work self-IPI that is what one
interrupts-disabled region could stage, normally a single callback; where
arch_irq_work_has_interrupt() is false the drain waits for the tick, so
several regions can accumulate first.
The drain clears ->next before re-issuing. A double call_rcu() on a head
that is already debug-object-active self-links the staged node, and
rcu_do_enqueue()'s duplicate path returns without clearing it, so the
drain would spin. A re-add behind other staged callbacks makes a longer
cycle, which that does not bound; a double call_rcu() stays undefined.
llist_del_all() yields newest-first, so a batch is re-issued in reverse
call order; nothing depends on call_rcu() ordering. The re-issue drops
the lazy hint, since staging records only ->func, so a deferred callback
loses its batching on CONFIG_RCU_LAZY. kasan_record_aux_stack() moves to
__call_rcu_common() so a use-after-free report names the caller rather
than the irq_work.
The re-issue runs with interrupts disabled, so instrumentation on the
enqueue path can re-enter call_rcu(), stage another callback and re-raise
the irq_work, livelocking the drain. A per-CPU flag guards it: a deferral
that arrives while this CPU is draining, and is not from an NMI, is
dropped. The WARN_ONCE() is under CONFIG_PROVE_RCU, so a production kernel
drops it silently. That leaks the callback and can strand state
the caller tied to it, since a one-shot flag only the callback clears
never resets, but the alternative is an unbounded loop.
A callback deferred past the CPUHP_AP_SMPCFD_DYING irq_work flush leaves
->defer_work claimed with its self-IPI lost. rcutree_migrate_callbacks()
still re-issues the callback, but the first deferral after that CPU comes
back raises no IPI and waits for the next irq_work there, or for
rcu_barrier(). Unqueueing an irq_work is not something the API offers.
The irq_work is IRQ_WORK_INIT_HARD so the re-issue stays prompt on
PREEMPT_RT, where a non-HARD irq_work runs in a kthread that can be
delayed under load. A hidden CONFIG_RCU_DEFER gates the deferral code and
its IRQ_WORK dependency, though the rcu_data members are unconditional;
without it call_rcu() enqueues directly as before. Under
CONFIG_PROVE_RCU, warn if the direct path is reached from an NMI.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The WARN_ON() added by commit 78a38cbf6f20 ("srcu: Queue sdp->work
when the delay timer is successfully deleted") uses rcu_segcblist_n_cbs()
to detect callbacks that srcu_barrier() failed to wait for. However, the
->len counter is decremented only at the end of srcu_invoke_callbacks(),
after the invoking loop has finished. Since srcu_barrier() can return
right after the barrier callback is invoked, cleanup_srcu_struct() can see
a non-zero n_cbs even though the cblist is already physically empty,
falsely triggering the WARN_ON() together with a still-pending delay_work
timer.
This can be triggered as follows, as seen in the syzbot report against
kvm_destroy_vm() -> cleanup_srcu_struct(&kvm->srcu):
1. call_srcu(&kvm->srcu, &bus->rcu, __free_bus) starts SRCU grace
period GP1.
2. GP1 ends: a delay timer is armed and sdp->work is queued, but
sdp->work has not run yet.
3. Another call_srcu(&kvm->srcu, &bus->rcu, __free_bus) call invokes
srcu_segcblist_advance(), which moves the GP1 callback to
RCU_DONE_TAIL, and starts SRCU grace period GP2.
4. srcu_barrier() is called. It queues its barrier callback after the
GP2 callback and waits for srcu_invoke_callbacks() to invoke it.
5. GP2 ends: another delay timer is armed, and the sdp->work queued in
step 2 begins to run. Its srcu_invoke_callbacks() call invokes
srcu_segcblist_advance() again, moving the GP2 and barrier callbacks
to RCU_DONE_TAIL as well, and then invokes all of them. However,
rcu_segcblist_add_len(), which updates srcu_cblist's ->len, has not
run yet at this point.
6. srcu_barrier() returns once its callback has been invoked, and
cleanup_srcu_struct() starts running. It finds the delay timer
armed in step 5 still pending and srcu_cblist's ->len still
non-zero (because step 5 has not reached rcu_segcblist_add_len()
yet), and WARN_ON() fires even though every callback has actually
been invoked.
Use rcu_segcblist_empty(), which checks the actual head of the cblist,
instead of rcu_segcblist_n_cbs(), which checks the racy ->len counter.
Callbacks that have genuinely not been invoked yet still leave the list
non-empty, so the WARN_ON() still catches callers that skip srcu_barrier()
or queue callbacks after it.
Link: https://lore.kernel.org/rcu/e6350377085ddd85d6ef00d8e9a67bd50c762d3c@linux.dev/T/#t
Reported-by: syzbot+d4faf7db59e11f6fd1ab@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=d4faf7db59e11f6fd1ab
Fixes: 78a38cbf6f20 ("srcu: Queue sdp->work when the delay timer is successfully deleted")
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Sunho Park <shpark061104@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Tested-by: Sean Christopherson <seanjc@google.com>
|
|
cgroup_e_css() and cgroup_get_e_css() find the effective css by walking
up the hierarchy until they reach a cgroup that has the subsystem
enabled. Per-I/O users such as nvmet need this lookup to be O(1).
Track the effective css in each cgroup and make cgroup_e_css() and
cgroup_get_e_css() use it. When a css is brought online or offline,
update the effective css of its cgroup and of all descendants.
Signed-off-by: Peng Yu <yupeng0921@gmail.com>
Assisted-by: Claude:claude-fable-5 [Claude Code]
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
do_call_rcu_ttrace() returns early when call_rcu_ttrace_in_progress is set
and leaves the objects in free_by_rcu_ttrace. __free_rcu() frees
waiting_for_gp_ttrace only and clears the flag. Hence the objects that
free_bulk() or __free_by_rcu() added while RCU tasks trace GP was in flight
stay in free_by_rcu_ttrace until free_bulk() or alloc_bulk() is called for
the same bpf_mem_cache again, which may never happen. The number of such
objects is not bounded.
Turn call_rcu_ttrace_in_progress into three states:
0 - idle
1 - __free_rcu() is queued
2 - __free_rcu() is queued and free_by_rcu_ttrace got more objects since
do_call_rcu_ttrace() sets 2. __free_rcu() does cmpxchg(1 -> 0) and starts
the next GP when it fails. It cannot clear the flag first and check
free_by_rcu_ttrace later, since bpf_mem_alloc_destroy() frees bpf_mem_cache
without waiting for RCU callbacks when the flag is zero.
Now __free_rcu() queues itself, so the one that didn't see 'draining' may
do call_rcu_tasks_trace() after rcu_barrier_tasks_trace() in
free_mem_alloc(). Queue it under rcu_read_lock() and do synchronize_rcu()
before the barriers. Calling rcu_barrier_tasks_trace() twice works too, but
creating and destroying hash maps in a loop on many cpus slows down to one
free_mem_alloc() per GP and kworkers pile up.
Fixes: 8d5a8011b35d ("bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU.")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20260930095920.601738-3-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Move the part of do_call_rcu_ttrace() that runs after
call_rcu_ttrace_in_progress is set into __do_call_rcu_ttrace().
The next patch will call it from __free_rcu().
No functional change.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20260930095920.601738-2-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Since commit 022ac0750883 ("bpf: use reg->var_off instead of reg->off
for pointers"), find_good_pkt_pointers() sets the range of all packet
pointers sharing an id from the umax of the compared pointer, and
check_packet_access() requires umax + off + size <= range. That assumes
the umax of two such pointers differ by exactly their constant distance.
reg_bounds_sync() breaks it when var_off tightens one umax and not the
other:
r4 &= 0x38
if r4 > 50 goto exit ; umax 50, var_off (0x0; 0x38)
r5 = pkt + r4 ; umax 50
r6 = r5
r6 += 8 ; umax 56, not 58
Comparing r6 with pkt_end sets the range to 56, and the valid 8-byte
load at r5 is rejected (50 + 8 > 56). Comparing r5 sets it to 50, and
the out-of-bounds 1-byte load at r6 - 7, i.e. r5 + 1, is accepted
(56 - 7 + 1 <= 50).
Don't call reg_bounds_sync() on a packet pointer that keeps its id (a
constant was added or subtracted) or its range (an unknown non-negative
value was subtracted), so that var_off cannot tighten its umax. Only
update the 32-bit bounds from var_off: reg_bounds_sanity_check() wants
them constant when the lower half of var_off is, e.g. for pkt + 8.
This relies on nothing else changing the 64-bit bounds of a packet
pointer, which holds today.
var_off of such a pointer is no longer narrowed by its bounds. Adjust
three verifier_align expectations; the low bits, which the alignment
checks use, don't change. veristat on the selftests shows no verdict
changes and +0.8% insns in test_cls_redirect_subprogs.
Fixes: 022ac0750883 ("bpf: use reg->var_off instead of reg->off for pointers")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261001145255.855630-1-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Two things in __cgroup_bpf_query() work only because CGROUP_TCP_SOCK_OPS is
the only struct_ops attach type.
It calls cgroup_bpf_enabled(atype) with an atype that
find_atype_by_struct_ops_id() works out at runtime. That macro is an asm
goto and needs a constant. Today the compiler can see there is only one
value; add a second type and the build breaks with "impossible constraint
in 'asm'". The check is not needed anyway. When the key is off nothing is
attached, so progs[atype] and effective[atype] are empty and the count loop
leaves total_cnt at 0. The other attach types already use that loop with
no such check. Drop the check and the skip_count label.
And find_atype_by_struct_ops_id() matches on type_id alone. An attach type
whose subsystem is not built keeps type_id 0, so a query for type 0 finds
it and returns success with nothing instead of -ENOENT. Skip such slots.
Fixes: 369d9dcd8fb8 ("bpf: Add infrastructure to support attaching struct_ops to cgroups")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Yafang Shao <laoar.shao@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20261001141901.3225830-1-shakeel.butt@linux.dev
|
|
linux/module.h appears in roughly 15k #include directives across the
kernel. This makes it a "hot" header, so it should avoid pulling in
unnecessary definitions.
The header currently includes linux/error-injection.h to obtain the
definition of `struct error_injection_entry`. However, this is unnecessary
because the type is only referenced in the file as a pointer, for which an
incomplete type is sufficient.
Remove the linux/error-injection.h include from linux/module.h and add it
to kernel/module/main.c instead, where
`sizeof(struct error_injection_entry)` is actually needed.
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl
Pull sysctl fixes from Joel Granados:
"Fix sysctl jiffies conversions errors introduced in 2dc164a48e6f
("sysctl: Create converter functions with two new macros")
- Ensure that mult_hz does *not* wrap
- Ensure we pass just the magnitude for the negative branch in
proc_int_k2u_conv_kop"
* tag 'sysctl-7.03-fixes-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl:
time/jiffies: Saturate in mult_hz() instead of wrapping
sysctl: Negate before converting in the int read path
|
|
Active references on the namespaces of an nsproxy are taken when the
nsproxy is installed into a task in switch_task_namespaces() and
copy_namespaces() and dropped again by put_nsproxy() through
deactivate_nsproxy().
But ksys_unshare() calls put_nsproxy() on an nsproxy that was never
installed when set_cred_ucounts() fails. The new namespaces of that
nsproxy go from zero to minus one and the namespaces shared with the
caller lose a reference that belongs to the nsproxy the caller keeps
using:
WARNING: kernel/nscommon.c:171 at __ns_ref_active_put+0x1cd/0x230
nsproxy_ns_active_put
deactivate_nsproxy
ksys_unshare
Afterwards the namespaces the caller lives in aren't listed by listns()
anymore. set_cred_ucounts() only fails when alloc_ucounts() can't
allocate and it only allocates when the real uid of the caller differs
from its effective uid.
Commit cefd55bd2159 ("nsproxy: fix free_nsproxy() and simplify
create_new_namespaces()") separated the two cases on purpose.
nsproxy_free() frees an nsproxy that was prepared but never installed
and that's what a failed setns() uses in put_nsset(). Export it and use
it for a failed unshare() as well.
Fixes: a98621a0f187 ("unshare: fix nsproxy leak in ksys_unshare() on set_cred_ucounts() failure")
Cc: stable@vger.kernel.org # v7.1+
Link: https://patch.msgid.link/20260930-work-mount-fixes-3-v1-16-be34c83956ae@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Perf events with exclude_kernel=1 can be opened without kernel perf
access. However, exclude_kernel does not suppress text-poke sideband
records. Every PERF_RECORD_TEXT_POKE is marked PERF_RECORD_MISC_KERNEL
and contains a raw kernel instruction address.
An unprivileged task can therefore open and mmap a task-local software
event with text_poke=1. Both opening a count-only tracepoint event and
configuring UDP GRO for ESP-in-UDP cause updates to inline static calls;
the observer receives the relocated addresses of the modified instructions.
For a known kernel image, any such address reveals the runtime kernel
text base despite KASLR.
Call perf_allow_kernel() whenever attr.text_poke is set, regardless of
exclude_kernel. Events that neither monitor kernel execution nor request
text-poke records retain their existing permissions.
Fixes: e17d43b93e54 ("perf: Add perf text poke event")
Assisted-by: LLM
Signed-off-by: Zhengchuan Liang <zcliangcn@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/99131354c41e23188f778b92f90363775b482395.1790573390.git.zcliangcn@gmail.com
|
|
perf_iterate_sb() invokes its callback for each matching perf_event on
the CPU and task context, passing a shared caller-allocated event
structure.
perf_event_header__init_id() mutated header->size in place by adding
event->id_header_size, requiring sideband output callbacks to save and
restore header fields across iterations. Three sideband callbacks failed
to save and restore header.size around perf_event_header__init_id():
- perf_event_ksymbol_output()
- perf_event_bpf_output()
- perf_event_text_poke_output()
When multiple events with attr.ksymbol, attr.bpf_event, or
attr.text_poke and sample_id_all are active on the same CPU, each
subsequent event receives a record whose header.size is inflated by all
preceding events' id_header_size values while only a single id_sample is
written, leaving uninitialized ring-buffer bytes at the end of the
record and causing userspace perf to fail with -EFAULT ("Bad address")
when parsing the sample_id trailer.
Similarly, perf_event_mmap_output() set PERF_RECORD_MISC_MMAP_BUILD_ID
in mmap_event->event_id.header.misc when event->attr.build_id was
enabled, but only saved and restored header.size and header.type. If an
event with attr.build_id was followed by an event with attr.mmap2 and
!attr.build_id, the second event received PERF_RECORD_MISC_MMAP_BUILD_ID
in header.misc while its payload contained maj/min/ino/ino_generation
instead of a build ID.
Rather than splitting header initialization between callers and output
callbacks and saving/restoring mutated header fields, replace
perf_event_header__init_id() with perf_event_header__init(), which
initializes header->type, header->misc, and header->size alongside the
sample_id fields on each invocation.
Fixes: 76193a94522f ("perf, bpf: Introduce PERF_RECORD_KSYMBOL")
Fixes: 6ee52e2a3fe4 ("perf, bpf: Introduce PERF_RECORD_BPF_EVENT")
Fixes: e17d43b93e54 ("perf: Add perf text poke event")
Fixes: 88a16a130933 ("perf: Add build id data in mmap2 event")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260929222332.973435-1-irogers@google.com
Cc: stable@vger.kernel.org
|
|
A race condition exists between perf_event_exit_task() and
perf_pending_task() during begin_new_exec().
During begin_new_exec(), perf_event_exit_task() may be called, and the
PF_EXITING flag is not set on task. So perf_sigtrap() continues to execute
and triggers WARN_ON_ONCE(event->ctx->task != current).
Since both task exit and exec paths can call perf_event_exit_task() which
sets ctx->task to TASK_TOMBSTONE, fix this by explicitly checking if
event->ctx->task equals TASK_TOMBSTONE and dropping the redundant
PF_EXITING check.
Fixes: 97ba62b27867 ("perf: Add support for SIGTRAP on perf events")
Signed-off-by: Luo Gengkun <luogengkun2@huawei.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260920075026.990582-1-luogengkun2@huawei.com
|
|
Once a smpboot thread gets work assigned it receives a wake up. If the
thread had no chance to run and receives a parking requesst during a
CPU-hotplug event then it is parked without running the callback first.
As a result the enqueued work remains stuck until the CPU gets back
online.
There is nothing wrong with invoking the thread function first and
parking in the following iteration. This would ensure that the callbacks
are processed before the shutdown.
Honor the park request once thread function does not need to run.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260911143815.997254-4-bigeasy@linutronix.de
|
|
PREEMPT_RT invokes IRQ_WORK_LAZY callbacks and callbacks which are not
explicitly marked IRQ_WORK_HARD_IRQ from thread context. If a CPU gets
shutdown, the irq_work is flushed during smpcfd_dying_cpu() which is
invoked on the target CPU with disabled interrupts.
The callbacks enqueued on lazy_list on PREEMPT_RT are not flushed because
they require thread context. The callbacks remain and get processed once
the CPU gets back online.
per-CPU irq_work items continue to work for enqueues on their local-CPU,
only the work on the "offline" CPU get stuck. "Single" irq_work are
worse because they remain "claimed" and can not be used again. One
"symptom" is the late printk() in CPU down path which enqueues the
irq_work for printing but it never gets scheduled. Further prints don't
print to the console because the irq_work is "pending" of the offline
CPU.
Another problem is the canceling/ flushing of the irq_work which is
stuck. Since the irq_work will not continue, the task waiting for its
completion will block waiting.
While looking through the users of irq_work, it does not matter if the
callback is enqueued on another CPU. This makes it possible to invoke
the callbacks from a remote CPU by smpcfd_dead_cpu(). This is invoked
from the control CPU with enabled interrupts shortly after the
hotplugged CPU is shutdown so the llist can be accessed safely.
Add irq_work_run_cpu() which invokes the irq_work callbacks of the
specified dead CPU. Invoke it from smpcfd_dead_cpu().
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260911143815.997254-3-bigeasy@linutronix.de
|
|
hotplug_cfd() no longer exists and the flow changed a bit since the
rework of CPU hotplug as of commit 31487f8328f20 ("smp/cfd: Convert core
to hotplug state machine").
Update the comment to reflect reality as of today.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260911143815.997254-2-bigeasy@linutronix.de
|
|
Architectures can use SD_ASYM_PACKING to describe preferred CPU ordering
at the SMT scheduling domain. Some systems benefit from this policy, but
their firmware cannot describe the preference and inferring it from the
CPU model would embed a platform-specific policy in the kernel.
Add sched_smt_asym_packing=on boot option to force SD_ASYM_PACKING at
the SMT level. Omitting the option preserves the topology provided by
the architecture or firmware.
Apply the override centrally to domains with SD_SHARE_CPUCAPACITY so it
also covers architectures with custom SMT topology callbacks, including
powerpc.
When enabled, priority remains defined by arch_asym_cpu_priority(). The
weak default prefers lower-numbered logical CPUs, while architecture
overrides remain authoritative. Siblings with equal priorities remain
unordered. In particular, x86 with CONFIG_SCHED_MC_PRIO normally gives
both SMT siblings the same core priority, so enabling the option there
does not prioritize a sibling.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Breno Leitao <leitao@debian.org>
Link: https://patch.msgid.link/20260929165538.726616-3-arighi@nvidia.com
|
|
POWER7 uses SD_ASYM_PACKING at the shared-capacity SMT level to order
hardware threads, and NVIDIA Olympus benefits from the same policy. Idle
CPU selection does not consult that order, so a task can wake on an
arbitrary sibling and remain there until load balancing corrects the
placement. On these systems, that initial choice can prevent the core
from entering its preferred lower-thread resource mode and cause a large
and persistent performance loss.
When idle selection finds an available CPU in an SMT core, choose the
highest-priority available sibling. On SMT2 Olympus this only changes
selection on fully idle cores. A partially idle core has only one
available CPU. On wider SMT systems such as POWER7, it also fills
available siblings in priority order while the core is partially busy.
Apply the preference to idle-core and idle-CPU scans,
asymmetric-capacity scans, target, previous, recently-used CPU fast
paths and the slow path. Inspect the lowest scheduling domain directly,
but require both CPUs to share its span because isolcpus can split
hardware siblings across scheduling domains.
Keep physical-core capacity selection independent from SMT sibling
ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
maximum capacities, then SD_ASYM_PACKING selects the preferred available
sibling inside the chosen core, whose siblings continue to share equal
capacity.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Tested-by: Breno Leitao <leitao@debian.org>
Link: https://patch.msgid.link/20260929165538.726616-2-arighi@nvidia.com
|
|
set_tsk_need_resched() tests TIF_NEED_RESCHED, calls
__trace_set_need_resched() if the flag is clear, then sets it via
set_tsk_thread_flag(). A BPF raw_tp program attached to
sched_set_need_resched executes synchronously inside __bpf_trace_run().
On return, __bpf_trace_run() drops the RCU lock with
rcu_read_unlock_migrate(), which on the preempt-or-BH-disabled path
calls set_need_resched_current() -> set_tsk_need_resched() again.
set_tsk_thread_flag() follows the tracepoint call, so every re-entrant
frame sees TIF_NEED_RESCHED clear and calls __trace_set_need_resched()
again:
BUG: TASK stack guard page was hit at ffffc9001224ff98
Oops: stack guard page: 0000 [#1] SMP KASAN PTI
RIP: 0010:__bpf_trace_sched_set_need_resched_tp+0x1c/0x190
Call Trace:
trace_sched_set_need_resched_tp+0x110/0x130
set_tsk_need_resched include/linux/sched.h:2076
set_need_resched_current include/linux/sched.h:2094
rcu_read_unlock_special+0x43a/0x440
__rcu_read_unlock+0x9e/0x120
rcu_read_unlock_migrate+0xa9/0x240
__bpf_trace_run+0x131/0x180
bpf_trace_run3+0x333/0x430
__bpf_trace_sched_set_need_resched_tp+0x13a/0x190
trace_sched_set_need_resched_tp+0x110/0x130
set_tsk_need_resched include/linux/sched.h:2076
...
__resched_curr() has the same ordering, firing the tracepoint before
setting the flag via set_ti_thread_flag() or set_nr_and_not_polling().
Fix it for consistency.
Replace the separate test_tsk_thread_flag() + set_tsk_thread_flag() pair
in set_tsk_need_resched() with test_and_set_tsk_thread_flag(). In
__resched_curr(), move the tracepoint call after the flag is set in
each path.
Fixes: adcc3bfa8806 ("sched: Adapt sched tracepoints for RV task model")
Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Acked-by: Gabriele Monaco <gmonaco@redhat.com>
Link: https://patch.msgid.link/20260630084750.2792851-1-rhkrqnwk98@gmail.com
|
|
Nathan reported build failure with clang23 when CONFIG_PREFERRED_CPU=y.
kernel/sched/core.c:11298:3: error: releasing raw_spinlock 'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis]
11298 | rq_unlock(rq, &rf);
| ^
kernel/sched/core.c:11303:1: error: raw_spinlock 'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis]
context_unsafe_alias(rq) was placed after rq_lock(), by which point it had
already associated the lock with the original rq alias.
Move the alias barrier before rq_lock() so that the changing rq alias is
accounted for before acquiring the lock.
Fixes: 74699f56ebcf ("sched/core: Push current task from non preferred CPU")
Closes: https://lore.kernel.org/all/20260929121838.GA1814129@ax162/
Reported-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Nathan Chancellor <nathan@kernel.org> # build
Link: https://patch.msgid.link/20260929164712.1054883-1-sshegde@linux.ibm.com
|
|
Commit d9e4c61a12dd ("rv: Use generic rv_this for the rv_monitor
variable in LTL") renamed the rv_monitor variable to rv_this. But
rtapp/wakeup monitor was late to the party and missed that.
So for this monitor, rv_this is an empty struct. The reactor fails to
work, because rv_react() checks monitor->react and sees NULL.
Correct to "rv_this" and make the reactor work again.
Signed-off-by: Nam Cao <namcao@linutronix.de>
Fixes: d9e4c61a12dd ("rv: Use generic rv_this for the rv_monitor variable in LTL")
Link: https://lore.kernel.org/r/20261001044218.3508990-1-namcao@linutronix.de
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
|
|
Add KUnit tests covering the reactor register/unregister lifecycle
(including duplicate and name-length rejection) and rv_react() dispatch
(a no-op without a callback, exactly one invocation with one). The
mdelay() callback keeps the CPU busy so a timer interrupt lands inside
rv_react()'s lockdep context, exercising the LD_WAIT_SPIN wait type
from the previous patch; a spurious lockdep splat there would show up
in the test output.
The dispatch tests rely on reacting_on being enabled, since rv_react()
returns early when it is off.
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Wen Yang <wen.yang@linux.dev>
Link: https://lore.kernel.org/r/7d213b110815da45595ad8cd7bcce5641fac3404.1789412799.git.wen.yang@linux.dev
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
|
|
rv_react() is exported to modules, but the reactor registration helpers
are not. Export them with EXPORT_SYMBOL_GPL() so reactor modules and
the tristate KUnit test module can register and unregister reactors
without hitting undefined symbol errors at link time(modpost).
Commit 3d3800b4f7f4 ("rv: Remove reactor's reference counter") noted
that module-based reactors, if supported, should be pinned with
try_module_get()/module_put() to prevent unloading under an active
monitor. This patch does not add that support. The only in-tree module
consumer, the tristate KUnit reactor test module, registers and
unregisters its reactor within a controlled test lifecycle.
The actual pinning is left to a follow-up series that adds support for
loadable-module reactors.
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Wen Yang <wen.yang@linux.dev>
Link: https://lore.kernel.org/r/94b7ec120a92b9368541ac050601e66be8fe653a.1789412799.git.wen.yang@linux.dev
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
|
|
Both register_react_printk() and register_react_panic() ignore the
return value of rv_register_reactor() and always return 0. If the
registration fails (e.g. a duplicate reactor name), the init functions
silently report success even though the reactor was not registered.
Propagate the error from rv_register_reactor() so a failed registration
is reported instead of being silently ignored.
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Reviewed-by: Nam Cao <namcao@linutronix.de>
Signed-off-by: Wen Yang <wen.yang@linux.dev>
Link: https://lore.kernel.org/r/1684c66520cee66ba86224f2b9e473c7dd2416ee.1789412799.git.wen.yang@linux.dev
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
|
|
rv_react() overrides the lockdep wait type to LD_WAIT_FREE to enforce
that reactor callbacks take no locks. But callbacks run in the context
of the triggering tracepoint, which can be preemptible task context on
any kernel. A timer interrupt firing during the callback makes the
interrupt-exit path schedule and take rq->__lock (LD_WAIT_SPIN) while
the LD_WAIT_FREE override is still held, producing a spurious
"Invalid wait context" warning:
[ BUG: Invalid wait context ]
context-{5:5}
1 lock held by kunit_try_catch/209:
#0: (rv_react_map-wait-type-override){+.+.}-{1:1}
kunit_try_catch/209 is trying to lock:
ffff8a743ed3e8a0 (&rq->__lock){-...}-{2:2}
Use LD_WAIT_SPIN instead of LD_WAIT_FREE, which causes false-positive
warnings in preemptible contexts due to scheduler preemption taking
rq->__lock.
Fixes: 69d8895cb9a9 ("rv: Add explicit lockdep context for reactors")
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Wen Yang <wen.yang@linux.dev>
Cc: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://lore.kernel.org/r/5586dd00789a0f904cc299cd8889744b8c6f8dc9.1789412799.git.wen.yang@linux.dev
Signed-off-by: Gabriele Monaco <gmonaco@redhat.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit
Pull audit fix from Paul Moore:
"A single audit fix for a potential UAF error in some audit filter
configurations"
* tag 'audit-pr-20260930' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/audit:
audit: fix exe mark UAF in kill_rules()
|
|
cpuset_update_flag() is only used by the legacy (v1) interface: its
callers are the write handlers of cpuset1_files via cpuset_write_u64(),
and cpuset_css_offline() which invokes it only on the legacy hierarchy
(!cpuset_v2() && is_sched_load_balance()).
Move it into cpuset-v1.c, where its sole helper
cpuset1_update_tasks_flags() already lives, so both can go back to being
static and drop out of cpuset-internal.h. As a result:
- dup_or_alloc_cpuset(), free_cpuset() and validate_change() are now
referenced from cpuset-v1.c, so they lose their static qualifier and
gain declarations in cpuset-internal.h.
- The v2 branch in cpuset_update_flag() that called
cpuset_force_rebuild() is dead on the v1-only call paths and is
dropped; the remaining rebuild_sched_domains_locked() path is kept.
callback_lock, a static in cpuset.c, is taken through the existing
cpuset_callback_lock_irq() wrapper.
- The open-coded "!cpuset_v2() && is_sched_load_balance()" test in
cpuset_css_offline() is wrapped in a new cpuset1_offline_css() helper
(a no-op stub when CONFIG_CPUSETS_V1=n), removing the last
cpuset_v2() branch from that path in cpuset.c.
No functional change intended.
Assisted-by: LLM
Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
Reviewed-by: Waiman Long <longman@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Hierarchical BPF schedulers may need to know how many CPUs a cgroup can
run on when distributing group-wide weights.
For example, fair's group share calculation scales a group's weight by
the smaller of its estimated runnable task count and the number of CPUs
in its effective cpuset. Without that count, a BPF scheduler can only
use the machine-wide CPU count and may give a group confined to a few
CPUs excessive shares.
A task's effective affinity can include per-task restrictions, so it
cannot reliably represent the group's cpuset. Reading the cpuset
directly from BPF would require CO-RE access to private cgroup and
cpuset structures, including inherited cpuset resolution, making the
approach fragile across kernel versions.
Add scx_bpf_cgroup_nr_cpus(), which returns the same count fair uses
through cpuset_num_cpus(). The count is independent of CPU or CID
numbering, so the kfunc is available to both cpu-form and cid-form
schedulers from any context. Also add a compat wrapper which falls
back to nr_cpu_ids on older kernels.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive cc07fae1de5d ("selftests/sched_ext: Add a test for
ops.dequeue() on remote local DSQ moves") and resolve its conflict with
f73ec6791424 ("sched_ext: Add selftest for blocked donor admission") here
so that for-next merges clean.
Conflicts: kept both auto-test-targets entries in the selftests Makefile,
where each side added one after dequeue_iter.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
When a task in the BPF scheduler's custody is moved to another CPU's
local DSQ, ops.dequeue() is only called once the task is picked for
execution, from set_next_task_scx() with SCX_DEQ_CORE_SCHED_EXEC, even
though no core-sched pick took place. It should instead be called
without flags when the task is inserted into the destination DSQ, as it
is for same-rq dispatches.
move_remote_task_to_local_dsq() sets p->scx.sticky_cpu across the
migration so that deactivate_task() on the source rq doesn't end custody.
Since commit b75aaea24c9f ("sched_ext: Properly mark SCX-internal
migrations via sticky_cpu"), enqueue_task_scx() only clears it after
inserting the task, so task_leave_custody() still sees the migration in
progress and skips the custody exit.
Clear p->scx.sticky_cpu as soon as enqueue_task_scx() has read it, as
was done before that commit. The source side is unaffected.
Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics")
Cc: stable@vger.kernel.org # 7.1.x: 18d62044cda7: sched_ext: Preserve rq tracking across local DSQ dispatch
Cc: stable@vger.kernel.org # 7.1.x
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
After seeing the patch [1] to guard is_in_v2_mode() with RCU, it makes
me realize that is_in_v2_mode() may be called in a context where a new
cgroup filesystem is being rebound with stale cpuset_cgrp_subsys.root
pointer. Avoid this potential UaF situation by adding a new cpuset_v2_mode
flag which is set when the cpuset_v2_mode mount option is used. This
flag is written into only when cpuset_bind() is being called with a
stable cpuset_cgrp_subsys.root value. The is_in_v2_mode() helper is
modified to read the new cpuset_v2_mode flag instead of accessing
cpuset_cgrp_subsys.root directly.
[1] https://lore.kernel.org/lkml/20260929084124.626693-2-arighi@nvidia.com
Fixes: b8d1b8ee93df ("cpuset: Allow v2 behavior in v1 cgroup")
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The cpuset code has 2 different helpers for determining if it is in a
v1 or v2 working environment - cpuset_v2() and is_in_v2_mode().
cpuset_v2() checks the binding of the cpuset subsystem is on a v1
or v2 hierarchy. It can also help to optimize out v1 specific code
if CONFIG_CPUSETS_V1 isn't defined.
is_in_v2_mode() layers on top of cpuset_v2() and returns true if it is in
a v2 hierarchy or in a v1 hierarchy with "cpuset_v2_mode" mount option
enabled to make "cpuset.cpus" and "cpuset.mems" behave in a way similar
to their v2 counterparts. However the cpuset hierarchy may still be a
standalone one separated from the other cgroup hierarchies.
Commit 90ac22ffef48 ("sched/fair: Add cgroup_mode: max") adds a new
cpuset_num_cpus() helper to return the number of effective CPUs in a given
cgroup. It is part of the "sched: Flatten the pick" patch series to help
flatten the cpu cgroup runqueue hierarchies. cpuset_num_cpus() currently
uses is_in_v2_mode() to determine if it is in a v2 environment. However
is_in_v2_mode() can be true in a v1 environment where the cpuset and cpu
cgroup subsystems can be in two separate hierarchies. The current use
case [1] for cpuset_num_cpus() assumes that it can return the number
of effective CPUs present in the current cpu cgroup which may not be
the case.
Fix that by switching to use cpuset_v2() which signifies that it is in
a unified hierarchy where both cpuset and cpu cgroup subsystems share
the same cgroup hierarchy if it returns true.
[1] https://lore.kernel.org/lkml/20260930081125.GF88198@noisy.programming.kicks-ass.net
Fixes: 90ac22ffef48 ("sched/fair: Add cgroup_mode: max")
Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Reviewed-by: Michal Koutný <mkoutny@suse.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
KCSAN's skip_watch, udelay_task and udelay_interrupt parameters can be
changed at runtime through /sys/module/kcsan/parameters/, where 0 means
watching every access or not delaying at all. With the default
CONFIG_KCSAN_SKIP_WATCH_RANDOMIZE=y and CONFIG_KCSAN_DELAY_RANDOMIZE=y,
writing 0 to any of them makes the next watchpoint setup divide by zero:
# echo 0 > /sys/module/kcsan/parameters/udelay_task
Oops: divide error: 0000 [#1] SMP NOPTI
CPU: 0 UID: 0 PID: 93 Comm: sh Not tainted 7.3.0-rc5 #3 PREEMPT(lazy)
RIP: 0010:kcsan_setup_watchpoint+0x1b5/0x410
Code: ... 65 89 05 ad 8d 98 02 d3 ee <f7> f6 29 d7 e8 22 0e 0f 01 ...
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
...
__show_trace_log_lvl+0x1bd/0x315
__die_body.cold+0x19/0x2b
die+0x34/0x50
...
BUG: TASK stack guard page was hit at ffffd3ab001fbfe8 (...)
reset_kcsan_skip() and delay_access() pass these values unchecked to
kcsan_prandom_u32_max(), which returns "state % ep_ro". delay_access()
halves the delay for compound and assert accesses, so
udelay_interrupt=1 is enough there.
The oops path runs instrumented code, which faults again until the stack
overflows. With skip_watch=0 the guest hangs without printing anything.
The division came in with commit 71a076f4a61a ("kcsan: Rewrite
kcsan_prandom_u32_max() without prandom_u32_state()"). Before it, the
helper scaled the random value with a multiply-shift, which yields 0 for
ep_ro == 0. Go back to that via reciprocal_scale().
Tested in an x86-64 KCSAN guest with virtme-ng: after this patch,
writing 0 to each parameter, or 1 to udelay_interrupt, no longer
oopses. skip_watch=0 now watches every access, which is very slow with
the default delays but completes when they are set to 0.
Fixes: 71a076f4a61a ("kcsan: Rewrite kcsan_prandom_u32_max() without prandom_u32_state()")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Reviewed-by: Marco Elver <elver@google.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Marco Elver <elver@google.com>
|
|
Program Structure reports have two attribution gaps. A missing jump
table is reported at the beginning of its subprogram rather than at
the gotox that needs the table, and the subprogram-layout checks run
before func_info and line_info are validated and installed, so their
reports cannot name the source function or line.
Report the missing jump table at the gotox instruction that looks it up,
rather than at the start of its subprogram.
func_info and line_info validation only needs the subprogram boundaries
found by add_subprogs() and the LD_ABS and tail-call properties that
check_subprogs() collects while scanning instructions; it does not
depend on the layout checks themselves. Split the property collection
into its own nested subprogram and instruction scan, together with the
program's callx marker, and run bpf_check_btf_info() before
check_subprogs(). The jump-boundary and fallthrough reports then carry
validated source information without changing either check.
CO-RE relocations are unaffected: commit c26e97721b17 ("bpf: Apply
CO-RE relocations before subprogram validation") applies them before
add_subprogs(), and they stay there. Only the func_info and line_info
validation moves.
Fixes: a8f427835394 ("bpf: Report Program Structure CFG errors")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/cf2f420c2b21de440a7dc51b1565c0f06d4b539640ee5c03384e5d77bcfb5686@mail.kernel.org/
Link: https://lore.kernel.org/bpf/20260926133048.2962553-3-memxor@gmail.com
|
|
unregister_fprobe() and unregister_fprobe_async() (used by BPF
kprobe-multi) rely on standard RCU grace periods (synchronize_rcu()
and call_rcu()) to wait until in-flight fprobe handlers complete before
freeing the fprobe.
However, if an fprobe handler executes while RCU is not watching (such
as in the idle loop or nohz_full extended quiescent states), standard
RCU does not track preemption-disabled sections. Consequently,
synchronize_rcu() does not wait for those executions, which can lead
to a use-after-free if the fprobe is freed immediately after
unregistration. Ensure handlers exit early when !rcu_is_watching().
Furthermore, fprobe_fgraph_entry() and fprobe_ftrace_entry() previously
used guard(rcu)() and rcu_read_lock(), which invoke lockdep on every
hit under CONFIG_PROVE_LOCKING. This adds overhead and can cause lockdep
recursion if probed functions interact with lockdep.
Since rhltable_lookup() and rhl_for_each_entry_rcu() use
rcu_dereference_all_check() (which checks rcu_read_lock_any_held()),
holding preemption disabled via rcu_read_lock_sched_notrace() is fully
valid and sufficient so long as rcu_is_watching() is true.
Define and use guard(rcu_sched_notrace)() across fprobe_ftrace_entry(),
fprobe_fgraph_entry(), and fprobe_return(). This eliminates fast-path
rcu_read_lock() and lockdep overhead while guaranteeing safe grace
period synchronization.
Link: https://lore.kernel.org/all/179064115227.394389.16910234241400391996.stgit@devnote2/
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-e46bcd68-4a56-4f19-a255-e3772980e5e3
Fixes: 657b594b2084 ("fprobe: Fix unregister_fprobe() to wait for RCU grace period")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
|
|
from BTF
With clang and pahole 1.32, x86-64 vmlinux BTF lacks
scx_bpf_task_set_lazy_resched() and sched_ext fails to initialize:
sched_ext: Failed to register kfunc sets (-22)
clang describes the lazy argument only after the prologue moves it into a
callee-saved register, which pahole 1.32 rejects as not being the argument
register. Released pahole is affected whatever the eventual fix. Keep the
argument on the stack with barrier_data() so that its location names no
register. OPTIMIZER_HIDE_VAR() would fix x86-64 but break arm64.
Fixes: f8e5a4e3f3be ("sched_ext: Add lazy preemption support")
Link: https://lore.kernel.org/r/20260925213619.2187751-1-yonghong.song@linux.dev
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The existing date range test covers +/-80,000 years around the epoch,
far below the day count at which a 32-bit long wraps (2^31 days,
roughly 5.88 million years).
Add a case at exactly 2^31 days after the epoch. Without the preceding
fix a 32-bit kernel returns tm_year=-5879541 and wrong month, day,
day-of-year and weekday; with it the result matches 64-bit kernels.
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: LLM
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/20260925165702.51677-3-kmehltretter@gmail.com
|
|
time64_to_tm() accepts a 64-bit seconds value, but stores the quotient in
long. On a 32-bit kernel the day count therefore wraps when it reaches
2^31, even though the corresponding year remains representable in
struct tm.
This is reachable when formatting externally supplied timestamps. For
example, NILFS recovery copies the little-endian 64-bit ss_create field
from an on-disk segment summary into time64_t. Its sysfs attributes then
print that value with %ptTs, whose formatter calls time64_to_tm(). A
corrupted image can consequently produce an architecture-dependent printed
date.
Keep the day quotient in s64 while leaving the seconds-within-day remainder
as long. Use div_s64_rem() for the weekday calculation so 32-bit builds do
not require compiler runtime division helpers.
Fixes: e6c2682a1da3 ("time: Add time64_to_tm()")
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: LLM
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/20260925165702.51677-2-kmehltretter@gmail.com
|
|
The timekeeper update path uses a bulk memcpy() to synchronize the
timekeeper structure, which is not guaranteed to be atomic. This allows for
torn reads in ktime_get_seconds() which bypasses the sequence counter
protection for performance.
To prevent reading a torn ktime_sec value, enforce atomic-like
access by using WRITE_ONCE() for the critical field before the bulk
memcpy() in timekeeping_update_from_shadow(). Correspondingly, use
READ_ONCE() in ktime_get_seconds() to ensure a fresh, consistent load
from memory.
The same was done for xtime_sec and ktime_get_real_seconds() in commit
d7fc133bf91f ("timekeeping: Use READ_ONCE/WRITE_ONCE() for xtime_sec to
prevent tearing").
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Acked-by: John Stultz <jstultz@google.com>
Link: https://patch.msgid.link/20260903-timekeeping-ktime_sec-v1-1-b33017536aa9@linutronix.de
|
|
The tmigr_event structure's ignore field is sometimes accessed locklessly,
but the __tmigr_cpu_activate(), tmigr_cpu_new_timer(), and
__tmigr_cpu_deactivate() functions do not mark accesses to this field.
Therefore, use READ_ONCE() for the tmigr_cpu_new_timer() function's
lockless load and WRITE_ONCE() for the __tmigr_cpu_activate() and
__tmigr_cpu_deactivate() functions' stores.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260919002523.3133928-2-paulmck@kernel.org
|
|
hlist_unhashed_lockless() locklessly samples the hlist_node structure's
pprev field, but detach_timer(), hlist_move_list(), and hlist_splice_init()
all use plain C-language stores to update this field, despite the "The
READ_ONCE() is paired with the various WRITE_ONCE() in hlist helpers that
are defined below" in the hlist_unhashed_lockless() header comment.
Therefore use WRITE_ONCE() for these hlist_node::pprev updates.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260919002523.3133928-1-paulmck@kernel.org
|
|
Updates to base->running are protected by the hrtimer base lock, but some
loads are lockless. Therefore, prevent compiler mischief by applying
READ_ONCE() to the lockless loads.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260919001428.3133388-12-paulmck@kernel.org
|
|
Currently, the hrtimer_switch_to_hres() function unconditionally updates
hrtimer_resolution to HIGH_RES_NSEC. This causes KCSAN false positives
in kernels built with CONFIG_KCSAN_STRICT=y. It also causes unnecessary
cache-line traffic on the many code paths that load hrtimer_resolution,
which is especially regrettable because the value of hrtimer_resolution
normally changes only once during the lifetime of a given kernel.
Therefore, update the value of hrtimer_resolution to HIGH_RES_NSEC only
if it does not already have that value.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260919001428.3133388-10-paulmck@kernel.org
|
|
The hrtimer_sleeper structure's ->task field is used as a flag to indicate
that the associated hrtimer has expired. This means that the hrtimer
handler can be storing to this field while other code is loading from it
to check for expiry. Note that additional races appear for hrtimers that
can be restarted, which could be argued to be a user error. However, that
is no reason to let the compiler introduce additional confusion, and to
this end, the hrtimer_sleeper_task_get() was introduced, use of which also
has the benefit of avoiding open-code access to hrtimer_sleeper innards.
Therefore, apply this accessor to handle_early_requeue_pi_wakeup().
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: André Almeida <andrealmeid@igalia.com>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Link: https://patch.msgid.link/20260919001428.3133388-9-paulmck@kernel.org
|
