| Age | Commit message (Collapse) | Author |
|---|
|
misc.2026.10.06a: Miscellaneous fixes.
torture.2026.10.12a: Make call_{s,}rcu() invocable from any environment,
add atomic SRCU, additional SRCU updates, and torture-test updates.
|
|
Add testtype 7 to verify that lockdep detects an IRQ-context
mismatch when an atomic SRCU read-side critical section is held
with IRQs enabled on one CPU and synchronize_srcu_atomic() is
called from an IPI handler on another CPU.
This covers the cross-CPU case that cannot be detected by
checking the current task's held locks.
Co-developed-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Acked-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Add three testtypes covering atomic SRCU lockdep behavior:
testtype 4: atomic SRCU same-type deadlock
testtype 5: atomic SRCU + raw spinlock dependency cycle
testtype 6: synchronize_srcu_atomic() inside rcu_read_lock()
Co-developed-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Acked-by: Boqun Feng <boqun@kernel.org>
Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The rcu_torture_reader() drives RCU readers from interrupt context via
smp_call_function_single(cpu, rcu_torture_irq, NULL, 0) with wait=0, to
runs rcu_torture_irq() on a remote CPU. this is async, nothing waits for
the remote handler to run.
On shutdown, torture_stop_kthread() only waits for each reader kthread to
return, and the reader's timer_delete_sync() only drains its timer. Neither
waits for a rcu_torture_irq() which still pending or executing on a remote
CPU, so it can run after all readers have exited and rcu_torture_cleanup()
has already advanced.
1. rcu_torture_irq() may issue cur_ops->call(rhp, rcu_torture_timer_cb)
after cur_ops->cb_barrier() has been waiting for all outstanding
callbacks complete. once the module is unloaded, fires into freed
module text, a use-after-free happen.
2. rcu_torture_irq() may still be inside rcu_torture_one_read(), holding
a read-side critical section, when cur_ops->cleanup() tears the flavor
down (e.g. cleanup_srcu_struct()), triggering an active-reader warning
or use-after-free of the torn-down structure.
This commit therefore issue a kick_all_cpus_sync() after all readers
kthread have returned and before cur_ops->cb_barrier(), synchronous IPI
round trip to every CPU guarantees that every rcu_torture_irq() which
previously issued by any reader has completed.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
When fwd_progress_div=1, the forward-progress test computes:
sd4 = (sd + div - 1) / div = sd
dur = sd4 + torture_random(&trs) % (sd - sd4) = sd4 + % 0
The modulo operation with a zero divisor triggers an integer
division by zero (undefined behavior at the C level, #DE trap on x86),
causing a kernel Oops and panic. On x86_64, this manifests as:
rcu_torture_fwd_prog_nr: Starting forward-progress test 0
Oops: divide error: 0000 [#1] SMP PTI
RIP: 0010:rcu_torture_fwd_prog+0x90b/0x1160
R12: 0000000000000000
The existing guard only handles non-positive values. However,
fwd_progress_div=1 also makes the random range empty because
sd4 == sd.
Change the guard to reject values below 2. The forward-progress test
only reaches this calculation when stall_dur() is positive, so
sd = stall_dur() + 1 >= 2. For fwd_progress_div >= 2, sd4 < sd,
ensuring that sd - sd4 is at least 1.
Keep the existing fallback to the default value of 4 for invalid
values.
Verified with QEMU/KVM: a 138-second run with fwd_progress_div=1
completed 81 forward-progress test cycles without a crash.
Fixes: 1b27291b1ea4f ("rcutorture: Add forward-progress tests for RCU grace periods")
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
synchronize_srcu_atomic() may be called with interrupts disabled, but
its callchain uses raw_spin_lock_irq_rcu_node() /
raw_spin_unlock_irq_rcu_node(), which unconditionally enables
interrupts on unlock. This can incorrectly change the caller's IRQ
state.
Use the irqsave/irqrestore variants for the locks in this callchain
so that the caller's IRQ state is preserved.
Co-developed-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Acked-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Add the RCU_LOCKDEP_WARN and early-boot check that the Tree version
gained, srcu_lock_sync() was already present.
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Add the lockdep annotation, same-type SRCU nesting warning, and
early-boot check used by __synchronize_srcu().
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Atomic SRCU does not use the srcu_node combining tree. Exclude it
from the SRCU_SIZING_IS_TORTURE() transition in
srcu_torture_stats_print(), and warn if such a transition is requested.
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Atomic SRCU must remain in the small size state. Warn if this
invariant is violated and avoid transitioning to big in that case.
[ paulmck: Folded "if" onto one line for 100-character limit. ]
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
|
|
The is_atomic parameter of init_srcu_struct_fields() exists so that
atomic SRCU never transitions to big, but neither
init_srcu_struct_atomic() nor its lockdep counterpart
__init_srcu_struct_atomic() sets it.
On systems where srcutree.convert_to_big selects SRCU_SIZING_INIT,
this needlessly allocates a full srcu_node combining tree for any
dynamically initialized atomic srcu_struct, despite atomic SRCU
having neither callbacks nor srcu_barrier() operations.
Pass true from both atomic entry points, adding an is_atomic parameter
to __init_srcu_struct_common().
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The atomic-SRCU path in srcu_barrier() sleeps for 100 milliseconds just
in case there are callbacks to wait for. But call_srcu() refuses
atomic SRCU with a WARN_ON_ONCE() before reaching the deferred-enqueue
path, so there can be no callbacks, deferred or otherwise.
Drop the sleep.
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
call_srcu() is forbidden on atomic srcu_struct, so srcu_gp_end() never
has callbacks to invoke for them. Yet it schedules callback invocation,
which for atomic SRCU's SRCU_SIZE_SMALL state arms the boot CPU's
->delay_work timer every grace period, only for srcu_invoke_callbacks()
to find nothing to do.
Skip this for atomic SRCU.
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
srcu_read_unlock_atomic(), whose read-side critical sections disable
preemption. In the common case where there are no readers at all, the
grace period therefore need not do the index flip. Add a fastpath
that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
lock counts match the unlock counts on both ranks, ends the grace
period immediately, skipping the srcu_advance_state() scans, mirroring
the similar Tiny SRCU fastpath.
Correctness requires the counter-sum proof to follow the grace-period
anchor written by srcu_gp_start(); placing it before the anchor could
let this grace period miss a pre-existing reader and return without
waiting for it. The smp_mb() between the unlock and lock sums pairs
with the smp_mb() in __srcu_read_lock(). The grace period is ended
manually under ->lock and ->srcu_atomic_gp_flag.
In theory, this is slower than David Woodhouse's earlier patch, but
David's measurements showed that the performance was close enough
that it makes sense to keep the get_state_synchronize_srcu() and
poll_state_synchronize_srcu semantics.
Link: https://lore.kernel.org/all/20260907075829.2073224-4-kunwu.chan@linux.dev/
Link: https://lore.kernel.org/all/0f4dea21bac43685d3286df329401177e4452b36.camel@infradead.org/
Link: https://lore.kernel.org/all/0d4af6318ac67486858be1df8d436147b444a2d2.camel@infradead.org/
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: David Woodhouse <dwmw2@infradead.org>
|
|
The fastpath of synchronize_srcu_atomic() should always be taken
because readers always disable preemption. Therefore, the fact that
synchronize_srcu_atomic() is running at all should mean that all readers
have ended.
But there are always bugs. And responding to a usage bug with a
too-short SRCU grace period, and thus possibly corrupting memory
is at best a sadistic response so such a bug. For this reason,
synchronize_srcu_atomic() explicitly waits for readers. Except that
it does so silently, possibly failing to flag this bug. This commit
therefore adds a splat if synchronize_srcu_atomic() fails to take the
early exit.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The irq_work in srcu is used to schedule a delayed work. The work item
is not scheduled directly because it is not always possible wake a
thread directly.
On PREEMPT_RT the default irq_work is initialized with IRQ_WORK_LAZY and
is delayed to the irq_work thread. A system booted with the command line
"trace_event=…" will freeze during boot because the irq_work thread is
not yet deployed (and tracing uses synchronize_srcu() in
tp_rcu_cond_sync()).
The irq_work performs just a wakeup a thread, there is nothing wrong
with doing this from hardirq context on PREEMPT_RT.
Use IRQ_WORK_INIT_HARD for srcu's irq_work.
Fixes: 7c405fb3279b3 ("rcu: Use an intermediate irq_work to start process_srcu()")
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Because synchronize_srcu_atomic() cannot sleep, if a pair of them run
concurrently pinned to the same CPU, it is possible that one will spin
uselessly waiting for the other while at the same time preventing that
other from running. This commit therefore disables preemption to prevent
this failure mode.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds support for the value 0x10 for reader_flavor, which
specifies SRCU_READ_FLAVOR_ATOMIC, that is, srcu_read_lock_atomic(),
srcu_read_unlock_atomic(), and synchronize_srcu_atomic().
[ paulmck: Apply Kunwu Chan feedback. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds the Tiny SRCU counterpart to Tree SRCU's
synchronize_srcu_atomic(). One might hope that this could be as trivial
as Tiny RCU's synchronize_rcu(), and there was a time when it would
have been. But lazy preemption really can preempt an SRCU read-side
critical section, which means that synchronize_srcu_atomic() really must
be prepared to spin waiting for it.
This spinning currently consists of cond_resched_tasks_rcu_qs()
and cpu_relax(). It would be better to have some way of telling the
scheduler that there is nothing useful for us to do. We cannot use
the traditional wait_event() approach because synchronize_srcu_atomic()
is not permitted to block.
[ paulmck: Apply Kunwu Chan and Joel Fernandes feedback. ]
Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Claude:claude-mythos-5
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Some dedicated srcu_struct users have read-side critical sections
which are short, never sleep, and never block on anything which may
itself depend on memory allocation — because they were, until
recently, spinlock or rwlock critical sections. For such a domain the
update side can safely wait for readers by spinning, from contexts
where sleeping is undesirable or the grace-period machinery's
latency (workqueue scheduling, jiffy-paced retries) dominates the
actual reader drain time.
But that is only safe if every reader keeps the promise. Make the
promise explicit and machine-checkable:
- srcu_read_lock_atomic() / srcu_read_unlock_atomic() enter the
usual (smp_mb-based) read-side critical section with preemption
disabled and (except in hardirq, where it is redundant)
non_block_start() armed, recording SRCU_READ_FLAVOR_ATOMIC in the
per-CPU reader flavor. Disabling preemption enforces the
no-sleeping promise on every configuration and bounds the section,
so it is always running on some CPU; non_block_start() extends the
enforcement to even potentially-sleeping calls on paths which
happen not to block. The existing reader-flavor consistency checks
complain about any mixing with other flavors.
- synchronize_srcu_atomic() waits for all pre-existing readers by
repeating the try_synchronize_srcu() both-epoch counter proof with
cpu_relax() until it succeeds: no sleeping, no index flip, no
grace-period sequence update, and therefore no interaction with
concurrent call_srcu(), synchronize_srcu() or srcu_barrier(). It
always provides the full grace-period guarantee: if the domain
turns out to have had readers of any other flavor — a caller bug,
since such a reader may be asleep and spinning on it would be
unbounded — it complains and falls back to a real (sleeping) grace
period internally, that being the only correct wait for a
possibly-sleeping reader. The flavor mask is rechecked on every
iteration so a first non-atomic reader appearing mid-spin takes
the same path.
The immediate motivation is the proposed conversion of KVM's
gfn_to_pfn_cache to SRCU¹, whose mmu_notifier invalidation path drains
readers before the primary MMU zaps a page. With the readers declared
atomic, that drain becomes spin-only: no sleeping at all in the
notifier, bounded by the longest reader section, satisfying even the
strictest reading of the OOM-reaper non-blocking requirement without
needing to touch the non_block_start() annotation².
This commit also adds atomic-SRCU-specific initializers:
init_srcu_struct_atomic(), DEFINE_SRCU_ATOMIC(), and
DEFINE_STATIC_SRCU_ATOMIC().
This commit implements only Tree SRCU. Tiny SRCU will follow.
¹ https://lore.kernel.org/all/20260811132237.102400-1-dwmw2@infradead.org/
² https://lore.kernel.org/all/20260812134934.GC662699@ziepe.ca/
[ paulmck: Apply Kunwu Chan feedback. ]
Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: Claude:claude-mythos-5
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
New workloads can do large bursts of call_rcu_tasks() invocations in a
short time period, followed by a quiet time period long enough to drain
all of the callbacks, followed by another burst of call_rcu_tasks()
invocations. This can cause RCU Tasks to switch back and forth between
queuing callbacks only on CPU 0 (during quiet periods) and on all CPUs
(during bursts).
Which is fine. Except for the fact that each cycle from CPU-0-only to
all-CPUs queuing and back generates three console messages, one announcing
the shift to all-CPUs queuing, another announcing the start of the shift
back to CPU-0-only queuing, and the third announcing completion of this
shift after an RCU grace period. And these console messages can overrun
console-log communications channels and obscure other console-message-based
debugging information. And the only known use for these console messages
is debugging RCU Tasks itself.
This commit therefore adds a rcupdate.rcu_task_collapse_debug module
parameter that defaults to false (suppressing these console messages).
Those debugging or otherwise playing with RCU Tasks callback queuing
auto-adjustment can set this parameter to the value true.
[ paulmck: Apply Breno Leitao feedback. ]
Reported-by: Breno Leitao <leitao@debian.org>
Reported-by: David Dai <david.dai@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Breno Leitao <leitao@debian.org>
|
|
The bcachefs ktest allocation-leak check writes rcutree.do_rcu_barrier
before reading /proc/allocinfo. While testing bcachefs performance
changes, small objects released with kfree_rcu() remained visible after
repeated writes to the hook and 20 seconds of waiting, causing otherwise
clean tests to fail their leak check.
The test assumes a stronger contract than the hook currently documents:
rcu_barrier() waits for ordinary callbacks, but does not flush objects
still held in kfree_rcu() batching or per-CPU SLUB sheaves. The retained
population eventually fell as a sheaf filled; there is no evidence here
of unbounded growth or OOM.
Changing the hook to drain kvfree_rcu() work let the same unmodified
bcachefs workload pass its allocation check. All eight checkpoints in
one VM, after 50 through 400 option changes, reported zero retained
reconcile_scan objects. This motivated the separate private-cache test
used to isolate the incomplete drain from bcachefs.
Calling kvfree_rcu_barrier() from rcu_barrier_throttled() was proposed
when kvfree_rcu_barrier() was added in 2024, to restore a clean baseline
between userspace benchmark runs. The discussion concluded that keeping
the existing hook name, adding the second operation and documenting both
was the safest compatibility choice, but the follow-up was not added.
Add that drain and document the stronger test interface. Always retain
the existing start-rate limit and perform the kvfree_rcu() drain: an
unrelated ordinary barrier does not establish that this work completed.
Retain the entry ordinary-barrier sequence snapshot. After draining,
skip the final ordinary barrier only if that snapshot is complete,
preserving the memory barrier on the completion path. Otherwise, invoke
rcu_barrier() explicitly. This keeps the ordinary-callback guarantee
independent of whether kvfree_rcu_barrier() embeds an ordinary barrier.
Clarify that the documented completion guarantee covers work queued
before the request, without preventing new work from being queued.
Earlier validation of the unconditional-drain version used four fresh
VM pairs with a private-cache fixture: controls retained the queued
object (60 to 60 active objects), and treatments drained it (60 to 59).
An ordinary-callback test passed on both kernels. Those runs predated
the guarded skip and do not validate that change. No elapsed-time
improvement is claimed.
Link: https://lore.kernel.org/all/20240820155935.1167988-1-urezki@gmail.com/
Signed-off-by: Matthias Goergens <matthias.goergens@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Nothing in the tree.c translation unit uses anything from the
private timekeeping header ../time/tick-internal.h, included
since commit 48d07c04b4cc ("rcu: Enable elimination of
Tree-RCU softirq processing"). The tick symbols RCU uses,
tick_dep_set(), tick_dep_clear(), their _cpu and _task
variants, tick_nohz_full_cpu() and TICK_DEP_BIT_RCU, are all
declared in the public linux/tick.h, which tree.c already
includes.
Drop the include.
Signed-off-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Correct "upto" to "up to", reported by scripts/checkpatch.pl using the
misspelling list in scripts/spelling.txt. Only touches comments, no code
changes.
Assisted-by: Cursor:claude-opus-5
Signed-off-by: Hemanth Selam <hemanth.selam@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, rcu_print_task_stall() will dump out the PID, RCU reader
nesting level, the rcu_special structure's flags, and whether or not
that reader is on the ->blkd_tasks list. When debugging RCU priority
boosting, it is also good to know whether the stalled RCU reader is
currently running and whether it is currently being RCU priority boosted.
This commit therefore adds this information to the output.
[ paulmck: Apply kernel test robot feedback. ]
[ paulmck: Apply feedback from Arnd Bergmann and kernel test robot. ]
Link: https://lore.kernel.org/all/20260925133922.1356404-1-arnd@kernel.org/
Link: https://lore.kernel.org/all/202609261005.GrnaXBXT-lkp@intel.com/
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Reviewed-by: Kunwu Chan <kunwu.chan@gmail.com>
Tested-by: Kunwu Chan <kunwu.chan@gmail.com>
|
|
Initially (and perhaps forever), call_srcu() will not be available for
atomic srcu_struct structures. There is therefore no reason to transition
such a structure to big, because the main purpose of such a transition
is to reduce lock contention for concurrent SRCU callback queueing.
This commit therefore adds an is_atomic parameter to both the
check_init_srcu_struct() and init_srcu_struct_fields() functions, which
suppresses the initialization-time transition to big that is enabled by
default on large systems.
It will still be possible to force a transition using rcutorture as a
destructive test. This might (or might not) be adjusted later.
[ paulmck: Apply feedback from Kunwu Chan. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The new SRCU_READ_FLAVOR_ATOMIC prohibits sleeping in both readers and
grace-period waits in order to permit waiting on an SRCU grace period
within an OOM notifier. This means that srcu_advance_state() cannot
acquire ->srcu_gp_mutex in this case, because mutex_lock() can sleep.
This commit therefore adds an is_atomic parameter to srcu_advance_state().
When this parameter is false, current behavior is maintained, in other
words, ->srcu_gp_mutex is acquired and released as before. But when this
is_atomic parameter is false, the caller is responsible for excluding
other callers.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
In non-preemptible kernels, a Tiny SRCU grace period implies an RCU
grace period because any context switch suffices. But in preemptible
kernels, it is possible for a Tiny SRCU grace period to elapse without
a corresponding RCU grace period.
Which was OK until RCU Tasks Trace was re-implemented in terms
of SRCU-fast, which in Tiny SRCU is implemented as SRCU, which is
already fast. And RCU Tasks Trace grace periods are required to imply
RCU grace periods. This commit therefore adds a synchronize_rcu(),
but only in preemptible kernels.
Because preemptible Tiny SRCU is not in mainline, this added call to
synchronize_rcu() will not slow anything down: The comparison would
instead be with TREE SRCU. But if this added call ever becomes a problem,
the Tiny SRCU srcu_struct structure could track whether or not this is for
SRCU-fast, and to add the synchronize_rcu() only in the SRCU-fast case.
However, at the moment, this is seen as unnecessary complexity.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
call_rcu() and call_srcu() are now safe to invoke from NMI, but
rcutorture never does, leaving the deferral path untested.
Add an ->nmi_capable flag to rcu_torture_ops. For flavors that set it,
arm a per-CPU hardware perf counter whose overflow handler submits a
callback via ->call(). The handler acts only when in_nmi(), so only a
genuine NMI exercises the deferral path. One preallocated callback per
CPU is kept in flight, guarded by an atomic, to avoid allocating in NMI.
The counter uses a fixed sample period rather than a frequency: a
frequency-based event sets TICK_DEP_BIT_PERF_EVENTS and would pin the tick
for the whole run on NO_HZ_FULL kernels.
Report the count issued from NMI ("nmi-calls:") and the count invoked
("nmi-cbs:"). rcu_torture_cleanup() disables the counters and then calls
cb_barrier(), which drains every deferred callback, so the two counts must
then match; a mismatch fails the test. This relies on
rcu_barrier()/srcu_barrier() flushing deferred callbacks, as added earlier
in the series.
Set ->nmi_capable on the NMI-safe flavors: rcu, srcu, srcud, and
tasks-tracing (call_srcu() under the hood). Tasks and Tasks Rude are left
alone, as call_rcu_tasks_generic() is not yet NMI-safe.
Enabled by default; the nmi_calls parameter disables it, which helps rule
NMI handling in or out when triaging a failure. Requires
CONFIG_PERF_EVENTS and a hardware PMU: without one nothing is issued from
NMI and the end-of-test check compares zero against zero, so a pass does
not by itself mean the path ran. That is the case under kvm.sh, which
boots qemu with -cpu kvm64 and no vPMU; rcu_torture_nmi_cleanup() says so
on the console.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Vanilla RCU readers, when either non-preemptible or preemptible with
priority-boosting enabled, can handle heavy overload conditions.
Other RCU implementations are more fragile. This commit adds an
rcu_torture_ops structure field named ->rdrs_handle_load that is set
for non-fragile configurations of vanilla RCU, but cleared otherwise.
It also refrains from running fragile readers concurrently with overload
testing, thus avoiding false-positive overload-testing failures.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Give Tiny call_srcu() the same treatment as Tree SRCU. When interrupts
are disabled and the scheduler is up, stage the callback on the
srcu_struct's lockless list for an irq_work to re-issue later. Tiny SRCU
is uniprocessor, so there is no CPU-offline drain. A draining flag drops
a deferring call_srcu() that re-enters mid-drain (unless from an NMI), as
in Tree SRCU; such a drop can strand state the caller tied to the callback,
not just the callback itself.
Interrupts stay off for the whole batch. TINY_SRCU implies !SMP, where
arch_irq_work_has_interrupt() is false, so the drain always waits for the
tick and a batch is whatever one tick's worth of interrupts-disabled
call_srcu()s staged. Unlike the other three flavors srcu_do_enqueue() here
has no debug_rcu_head_queue(), so nothing reports a double call_srcu();
termination of the drain rests on srcu_do_enqueue() clearing ->next, and
the callback list self-links at the tail exactly as a double call_srcu()
made it before.
srcu_barrier() (now out of line) and cleanup_srcu_struct() drain the
deferred list first, so a deferred callback is re-issued onto the callback
list and invoked by the grace-period work that cleanup_srcu_struct()
flushes, rather than stranded on a soon-to-be-freed srcu_struct.
cleanup_srcu_struct() also syncs ->defer_iw, since that irq_work is
embedded in the srcu_struct the caller is about to free.
The draining flag is global rather than per-srcu_struct: a re-entrant
call_srcu(B) inside a drain of A raises B's own ->defer_iw, whose drain can
stage back onto A, so a per-srcu_struct flag would not break the chain.
The cost is that a drain of A also drops a non-NMI call_srcu() to any
other srcu_struct for its duration.
Gated by CONFIG_RCU_DEFER like Tree SRCU, though the srcu_struct members
are unconditional.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
call_srcu() has the same constraint as call_rcu():
srcu_gp_start_if_needed() enqueues under raw_spin_lock_irqsave() and may
walk the srcu_node tree, as do callback invocation and grace-period work,
so a call_srcu() with interrupts already disabled can race an operation in
flight on this CPU. call_rcu_tasks_trace() is call_srcu() under the hood,
so a sleepable BPF program freeing an object can reach this.
Defer as call_rcu() does: stage the callback on the srcu_data's
->defer_cbs, chain that srcu_data onto a per-CPU list, and raise a per-CPU
irq_work that re-issues it straight to the enqueue helper, never back
through __call_srcu(). The irqs-enabled path is unchanged; as for
call_rcu() the gate is bare irqs_disabled(), so callers that merely hold
interrupts off are deferred too and pay one irq_work hop, including
call_rcu_tasks_trace() from the BPF memalloc irq_work.
The irq_work is per-CPU rather than per-srcu_struct and statically
initialized, so deferral never runs check_init_srcu_struct(); it is
IRQ_WORK_INIT_HARD as for call_rcu(). srcu_barrier() flushes it first, and
rcutree_migrate_callbacks() calls srcu_offline_drain() for an outgoing CPU.
cleanup_srcu_struct() drains before its "just leak it" early returns, and
srcu_module_going() before freeing any ->sda, since a staged srcu_data left
chained on a per-CPU list would dangle. Staging is two steps, the callback
onto
->defer_cbs and then the srcu_data onto the per-CPU list, so a flusher can
find the per-CPU list empty while a callback whose call_srcu() has not
returned sits on ->defer_cbs; the staging CPU's own irq_work takes that
one.
The per-CPU srcu_defer ->lock, not any srcu_data's, is held with interrupts
off across the whole nested drain: the chain of srcu_datas staged on that
CPU and, for each, its callbacks, with srcu_do_enqueue() taking that
srcu_data's ->lock and possibly starting a grace period for every one.
That is what serializes the drainers. The bound is as for call_rcu(): what
one interrupts-disabled region could stage, normally a single callback, or
whatever accumulates before the tick where arch_irq_work_has_interrupt() is
false. Both lists are drained newest-first; nothing depends on call_srcu()
ordering.
The drain clears ->next before re-issuing, which bounds a node self-linked
by a double call_srcu(); a longer cycle is not bounded, and a double
call_srcu() stays undefined, as for call_rcu().
A callback deferred past the CPUHP_AP_SMPCFD_DYING irq_work flush leaves
that CPU's srcu_defer ->iw claimed with its self-IPI lost, as for
call_rcu(). srcu_offline_drain() still re-issues the callback, but the
irq_work cannot be un-queued, and here the claim is shared by every
srcu_struct on the CPU.
As in call_rcu(), the re-issue runs with interrupts disabled and can be
re-entered by instrumentation, so a per-CPU flag, set only while that CPU
is inside its own irq_work drain, drops a deferring call_srcu() seen
mid-drain unless it comes from an NMI. Such a drop can strand state the
caller associated with the callback, not just the callback itself.
Staging records only the callback, so a deferred expedited call_srcu()
completes as a normal grace period. Only srcu_expedite_current() can hit
that, and only when invoked with interrupts already disabled.
Gated by CONFIG_RCU_DEFER, though the srcu_data members and the per-CPU
srcu_defer are unconditional. Under CONFIG_PROVE_RCU, warn if the direct
path is reached from an NMI.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Give Tiny call_rcu() the same treatment as Tree RCU. When interrupts are
disabled and the scheduler is up, stage the callback on a lockless list
that an irq_work re-issues later. One global list and irq_work suffice
since Tiny RCU is uniprocessor, and there is no CPU-offline drain.
The re-issue runs with interrupts disabled and can be re-entered by
instrumentation, so a draining flag drops a deferring call_rcu() seen
mid-drain (unless from an NMI), as in Tree RCU. Gated by CONFIG_RCU_DEFER,
though the deferral state is unconditional.
Interrupts stay off for the whole batch, but the re-issue is a tail append
with no locks. TINY_RCU implies !SMP, where arch_irq_work_has_interrupt()
is false, so the drain always waits for the tick and a batch is whatever
one tick's worth of interrupts-disabled call_rcu()s staged. As in Tree
RCU the drain clears ->next before re-issuing, which bounds a node
self-linked by a double call_rcu(): rcu_do_enqueue()'s duplicate path
returns without clearing it. A longer cycle is not bounded; a double
call_rcu() stays undefined.
The idle-task reschedule moves out of the enqueue helper so that a drain
does it once for the batch rather than once per callback, which would
otherwise take the runqueue lock N times with interrupts disabled.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
RCU's per-CPU callback list is only touched with interrupts disabled: the
enqueue runs under local_irq_save() (and the nocb locks when offloaded),
as do callback invocation and grace-period work. A call_rcu() that
arrives with interrupts already disabled, whether from an NMI or from
instrumentation that re-enters RCU, can interrupt one of those and corrupt
the list or deadlock.
Defer instead: stage the callback on a per-CPU llist and raise an irq_work
that re-issues it once interrupts are on, straight to the enqueue so it
cannot defer again. The gate is bare irqs_disabled(), so callers that
merely hold interrupts off are deferred too and pay one irq_work hop.
Skip it while the scheduler is down (RCU_SCHEDULER_INACTIVE): irq_work is
not usable that early, rcu_init() already calls call_rcu(), and the per-CPU
deferral state is not initialised until rcu_init_one() runs later in it.
rcu_barrier() drains every CPU's ->defer_head before it scans the lists,
and rcutree_migrate_callbacks() drains an outgoing CPU's. A drain
re-issues onto the draining CPU, so a barrier moves other CPUs' staged
callbacks onto
its own ->cblist; call_rcu() promises no CPU affinity for invocation.
->defer_lock is held across llist_del_all() and the whole re-issue so the
drainers
serialize: one that finds the list empty can conclude that everything
staged before it is already on a callback list. Interrupts stay off for
the batch. Where the arch has an irq_work self-IPI that is what one
interrupts-disabled region could stage, normally a single callback; where
arch_irq_work_has_interrupt() is false the drain waits for the tick, so
several regions can accumulate first.
The drain clears ->next before re-issuing. A double call_rcu() on a head
that is already debug-object-active self-links the staged node, and
rcu_do_enqueue()'s duplicate path returns without clearing it, so the
drain would spin. A re-add behind other staged callbacks makes a longer
cycle, which that does not bound; a double call_rcu() stays undefined.
llist_del_all() yields newest-first, so a batch is re-issued in reverse
call order; nothing depends on call_rcu() ordering. The re-issue drops
the lazy hint, since staging records only ->func, so a deferred callback
loses its batching on CONFIG_RCU_LAZY. kasan_record_aux_stack() moves to
__call_rcu_common() so a use-after-free report names the caller rather
than the irq_work.
The re-issue runs with interrupts disabled, so instrumentation on the
enqueue path can re-enter call_rcu(), stage another callback and re-raise
the irq_work, livelocking the drain. A per-CPU flag guards it: a deferral
that arrives while this CPU is draining, and is not from an NMI, is
dropped. The WARN_ONCE() is under CONFIG_PROVE_RCU, so a production kernel
drops it silently. That leaks the callback and can strand state
the caller tied to it, since a one-shot flag only the callback clears
never resets, but the alternative is an unbounded loop.
A callback deferred past the CPUHP_AP_SMPCFD_DYING irq_work flush leaves
->defer_work claimed with its self-IPI lost. rcutree_migrate_callbacks()
still re-issues the callback, but the first deferral after that CPU comes
back raises no IPI and waits for the next irq_work there, or for
rcu_barrier(). Unqueueing an irq_work is not something the API offers.
The irq_work is IRQ_WORK_INIT_HARD so the re-issue stays prompt on
PREEMPT_RT, where a non-HARD irq_work runs in a kthread that can be
delayed under load. A hidden CONFIG_RCU_DEFER gates the deferral code and
its IRQ_WORK dependency, though the rcu_data members are unconditional;
without it call_rcu() enqueues directly as before. Under
CONFIG_PROVE_RCU, warn if the direct path is reached from an NMI.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The WARN_ON() added by commit 78a38cbf6f20 ("srcu: Queue sdp->work
when the delay timer is successfully deleted") uses rcu_segcblist_n_cbs()
to detect callbacks that srcu_barrier() failed to wait for. However, the
->len counter is decremented only at the end of srcu_invoke_callbacks(),
after the invoking loop has finished. Since srcu_barrier() can return
right after the barrier callback is invoked, cleanup_srcu_struct() can see
a non-zero n_cbs even though the cblist is already physically empty,
falsely triggering the WARN_ON() together with a still-pending delay_work
timer.
This can be triggered as follows, as seen in the syzbot report against
kvm_destroy_vm() -> cleanup_srcu_struct(&kvm->srcu):
1. call_srcu(&kvm->srcu, &bus->rcu, __free_bus) starts SRCU grace
period GP1.
2. GP1 ends: a delay timer is armed and sdp->work is queued, but
sdp->work has not run yet.
3. Another call_srcu(&kvm->srcu, &bus->rcu, __free_bus) call invokes
srcu_segcblist_advance(), which moves the GP1 callback to
RCU_DONE_TAIL, and starts SRCU grace period GP2.
4. srcu_barrier() is called. It queues its barrier callback after the
GP2 callback and waits for srcu_invoke_callbacks() to invoke it.
5. GP2 ends: another delay timer is armed, and the sdp->work queued in
step 2 begins to run. Its srcu_invoke_callbacks() call invokes
srcu_segcblist_advance() again, moving the GP2 and barrier callbacks
to RCU_DONE_TAIL as well, and then invokes all of them. However,
rcu_segcblist_add_len(), which updates srcu_cblist's ->len, has not
run yet at this point.
6. srcu_barrier() returns once its callback has been invoked, and
cleanup_srcu_struct() starts running. It finds the delay timer
armed in step 5 still pending and srcu_cblist's ->len still
non-zero (because step 5 has not reached rcu_segcblist_add_len()
yet), and WARN_ON() fires even though every callback has actually
been invoked.
Use rcu_segcblist_empty(), which checks the actual head of the cblist,
instead of rcu_segcblist_n_cbs(), which checks the racy ->len counter.
Callbacks that have genuinely not been invoked yet still leave the list
non-empty, so the WARN_ON() still catches callers that skip srcu_barrier()
or queue callbacks after it.
Link: https://lore.kernel.org/rcu/e6350377085ddd85d6ef00d8e9a67bd50c762d3c@linux.dev/T/#t
Reported-by: syzbot+d4faf7db59e11f6fd1ab@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=d4faf7db59e11f6fd1ab
Fixes: 78a38cbf6f20 ("srcu: Queue sdp->work when the delay timer is successfully deleted")
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Sunho Park <shpark061104@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Tested-by: Sean Christopherson <seanjc@google.com>
|
|
The current RCU CPU stall-warning code can say "INFO: Stall ended before
state dump start" but then panic the system anyway. This commit therefore
avoids panicking the system if the stall has already ended.
However, the sched_ext scheduler will still be ejected, as it might have
been the cause of the almost stall. This can easily be changed if needed.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The lazy RCU shrinker decremented sc->nr_to_scan (unsigned long)
and then tested the result with <= 0. When a single CPU's lazy
callback count exceeds the remaining budget, the subtraction wraps
to a large positive value and the <= 0 comparison, which is
equivalent to == 0 for an unsigned type, never fires again. The
scan loop then iterates through every nocb CPU instead of honouring
the reclaim budget.
Accumulate into count and stop once count >= nr_to_scan.
Fixes: c945b4da7a448 ("rcu: Shrinker for lazy rcu")
Assisted-by: Zcode:GLM-5.2
Signed-off-by: Longlong Xia <xialonglong@kylinos.cn>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux
Pull RCU updates from Paul McKenney:
"Make expedited grace periods expedite normal RCU callbacks
Miscellaneous fixes:
- Improve diagnostic output with character task states
- Mark accesses to inform KCSAN of concurrency design
- Move from kmalloc() to kmalloc_obj()
- Documentation updates
- Improve handling of RCU deferred quiescent states
- Clean up unused function arguments and structure fields
- Reduce show_rcu_gp_kthreads() stack space
Tasks RCU updates:
- Clean up after SRCU re-implementation of Tasks Trace RCU
- Mark accesses to inform KCSAN of concurrency design
- Add ->lazy_timer status to diagnostic output
- Remove an unnecessary memory barrier
- Fix a data race, courtesy of KCSAN
- Documentation updates
- Convert cond_resched_tasks_rcu_qs() from macro to static inline
function
SRCU updates:
- Add Rust helpers for SRCU
- Avoid losing queued work at cleanup_srcu_struct() time
Torture-test updates:
- Preparation work for immediate RCU priority deboosting
- Test RCU readers from real interrupt handlers (as opposed to
softirq)
- Simplify code through use of cpumask_next_wrap()
- Improve diagnostic output with character task states
- Add rcutorture.nwriters parameter to allow lightweight stall
testing, and rcutorture.stall_only to make doing so easier
- Test an RCU Tasks Trace grace period implying an RCU grace period
- Make RCU Tasks Trace torturing track reader batches
- Fix a data race, courtesy of KCSAN
- Plug a shuffle_tmp_mask memory leak on kthread spawn failure"
* tag 'rcu.2026.08.18a' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux: (59 commits)
rcu: Add closing parenthesis in comment in rcu_read_unlock_strict()
rcutorture: Make {,s}rcu_read_delay() better handle forward-progress testing
rcutorture: Announce declining to forward-progress test
torture: Don't leak shuffle_tmp_mask when shuffler kthread fails to start
rcutorture: Use this_cpu_inc() for rcu_torture_count[] and rcu_torture_batch[]
rcutorture: Make RCU Tasks Trace track Reader Batches
rcutorture: Test RCU Tasks Trace GP implying RCU GP
rcutorture: Add a stall_only module parameter
rcutorture: Add nwriters module parameter
rcutorture: Use task_state_to_char() for task-state reporting
rcutorture: Use cpumask_next_wrap() in rcu_torture_preempt()
rcutorture: Test RCU readers from hardware interrupt handlers
rcutorture: Check for immediate deboosting at reader end
srcu: Queue sdp->work when the delay timer is successfully deleted
rcu-tasks: Convert cond_resched_tasks_rcu_qs() to static inline
rcu-tasks: Fix some comments for call_rcu_tasks() and call_rcu_tasks_rude()
rcu-tasks: Rename tasks_rcu_exit_srcu_stall_timer to tasks_rcu_exit_stall_timer
rcu: Mark interrupts-enabled accesses to rdp->cpu_no_qs.s
rcu: Reduce stack usage in show_rcu_gp_kthreads()
rcu: Mark accesses to ->rcu_urgent_qs and ->rcu_need_heavy_qs
...
|
|
'rcu-tasks.2026.07.30a', 'srcu.2026.08.11a' and 'torture.2026.08.14a' into HEAD
Changes:
Make expedited grace periods expedite normal RCU callbacks
Miscellaneous fixes:
* Improve diagnostic output with character task states.
* Mark accesses to inform KCSAN of concurrency design.
* Move from kmalloc() to kmalloc_obj().
* Documentation updates.
* Improve handling of RCU deferred quiescent states.
* Clean up unused function arguments and structure fields.
* Reduce show_rcu_gp_kthreads() stack space.
Tasks RCU updates:
* Clean up after SRCU re-implementation of Tasks Trace RCU.
* Mark accesses to inform KCSAN of concurrency design.
* Add ->lazy_timer status to diagnostic output.
* Remove an unnecessary memory barrier.
* Fix a data race, courtesy of KCSAN.
* Documentation updates.
* Convert cond_resched_tasks_rcu_qs() from macro to static inline
function.
SRCU updates:
* Add Rust helpers for SRCU.
* Avoid losing queued work at cleanup_srcu_struct() time.
Torture-test updates:
* Preparation work for immediate RCU priority deboosting.
* Test RCU readers from real interrupt handlers (as opposed to softirq).
* Simplify code through use of cpumask_next_wrap().
* Improve diagnostic output with character task states.
* Add rcutorture.nwriters parameter to allow lightweight stall testing,
and rcutorture.stall_only to make doing so easier.
* Test an RCU Tasks Trace grace period implying an RCU grace period.
* Make RCU Tasks Trace torturing track reader batches.
* Fix a data race, courtesy of KCSAN.
* Plug a shuffle_tmp_mask memory leak on kthread spawn failure.
|
|
This comment is missing a closing parenthesis:
* The in_atomic_preempt_off() check ensures that we come here holding
* the last preempt_count (which will get dropped once we return to
* __rcu_read_unlock().
This commit therefore adds it at the end of the sentence.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The rcu_read_delay() and srcu_read_delay() functions are suppose to
avoid (or at least to minimize) read-side delays during call_rcu()-based
forward-progress testing. Although rcu_read_delay() does account for
this by testing rcu_fwd_cb_nodelay, it only does so in some cases, and
srcu_read_delay() does not bother at all. And rcutorture testing does
occasionally turn up the MIN_FWD_CBS_LAUNDERED failure when running
forward-progress tests on Tree SRCU flavors.
This commit therefore makes both rcu_read_delay() and srcu_read_delay()
take an immediate exit if rcu_fwd_cb_nodelay is set.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently, rcu_torture_fwd_prog_cr() announces function entry,
but silently exits if rcu_fwd_emergency_stop is set or if there is no
->call() function available. This could cause confusion, leading people
to believe that rcu_torture_fwd_prog_cr() is on the job when it is not.
This commit therefore also announces the early exits from this function.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Currently __this_cpu_inc() is used to increment elements of both
the rcu_torture_count[] and rcu_torture_batch[] arrays. However,
this can fail when the increments can happen in interrupt handlers,
as recently became possible. This commit therefore upgrades the uses
of __this_cpu_inc() to the interrupt-safe this_cpu_inc().
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds the ->get_sp_seq and ->gp_diff fields to the
tasks_tracing_ops structure so that RCU Tasks Trace rcutorture runs will
track Reader Batch.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
An RCU Tasks Trace grace period is supposed to imply an RCU grace period,
and this implication is relied on by BPF. But this is not currently
tested. This commit therefore makes tasks_tracing_torture_read_lock()
sometimes use rcu_read_lock() instead of rcu_read_lock_trace(), thus
testing the required implication.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
This commit adds a stall_only module parameter that shuts off all
rcutorture kthreads other than the RCU CPU stall-warning test kthreads.
The purpose of this is to test production applictions' reactions to
CPU stalls, and with minimal additional overhead. Or you can omit the
stall-warning tests as well and get a heavy no-op, your choice!
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Believe it or not, there are people who would like to run rcutorture
without actually torturing RCU. For example, some people would like to
induce various types of stall warnings without placing any unnecessary
additional overhead on their systems running in production. And
rcutorture provides the stall_cpu, stall_cpu_holdoff, stall_no_softlockup,
stall_cpu_irqsoff, stall_cpu_block, and stall_cpu_repeat module parameters
in order to allow the user to force numerous types of stalls. In addition,
rcutorture provides a great number of other module parameters to allow the
user to reduce other overhead.
But unfortunately, there is no way to turn of the rcu_torture_writer()
portion of this torture test, which on my x86 laptop consumes somewhere
between 40% and 45% of a CPU. Although this is quite lightweight for a
torture test, it is not welcome on systems running production workloads.
This commit therefore adds an nwriters module parameter that defaults
to 1 but can be set to 0 in order to disable the rcu_torture_writer()
portion of the torture test, but that cannot be set to any other value
(that is what the fakewriters module parameter is for!). This reduces
the overhead to well under 1% of a CPU, which is much more likely to be
compatible with production workloads.
Reported-by: Breno Leitao <leitao@debian.org>
Reported-by: Puranjay Mohan <puranjay@kernel.org>
Reported-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Use the kernel's standard symbolic task-state representation instead of
printing raw hexadecimal task-state values.
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Co-developed-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The rcu_torture_preempt() function uses cpumask_next(), and if that
returns an out-of-bounds result, re-invokes cpumask_next() on -1.
Which is exactly what cpumask_next_wrap() does. This commit therefore
saves a couple of lines by instead using cpumask_next_wrap().
This was reported by metacode when asked to look for opportunities
to use cpumask_next_wrap() in kernel/rcu.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Although rcutorture has long had the irqreader module parameter, this
parameter results only in RCU readers in softirq handlers, specifically,
timers. This commit therefore uses smp_call_function_single() to test
RCU readers in real hardware interrupt handlers, thus providing the full
effect from the irqreader module parameter.
However, consistency/debug checks must account for the possibility that
the smp_call_function_single() handler function is directly invoked
from the idle loop, in which case, for example, in_hardirq() will
return false. This commit uses a per-CPU variable to record being in
the rcu_torture_irq() smp_call_function_single() handler function.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
