| Age | Commit message (Collapse) | Author |
|---|
|
Rust's core::fmt keeps a flag in the low bit of a pointer:
r2 = *(u64 *)(r1 + 0)
r3 = r2
r3 &= 1
r2 >>= 1
The verifier rejects it:
r0 &= 8
R0 bitwise operator &= on pointer prohibited
Negation and byte swap of a pointer already produce a number when
allow_ptr_leaks is set. Do the same for bitwise ops, shifts, *=, /= and
%=, 64-bit and 32-bit. The result is an unknown number. With CAP_PERFMON
the program can store the pointer and load it back as a number already.
+= and -= are not changed: they keep the pointer, 32-bit += is rejected.
Pointers that allow no arithmetic and pointers that may be NULL are
still rejected.
Two tests in verifier_value_illegal_alu store through the result of
&= and /=. They are rejected at the store now. Update expected messages.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261002124714.180012-2-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
A freplace attach claims the target prog by bumping
tgt_prog->aux->freplace_link_cnt and setting tr->extension_prog.
bpf_arch_text_poke() then makes the extension take effect. If the
poke fails, the claims are never released: the attach unwinds
through bpf_link_cleanup(), which clears link->prog, so
bpf_trampoline_unlink_prog() never runs.
Drop the link count under ext_mutex on the error path, and set
tr->extension_prog only after the poke succeeded. The count is
still bumped before the poke: it blocks prog_array updates while
the entry is patched.
Fixes: d6083f040d5d ("bpf: Prevent tailcall infinite loop caused by freplace")
Fixes: be8704ff07d2 ("bpf: Introduce dynamic program extensions")
Suggested-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Yuan Chen <chenyuan@kylinos.cn>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://patch.msgid.link/20260928091121.219333-1-chenyuan_fl@163.com
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull probes fixes from Masami Hiramatsu:
- fprobe: Use guard(rcu_sched_notrace) and check rcu_is_watching()
Exit handlers early if !rcu_is_watching() to prevent potential
use-after-free during unregistration in idle/quiescent states. Switch
to guard(rcu_sched_notrace) to avoid fast-path lockdep overhead and
recursion while ensuring safe grace period synchronization.
- kprobes: Skip disarmed probes when checking optkprobe overlap
Continue past disarmed or unprepared probes in get_optimized_kprobe()
to find active optimized probes. This avoids overwriting active jump
displacements which can lead to panic.
* tag 'probes-fixes-v7.3-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
kprobes: Skip disarmed probes when checking optkprobe overlap
fprobe: Use guard(rcu_sched_notrace) and check rcu_is_watching()
|
|
Currently, kprobe_optimizer() holds kprobe_mutex, text_mutex, and
cpus_read_lock() simultaneously while executing synchronize_rcu_tasks().
Under PREEMPT_LAZY and server workloads with long-running CPU-bound kernel
tasks or heavy cgroup writeback loops, synchronize_rcu_tasks() can block
for seconds to minutes. Because text_mutex and cpus_read_lock() are held
during this entire wait, any concurrent static key updates, module
loading/unloading, CPU hotplug, or tracing updates stall, frequently
triggering hung-task detector panics.
Instead of penalizing the kernel preemption path with invasive hooks and
global hash lookups, decouple kprobe jump optimization from synchronous
waiting entirely:
1. Replace synchronize_rcu_tasks() with call_rcu_tasks(). Locks
(text_mutex and cpus_read_lock()) are held only for the brief moment
needed to patch instructions via arch_unoptimize_kprobes() and
arch_optimize_kprobes() (microseconds), and are completely released
while waiting for the Tasks RCU grace period.
2. Introduce a fixed ring of generations (optprobe_gens[OPTPROBE_GEN_MAX])
to avoid any dynamic memory allocation (kmalloc) or -ENOMEM failure
modes.
3. The last generation slot in the array is reserved as a "waiting room"
and is not dispatched to RCU until another in-flight generation has
finished. While a generation is waiting for its Tasks RCU grace period,
any newly registered or unregistered probes accumulate in the waiting
room generation without blocking or requiring additional slots.
4. When the Tasks RCU callback fires, it marks the generation as ready
and wakes up the optimizer thread to finalize optimization (poking
the jump instructions) and free cleaned probe slots. Once finalized,
the generation is marked idle, allowing the waiting room generation
to be dispatched next.
5. Flushing via wait_for_kprobe_optimizer() waits asynchronously for all
in-flight and queued generations to drain without stalling other
kernel subsystems.
On non-preemptive kernels or configs where CONFIG_TASKS_RCU=n,
call_rcu_tasks() transparently aliases to call_rcu(), preserving full
portability.
Link: https://lore.kernel.org/all/179091652953.526694.3402334249215311330.stgit@devnote2/
Assisted-by: LLM
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
In preparation for converting the kmalloc family of allocators to the
type-aware kmalloc_obj family, we need to make sure that the returned
type from the allocation matches the type of the variable being
assigned. (The kmalloc family returns "void *", which can be implicitly
cast to any pointer type.)
The assigned type is "const char **", but the converted allocation type
would be "char **", which is the same type without the const qualifier.
As there is no general way to safely add const qualifiers, take the size
from the assignment target instead. No change in allocation size
results.
Build tested ARCH=x86_64 allmodconfig with GCC 16.2.0:
kernel/trace/trace_probe.o
Link: https://lore.kernel.org/all/20260917211022.i.677-kees@kernel.org/
Assisted-by: LLM coccinelle
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
This enabled knob is a disgusting terrible hack. It has rolled its
own read/write pair since 2007, writing '1' or '0' into a three byte
buffer by hand just to print a single character, with an XXX comment
begging debugfs for write callbacks on bool files.
DEFINE_DEBUGFS_ATTRIBUTE showed up in 2016 and does exactly that, so
the disgusting terrible hack has outlived its excuse for nine years.
Kill it, and the stale comment with it.
The behavior does not change, except the write only accepts 0/1 now
instead of y/n/on/off, and nothing uses anything else.
Link: https://lore.kernel.org/all/20260818015318.24103-1-include@grrlz.net/
Signed-off-by: Bradley Morgan <include@grrlz.net>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
Cross-merge BPF and other fixes after downstream PR.
Conflicts:
arch/arm64/net/bpf_jit_comp.c
arch/x86/net/bpf_jit_comp.c
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Pull bpf fixes from Alexei Starovoitov:
- Fix overflow of backward jump offset in constant blinding
(Alexei Starovoitov)
- Fix packet range of packet pointers sharing an id when var_off
tightens umax of one pointer and not the other (Alexei Starovoitov)
- Fix objects stuck in free_by_rcu_ttrace list of bpf memalloc
(Alexei Starovoitov)
- Fix use-after-free of progs detached from busy trampolines: wait for
an RCU tasks grace period before freeing trampoline progs, and patch
detached progs out of trampoline images that are still in use
(Florent Revest)
- Hold map BTF for the memory allocator destructor record to fix UAF in
deferred bpf_mem_alloc destruction (Kumar Kartikeya Dwivedi)
- Fix missing migration protection in resizable hashtab
lookup_and_delete batch operation (Ömer Mete Kaya)
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf:
bpf: Fix missing migration protection in __rhtab_map_lookup_and_delete_batch()
selftests/bpf: Add a test for objects stuck in free_by_rcu_ttrace
bpf: Fix objects stuck in free_by_rcu_ttrace
bpf: Factor out __do_call_rcu_ttrace()
selftests/bpf: Test packet range of pointers sharing an id
bpf: Fix packet range of pointers sharing an id
selftests/bpf: Detach a trampoline prog while a task sleeps before it
bpf: Skip detached progs in trampoline images that are still in use
bpf: Wait for an RCU tasks grace period before freeing trampoline progs
bpf: Hold map BTF for the memory allocator destructor record
bpf: Fix overflow of jump offset in constant blinding
|
|
With commit 2125c0034c5d ("cgroup/cpuset: Make cpuset hotplug processing
synchronous"), the cpuset hotplug operation becomes synchronous. That
commit also removes the code that handles race between cpuset_hotplug_work
and cpu hotplug notifier with the assumption that race is now gone.
Later commit 7a0aabd9ce69 ("cgroup/cpuset: Always use cpu_active_mask")
updates the cpuset code to always use cpu_active_mask instead of
cpu_online_mask in some places including guarantee_online_cpus() which
is renamed to guarantee_active_cpus() in that commit.
In the case of CPU offline operation, cpu_active_mask is updated
first in sched_cpu_deactivate() to remove the offline CPU before
cpuset_handle_hotplug() is called to update the effective_cpus of the
affected cpusets. The cpu_online_mask is updated after that near the end
of the offline operation to remove the offline CPU. As a result, the race
comes back and the top cpuset may not have any active CPU leading to NULL
pointer dereference during the race window when guarantee_active_cpus()
is called after cpu_active_mask is updated to remove the CPU to be torn
down but before cpuset_handle_hotplug() is able to properly update the
effective_cpus of the top cpuset.
Fix this by adding back the NULL cs check to avoid this problem.
Fixes: 7a0aabd9ce69 ("cgroup/cpuset: Always use cpu_active_mask")
Cc: stable@vger.kernel.org # v6.16+
Reported-by: Farhad Alemi <farhad.alemi@berkeley.edu>
Link: https://lore.kernel.org/lkml/CA+0ovChh3VjsKN1g+ZGjwwY2fGTpP7uD+aCCByLj5Qbymw=bfQ@mail.gmail.com
Signed-off-by: Waiman Long <longman@redhat.com>
Tested-by: Farhad Alemi <farhad.alemi@berkeley.edu>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Add a hook to allow arch code disable hibernation. On arm64, with an RMM
active, there could be pages that are delegated to the "Realm" security
state. Any access to them are fatal (e.g., hibernate save image or
restoring an image to a delegated page). pKVM host also has its
restrictions on enabling hibernation.
Add a weak arch hook to let the architectures have a say.
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: Len Brown <lenb@kernel.org>
Cc: Pavel Machek <pavel@kernel.org>
Cc: linux-pm@vger.kernel.org
Cc: Sudeep Holla <sudeep.holla@kernel.org>
Suggested-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Sudeep Holla <sudeep.holla@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
|
|
This fixes a regression when a console option includes ':'. Commit
7640f1a44eba ("printk: Add match_devname_and_update_preferred_console()")
introduced console=DEVNAME:0.0 hardware style addressing by looking for a
colon. If the colon is part of an option the name is handled as devname
instead of ttyname.
Fix by handling the options first which will add a NULL terminator to the
string. Note that parsing the options first means that also idx parsing
needs changing. Handle the idx parsing by refactoring the code (thanks to
Petr Mladek).
Signed-off-by: David Engraf <david.engraf@sysgo.com>
Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: John Ogness <john.ogness@linutronix.de>
Link: https://patch.msgid.link/20260924070153.250783-2-david.engraf@sysgo.com
[pmladek@suse.com: Updated the commit message as suggested by Tony Lindgren]
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
specialize_kfunc() currently updates the canonical kfunc descriptor in
place. Different call sites therefore cannot select different
specializations of the same kfunc, and the selected target depends on
verification order.
Keep canonical descriptors unchanged for verifier lookups. Specialize a
copy for each call site, then reuse or append an immutable target
descriptor and store its index in the finalized call instruction.
Allow space for one canonical and one specialized target per kfunc. This
is conservative because most kfuncs do not specialize. Adding specialized
kfunc versions does not require resorting the array because lookups are
only used for deduplication and func_model lookup. Deduplication for
specialized kfuncs is handled by the specialization process itself,
while all specializations of a function share the same proto and
func_model.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-7-emil@etsalapatis.com
|
|
Currently, the verifier sort the kfunc desc table twice: Once during
kfunc collection by function ID, useful during verification, and
once by immediate address, useful during JIT bytecode lowering
time. The latter sort can be removed by storing in the instruction
the kfunc desc array index the desc is in and using that instead.
Modify the JITs to follow the same convention.
This change simplifies the subsequent patch that adds per-call site
function specialization.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-6-emil@etsalapatis.com
|
|
The bpf_arena_alloc_pages() function currently only allocates pages
inside a spinlock critical section with IRQs off. This forces the use of
alloc_pages_nolock() in the BPF allocator, even when the caller is a
sleepable BPF function. This in turn causes allocation failures even in
cases where falling into the allocator slow path and possibly sleeping
would eventually succeed. This can be triggered consistently by heavy
BPF arena users like scx.
Allocate the arena pages before taking the critical section and pass
whether the caller can sleep to bpf_alloc_pages(). This lets sleepable
callers use the blocking allocator while non-sleepable callers retain
the no-lock allocation behavior.
Fixes: b8467290edab ("bpf: arena: make arena kfuncs any context safe")
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-4-emil@etsalapatis.com
|
|
bpf_alloc_pages() currently decides whether it may use the
blocking page allocator from the current execution context
alone. Let callers further restrict that choice by passing
whether their context is sleepable.
Use the blocking allocator only when both the caller and
runtime context allow sleeping. Add __GFP_RETRY_MAYFAIL so
this path can reclaim without invoking the OOM killer when
the allocation is charged to another memcg. Existing
non-sleepable callers retain the no-lock allocation behavior.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-3-emil@etsalapatis.com
|
|
bpf_map_alloc_pages() does not use its map argument. Storing allocated
pages in an array also forces arena callers to allocate a separate
pointer array.
Expose the single-page allocator as bpf_alloc_page(), rename the bulk
helper to bpf_alloc_pages(), and return bulk allocations through an
llist using page->pcp_llist. Add bpf_free_pages() to safely release all
pages remaining on such a list.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-2-emil@etsalapatis.com
|
|
bpf_mem_cache_free_rcu() uses this_cpu_ptr() which requires migration
to be disabled. All callers of rhtab_delete_elem() disable migration
except __rhtab_map_lookup_and_delete_batch(), which calls it under
rcu_read_lock() only.
On CONFIG_PREEMPT_RCU, rcu_read_lock() does not disable preemption or
migration, so the task can migrate between CPUs during the delete loop,
causing this_cpu_ptr() to trigger:
BUG: using smp_processor_id() in preemptible [00000000] code
Fix by wrapping the delete loop in migrate_disable()/migrate_enable()
in __rhtab_map_lookup_and_delete_batch(), matching the migration
protection that the other callers already provide.
Fixes: 818e00848227 ("bpf: Implement iteration ops for resizable hashtab")
Reported-by: syzbot+fd7e415d891073b83e1f@syzkaller.appspotmail.com
Signed-off-by: Ömer Mete Kaya <omermetekaya0@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://patch.msgid.link/20260929081609.557899-1-omermetekaya0@gmail.com
Closes: https://syzkaller.appspot.com/bug?extid=fd7e415d891073b83e1f
|
|
To confirm the max nesting level of local_interrupt_disable() works, a
kunit test is added to the whole test suite.
Note that when DEBUG_PREEMPT=y, it'll generate a warning which is
expected.
Suggested-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Link: https://patch.msgid.link/20261001082519.16195-6-boqun@kernel.org
|
|
poll_state_synchronize_rcu(mm->futex.phash.batches) is used by
futex_ref_drop() to check that a grace period has passed since the
current hash was published. This relies on batches referencing a grace
period which started after the hash pointer was assigned.
__futex_pivot_hash() sets mmph->batches before it replaces mmph->hash:
scoped_guard(rcu) {
mmph->batches = get_state_synchronize_rcu();
rcu_assign_pointer(mmph->hash, new);
}
The scoped_guard(rcu) doesn't stop new grace periods from starting, and
if one starts between those two assignments, futex_ref_drop() can move
forward while a reader still holds a pointer to the old hash.
Fix things by setting mmph->batches after assigning mmph->hash. The
scoped_guard(rcu) isn't needed, so let's drop that as well.
Fixes: 56180dd20c19 ("futex: Use RCU-based per-CPU reference counting instead of rcuref_t")
Assisted-by: kres
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Link: https://patch.msgid.link/20261001135022.2220288-1-mason@kernel.org
|
|
Currently, refcount_interrupt_test is built unconditionally when
CONFIG_KUNIT is enabled, causing it to run unexpectedly during boot.
Fix this by introducing CONFIG_REFCOUNT_INTERRUPT_KUNIT_TEST so the
test can be configured independently, following standard kunit
practices.
Fixes: 07a88e2bcd5b ("irq: Add KUnit test for refcounted interrupt enable/disable")
Signed-off-by: Kuan-Wei Chiu <visitorckw@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Lyude Paul <lyude@redhat.com>
Reviewed-by: Radu Rendec <radu@rendec.net>
Reviewed-by: Boqun Feng <boqun@kernel.org>
Link: https://patch.msgid.link/20261001082519.16195-2-boqun@kernel.org
|
|
The Braille console is not registered in console_list. Instead, it is
integrated with the virtual terminal (VT) and shows what is displayed
on the terminal. It writes the data using con->write*() callback
of the associated serial console driver.
braille_write() is called from the VT code under console_lock().
The associated serial console driver can be converted to the NBCON API
though. The situation is similar to flushing nbcon consoles in the legacy
loop when some boot consoles are still registered, see
nbcon_legacy_emit_next_record().
But there is a big difference though. braille_write() is not directly
called from the code paths flushing registered consoles. The VT code
expects that braille_write() succeeds. It does not replay the message
when it can't acquire the ownership. As a result, braille_write():
+ must try harder to get the ownership.
+ has to be synchronized only against non-printk serial console
which depend nbcon_device_try_acquire() using NBCON_PRIO_NORMAL.
Let's look at it from another side and try to simulate the original
locking using the NBCON API:
1. Take con->device_lock(), aka the port->lock in the legacy serial
console driver.
2. Acquire nbcon context to provide some synchronization for a panic()
context. Use NBCON_PRIO_NORMAL because it contends only with
nbcon_device_try_acquire() users. Do it in a busy loop. It should
always succeed when con->device_lock() succeeded because all other
users do the same. The only exception is when the context get acquired
by a CPU handling panic.
3. In panic, disable interrupts and try to acquire the nbcon context.
Use NBCON_PRIO_PANIC. And try even an unsafe takeover because
otherwise the Braille console won't see the text shown during panic().
It is similar to the oops_in_progress/trylock handling in the legacy
serial console driver.
Finally, avoid the newline prepending logic in the existing serial console
drivers when they are used as a Braille console. As explained above,
the Braille console shows the last modified line on the terminal (VT).
braille_write() is called when single characters are added. Most
messages are not ended by newline. Anyway, the VT code does not have
logic to reply partially printed messages.
Fixes: 13189fa73afa ("printk: nbcon: Rely on kthreads for normal operation")
Reviewed-by: John Ogness <john.ogness@linutronix.de>
[pmladek@suse.com: Remove white space changes in braille_register_console(). Typo fix.]
Link: https://patch.msgid.link/20261001140727.124398-2-pmladek@suse.com
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
On x86, an optkprobe at A replaces five bytes with a jump. If a disabled
probe B is at A+2, get_optimized_kprobe() stops at B when arming a new
probe C at A+4. It leaves A optimized:
A A+1 A+2 A+3 A+4
A's jump | e9 | d0 | d1 | d2 | d3 |
after C | e9 | d0 | d1 | d2 | cc |
The INT3 for C overwrites the last byte of A's jump displacement, so
execution can jump to the wrong address. B can have prepared optinsns
while disarmed, but has no jump to unoptimize.
Continue past disarmed and unprepared probes to find the active optimized
probe before arming a probe in its jump.
Link: https://lore.kernel.org/all/20260930053618.104498-1-leon.hwang@linux.dev/
Fixes: afd66255b9a4 ("kprobes: Introduce kprobes jump optimization")
Cc: stable@vger.kernel.org
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
Initially (and perhaps forever), call_srcu() will not be available for
atomic srcu_struct structures. There is therefore no reason to transition
such a structure to big, because the main purpose of such a transition
is to reduce lock contention for concurrent SRCU callback queueing.
This commit therefore adds an is_atomic parameter to both the
check_init_srcu_struct() and init_srcu_struct_fields() functions, which
suppresses the initialization-time transition to big that is enabled by
default on large systems.
It will still be possible to force a transition using rcutorture as a
destructive test. This might (or might not) be adjusted later.
[ paulmck: Apply feedback from Kunwu Chan. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The new SRCU_READ_FLAVOR_ATOMIC prohibits sleeping in both readers and
grace-period waits in order to permit waiting on an SRCU grace period
within an OOM notifier. This means that srcu_advance_state() cannot
acquire ->srcu_gp_mutex in this case, because mutex_lock() can sleep.
This commit therefore adds an is_atomic parameter to srcu_advance_state().
When this parameter is false, current behavior is maintained, in other
words, ->srcu_gp_mutex is acquired and released as before. But when this
is_atomic parameter is false, the caller is responsible for excluding
other callers.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
In non-preemptible kernels, a Tiny SRCU grace period implies an RCU
grace period because any context switch suffices. But in preemptible
kernels, it is possible for a Tiny SRCU grace period to elapse without
a corresponding RCU grace period.
Which was OK until RCU Tasks Trace was re-implemented in terms
of SRCU-fast, which in Tiny SRCU is implemented as SRCU, which is
already fast. And RCU Tasks Trace grace periods are required to imply
RCU grace periods. This commit therefore adds a synchronize_rcu(),
but only in preemptible kernels.
Because preemptible Tiny SRCU is not in mainline, this added call to
synchronize_rcu() will not slow anything down: The comparison would
instead be with TREE SRCU. But if this added call ever becomes a problem,
the Tiny SRCU srcu_struct structure could track whether or not this is for
SRCU-fast, and to add the synchronize_rcu() only in the SRCU-fast case.
However, at the moment, this is seen as unnecessary complexity.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
call_rcu() and call_srcu() are now safe to invoke from NMI, but
rcutorture never does, leaving the deferral path untested.
Add an ->nmi_capable flag to rcu_torture_ops. For flavors that set it,
arm a per-CPU hardware perf counter whose overflow handler submits a
callback via ->call(). The handler acts only when in_nmi(), so only a
genuine NMI exercises the deferral path. One preallocated callback per
CPU is kept in flight, guarded by an atomic, to avoid allocating in NMI.
The counter uses a fixed sample period rather than a frequency: a
frequency-based event sets TICK_DEP_BIT_PERF_EVENTS and would pin the tick
for the whole run on NO_HZ_FULL kernels.
Report the count issued from NMI ("nmi-calls:") and the count invoked
("nmi-cbs:"). rcu_torture_cleanup() disables the counters and then calls
cb_barrier(), which drains every deferred callback, so the two counts must
then match; a mismatch fails the test. This relies on
rcu_barrier()/srcu_barrier() flushing deferred callbacks, as added earlier
in the series.
Set ->nmi_capable on the NMI-safe flavors: rcu, srcu, srcud, and
tasks-tracing (call_srcu() under the hood). Tasks and Tasks Rude are left
alone, as call_rcu_tasks_generic() is not yet NMI-safe.
Enabled by default; the nmi_calls parameter disables it, which helps rule
NMI handling in or out when triaging a failure. Requires
CONFIG_PERF_EVENTS and a hardware PMU: without one nothing is issued from
NMI and the end-of-test check compares zero against zero, so a pass does
not by itself mean the path ran. That is the case under kvm.sh, which
boots qemu with -cpu kvm64 and no vPMU; rcu_torture_nmi_cleanup() says so
on the console.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Vanilla RCU readers, when either non-preemptible or preemptible with
priority-boosting enabled, can handle heavy overload conditions.
Other RCU implementations are more fragile. This commit adds an
rcu_torture_ops structure field named ->rdrs_handle_load that is set
for non-fragile configurations of vanilla RCU, but cleared otherwise.
It also refrains from running fragile readers concurrently with overload
testing, thus avoiding false-positive overload-testing failures.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Give Tiny call_srcu() the same treatment as Tree SRCU. When interrupts
are disabled and the scheduler is up, stage the callback on the
srcu_struct's lockless list for an irq_work to re-issue later. Tiny SRCU
is uniprocessor, so there is no CPU-offline drain. A draining flag drops
a deferring call_srcu() that re-enters mid-drain (unless from an NMI), as
in Tree SRCU; such a drop can strand state the caller tied to the callback,
not just the callback itself.
Interrupts stay off for the whole batch. TINY_SRCU implies !SMP, where
arch_irq_work_has_interrupt() is false, so the drain always waits for the
tick and a batch is whatever one tick's worth of interrupts-disabled
call_srcu()s staged. Unlike the other three flavors srcu_do_enqueue() here
has no debug_rcu_head_queue(), so nothing reports a double call_srcu();
termination of the drain rests on srcu_do_enqueue() clearing ->next, and
the callback list self-links at the tail exactly as a double call_srcu()
made it before.
srcu_barrier() (now out of line) and cleanup_srcu_struct() drain the
deferred list first, so a deferred callback is re-issued onto the callback
list and invoked by the grace-period work that cleanup_srcu_struct()
flushes, rather than stranded on a soon-to-be-freed srcu_struct.
cleanup_srcu_struct() also syncs ->defer_iw, since that irq_work is
embedded in the srcu_struct the caller is about to free.
The draining flag is global rather than per-srcu_struct: a re-entrant
call_srcu(B) inside a drain of A raises B's own ->defer_iw, whose drain can
stage back onto A, so a per-srcu_struct flag would not break the chain.
The cost is that a drain of A also drops a non-NMI call_srcu() to any
other srcu_struct for its duration.
Gated by CONFIG_RCU_DEFER like Tree SRCU, though the srcu_struct members
are unconditional.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
call_srcu() has the same constraint as call_rcu():
srcu_gp_start_if_needed() enqueues under raw_spin_lock_irqsave() and may
walk the srcu_node tree, as do callback invocation and grace-period work,
so a call_srcu() with interrupts already disabled can race an operation in
flight on this CPU. call_rcu_tasks_trace() is call_srcu() under the hood,
so a sleepable BPF program freeing an object can reach this.
Defer as call_rcu() does: stage the callback on the srcu_data's
->defer_cbs, chain that srcu_data onto a per-CPU list, and raise a per-CPU
irq_work that re-issues it straight to the enqueue helper, never back
through __call_srcu(). The irqs-enabled path is unchanged; as for
call_rcu() the gate is bare irqs_disabled(), so callers that merely hold
interrupts off are deferred too and pay one irq_work hop, including
call_rcu_tasks_trace() from the BPF memalloc irq_work.
The irq_work is per-CPU rather than per-srcu_struct and statically
initialized, so deferral never runs check_init_srcu_struct(); it is
IRQ_WORK_INIT_HARD as for call_rcu(). srcu_barrier() flushes it first, and
rcutree_migrate_callbacks() calls srcu_offline_drain() for an outgoing CPU.
cleanup_srcu_struct() drains before its "just leak it" early returns, and
srcu_module_going() before freeing any ->sda, since a staged srcu_data left
chained on a per-CPU list would dangle. Staging is two steps, the callback
onto
->defer_cbs and then the srcu_data onto the per-CPU list, so a flusher can
find the per-CPU list empty while a callback whose call_srcu() has not
returned sits on ->defer_cbs; the staging CPU's own irq_work takes that
one.
The per-CPU srcu_defer ->lock, not any srcu_data's, is held with interrupts
off across the whole nested drain: the chain of srcu_datas staged on that
CPU and, for each, its callbacks, with srcu_do_enqueue() taking that
srcu_data's ->lock and possibly starting a grace period for every one.
That is what serializes the drainers. The bound is as for call_rcu(): what
one interrupts-disabled region could stage, normally a single callback, or
whatever accumulates before the tick where arch_irq_work_has_interrupt() is
false. Both lists are drained newest-first; nothing depends on call_srcu()
ordering.
The drain clears ->next before re-issuing, which bounds a node self-linked
by a double call_srcu(); a longer cycle is not bounded, and a double
call_srcu() stays undefined, as for call_rcu().
A callback deferred past the CPUHP_AP_SMPCFD_DYING irq_work flush leaves
that CPU's srcu_defer ->iw claimed with its self-IPI lost, as for
call_rcu(). srcu_offline_drain() still re-issues the callback, but the
irq_work cannot be un-queued, and here the claim is shared by every
srcu_struct on the CPU.
As in call_rcu(), the re-issue runs with interrupts disabled and can be
re-entered by instrumentation, so a per-CPU flag, set only while that CPU
is inside its own irq_work drain, drops a deferring call_srcu() seen
mid-drain unless it comes from an NMI. Such a drop can strand state the
caller associated with the callback, not just the callback itself.
Staging records only the callback, so a deferred expedited call_srcu()
completes as a normal grace period. Only srcu_expedite_current() can hit
that, and only when invoked with interrupts already disabled.
Gated by CONFIG_RCU_DEFER, though the srcu_data members and the per-CPU
srcu_defer are unconditional. Under CONFIG_PROVE_RCU, warn if the direct
path is reached from an NMI.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Give Tiny call_rcu() the same treatment as Tree RCU. When interrupts are
disabled and the scheduler is up, stage the callback on a lockless list
that an irq_work re-issues later. One global list and irq_work suffice
since Tiny RCU is uniprocessor, and there is no CPU-offline drain.
The re-issue runs with interrupts disabled and can be re-entered by
instrumentation, so a draining flag drops a deferring call_rcu() seen
mid-drain (unless from an NMI), as in Tree RCU. Gated by CONFIG_RCU_DEFER,
though the deferral state is unconditional.
Interrupts stay off for the whole batch, but the re-issue is a tail append
with no locks. TINY_RCU implies !SMP, where arch_irq_work_has_interrupt()
is false, so the drain always waits for the tick and a batch is whatever
one tick's worth of interrupts-disabled call_rcu()s staged. As in Tree
RCU the drain clears ->next before re-issuing, which bounds a node
self-linked by a double call_rcu(): rcu_do_enqueue()'s duplicate path
returns without clearing it. A longer cycle is not bounded; a double
call_rcu() stays undefined.
The idle-task reschedule moves out of the enqueue helper so that a drain
does it once for the batch rather than once per callback, which would
otherwise take the runqueue lock N times with interrupts disabled.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
RCU's per-CPU callback list is only touched with interrupts disabled: the
enqueue runs under local_irq_save() (and the nocb locks when offloaded),
as do callback invocation and grace-period work. A call_rcu() that
arrives with interrupts already disabled, whether from an NMI or from
instrumentation that re-enters RCU, can interrupt one of those and corrupt
the list or deadlock.
Defer instead: stage the callback on a per-CPU llist and raise an irq_work
that re-issues it once interrupts are on, straight to the enqueue so it
cannot defer again. The gate is bare irqs_disabled(), so callers that
merely hold interrupts off are deferred too and pay one irq_work hop.
Skip it while the scheduler is down (RCU_SCHEDULER_INACTIVE): irq_work is
not usable that early, rcu_init() already calls call_rcu(), and the per-CPU
deferral state is not initialised until rcu_init_one() runs later in it.
rcu_barrier() drains every CPU's ->defer_head before it scans the lists,
and rcutree_migrate_callbacks() drains an outgoing CPU's. A drain
re-issues onto the draining CPU, so a barrier moves other CPUs' staged
callbacks onto
its own ->cblist; call_rcu() promises no CPU affinity for invocation.
->defer_lock is held across llist_del_all() and the whole re-issue so the
drainers
serialize: one that finds the list empty can conclude that everything
staged before it is already on a callback list. Interrupts stay off for
the batch. Where the arch has an irq_work self-IPI that is what one
interrupts-disabled region could stage, normally a single callback; where
arch_irq_work_has_interrupt() is false the drain waits for the tick, so
several regions can accumulate first.
The drain clears ->next before re-issuing. A double call_rcu() on a head
that is already debug-object-active self-links the staged node, and
rcu_do_enqueue()'s duplicate path returns without clearing it, so the
drain would spin. A re-add behind other staged callbacks makes a longer
cycle, which that does not bound; a double call_rcu() stays undefined.
llist_del_all() yields newest-first, so a batch is re-issued in reverse
call order; nothing depends on call_rcu() ordering. The re-issue drops
the lazy hint, since staging records only ->func, so a deferred callback
loses its batching on CONFIG_RCU_LAZY. kasan_record_aux_stack() moves to
__call_rcu_common() so a use-after-free report names the caller rather
than the irq_work.
The re-issue runs with interrupts disabled, so instrumentation on the
enqueue path can re-enter call_rcu(), stage another callback and re-raise
the irq_work, livelocking the drain. A per-CPU flag guards it: a deferral
that arrives while this CPU is draining, and is not from an NMI, is
dropped. The WARN_ONCE() is under CONFIG_PROVE_RCU, so a production kernel
drops it silently. That leaks the callback and can strand state
the caller tied to it, since a one-shot flag only the callback clears
never resets, but the alternative is an unbounded loop.
A callback deferred past the CPUHP_AP_SMPCFD_DYING irq_work flush leaves
->defer_work claimed with its self-IPI lost. rcutree_migrate_callbacks()
still re-issues the callback, but the first deferral after that CPU comes
back raises no IPI and waits for the next irq_work there, or for
rcu_barrier(). Unqueueing an irq_work is not something the API offers.
The irq_work is IRQ_WORK_INIT_HARD so the re-issue stays prompt on
PREEMPT_RT, where a non-HARD irq_work runs in a kthread that can be
delayed under load. A hidden CONFIG_RCU_DEFER gates the deferral code and
its IRQ_WORK dependency, though the rcu_data members are unconditional;
without it call_rcu() enqueues directly as before. Under
CONFIG_PROVE_RCU, warn if the direct path is reached from an NMI.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The WARN_ON() added by commit 78a38cbf6f20 ("srcu: Queue sdp->work
when the delay timer is successfully deleted") uses rcu_segcblist_n_cbs()
to detect callbacks that srcu_barrier() failed to wait for. However, the
->len counter is decremented only at the end of srcu_invoke_callbacks(),
after the invoking loop has finished. Since srcu_barrier() can return
right after the barrier callback is invoked, cleanup_srcu_struct() can see
a non-zero n_cbs even though the cblist is already physically empty,
falsely triggering the WARN_ON() together with a still-pending delay_work
timer.
This can be triggered as follows, as seen in the syzbot report against
kvm_destroy_vm() -> cleanup_srcu_struct(&kvm->srcu):
1. call_srcu(&kvm->srcu, &bus->rcu, __free_bus) starts SRCU grace
period GP1.
2. GP1 ends: a delay timer is armed and sdp->work is queued, but
sdp->work has not run yet.
3. Another call_srcu(&kvm->srcu, &bus->rcu, __free_bus) call invokes
srcu_segcblist_advance(), which moves the GP1 callback to
RCU_DONE_TAIL, and starts SRCU grace period GP2.
4. srcu_barrier() is called. It queues its barrier callback after the
GP2 callback and waits for srcu_invoke_callbacks() to invoke it.
5. GP2 ends: another delay timer is armed, and the sdp->work queued in
step 2 begins to run. Its srcu_invoke_callbacks() call invokes
srcu_segcblist_advance() again, moving the GP2 and barrier callbacks
to RCU_DONE_TAIL as well, and then invokes all of them. However,
rcu_segcblist_add_len(), which updates srcu_cblist's ->len, has not
run yet at this point.
6. srcu_barrier() returns once its callback has been invoked, and
cleanup_srcu_struct() starts running. It finds the delay timer
armed in step 5 still pending and srcu_cblist's ->len still
non-zero (because step 5 has not reached rcu_segcblist_add_len()
yet), and WARN_ON() fires even though every callback has actually
been invoked.
Use rcu_segcblist_empty(), which checks the actual head of the cblist,
instead of rcu_segcblist_n_cbs(), which checks the racy ->len counter.
Callbacks that have genuinely not been invoked yet still leave the list
non-empty, so the WARN_ON() still catches callers that skip srcu_barrier()
or queue callbacks after it.
Link: https://lore.kernel.org/rcu/e6350377085ddd85d6ef00d8e9a67bd50c762d3c@linux.dev/T/#t
Reported-by: syzbot+d4faf7db59e11f6fd1ab@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=d4faf7db59e11f6fd1ab
Fixes: 78a38cbf6f20 ("srcu: Queue sdp->work when the delay timer is successfully deleted")
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Sunho Park <shpark061104@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Tested-by: Sean Christopherson <seanjc@google.com>
|
|
cgroup_e_css() and cgroup_get_e_css() find the effective css by walking
up the hierarchy until they reach a cgroup that has the subsystem
enabled. Per-I/O users such as nvmet need this lookup to be O(1).
Track the effective css in each cgroup and make cgroup_e_css() and
cgroup_get_e_css() use it. When a css is brought online or offline,
update the effective css of its cgroup and of all descendants.
Signed-off-by: Peng Yu <yupeng0921@gmail.com>
Assisted-by: Claude:claude-fable-5 [Claude Code]
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
do_call_rcu_ttrace() returns early when call_rcu_ttrace_in_progress is set
and leaves the objects in free_by_rcu_ttrace. __free_rcu() frees
waiting_for_gp_ttrace only and clears the flag. Hence the objects that
free_bulk() or __free_by_rcu() added while RCU tasks trace GP was in flight
stay in free_by_rcu_ttrace until free_bulk() or alloc_bulk() is called for
the same bpf_mem_cache again, which may never happen. The number of such
objects is not bounded.
Turn call_rcu_ttrace_in_progress into three states:
0 - idle
1 - __free_rcu() is queued
2 - __free_rcu() is queued and free_by_rcu_ttrace got more objects since
do_call_rcu_ttrace() sets 2. __free_rcu() does cmpxchg(1 -> 0) and starts
the next GP when it fails. It cannot clear the flag first and check
free_by_rcu_ttrace later, since bpf_mem_alloc_destroy() frees bpf_mem_cache
without waiting for RCU callbacks when the flag is zero.
Now __free_rcu() queues itself, so the one that didn't see 'draining' may
do call_rcu_tasks_trace() after rcu_barrier_tasks_trace() in
free_mem_alloc(). Queue it under rcu_read_lock() and do synchronize_rcu()
before the barriers. Calling rcu_barrier_tasks_trace() twice works too, but
creating and destroying hash maps in a loop on many cpus slows down to one
free_mem_alloc() per GP and kworkers pile up.
Fixes: 8d5a8011b35d ("bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU.")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20260930095920.601738-3-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Move the part of do_call_rcu_ttrace() that runs after
call_rcu_ttrace_in_progress is set into __do_call_rcu_ttrace().
The next patch will call it from __free_rcu().
No functional change.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20260930095920.601738-2-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Since commit 022ac0750883 ("bpf: use reg->var_off instead of reg->off
for pointers"), find_good_pkt_pointers() sets the range of all packet
pointers sharing an id from the umax of the compared pointer, and
check_packet_access() requires umax + off + size <= range. That assumes
the umax of two such pointers differ by exactly their constant distance.
reg_bounds_sync() breaks it when var_off tightens one umax and not the
other:
r4 &= 0x38
if r4 > 50 goto exit ; umax 50, var_off (0x0; 0x38)
r5 = pkt + r4 ; umax 50
r6 = r5
r6 += 8 ; umax 56, not 58
Comparing r6 with pkt_end sets the range to 56, and the valid 8-byte
load at r5 is rejected (50 + 8 > 56). Comparing r5 sets it to 50, and
the out-of-bounds 1-byte load at r6 - 7, i.e. r5 + 1, is accepted
(56 - 7 + 1 <= 50).
Don't call reg_bounds_sync() on a packet pointer that keeps its id (a
constant was added or subtracted) or its range (an unknown non-negative
value was subtracted), so that var_off cannot tighten its umax. Only
update the 32-bit bounds from var_off: reg_bounds_sanity_check() wants
them constant when the lower half of var_off is, e.g. for pkt + 8.
This relies on nothing else changing the 64-bit bounds of a packet
pointer, which holds today.
var_off of such a pointer is no longer narrowed by its bounds. Adjust
three verifier_align expectations; the low bits, which the alignment
checks use, don't change. veristat on the selftests shows no verdict
changes and +0.8% insns in test_cls_redirect_subprogs.
Fixes: 022ac0750883 ("bpf: use reg->var_off instead of reg->off for pointers")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261001145255.855630-1-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Two things in __cgroup_bpf_query() work only because CGROUP_TCP_SOCK_OPS is
the only struct_ops attach type.
It calls cgroup_bpf_enabled(atype) with an atype that
find_atype_by_struct_ops_id() works out at runtime. That macro is an asm
goto and needs a constant. Today the compiler can see there is only one
value; add a second type and the build breaks with "impossible constraint
in 'asm'". The check is not needed anyway. When the key is off nothing is
attached, so progs[atype] and effective[atype] are empty and the count loop
leaves total_cnt at 0. The other attach types already use that loop with
no such check. Drop the check and the skip_count label.
And find_atype_by_struct_ops_id() matches on type_id alone. An attach type
whose subsystem is not built keeps type_id 0, so a query for type 0 finds
it and returns success with nothing instead of -ENOENT. Skip such slots.
Fixes: 369d9dcd8fb8 ("bpf: Add infrastructure to support attaching struct_ops to cgroups")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Yafang Shao <laoar.shao@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20261001141901.3225830-1-shakeel.butt@linux.dev
|
|
linux/module.h appears in roughly 15k #include directives across the
kernel. This makes it a "hot" header, so it should avoid pulling in
unnecessary definitions.
The header currently includes linux/error-injection.h to obtain the
definition of `struct error_injection_entry`. However, this is unnecessary
because the type is only referenced in the file as a pointer, for which an
incomplete type is sufficient.
Remove the linux/error-injection.h include from linux/module.h and add it
to kernel/module/main.c instead, where
`sizeof(struct error_injection_entry)` is actually needed.
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl
Pull sysctl fixes from Joel Granados:
"Fix sysctl jiffies conversions errors introduced in 2dc164a48e6f
("sysctl: Create converter functions with two new macros")
- Ensure that mult_hz does *not* wrap
- Ensure we pass just the magnitude for the negative branch in
proc_int_k2u_conv_kop"
* tag 'sysctl-7.03-fixes-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl:
time/jiffies: Saturate in mult_hz() instead of wrapping
sysctl: Negate before converting in the int read path
|
|
Active references on the namespaces of an nsproxy are taken when the
nsproxy is installed into a task in switch_task_namespaces() and
copy_namespaces() and dropped again by put_nsproxy() through
deactivate_nsproxy().
But ksys_unshare() calls put_nsproxy() on an nsproxy that was never
installed when set_cred_ucounts() fails. The new namespaces of that
nsproxy go from zero to minus one and the namespaces shared with the
caller lose a reference that belongs to the nsproxy the caller keeps
using:
WARNING: kernel/nscommon.c:171 at __ns_ref_active_put+0x1cd/0x230
nsproxy_ns_active_put
deactivate_nsproxy
ksys_unshare
Afterwards the namespaces the caller lives in aren't listed by listns()
anymore. set_cred_ucounts() only fails when alloc_ucounts() can't
allocate and it only allocates when the real uid of the caller differs
from its effective uid.
Commit cefd55bd2159 ("nsproxy: fix free_nsproxy() and simplify
create_new_namespaces()") separated the two cases on purpose.
nsproxy_free() frees an nsproxy that was prepared but never installed
and that's what a failed setns() uses in put_nsset(). Export it and use
it for a failed unshare() as well.
Fixes: a98621a0f187 ("unshare: fix nsproxy leak in ksys_unshare() on set_cred_ucounts() failure")
Cc: stable@vger.kernel.org # v7.1+
Link: https://patch.msgid.link/20260930-work-mount-fixes-3-v1-16-be34c83956ae@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Perf events with exclude_kernel=1 can be opened without kernel perf
access. However, exclude_kernel does not suppress text-poke sideband
records. Every PERF_RECORD_TEXT_POKE is marked PERF_RECORD_MISC_KERNEL
and contains a raw kernel instruction address.
An unprivileged task can therefore open and mmap a task-local software
event with text_poke=1. Both opening a count-only tracepoint event and
configuring UDP GRO for ESP-in-UDP cause updates to inline static calls;
the observer receives the relocated addresses of the modified instructions.
For a known kernel image, any such address reveals the runtime kernel
text base despite KASLR.
Call perf_allow_kernel() whenever attr.text_poke is set, regardless of
exclude_kernel. Events that neither monitor kernel execution nor request
text-poke records retain their existing permissions.
Fixes: e17d43b93e54 ("perf: Add perf text poke event")
Assisted-by: LLM
Signed-off-by: Zhengchuan Liang <zcliangcn@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/99131354c41e23188f778b92f90363775b482395.1790573390.git.zcliangcn@gmail.com
|
|
perf_iterate_sb() invokes its callback for each matching perf_event on
the CPU and task context, passing a shared caller-allocated event
structure.
perf_event_header__init_id() mutated header->size in place by adding
event->id_header_size, requiring sideband output callbacks to save and
restore header fields across iterations. Three sideband callbacks failed
to save and restore header.size around perf_event_header__init_id():
- perf_event_ksymbol_output()
- perf_event_bpf_output()
- perf_event_text_poke_output()
When multiple events with attr.ksymbol, attr.bpf_event, or
attr.text_poke and sample_id_all are active on the same CPU, each
subsequent event receives a record whose header.size is inflated by all
preceding events' id_header_size values while only a single id_sample is
written, leaving uninitialized ring-buffer bytes at the end of the
record and causing userspace perf to fail with -EFAULT ("Bad address")
when parsing the sample_id trailer.
Similarly, perf_event_mmap_output() set PERF_RECORD_MISC_MMAP_BUILD_ID
in mmap_event->event_id.header.misc when event->attr.build_id was
enabled, but only saved and restored header.size and header.type. If an
event with attr.build_id was followed by an event with attr.mmap2 and
!attr.build_id, the second event received PERF_RECORD_MISC_MMAP_BUILD_ID
in header.misc while its payload contained maj/min/ino/ino_generation
instead of a build ID.
Rather than splitting header initialization between callers and output
callbacks and saving/restoring mutated header fields, replace
perf_event_header__init_id() with perf_event_header__init(), which
initializes header->type, header->misc, and header->size alongside the
sample_id fields on each invocation.
Fixes: 76193a94522f ("perf, bpf: Introduce PERF_RECORD_KSYMBOL")
Fixes: 6ee52e2a3fe4 ("perf, bpf: Introduce PERF_RECORD_BPF_EVENT")
Fixes: e17d43b93e54 ("perf: Add perf text poke event")
Fixes: 88a16a130933 ("perf: Add build id data in mmap2 event")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260929222332.973435-1-irogers@google.com
Cc: stable@vger.kernel.org
|
|
A race condition exists between perf_event_exit_task() and
perf_pending_task() during begin_new_exec().
During begin_new_exec(), perf_event_exit_task() may be called, and the
PF_EXITING flag is not set on task. So perf_sigtrap() continues to execute
and triggers WARN_ON_ONCE(event->ctx->task != current).
Since both task exit and exec paths can call perf_event_exit_task() which
sets ctx->task to TASK_TOMBSTONE, fix this by explicitly checking if
event->ctx->task equals TASK_TOMBSTONE and dropping the redundant
PF_EXITING check.
Fixes: 97ba62b27867 ("perf: Add support for SIGTRAP on perf events")
Signed-off-by: Luo Gengkun <luogengkun2@huawei.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260920075026.990582-1-luogengkun2@huawei.com
|
|
Once a smpboot thread gets work assigned it receives a wake up. If the
thread had no chance to run and receives a parking requesst during a
CPU-hotplug event then it is parked without running the callback first.
As a result the enqueued work remains stuck until the CPU gets back
online.
There is nothing wrong with invoking the thread function first and
parking in the following iteration. This would ensure that the callbacks
are processed before the shutdown.
Honor the park request once thread function does not need to run.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260911143815.997254-4-bigeasy@linutronix.de
|
|
PREEMPT_RT invokes IRQ_WORK_LAZY callbacks and callbacks which are not
explicitly marked IRQ_WORK_HARD_IRQ from thread context. If a CPU gets
shutdown, the irq_work is flushed during smpcfd_dying_cpu() which is
invoked on the target CPU with disabled interrupts.
The callbacks enqueued on lazy_list on PREEMPT_RT are not flushed because
they require thread context. The callbacks remain and get processed once
the CPU gets back online.
per-CPU irq_work items continue to work for enqueues on their local-CPU,
only the work on the "offline" CPU get stuck. "Single" irq_work are
worse because they remain "claimed" and can not be used again. One
"symptom" is the late printk() in CPU down path which enqueues the
irq_work for printing but it never gets scheduled. Further prints don't
print to the console because the irq_work is "pending" of the offline
CPU.
Another problem is the canceling/ flushing of the irq_work which is
stuck. Since the irq_work will not continue, the task waiting for its
completion will block waiting.
While looking through the users of irq_work, it does not matter if the
callback is enqueued on another CPU. This makes it possible to invoke
the callbacks from a remote CPU by smpcfd_dead_cpu(). This is invoked
from the control CPU with enabled interrupts shortly after the
hotplugged CPU is shutdown so the llist can be accessed safely.
Add irq_work_run_cpu() which invokes the irq_work callbacks of the
specified dead CPU. Invoke it from smpcfd_dead_cpu().
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260911143815.997254-3-bigeasy@linutronix.de
|
|
hotplug_cfd() no longer exists and the flow changed a bit since the
rework of CPU hotplug as of commit 31487f8328f20 ("smp/cfd: Convert core
to hotplug state machine").
Update the comment to reflect reality as of today.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260911143815.997254-2-bigeasy@linutronix.de
|
|
Architectures can use SD_ASYM_PACKING to describe preferred CPU ordering
at the SMT scheduling domain. Some systems benefit from this policy, but
their firmware cannot describe the preference and inferring it from the
CPU model would embed a platform-specific policy in the kernel.
Add sched_smt_asym_packing=on boot option to force SD_ASYM_PACKING at
the SMT level. Omitting the option preserves the topology provided by
the architecture or firmware.
Apply the override centrally to domains with SD_SHARE_CPUCAPACITY so it
also covers architectures with custom SMT topology callbacks, including
powerpc.
When enabled, priority remains defined by arch_asym_cpu_priority(). The
weak default prefers lower-numbered logical CPUs, while architecture
overrides remain authoritative. Siblings with equal priorities remain
unordered. In particular, x86 with CONFIG_SCHED_MC_PRIO normally gives
both SMT siblings the same core priority, so enabling the option there
does not prioritize a sibling.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Breno Leitao <leitao@debian.org>
Link: https://patch.msgid.link/20260929165538.726616-3-arighi@nvidia.com
|
|
POWER7 uses SD_ASYM_PACKING at the shared-capacity SMT level to order
hardware threads, and NVIDIA Olympus benefits from the same policy. Idle
CPU selection does not consult that order, so a task can wake on an
arbitrary sibling and remain there until load balancing corrects the
placement. On these systems, that initial choice can prevent the core
from entering its preferred lower-thread resource mode and cause a large
and persistent performance loss.
When idle selection finds an available CPU in an SMT core, choose the
highest-priority available sibling. On SMT2 Olympus this only changes
selection on fully idle cores. A partially idle core has only one
available CPU. On wider SMT systems such as POWER7, it also fills
available siblings in priority order while the core is partially busy.
Apply the preference to idle-core and idle-CPU scans,
asymmetric-capacity scans, target, previous, recently-used CPU fast
paths and the slow path. Inspect the lowest scheduling domain directly,
but require both CPUs to share its span because isolcpus can split
hardware siblings across scheduling domains.
Keep physical-core capacity selection independent from SMT sibling
ordering. SD_ASYM_CPUCAPACITY first selects among cores with different
maximum capacities, then SD_ASYM_PACKING selects the preferred available
sibling inside the chosen core, whose siblings continue to share equal
capacity.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Tested-by: Breno Leitao <leitao@debian.org>
Link: https://patch.msgid.link/20260929165538.726616-2-arighi@nvidia.com
|
|
set_tsk_need_resched() tests TIF_NEED_RESCHED, calls
__trace_set_need_resched() if the flag is clear, then sets it via
set_tsk_thread_flag(). A BPF raw_tp program attached to
sched_set_need_resched executes synchronously inside __bpf_trace_run().
On return, __bpf_trace_run() drops the RCU lock with
rcu_read_unlock_migrate(), which on the preempt-or-BH-disabled path
calls set_need_resched_current() -> set_tsk_need_resched() again.
set_tsk_thread_flag() follows the tracepoint call, so every re-entrant
frame sees TIF_NEED_RESCHED clear and calls __trace_set_need_resched()
again:
BUG: TASK stack guard page was hit at ffffc9001224ff98
Oops: stack guard page: 0000 [#1] SMP KASAN PTI
RIP: 0010:__bpf_trace_sched_set_need_resched_tp+0x1c/0x190
Call Trace:
trace_sched_set_need_resched_tp+0x110/0x130
set_tsk_need_resched include/linux/sched.h:2076
set_need_resched_current include/linux/sched.h:2094
rcu_read_unlock_special+0x43a/0x440
__rcu_read_unlock+0x9e/0x120
rcu_read_unlock_migrate+0xa9/0x240
__bpf_trace_run+0x131/0x180
bpf_trace_run3+0x333/0x430
__bpf_trace_sched_set_need_resched_tp+0x13a/0x190
trace_sched_set_need_resched_tp+0x110/0x130
set_tsk_need_resched include/linux/sched.h:2076
...
__resched_curr() has the same ordering, firing the tracepoint before
setting the flag via set_ti_thread_flag() or set_nr_and_not_polling().
Fix it for consistency.
Replace the separate test_tsk_thread_flag() + set_tsk_thread_flag() pair
in set_tsk_need_resched() with test_and_set_tsk_thread_flag(). In
__resched_curr(), move the tracepoint call after the flag is set in
each path.
Fixes: adcc3bfa8806 ("sched: Adapt sched tracepoints for RV task model")
Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Acked-by: Gabriele Monaco <gmonaco@redhat.com>
Link: https://patch.msgid.link/20260630084750.2792851-1-rhkrqnwk98@gmail.com
|
|
Nathan reported build failure with clang23 when CONFIG_PREFERRED_CPU=y.
kernel/sched/core.c:11298:3: error: releasing raw_spinlock 'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis]
11298 | rq_unlock(rq, &rf);
| ^
kernel/sched/core.c:11303:1: error: raw_spinlock 'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis]
context_unsafe_alias(rq) was placed after rq_lock(), by which point it had
already associated the lock with the original rq alias.
Move the alias barrier before rq_lock() so that the changing rq alias is
accounted for before acquiring the lock.
Fixes: 74699f56ebcf ("sched/core: Push current task from non preferred CPU")
Closes: https://lore.kernel.org/all/20260929121838.GA1814129@ax162/
Reported-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Nathan Chancellor <nathan@kernel.org> # build
Link: https://patch.msgid.link/20260929164712.1054883-1-sshegde@linux.ibm.com
|
