| Age | Commit message (Collapse) | Author |
|---|
|
pick_idle_cpu_from_online_nodes() uses the per-CPU per_cpu_unvisited
nodemask as scratch while walking the online nodes, protected only by
preempt_disable(). preempt_disable() does not mask IRQs and the idle
kfuncs are callable from IRQ-enabled contexts, so a nested invocation
on the same CPU can overwrite the mask with nodes_copy() while the
interrupted invocation is still iterating it, leading to a wrong node
traversal and a wrong idle CPU pick.
Switch to irqsave so a nested invocation can't run on the same CPU.
A stack-allocated nodemask would also close the race, but that would
enlarge the diff to fix a race that is already rare (per-node idle,
CONFIG_NUMA and a cross-node search all at once); irqsave is the
minimal fix for the context that actually triggers it. The NMI case is
deliberately not addressed: there is no legitimate reason to
call pick_idle from NMI and doing so poses no crash risk, so such a
caller is on its own.
Suggested-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/d84b31727f04e1ed0d40042ba1c09e61@kernel.org
Signed-off-by: Wanwu Li <liwanwu@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_locked_rq() reads the per-CPU scx_locked_rq_state, which tracks the
rq locked by the context running on the CPU. Tracing progs can call
kfuncs from NMI, and an NMI interrupts - rather than replaces - the
context that set scx_locked_rq_state, so a non-NULL read from NMI
falsely tells the caller that it holds the interrupted context's rq
lock.
Three "any"-category kfuncs read scx_locked_rq() on their success path
and take an unsafe fast path on a non-NULL return:
- scx_bpf_task_set_slice() writes p->scx.slice directly, racing
update_curr_scx()'s non-atomic read-modify-write of the same field.
- scx_bpf_dsq_nr_queued() resolves %SCX_DSQ_LOCAL to
(scx_locked_rq() ?: this_rq()) and can report the interrupted
context's local DSQ length instead of the caller's.
- scx_bpf_locked_rq() hands the interrupted context's rq to the BPF
program, which may then operate on it as if it owned the rq lock.
Make scx_locked_rq() return NULL from NMI so that all three take their
unlocked paths: scx_bpf_task_set_slice() stashes the request into the
atomic p->scx.slice_oob for application under the rq lock,
scx_bpf_dsq_nr_queued() falls back to this_rq(), and
scx_bpf_locked_rq() reports an error and aborts the scheduler through
the NMI-safe exit path.
The kfuncs that take scheduler locks reject NMI calls through
scx_kf_allowed_ctx() before reaching scx_locked_rq(), and the internal
callers only run from struct_ops callbacks, which never run in NMI, so
no other caller is affected.
Suggested-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/d84b31727f04e1ed0d40042ba1c09e61@kernel.org
Signed-off-by: Wanwu Li <liwanwu@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The CSD lock debug output is a useful way to catch IPI stalls, but when
the lock finally recovers it only says that it did:
smp: csd: CSD lock (#1) got unstuck on CPU#32, CPU#123 released the lock.
How long the target took to answer is left out, even though
csd_lock_wait_toolong() already has the timestamp the wait started from.
At Meta's fleet, that line fired 211K times in the last 24 hours, so
plenty of stalls get reported with no indication of how long they
lasted.
Print how long the lock was stuck, and, when an IPI was re-sent, how
long after that re-send the target released, as:
smp: csd: CSD lock (#1) got unstuck on CPU#00, CPU#01 released the lock after 8000854660 ns, 3000825778 ns after the last IPI re-send.
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
|
|
csd_lock_wait_toolong() has some fields and they are being expanded now,
separate them into a structure, that can be easily digestible.
This simplify the function aslo, given the fields were passed by
reference, and the ts0/ts1 names say nothing about what the two
timestamps hold.
Pack them into struct csd_wait_state and name the timestamps for what
they store, ts_start and ts_report. The local ts2 becomes ts_now.
Reporting a further timestamp, such as next patch, then costs a struct
member rather than another argument.
No functional change.
Suggested-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
|
|
When the root scheduler has sub-scheds attached, the COMPAT kfunc
wrappers scx_bpf_select_cpu_and() and scx_bpf_dsq_insert_vtime() refuse
the call and report to @p's scheduler:
scx_error(scx_task_sched(p), "... must be used");
The wrappers are reachable with tasks that have no scheduler.
scx_bpf_select_cpu_and() is in the select_cpu kfunc group, which
scx_kfunc_context_filter() opens to BPF_PROG_TYPE_SYSCALL programs;
scx_bpf_dsq_insert_vtime() is in the enqueue_dispatch group, which
ops.enqueue() and ops.dispatch() may call with any KF_RCU task -- the
group has no kf_tasks validation, and scx_dsq_insert_preamble() checks
task ownership with scx_task_on_sched() precisely because @p may be an
arbitrary task.
scx_task_sched(p) is p->scx.sched, which is NULL for tasks past
sched_ext_dead() -- which clears it via scx_disable_and_exit_task() on
exit -- and for idle tasks, which the enable paths skip as they are
never scheduled through SCX. It is also an rcu_dereference_protected()
that expects @p's pi_lock or rq lock, which neither wrapper holds.
Passing NULL to scx_error() reaches scx_vexit(), which dereferences
sch->exit_info, oopsing the kernel.
One concrete trigger exercised while developing the fix: a
BPF_PROG_TYPE_SYSCALL program calling the select_cpu_and wrapper on an
exited-but-not-reaped task while a sub-scheduler was attached (its pid
stays findable while the zombie is unreaped; faulting instruction is
the scx_vexit() prologue "mov r15,[rdi+0x398]" with RDI=NULL and 0x398
the offset of sch->exit_info):
sched_ext: BPF scheduler "kfunc_subsched_null" enabled
sched_ext: BPF sub-scheduler "kfunc_subsched_null" enabled
sched_ext: Unassociated program run_select_cpu_ (id 76)
BUG: kernel NULL pointer dereference, address: 0000000000000398
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: Oops: 0000 [#1] SMP NOPTI
CPU: 7 UID: 0 PID: 8201 Comm: kfunc_test_runn Tainted: G W
RIP: 0010:scx_vexit+0x25/0xa0
Code: ... <4c> 8b bf 98 03 00 00 ...
CR2: 0000000000000398
Call Trace:
<TASK>
__scx_exit+0x4f/0x70
scx_bpf_select_cpu_and+0xab/0xb0
bpf_prog_430ed61a7b66e03a_run_select_cpu_and+0x9c/0xe7
? __x64_sys_bpf+0x2c/0x40
bpf_prog_test_run_syscall+0x130/0x2f0
__sys_bpf+0x930/0x10d0
? __x64_sys_bpf+0x2c/0x40
__x64_sys_bpf+0x2c/0x40
do_syscall_64+0xbc/0x460
entry_SYSCALL_64_after_hwframe+0x76/0x7e
</TASK>
Read @p's scheduler under RCU instead, which the wrappers can do from
their guard(rcu)(): fault it when it can be determined, and when it
can't be determined -- @p is a task past sched_ext_dead() or an idle
task -- there is nothing obviously wrong to report, so just refuse the
call as before without faulting any scheduler.
These COMPAT wrappers are scheduled for eventual removal once the
deprecation grace period elapses, but until then -- and regardless of
their removal timeline -- they must not oops the kernel on a task they
are handed.
Cc: stable@vger.kernel.org # v7.1+
Fixes: a5fa0708cbfd ("sched_ext: Enforce scheduling authority in dispatch and select_cpu operations")
Suggested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Wanwu Li <liwanwu@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
bpf_sys_bpf() executes the bpf(2) syscall body, which can take mutexes,
allocate with GFP_KERNEL, and wait for an RCU grace period.
bpf_sys_close() reaches close_fd() and filp_close(), which can sleep as
well.
Both helpers are limited to BPF_PROG_TYPE_SYSCALL, whose main program is
sleepable. That does not make every callback sleepable: a syscall program
can register a bpf_timer callback, and the verifier checks that callback
in a non-sleepable context while retaining the syscall helper set.
Without .might_sleep on the prototypes, such a callback can invoke
bpf_sys_bpf() from hrtimer softirq context and trigger a
scheduling-while-atomic failure. bpf_sys_close() is exposed through the
same missing context check.
Set .might_sleep on both prototypes so the existing helper-context check
rejects them from timer callbacks and other atomic regions. Calls from the
sleepable main body remain valid.
Fixes: 79a7f8bdb159 ("bpf: Introduce bpf_sys_bpf() helper and program type.")
Fixes: 3abea089246f ("bpf: Add bpf_sys_close() helper.")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903144433.1716731-10-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
do_wait() passes wo->wo_pid to the sched_process_wait tracepoint.
kernel_wait4() leaves wo_pid NULL for wait4(-1), and
kernel_waitid_prepare() does likewise for waitid(P_ALL).
btf_ctx_access() currently types argument 0 as PTR_TO_BTF_ID |
PTR_TRUSTED. Without PTR_MAYBE_NULL, the verifier accepts an unchecked
dereference. Trusted pointer loads have no fault protection, so a wait for
any child can then cause a NULL pointer dereference in JITed BPF code.
Add sched_process_wait to raw_tp_null_args[] with argument 0 marked
nullable. The verifier rejects an unchecked dereference while preserving
access after the program checks the pointer for NULL.
Fixes: 838a10bd2ebf ("bpf: Augment raw_tp arguments with PTR_MAYBE_NULL")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903144433.1716731-8-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
__bpf_rbtree_add() keeps parent and link pointers live across calls to the
program-supplied comparison callback. The verifier therefore requires the
root's lock to remain held throughout the callback.
The helper path enforces this rule for bpf_spin_lock() and
bpf_spin_unlock(), but the resilient lock kfunc argument path does not.
Since resilient locks may protect BPF rbtree roots, a callback can release
the root lock and let another CPU remove and free the node referenced by
the in-progress tree walk. The walk then resumes using freed pointers.
Reject resilient lock kfuncs in an rbtree comparison callback, matching
the existing policy for the spin lock helpers. Resilient-lock-protected
trees remain valid when their comparison callbacks leave lock state alone.
Fixes: 0de2046137f9 ("bpf: Implement verifier support for rqspinlock")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903144433.1716731-6-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
A tail call from a non-zero frame is modeled as a return from that frame.
The verifier makes R0 unknown and calls prepare_func_exit() for the taken
branch.
When the current frame is a synchronous callback, prepare_func_exit()
enforces the callback return-value contract and marks R0 precise. Since the
tail-call path synthesized R0 rather than deriving it from an instruction,
precision backtracking reaches the callback-calling instruction with R0
still requested and triggers the "callback unexpected regs" verifier bug.
A CAP_BPF task can therefore cause a WARN and an -EFAULT BPF_PROG_LOAD.
Tail calls reachable from callbacks are already rejected later by
check_max_stack_depth(). Reject a tail call made directly by a callback
before constructing the inconsistent return state, using the existing
diagnostic. Tail calls from ordinary subprograms keep their current
behavior.
Fixes: e3245f899043 ("bpf: properly verify tail call behavior")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903144433.1716731-4-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
The signal_generate and signal_deliver tracepoints declare their info
argument as a struct kernel_siginfo pointer. btf_ctx_access() therefore
treats it as a trusted pointer for tp_btf programs.
Signal delivery also uses SEND_SIG_NOINFO and SEND_SIG_PRIV as special
values for this argument. Those values are zero and one respectively,
and are not pointers. A tp_btf program can currently dereference either
value and fault the kernel. In particular, signal_generate can run from
timer interrupt context, turning the fault into a kernel panic.
Record both tracepoints in raw_tp_null_args[] and mark argument one as
a non-pointer. This preserves scalar access to the cookie while rejecting
direct and helper-mediated pointer use. Merely marking it nullable would
not suffice because SEND_SIG_PRIV is nonzero.
Fixes: 838a10bd2ebf ("bpf: Augment raw_tp arguments with PTR_MAYBE_NULL")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903144433.1716731-2-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
btf_var_show() calls btf_type_id_resolve() unconditionally, which
dereferences btf->resolved_ids. That is NULL for a base BTF - e.g. the
vmlinux BTF that bpf_snprintf_btf() renders against - since base BTF is
not resolved during parsing. btf_modifier_show() guards this with
'if (btf->resolved_ids)', but btf_var_show() does not.
A BPF program that passes the type_id of a BTF_KIND_VAR from the vmlinux
BTF to bpf_snprintf_btf() thus NULL-derefs:
KASAN: probably user-memory-access in range [0x46638-0x4663f]
RIP: 0010:btf_var_show (kernel/bpf/btf.c:2929)
Call Trace:
<TASK>
btf_type_show (kernel/bpf/btf.c:8259)
btf_type_snprintf_show (kernel/bpf/btf.c:8329)
bpf_snprintf_btf (kernel/trace/bpf_trace.c:1047)
bpf_prog_test_run_raw_tp (net/bpf/test_run.c:829)
__sys_bpf (kernel/bpf/syscall.c:4804)
do_syscall_64 (arch/x86/entry/syscall_64.c:84)
entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121)
</TASK>
Resolve the var's type directly with btf_type_skip_modifiers() when
resolved_ids is NULL, mirroring btf_modifier_show().
Fixes: c4d0bfb45068 ("bpf: Add bpf_snprintf_btf helper")
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/r/20260901104924.346187-4-jiayuan.chen@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
btf_modifier_show() resolves the modifier and then calls
btf_type_ops(t)->show() unconditionally. For the void type (type_id 0,
BTF_KIND_UNKN) kind_ops[] has no entry, so ->show is NULL.
A "const void" (a modifier resolving to void) cannot be a map key or
value - map_check_btf() rejects it because void has no size - so the map
dump path does not reach it. But bpf_snprintf_btf() takes a type_id
straight from the BPF program, and passing such a "const void" from the
vmlinux BTF NULL-derefs:
KASAN: null-ptr-deref in range [0x0000000000000028-0x000000000000002f]
RIP: 0010:btf_modifier_show (kernel/bpf/btf.c:2914)
Call Trace:
<TASK>
btf_type_show (kernel/bpf/btf.c:8251)
btf_type_snprintf_show (kernel/bpf/btf.c:8321)
bpf_snprintf_btf (kernel/trace/bpf_trace.c:1047)
bpf_prog_test_run_raw_tp (net/bpf/test_run.c:829)
__sys_bpf (kernel/bpf/syscall.c:4804)
do_syscall_64 (arch/x86/entry/syscall_64.c:94)
entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121)
</TASK>
Fall back to btf_df_show() when the resolved type has no show op; it
emits the "<unsupported kind:N>" placeholder already used for kinds like
FWD and FUNC. bpf_snprintf_btf() then returns the length as usual.
Fixes: c4d0bfb45068 ("bpf: Add bpf_snprintf_btf helper")
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/r/20260901104924.346187-3-jiayuan.chen@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
map_check_btf() allows a key-less BTF (btf_key_type_id == 0) only for
maps that have a ->map_check_btf callback, and leaves the actual
decision to that callback. Hash maps used to have no ->map_check_btf,
so a key-less BTF was rejected outright.
That changed when htab and rhtab gained a ->map_check_btf to register a
dtor - htab in commit 1df97a7453ee ("bpf: Register dtor for freeing
special fields") and rhtab in commit 6905f8601298 ("bpf: Allow special
fields in resizable hashtab"). Neither looks at the key, so a key-less
hash map now passes map_check_btf() and gets created. Reading it back
through bpffs feeds the key type_id 0 into btf_type_seq_show();
btf_type_by_id() returns the void type, kind_ops[BTF_KIND_UNKN] is NULL,
and btf_type_show() dereferences it:
RIP: 0010:btf_type_show+0x223/0x2e0 kernel/bpf/btf.c:8232
RSP: 0018:ffffc9000399f868 EFLAGS: 00010206
RAX: dffffc0000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000005 RSI: 0000000000000000 RDI: 0000000000000028
RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
R10: ffffc9000399f970 R11: 0000000000000001 R12: ffffffff9b96b140
R13: ffffc9000399f8e0 R14: ffff88803d393c00 R15: 0000000000000003
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000200000000000 CR3: 000000003d213000 CR4: 0000000000352ef0
DR0: 0000000039ae8f55 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
btf_type_seq_show_flags+0xca/0x120 kernel/bpf/btf.c:8250
htab_map_seq_show_elem+0x12e/0x350 kernel/bpf/hashtab.c:1669
map_seq_show+0x13d/0x1e0 kernel/bpf/inode.c:293
traverse.part.0.constprop.0+0x107/0x650 fs/seq_file.c:112
traverse fs/seq_file.c:99 [inline]
seq_read_iter+0x93f/0x1270 fs/seq_file.c:196
seq_read+0x344/0x4d0 fs/seq_file.c:163
vfs_read+0x1e4/0xb40 fs/read_write.c:572
ksys_pread64 fs/read_write.c:764 [inline]
__do_sys_pread64 fs/read_write.c:772 [inline]
__se_sys_pread64 fs/read_write.c:769 [inline]
__x64_sys_pread64+0x1eb/0x250 fs/read_write.c:769
do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline]
do_syscall_64+0x123/0x790 arch/x86/entry/syscall_64.c:84
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Reject a key-less BTF in htab_map_check_btf() and rhtab_map_check_btf(),
restoring the previous behavior.
Fixes: 1df97a7453ee ("bpf: Register dtor for freeing special fields")
Fixes: 6905f8601298 ("bpf: Allow special fields in resizable hashtab")
Reported-by: syzbot+37b56485bbbf90ad8489@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/6a8f4e88.27659fcc.2ceef7.0008.GAE@google.com/T/
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/r/20260901104924.346187-2-jiayuan.chen@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
The sole caller, create_new_namespaces(), always passes a valid cgroup
namespace pointer from the task's nsproxy. The other copy_*_ns()
helpers, e.g. copy_time_ns() and copy_pid_ns(), don't perform such a
check either. Drop the BUG_ON() for consistency with them.
Signed-off-by: Shaojie Sun <sunshaojie@kylinos.cn>
Reviewed-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
|
|
ring_buffer_read_page() rejects splice (full=1) reads on static buffers
(that is user-mapped, persistent or remote) because !read check assumes
unread pages must be swapped. However for those buffers we have no other
choice than memcpy the data.
For the memcpy case, only return an error when the writer is still on
the reader page for the splice interface to wait.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260901155445.1475405-2-vdonnefort@google.com
Fixes: 117c39200d9d ("ring-buffer: Introducing ring-buffer mapping functions")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Boot consoles are temporary and should be removed once a real console is
available. However, the late init cleanup currently only unregisters boot
consoles that use init section memory. Other boot consoles are expected
to be removed when the real preferred console is registered.
This does not cover cases where a real console has registered, but the
boot console was not removed because the real console did not become the
preferred console. For example, with multiple console= parameters using
the same driver, a real 8250 console may be enabled while the early
console remains registered. The result is duplicate printk output from
both consoles.
In the mailing list discussion, two possible approaches were suggested
to fix this problem [1]. This patch implements the first one: during
printk_late_init(), check whether at least one real console is already
registered. If so, unregister all remaining boot consoles. If no real
console exists yet, keep the existing behavior and unregister only boot
consoles that reference init section memory, avoiding a period with no
console output while waiting for a deferred or modular real console.
Suggested-by: Petr Mladek <pmladek@suse.com>
Link: https://lore.kernel.org/lkml/ahBsSW2bB2CRfW-k@pathway.suse.cz/ # [1]
Signed-off-by: Xiaochun Li <lixiaochun@open-hieco.net>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Tested-by: Petr Mladek <pmladek@suse.com>
Link: https://patch.msgid.link/20260821071721.178517-1-lixiaochun@open-hieco.net
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
fixups.c:jit_subprogs() rewrites BPF_PSEUDO_FUNC loads to contain real
function addresses. This function is invoked from bpf_jit_subprogs()
only when env->subprog_cnt > 1. Meaning that for any program like
below:
int main(void *ctx) {
void *ptr = main;
...
bpf_timer_set_callback(..., ptr);
...
}
The 'ptr' won't be ever converted to contain an address.
In combination with e.g. bpf_timer_set_callback() this would lead to a
function call at a bogus address.
Instead of complicating the implementation, just assume that no useful
program needs main to be a sync or async callback and reject
BPF_PSEUDO_FUNC loads for the main subprogram.
Fixes: 69c087ba6225 ("bpf: Add bpf_for_each_map_elem() helper")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260902233658.1186477-1-eddyz87@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
__within_kprobe_blacklist() traverses kprobe_blacklist without holding
kprobe_mutex. When a module is unloaded, kprobe_remove_area_blacklist()
removes blacklist entries and immediately frees them with kfree().
A concurrent call to within_kprobe_blacklist() can therefore dereference
freed memory.
Furthermore, within_kprobe_blacklist() can be called in atomic or
non-preemptible contexts where the sleeping kprobe_mutex cannot be taken.
Protect kprobe_blacklist with RCU. Use guard(rcu)() and
list_for_each_entry_rcu() for traversal, list_add_tail_rcu() for
insertions, list_del_rcu() for deletions, and kfree_rcu() to reclaim
entries safely after a grace period.
Link: https://lore.kernel.org/all/178810004323.64882.16493230858653316962.stgit@devnote2/
Fixes: 376e242429bf ("kprobes: Introduce NOKPROBE_SYMBOL() macro to maintain kprobes blacklist")
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260807155802.F06041F000E9@smtp.kernel.org/
Assisted-by: Antigravity:gemini-3.7-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
multiple probes
The fields of a probe-based dynamic event (kprobe, uprobe, eprobe and
fprobe events) are created in traceprobe_define_arg_fields() by handing
the probe_arg name/type strings to trace_define_field(), which only
stores the pointers without copying. Those strings are owned by the
trace_probe and are freed when that probe is removed.
An event can have several probes attached. The field list is defined
only once, by the first probe that registers the event, but it is kept
alive by any surviving sibling probe. Deleting just that first probe by
symbol -
# primary A: fields are defined from A's args
echo 'p:kprobes/ev vfs_read a1=$arg1' > kprobe_events
# append B: shares A's event call
echo 'p:kprobes/ev vfs_write a1=$arg1' >> kprobe_events
# delete only A (matched by symbol), B survives
echo '-:kprobes/ev vfs_read' >> kprobe_events
frees A's args (trace_probe_cleanup() -> traceprobe_free_probe_arg()),
but trace_probe_unlink() keeps the trace_probe_event because the probe
list is not empty. The event call stays registered via B while its
fields now reference freed memory. Any field lookup then reads it, e.g.
echo 'a1 == 1' > events/kprobes/ev/filter
BUG: KASAN: slab-use-after-free in strcmp+0xa7/0xb0
Call Trace:
strcmp
trace_find_event_field
parse_pred
process_preds
create_filter
apply_event_filter
event_filter_write
field->name references parg->name (kstrdup'd, freed with the probe) and,
for array arguments, field->type references parg->fmt (kmalloc'd, freed
with the probe) - the scalar type otherwise points at the static
fmttype rodata, which is safe.
Have traceprobe_define_arg_fields() duplicate the name and type strings
and anchor the copies on the trace_probe_event, which embeds the event
call and outlives every individual probe; trace_probe_event_free()
releases them.
The reproducer above triggers reliably; the field lookup and the delete
both run under event_mutex, so this is a dangling reference after
removal rather than a race.
The issue was found by the autokbug dynamic kernel fuzzer at Tencent
Yunding Lab.
Link: https://lore.kernel.org/all/20260826030009.1855331-1-bsdhenrymartin@gmail.com/
Fixes: ca89bc071d5e4 ("tracing/kprobe: Add multi-probe per event support")
Signed-off-by: Henry Martin <bsdhenrymartin@gmail.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
Fix code block indentation introduced by commit f21834524025
("tracing/probes: Support field specifier option for typecast").
Link: https://lore.kernel.org/all/178827252027.123716.7095571176291547259.stgit@devnote2/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
|
|
btf_find_struct_member() traverses into nested anonymous structures and
unions to find a struct member. However, get_bitoffset_of_field() in
trace_probe.c checked btf_type_kflag(type) using the outer parent type
instead of the actual anonymous structure/union that directly contains
the found member.
If the parent structure and anonymous structure have mismatched kflags
(e.g., the parent has kflag=0 while the anonymous structure has kflag=1
because it contains bitfields), the bitfield size encoded in the upper
8 bits of member->offset is erroneously treated as part of the byte/bit
offset, corrupting the resolved offset and failing to set last_bitsize.
Similarly, btf_find_struct_member() pushed anonymous member offsets
onto anon_stack without masking BTF_MEMBER_BIT_OFFSET() when kflag is set.
To fix this problem, update btf_find_struct_member() to return actual
containing structure/union type via member_type, use appropriate
__btf_member_bit_offset() to get bit offset, and use member_type for
btf_type_kflag() in get_bitoffset_of_field().
Link: https://lore.kernel.org/all/178827250904.123716.17452648791331881284.stgit@devnote2/
Fixes: c440adfbe302 ("tracing/probes: Support BTF based data structure field access")
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260822095110.0772E1F000E9@smtp.kernel.org/
Assisted-by: Antigravity:gemini-3.7-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
|
|
btf_find_struct_member
btf_find_struct_member() traverses into nested anonymous structures
and unions by pushing members with !member->name_off onto anon_stack.
However, it does not consider the unnamed bitfields (e.g. `int : 5`
or `unsigned int : 0`) which also have member->name_off == 0.
If such an unnamed bitfield is pushed to anon_stack, the
btf_find_struct_member() return an error even if there are other
valid entries in anon_stack.
To fix this, only push unnamed struct/union members to anon_stack.
Also move the btf_type_is_struct() check to the entry of this function
because now it is sure only struct/union are pushed to anon_stack.
Link: https://lore.kernel.org/all/178827249775.123716.7813217688423513612.stgit@devnote2/
Fixes: 302db0f5b3d8 ("tracing/probes: Add a function to search a member of a struct/union")
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260830143859.D56991F00A3D@smtp.kernel.org/
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The queue/stack map addresses elements[] with the product of a u32
head/tail index and value_size, but the storage itself is allocated in
64-bit arithmetic. When max_entries * value_size reaches or exceeds
U32_MAX, the product wraps and push/peek/pop operate on the wrong
element, corrupting map data and leaking stale values to user space.
max_entries == U32_MAX would also make the u32 capacity counter
qs->size (max_entries + 1) wrap to 0 and permanently break the map.
The original bound check was removed by commit a37fb7ef24a4 ("bpf:
Eliminate rlimit-based memory accounting for queue_stack_maps maps"),
which deleted the bpf_map_charge_init() call and with it the
U32_MAX - PAGE_SIZE check that had earlier been moved into
bpf_map_charge_init() by c85d69135a91. Oversized queue/stack maps can
therefore be created again.
Restore the bound in queue_stack_map_alloc_check() with a single
comparison that rejects any max_entries/value_size combination whose
element storage would reach or exceed U32_MAX bytes, keeping the u32
index multiplication overflow-free and the capacity counter valid.
Fixes: a37fb7ef24a4 ("bpf: Eliminate rlimit-based memory accounting for queue_stack_maps maps")
Signed-off-by: Yuan Chen <chenyuan@kylinos.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/20260831063226.621309-2-chenyuan_fl@163.com
|
|
commit e06ece82d7b0 ("sched_ext: Report NMI kicks with scx_error()") made
scx_bpf_kick_cpu() reject NMI calls, and its cover letter describes the
reachability: sched_ext kfuncs in the "any" category "are callable from
tracing progs that can attach to functions running in NMI", and an unlucky
call from there "could deadlock the machine". The fix in that series made
the error/exit path lock-free so scx_error() is safe to call from NMI. That
closes the *error* path of every kfunc, but not a kfunc's own
business-logic lock acquisition on its success path.
The remaining lock-taking kfuncs that scx_kfunc_context_filter() exposes to
BPF_PROG_TYPE_TRACING have the same hazard: if an NMI lands on a CPU whose
interrupted context already holds the lock, the kfunc's raw spinlock
acquisition spins forever and hard-locks the CPU:
- scx_bpf_destroy_dsq() -> dsq->lock
- scx_bpf_dsq_reenq() -> rq's deferred_reenq_lock
- scx_bpf_cpuperf_set() / scx_bpf_cidperf_set() -> rq->lock
- scx_bpf_sub_grant() / scx_bpf_sub_revoke() -> pshard lock
(via the shared sub_cap_preamble())
- bpf_iter_scx_dsq_next() / bpf_iter_scx_dsq_destroy() -> dsq->lock
(the reject is placed in bpf_iter_scx_dsq_new(), which is otherwise
lockless, so next() and destroy() are no-ops while it leaves kit->dsq
NULL)
As things stand, there is no scenario for reenqueueing, iterating a DSQ,
setting a performance target or granting sub-caps from NMI. The guards
defend against a buggy or malicious BPF program turning an "any"-category
kfunc into a machine-wide hard-lockup through the door that
scx_kfunc_context_filter() already opens. This matches the intent of
scx_bpf_kick_cpu()'s NMI check, which the commit cited above added not to
enable an NMI use case but to surface such a bug as a clean abort.
Route all of them through a new scx_kf_allowed_ctx() helper and reuse
scx_bpf_kick_cpu()'s existing in_nmi() check - now shared with its cid
equivalent scx_bpf_kick_cid() through scx_kick_cpu() - so the rule is
stated once and the coverage is auditable from one place. scx_error() is
already NMI-safe (commit f883dbb64ca5 ("sched_ext: Make exit claiming
lock-free")), so the reject-abort cannot deadlock the lock acquisition.
Kfuncs with an error return report -EDEADLK, the situation being avoided.
Read-only members of the reachable sets (dsq_peek, dsq_nr_queued,
cpuperf_cur/cap, sub_caps, the idle cpumask helpers and the cid lookups)
take no scheduler lock on the path a tracing program reaches them, and were
audited to that effect; they are correctly left unguarded. The select_cpu
kfuncs do take pi_lock, but scx_kfunc_context_filter() only exposes the
any/idle/cid sets to BPF_PROG_TYPE_TRACING, and struct_ops programs never
run in NMI, so no lock-taking path here is reachable from NMI.
Link: https://lore.kernel.org/r/20260901095652.1009104-1-liwanwu@kylinos.cn
Signed-off-by: Wanwu Li <liwanwu@kylinos.cn>
Acked-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The elfcorehdr reservation is now sized with NR_CPUS, but two comments
still name NR_CPUS_DEFAULT: the one above crash_handle_hotplug_event()
and the CRASH_MAX_MEMORY_RANGES help text.
Update both. Documentation only, no functional change.
Suggested-by: Jinjie Ruan <ruanjinjie@huawei.com>
Suggested-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Link: https://patch.msgid.link/20260901071041.16311-3-ionut.nechita@windriver.com
|
|
The options files do not take the trace_array reference for the options
they represent. This could cause a use-after-free kernel crash if one of
these files is opened by one task and another task removes the instance
that the option is for. Because it doesn't take a reference upon opening,
it will not stop the removal which will free the options descriptor that
is being used.
As the options are somewhat dynamic in their creation at boot up, each
file represents a flag in the trace_array. The trace_array has an array of
indexes to represent each of these flags that is stored in the
trace_flags_index array. The address of the index array element is used to
pass to the inode->i_private pointer. Then that element is read which
holds the index (which represents the flag) and then the index is used to
calculate the trace_array descriptor from its trace_flags_index array.
One issue is that the index element can not be referenced until the
trace_array's reference is taken. To handle this, create a new helper
function called: trace_array_options_get() that will iterate all the
existing trace_arrays in the ftrace_trace_arrays list (under the
trace_types_lock), and compare the passed in address of the index element
with the entire array of the trace_array's trace_flags_index array.
If it matches, then up the corresponding trace_array's reference and
return.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260902121918.5a9e9d1b@gandalf.local.home
Fixes: 577b785f55168 ("tracing: add tracer dependent options to options directory")
Reported-by: sashiko-bot@kernel.org
Closes: https://lore.kernel.org/linux-trace-kernel/20260828135858.2AC501F000E9@smtp.kernel.org/
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
When processing calls to bpf_loop() verifier marks R1 (and R4) as
precise. R1 tracks loop iterations number and because of the
'callback_depth < R1' mechanics in check_helper_call() must be marked
precise. However, precision propagation for R1 was broken,
when bpf_loop() call was verified on a second iteration.
Consider the following verification trace:
- main: bpf_loop(nr_loops, callback ...)
- callback: BPF_EXIT
- main: bpf_loop(nr_loops, callback ...)
- ...
While the first visit of the call to bpf_loop() propagated R1
precision as expected, the second call to mark_chain_precision() in
the check_helper_call() set R1, but it was immediately reset when
backtrack_insn() processed preceding BPF_EXIT in the loop deleted in
this patch.
Because of that, the second visit of the call to bpf_loop() injected
checkpoint with R1 not marked as precise. Which could trick the
verifier into accepting unsafe programs. See the next patch for an
example of such program.
Commit is structured in a way to minimize conflicts when
'bpf' would be eventually merged with 'bpf-next'.
Fixes: ab5cfac139ab ("bpf: verify callbacks as if they are called unknown number of times")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260831-bug-015-backtrack-cb-args-precise-v1-1-68a8e2a821e0@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
There's some internal state that ftrace_ops needs to have set, but since
it can be declared outside of the ftrace.c code, it calls
ftrace_ops_init() on the ops in every global function. The issue is that
if two tasks call it on the same ops at the same time it is possible to
have the initialization of one corrupt the initialization of the other
call.
Create a ops_mutex to use to synchronize every initialization of the
ftrace_ops. The mutex is taken within checking the ftrace_ops flag that
states it was initializied but the flag is checked again after the mutex
has been taken. Checking first outside the mutex allows it to shortcut
having to take the mutex. But then the check needs to be done again after
the mute is taken in case of races.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260902095501.6b59af20@gandalf.local.home
Fixes: f04f24fb7e48d ("ftrace, kprobes: Fix a deadlock on ftrace_regex_lock")
Reported-by: sashiko-bot@kernel.org
Close: https://lore.kernel.org/all/20260829025528.49A831F000E9@smtp.kernel.org/
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Nicholas Carlini reported a bug in precision backtracking mechanism
for BPF_LD | BPF_{IND,ABS} instructions. These instructions are
modelled as two branches:
- fallthrough;
- implicit exit from current subprogram.
The implicit exit case was not handled by the backtrack_insn()
function. When backtracking such a path backtrack_insn() did not
call bt_subprog_enter(), which meant that backtracking continued
manipulating precision marks in a caller frame, while looking at
instructions in a callee frame.
This lead to segmentation faults during verification (see the
selftest), or unsound state pruning.
Fixes: ee861486e377 ("bpf: Fix ld_{abs,ind} failure path analysis in subprogs")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260901-bug-016-backtrack-ld-abs-v1-1-59368f1be435@gmail.com
|
|
The upcoming patch will support x86 APX eGPRs sampling by using the
reclaimed XMM register space to represent eGPRs in sample_regs_* fields.
To differentiate between XMM and eGPRs in sample_regs_* fields, an
additional argument, simd_enabled, is introduced to the
perf_reg_validate() helper. If simd_enabled is set to 1, it indicates
that eGPRs are represented in sample_regs_* fields for the x86 platform;
otherwise, XMM registers are represented.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260824082731.1013973-18-dapeng1.mi@linux.intel.com
|
|
Support sampling of XMM registers using the sample_simd_vec_reg_* fields.
When sample_simd_regs_enabled is set, the original XMM space in the
sample_regs_* field is treated as reserved. An INVAL error will be
reported to user space if any bit is set in the original XMM space while
sample_simd_regs_enabled is set.
The perf_reg_value function requires ABI information to understand the
layout of sample_regs. To accommodate this, a new abi field is introduced
in the struct x86_perf_regs to represent ABI information.
Additionally, the x86 specific perf_simd_reg_value() function is
implemented to retrieve the XMM register values.
XMM sampling will be enabled in a subsequent patch that sets
PERF_PMU_CAP_SIMD_REGS.
Co-developed-by: Kan Liang <kan.liang@linux.intel.com>
Signed-off-by: Kan Liang <kan.liang@linux.intel.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260824082731.1013973-14-dapeng1.mi@linux.intel.com
|
|
Users may be interested in sampling SIMD registers during profiling.
The current sample_regs_* structure does not have sufficient space
for all SIMD registers.
To address this, new attribute fields sample_simd_{pred,vec}_reg_* are
added to struct perf_event_attr to represent the SIMD registers that are
expected to be sampled.
Currently, the perf/x86 code supports XMM registers in sample_regs_*.
To unify the configuration of SIMD registers and ensure a consistent
method for configuring XMM and other SIMD registers, a new event
attribute field, sample_simd_regs_enabled, is introduced. When
sample_simd_regs_enabled is set, it indicates that all SIMD registers,
including XMM, will be represented by the newly introduced
sample_simd_{pred|vec}_reg_* fields. The original XMM space in
sample_regs_* is reserved for future uses.
Since SIMD registers are wider than 64 bits, a new output format is
introduced. The number and width of SIMD registers are dumped first,
followed by the register values. The number and width are based on the
user's configuration.
A new ABI, PERF_SAMPLE_REGS_ABI_SIMD, is added to indicate the new format.
The enum perf_sample_regs_abi is now a bitmap. This change should not
impact existing tools, as the version and bitmap remain the same for
values 1 and 2.
Additionally, two new __weak functions are introduced:
- perf_simd_reg_value(): Retrieves the value of the requested SIMD
register.
- perf_simd_reg_validate(): Validates the configuration of the SIMD
registers.
A new flag, PERF_PMU_CAP_SIMD_REGS, is added to indicate that the PMU
supports SIMD register dumping. An error is generated if
sample_simd_{pred|vec}_reg_* is mistakenly set for a PMU that does not
support this capability.
Suggested-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Kan Liang <kan.liang@linux.intel.com>
Co-developed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260824082731.1013973-13-dapeng1.mi@linux.intel.com
|
|
Move has_extended_regs() to include/linux/perf_event.h so it can be used
by arch-specific code.
While moving it, enhance the check logic and rename it to
event_has_extended_regs() to match existing perf event helper naming.
Signed-off-by: Kan Liang <kan.liang@linux.intel.com>
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260824082731.1013973-9-dapeng1.mi@linux.intel.com
|
|
Define default common __weak functions for perf_reg_value(),
perf_reg_validate(), perf_reg_abi() and perf_get_regs_user(). This helps
to eliminate the duplicated arch-specific definitions.
No function changes intended.
Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260824082731.1013973-5-dapeng1.mi@linux.intel.com
|
|
We recently removed the enum values 'preempt_dynamic_none' and
'preempt_dynamic_voluntary', but forgot to update preempt_modes[]
accordingly. Due to this, preempt_model_str() and sched_dynamic_show()
produce incorrect output, with the wrong preemption model string in
backtraces, and empty output for /sys/kernel/debug/sched/preempt.
Update preempt_modes[] to account for the removed enum values. Given
that CONFIG_PREEMPT_DYNAMIC now depends on CONFIG_ARCH_HAS_PREEMPT_LAZY,
and given that only 'preempt_dynamic_full' and 'preempt_dynamic_lazy'
remain, sched_dynamic_show() no longer needs to skip any model strings.
Fixes: 9650ce11f2e3 ("sched: dynamic: Simplify preempt model accessors")
Reported-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260902101637.232129-1-mark.rutland@arm.com
|
|
Fix "succesful" -> "successful" (x2) in the kernel-doc for
dmem_cgroup_region::ref and dmem_cgroup_try_charge(). No functional
change.
Signed-off-by: Manush Prajwal <manushprajwal555@gmail.com>
Reviewed-by: Natalie Vock <nat@pixelcluster.dev>
Signed-off-by: Natalie Vock <nat@pixelcluster.dev>
Link: https://patch.msgid.link/20260813104236.1532-1-manushprajwal555@gmail.com
|
|
|
|
Change the nbcon_irq_work to be IRQ_WORK_LAZY, thus not raising
an IRQ upon irq_work queuing. The irq_work is then handled on the
next kernel tick. This additional delay is acceptable because
nbcon_irq_work is only responsible for non-emergency deferred
printing, which is delayed anyway. This has the benefit of not
needing to raise an IRQ for each printk() call.
On a side note, the Tegra20 and Tegra30 platforms can hang if an
irq_work IRQ is raised while entering cpuidle states. This problem
was reproducible by calling printk() while entering cpuidle. So
this change also provides a workaround for these platforms (as long
as they are not running tickless).
Link: https://lore.kernel.org/lkml/f3757a75-0ba1-4558-bf57-f19ab7e59a4c@nvidia.com
Fixes: 76f258bf3f2a ("printk: nbcon: Introduce printer kthreads")
Signed-off-by: John Ogness <john.ogness@linutronix.de>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Tested-by: Jon Hunter <jonathanh@nvidia.com>
Link: https://patch.msgid.link/20260901093245.344455-3-john.ogness@linutronix.de
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
Ensure any pending nbcon_irq_work is flushed before allowing the
console to be recycled.
Signed-off-by: John Ogness <john.ogness@linutronix.de>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Link: https://patch.msgid.link/20260901093245.344455-2-john.ogness@linutronix.de
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
It's a trivial and non-functional change, the removed explicit function
type cast does not convert the function type of struct balance_callback
member (*func), and the cast can be removed.
Signed-off-by: Vladimir Zapolskiy <vz@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260819080159.105433-1-vz@kernel.org
|
|
Replace ktime_us_delta() < 0 with ktime_before() so the past-expiry
test uses the same resolution as act/now.
Signed-off-by: Liang Hao <haohlliang@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juri Lelli <juri.lelli@redhat.com>
Link: https://patch.msgid.link/20260816032643.44969-1-haohlliang@gmail.com
|
|
Both raw static keys and static_key_true/false() have been deprecated.
Migrate to new API.
It's a bit tricky to convert the sched_feat array to the new API because
the new one has two different types for true and false keys. Use a union
to wrap it.
No functional change.
Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260819081312.12447-1-hongyan.xia@transsion.com
|
|
paravirt_steal_rq_enabled and paravirt_steal_enabled use raw static_key
APIs which are now deprecated. Use the new API instead.
No functional change.
Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/20260819081207.12150-1-hongyan.xia@transsion.com
|
|
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models.
When CONFIG_PREEMPT_DYNAMIC=y, the NONE and VOLUNTARY models cannot be
used, and neither CONFIG_PREEMPT_NONE nor CONFIG_PREEMPT_VOLUNTARY can
be selected.
Simplify the preempt model accessors accordingly: remove the out-of-line
implementations of preempt_model_none() and preempt_model_voluntary(),
and remove the preempt_dynamic_{none,voluntary} values.
There are no current users of preempt_model_voluntary(), but I've kept
it around for consistency, with every scheduler model having its own
preempt_model_*() accessor.
Suggested-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Tested-by: Mete Durlu <meted@linux.ibm.com>
Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260803191731.3244294-7-mark.rutland@arm.com
|
|
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models.
Switching model only changes the behaviour of dynamic_preempt_lazy(),
which uses a static key. There are no other static calls or static keys,
and so there's no need for HAVE_PREEMPT_DYNAMIC_CALL or
HAVE_PREEMPT_DYNAMIC_KEY.
The static key used by dynamic_preempt_lazy() is entirely local to
kernel/sched/core.c, and any architecture which selects
ARCH_HAS_PREEMPT_LAZY implements the necessary support. Remove
PREEMPT_DYNAMIC's dependencies on HAVE_PREEMPT_DYNAMIC_CALL and
HAVE_PREEMPT_DYNAMIC_KEY entirely, leaving the dependency on
ARCH_HAS_PREEMPT_LAZY.
For architectures which previously selected HAVE_PREEMPT_DYNAMIC_KEY,
PREEMPT_DYNAMIC will now be selected by default, matching x86. As the
runtime impact is limited to dynamic_preempt_lazy(), this shouldn't be
as concerning as previously (e.g. where calls to {cond,might}_resched()
stubs could introduce a measurable penalty).
As all of this can work without jump labels, JUMP_LABEL is not selected
explicitly. It is obviously preferable to have JUMP_LABEL enabled, but
this is not functionally necessary.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Tested-by: Mete Durlu <meted@linux.ibm.com>
Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260803191731.3244294-6-mark.rutland@arm.com
|
|
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models.
In either model, irqentry_exit_cond_resched() is always called and never
disabled.
Remove the unnecessary code for this when PREEMPT_DYNAMIC is selected.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Mete Durlu <meted@linux.ibm.com>
Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260803191731.3244294-5-mark.rutland@arm.com
|
|
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models.
In either model, both preempt_schedule() and preempt_schedule_notrace()
are always called and never disabled.
Remove the unnecessary code for these when PREEMPT_DYNAMIC is selected.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Mete Durlu <meted@linux.ibm.com>
Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260803191731.3244294-4-mark.rutland@arm.com
|
|
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models.
In either model, both cond_resched() and might_resched() are always
disabled and do nothing.
Remove the unnecessary code for these when PREEMPT_DYNAMIC is selected.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Mete Durlu <meted@linux.ibm.com>
Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260803191731.3244294-3-mark.rutland@arm.com
|
|
On architectures which select ARCH_HAS_PREEMPT_LAZY, it has not been
possible to select the NONE and VOLUNTARY preemption models since
commit:
7dadeaa6e851 ("sched: Further restrict the preemption modes")
... which was merged in v7.0.
All architectures which currently support PREEMPT_DYNAMIC select
ARCH_HAS_PREEMPT_LAZY:
[mark@lakrids:~/src/linux]% git describe HEAD
v7.2-rc1-1-g871a4586ea2e1
[mark@lakrids:~/src/linux]% git grep 'select HAVE_PREEMPT_DYNAMIC_' -- arch
arch/arm64/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY
arch/loongarch/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY
arch/powerpc/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY
arch/riscv/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY
arch/s390/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY
arch/x86/Kconfig: select HAVE_PREEMPT_DYNAMIC_CALL
[mark@lakrids:~/src/linux]% git grep 'select ARCH_HAS_PREEMPT_LAZY' -- arch
arch/arm64/Kconfig: select ARCH_HAS_PREEMPT_LAZY
arch/loongarch/Kconfig: select ARCH_HAS_PREEMPT_LAZY
arch/powerpc/Kconfig: select ARCH_HAS_PREEMPT_LAZY
arch/riscv/Kconfig: select ARCH_HAS_PREEMPT_LAZY
arch/s390/Kconfig: select ARCH_HAS_PREEMPT_LAZY
arch/x86/Kconfig: select ARCH_HAS_PREEMPT_LAZY
... and hence in practice PREEMPT_DYNAMIC no longer supports the NONE or
VOLUNTARY preemption models.
Make this official: have PREEMPT_DYNAMIC depend on
ARCH_HAS_PREEMPT_LAZY, and remove the trivially unreachable code.
Further simplifications will be made in subsequent patches.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Mete Durlu <meted@linux.ibm.com>
Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
Link: https://patch.msgid.link/20260803191731.3244294-2-mark.rutland@arm.com
|
