| Age | Commit message (Collapse) | Author |
|---|
|
unregister_fprobe() and unregister_fprobe_async() (used by BPF
kprobe-multi) rely on standard RCU grace periods (synchronize_rcu()
and call_rcu()) to wait until in-flight fprobe handlers complete before
freeing the fprobe.
However, if an fprobe handler executes while RCU is not watching (such
as in the idle loop or nohz_full extended quiescent states), standard
RCU does not track preemption-disabled sections. Consequently,
synchronize_rcu() does not wait for those executions, which can lead
to a use-after-free if the fprobe is freed immediately after
unregistration. Ensure handlers exit early when !rcu_is_watching().
Furthermore, fprobe_fgraph_entry() and fprobe_ftrace_entry() previously
used guard(rcu)() and rcu_read_lock(), which invoke lockdep on every
hit under CONFIG_PROVE_LOCKING. This adds overhead and can cause lockdep
recursion if probed functions interact with lockdep.
Since rhltable_lookup() and rhl_for_each_entry_rcu() use
rcu_dereference_all_check() (which checks rcu_read_lock_any_held()),
holding preemption disabled via rcu_read_lock_sched_notrace() is fully
valid and sufficient so long as rcu_is_watching() is true.
Define and use guard(rcu_sched_notrace)() across fprobe_ftrace_entry(),
fprobe_fgraph_entry(), and fprobe_return(). This eliminates fast-path
rcu_read_lock() and lockdep overhead while guaranteeing safe grace
period synchronization.
Link: https://lore.kernel.org/all/179064115227.394389.16910234241400391996.stgit@devnote2/
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-e46bcd68-4a56-4f19-a255-e3772980e5e3
Fixes: 657b594b2084 ("fprobe: Fix unregister_fprobe() to wait for RCU grace period")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
|
|
from BTF
With clang and pahole 1.32, x86-64 vmlinux BTF lacks
scx_bpf_task_set_lazy_resched() and sched_ext fails to initialize:
sched_ext: Failed to register kfunc sets (-22)
clang describes the lazy argument only after the prologue moves it into a
callee-saved register, which pahole 1.32 rejects as not being the argument
register. Released pahole is affected whatever the eventual fix. Keep the
argument on the stack with barrier_data() so that its location names no
register. OPTIMIZER_HIDE_VAR() would fix x86-64 but break arm64.
Fixes: f8e5a4e3f3be ("sched_ext: Add lazy preemption support")
Link: https://lore.kernel.org/r/20260925213619.2187751-1-yonghong.song@linux.dev
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The existing date range test covers +/-80,000 years around the epoch,
far below the day count at which a 32-bit long wraps (2^31 days,
roughly 5.88 million years).
Add a case at exactly 2^31 days after the epoch. Without the preceding
fix a 32-bit kernel returns tm_year=-5879541 and wrong month, day,
day-of-year and weekday; with it the result matches 64-bit kernels.
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: LLM
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/20260925165702.51677-3-kmehltretter@gmail.com
|
|
time64_to_tm() accepts a 64-bit seconds value, but stores the quotient in
long. On a 32-bit kernel the day count therefore wraps when it reaches
2^31, even though the corresponding year remains representable in
struct tm.
This is reachable when formatting externally supplied timestamps. For
example, NILFS recovery copies the little-endian 64-bit ss_create field
from an on-disk segment summary into time64_t. Its sysfs attributes then
print that value with %ptTs, whose formatter calls time64_to_tm(). A
corrupted image can consequently produce an architecture-dependent printed
date.
Keep the day quotient in s64 while leaving the seconds-within-day remainder
as long. Use div_s64_rem() for the weekday calculation so 32-bit builds do
not require compiler runtime division helpers.
Fixes: e6c2682a1da3 ("time: Add time64_to_tm()")
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: LLM
Reviewed-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/20260925165702.51677-2-kmehltretter@gmail.com
|
|
The timekeeper update path uses a bulk memcpy() to synchronize the
timekeeper structure, which is not guaranteed to be atomic. This allows for
torn reads in ktime_get_seconds() which bypasses the sequence counter
protection for performance.
To prevent reading a torn ktime_sec value, enforce atomic-like
access by using WRITE_ONCE() for the critical field before the bulk
memcpy() in timekeeping_update_from_shadow(). Correspondingly, use
READ_ONCE() in ktime_get_seconds() to ensure a fresh, consistent load
from memory.
The same was done for xtime_sec and ktime_get_real_seconds() in commit
d7fc133bf91f ("timekeeping: Use READ_ONCE/WRITE_ONCE() for xtime_sec to
prevent tearing").
Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Acked-by: John Stultz <jstultz@google.com>
Link: https://patch.msgid.link/20260903-timekeeping-ktime_sec-v1-1-b33017536aa9@linutronix.de
|
|
The tmigr_event structure's ignore field is sometimes accessed locklessly,
but the __tmigr_cpu_activate(), tmigr_cpu_new_timer(), and
__tmigr_cpu_deactivate() functions do not mark accesses to this field.
Therefore, use READ_ONCE() for the tmigr_cpu_new_timer() function's
lockless load and WRITE_ONCE() for the __tmigr_cpu_activate() and
__tmigr_cpu_deactivate() functions' stores.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260919002523.3133928-2-paulmck@kernel.org
|
|
hlist_unhashed_lockless() locklessly samples the hlist_node structure's
pprev field, but detach_timer(), hlist_move_list(), and hlist_splice_init()
all use plain C-language stores to update this field, despite the "The
READ_ONCE() is paired with the various WRITE_ONCE() in hlist helpers that
are defined below" in the hlist_unhashed_lockless() header comment.
Therefore use WRITE_ONCE() for these hlist_node::pprev updates.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260919002523.3133928-1-paulmck@kernel.org
|
|
Updates to base->running are protected by the hrtimer base lock, but some
loads are lockless. Therefore, prevent compiler mischief by applying
READ_ONCE() to the lockless loads.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260919001428.3133388-12-paulmck@kernel.org
|
|
Currently, the hrtimer_switch_to_hres() function unconditionally updates
hrtimer_resolution to HIGH_RES_NSEC. This causes KCSAN false positives
in kernels built with CONFIG_KCSAN_STRICT=y. It also causes unnecessary
cache-line traffic on the many code paths that load hrtimer_resolution,
which is especially regrettable because the value of hrtimer_resolution
normally changes only once during the lifetime of a given kernel.
Therefore, update the value of hrtimer_resolution to HIGH_RES_NSEC only
if it does not already have that value.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260919001428.3133388-10-paulmck@kernel.org
|
|
The hrtimer_sleeper structure's ->task field is used as a flag to indicate
that the associated hrtimer has expired. This means that the hrtimer
handler can be storing to this field while other code is loading from it
to check for expiry. Note that additional races appear for hrtimers that
can be restarted, which could be argued to be a user error. However, that
is no reason to let the compiler introduce additional confusion, and to
this end, the hrtimer_sleeper_task_get() was introduced, use of which also
has the benefit of avoiding open-code access to hrtimer_sleeper innards.
Therefore, apply this accessor to handle_early_requeue_pi_wakeup().
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: André Almeida <andrealmeid@igalia.com>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Link: https://patch.msgid.link/20260919001428.3133388-9-paulmck@kernel.org
|
|
The hrtimer_sleeper structure's ->task field is used as a flag to indicate
that the associated hrtimer has expired. This means that the hrtimer
handler can be storing to this field while other code is loading from it
to check for expiry. Note that additional races appear for hrtimers that
can be restarted, which could be argued to be a user error. However, that
is no reason to let the compiler introduce additional confusion, and to
this end, the hrtimer_sleeper_task_get() was introduced, use of which also
has the benefit of avoiding open-code access to hrtimer_sleeper innards.
Therefore, apply this accessor to rt_mutex_slowlock_block().
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Link: https://patch.msgid.link/20260919001428.3133388-8-paulmck@kernel.org
|
|
The hrtimer_sleeper structure's ->task field is used as a flag to indicate
that the associated hrtimer has expired. This means that the hrtimer
handler can be storing to this field while other code is loading from it
to check for expiry. Note that additional races appear for hrtimers that
can be restarted, which could be argued to be a user error. However, that
is no reason to let the compiler introduce additional confusion, and to
this end, the hrtimer_sleeper_task_get() was introduced, use of which also
has the benefit of avoiding open-code access to hrtimer_sleeper innards.
Therefore, apply this accessor to schedule_hrtimeout_range_clock().
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Link: https://patch.msgid.link/20260919001428.3133388-6-paulmck@kernel.org
|
|
The hrtimer_sleeper structure's ->task field is used as a flag to indicate
that the associated hrtimer has expired. This means that the hrtimer
handler can be storing to this field while other code is loading from it
to check for expiry. Note that additional races appear for hrtimers that
can be restarted, which could be argued to be a user error. However, that
is no reason to let the compiler introduce additional confusion, and to
this end, the hrtimer_sleeper_task_get() was introduced, use of which also
has the benefit of avoiding open-code access to hrtimer_sleeper innards.
Therefore, apply this accessor to kernel/futex/waitwake.c.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: André Almeida <andrealmeid@igalia.com>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Link: https://patch.msgid.link/20260919001428.3133388-5-paulmck@kernel.org
|
|
The hrtimer_sleeper structure's ->task field is used as a flag to indicate
that the associated hrtimer has expired. This means that the hrtimer
handler can be storing to this field while other code is loading from it
to check for expiry. Note that additional races appear for hrtimers that
can be restarted, which could be argued to be a user error. However,
that is no reason to let the compiler introduce additional confusion.
Therefore, mark data-racy accesses to the hrtimer_sleeper ->task field
using READ_ONCE() (using a new hrtimer_sleeper_task_get() access function)
and WRITE_ONCE() (using a new hrtimer_sleeper_task_set() access function).
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Link: https://patch.msgid.link/20260919001428.3133388-1-paulmck@kernel.org
|
|
tick_nohz_next_event() limits a CPU's sleep interval to the maximum
deferment supported by the current clocksource when that CPU owns the
do_timer() duty. If the duty is unassigned, the limit also applies when
the CPU's TS_FLAG_DO_TIMER_LAST flag is set.
After the early timer checks, the function currently reads the maximum
deferment unconditionally. It then replaces the result with KTIME_MAX
unless one of the two conditions above applies.
timekeeping_max_deferment() performs a seqcount-protected read of the
shared timekeeper and follows its clocksource pointer. Check the do_timer
state first and avoid this work when the result would be discarded. This
leaves the resulting expiry unchanged and reduces accesses to timekeeper
data that is modified regularly.
On x86-64 this removes 18-20 dynamically executed instructions, including
the call, from the common non-owner path when the seqcount does not retry.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Rik van Riel <riel@surriel.com>
Link: https://patch.msgid.link/20260918133408.2834751-1-usama.arif@linux.dev
|
|
tick_nohz_update_jiffies() is only reached through tick_irq_enter() from
hard interrupt entry, where local interrupts are already disabled.
tick_do_update_jiffies64() requires interrupts to be disabled. The
local_irq_save()/local_irq_restore() pair around it therefore provides no
additional interrupt exclusion.
Remove it and assert the caller invariant instead.
On x86-64, this avoids the redundant RFLAGS save, CLI, saved-IF test and
conditional branch on the tick-stopped path.
On a 252-CPU host running a Thrift RPC service, tick_irq_enter() ran
165,116 times/s out of 212-244k interrupts/s. Only calls with
TS_FLAG_STOPPED reach the optimized path, so this is an upper bound on
its execution rate.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Link: https://patch.msgid.link/20260916141530.3423310-1-usama.arif@linux.dev
|
|
|
|
The default LOG_BUF_SHIFT is 17, which means 128 KB, and
LOG_CPU_MAX_BUF_SHIFT is 12, which means 4 KB per CPU.
By default the ring buffer is increased when
(num_possible_cpus() - 1) * 4 KB > 64 KB, which requires more than 17
CPUs.
Remove the explicit subtraction of one, so the ring buffer is increased
for more than 16 CPUs, which is a power of two. Also fix the outdated
comment.
Suggested-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Reviewed-by: John Ogness <john.ogness@linutronix.de>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Link: https://patch.msgid.link/20260927091756.50831-1-ekffu200098@gmail.com
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
Since commit f244b4dc53e5 ("printk: ringbuffer: Improve prb_next_seq()
performance"), desc_read() allows @desc_out to be NULL. We can drop the
local `desc` variable in desc_reopen_last() and prb_first_seq(). Also
update the comment for desc_read().
v1->v2:
Update the comment suggested by John.
Signed-off-by: Wang Wensheng <wsw9603@163.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: John Ogness <john.ogness@linutronix.de>
Link: https://patch.msgid.link/20260926094502.251204-1-wsw9603@163.com
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
In commit b845b517b5e37 ("printk: robustify printk") initially got its
delayed wake up which is today implemented via the irq_work. Back then
it was a variable which was assigned to 1 if a pending wake up was
waiting. This variable was checked on each timer tick. Therefore it
makes sense to assign a per-CPU variable with this task to avoid cache
line bouncing when each CPU reads the variable on each timer tick
simultaneously.
Today we have a irq_work which is enqueued onto a list and handled if it
has work items assigned. This list is already per-CPU.
We could replace the per-CPU irq_work and the per-CPU printk_pending with
two irq_work structs: One for PRINTK_PENDING_WAKEUP and the other for
PRINTK_PENDING_OUTPUT.
While unrolling the conditions in __wake_up_klogd() it becomes obvious
that PRINTK_PENDING_WAKEUP is always passed as an argument but never
checked. The return value wq_has_sleeper() of does not really matter
it's sole purpose is to enforce the needed memory barrier.
Redo the irq_work by:
- Use one irq_work (pending_wakeup_work) for klogd wakes. Schedule it
only if there is a sleeper waiting.
- Use one irq_work (&pending_output_work) for output printing. Schedule
it only if PRINTK_PENDING_OUTPUT passed.
- Drop unused PRINTK_PENDING_WAKEUP.
- Remove the preempt_disable() section. Its purpose was to ensure that
printk_pending and wake_up_klogd_work is accessed from the same CPU.
Worst case is defer_console_output() where two irq_work are scheduled but
those get processed one after the other during the HZ tick so it
shouldn't get worse.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: John Ogness <john.ogness@linutronix.de>
Link: https://patch.msgid.link/20260914143412.1sMcp89V@linutronix.de
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
Ensure that vlen and flags combinations are consistent for
BTF_KIND_LOC_PARAMs.
Fixes: 33c5a3278bdb ("btf: Extend UAPI to support BTF location (inline site) info")
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260926175113.2368566-2-alan.maguire@oracle.com
|
|
All supported BPF-subprogram argument types now pass through
check_func_arg() from one guarded branch in the legacy validation loop.
Replace that loop and its outgoing-stack validation with
check_func_args().
The scratch prototype is indexed by BTF parameter and the call metadata
carries the BTF function prototype. The existing BTF argument iteration
therefore maps parameters to ABI slots for both kfunc and BPF subprogram
calls, including additional slots occupied by by-value aggregates.
The common checker can return non-EFAULT errors other than -EINVAL, as
the existing BTF-ID path already did. This is compatible with subprog
call handling: only -EFAULT is fatal. Any other error marks BTF
unreliable. Static subprogs can then fall back to inline verification,
while global subprogs reject the call.
Remove the caller-register plumbing that the dedicated loop required.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-12-ameryhung@gmail.com
|
|
Route fixed-size global ARG_PTR_TO_MEM arguments through
check_func_arg(). Preserve their read-write access check, nullable
contract, and support for BTF-defined allocated memory.
Recognize subprog calls explicitly in the common checker. Global
subprog stack liveness can prove that bytes in an argument are unused
by the callee. Retain the existing allowance for those poisoned stack
bytes when call metadata is present, and update the nullability log
expectation.
The verifier also rejects packet pointers when the callee may change
packet data. The concrete packet-backed register state is only available
while checking the call site; the independently verified callee sees
generic PTR_TO_MEM. Record this property as subprog_may_change_pkt in
subprog-only call metadata and retain the packet-pointer rejection in the
common fixed-memory path.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-11-ameryhung@gmail.com
|
|
Route global ARG_PTR_TO_BTF_ID arguments through check_func_arg(). Use
vmlinux BTF for their type match and retain the global-subprogram rules
instead of applying kfunc-only trusted-pointer validation or runtime type
resolution.
The common nullability check now rejects non-nullable global BTF-ID
arguments before register-type matching. Update the corresponding
verifier log expectations.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-10-ameryhung@gmail.com
|
|
Subprog and helper/kfunc dynptr arguments ultimately use the same
process_dynptr_func() validation. Route the subprog dynptr branch
through check_func_arg() so register type, offset, and dynptr state
are checked in the common order.
check_func_arg() validates the register against the PTR_TO_STACK and
CONST_PTR_TO_DYNPTR compatibility set before dispatching to
process_dynptr_func(). Once the subprog path uses the common checker,
process_dynptr_func() has no caller that bypasses this validation. Remove
its now-redundant register-type check.
The existing wrong-register-type test passed a NULL local to a callee
that did not use the argument. Clang left the context pointer in R1,
while GCC materialized zero. The common checker rejected the values at
different stages and emitted different diagnostics.
Obtain a PTR_TO_BTF_ID from bpf_get_current_task_btf() and keep its
callee argument live. This makes both compilers exercise the intended
register-type mismatch.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-9-ameryhung@gmail.com
|
|
ARG_PTR_TO_ARENA accepts both arena pointers and scalars. The existing
BPF-subprogram contract includes a constant-zero scalar; it is an arena
address value rather than a conventional pointer rejected by
nullability checking.
Kfunc prototype generation already records this zero acceptance with
PTR_MAYBE_NULL for both arena suffixes; separate JIT metadata determines
whether the kernel callee receives the arena base or NULL. Apply the same
call-site representation to the temporary BPF-subprogram prototype, then
extend the guarded check_func_arg() path to cover arena arguments.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-8-ameryhung@gmail.com
|
|
Subprog and helper/kfunc context arguments require PTR_TO_CTX with an
acceptable offset. Route subprog context arguments through
check_func_arg() and update the verifier-log expectation.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-7-ameryhung@gmail.com
|
|
Global subprogram arguments tagged __arg_untrusted deliberately
accept any caller value. The callee is verified separately with a
read-only untrusted pointer, whose accesses use protected probe-read
instructions.
Keep the untrusted type in the canonical bpf_subprog_info used to
prepare the callee state. Represent it as ARG_IGNORE only in the
temporary bpf_func_proto used for call-site checking, then extend the
guarded check_func_arg() path to cover it.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-6-ameryhung@gmail.com
|
|
BPF subprogram scalar arguments are checked separately even though the
common argument checker enforces the same SCALAR_VALUE requirement.
Start the migration by routing the scalar branch through check_func_arg()
and update the verifier-log expectation.
Use the BTF parameter index for metadata and the ABI slot index for
register lookup. Route extra slots of by-value aggregates through
check_arg_extra_slot() so every occupied slot follows the common path.
Following patches can extend this guarded common-check branch as each
remaining BPF-subprogram-specific implementation is removed.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-5-ameryhung@gmail.com
|
|
The common argument checker consumes a parameter-indexed bpf_func_proto,
while BPF subprogram argument metadata is cached by ABI slot in compact
bpf_subprog_info entries. Embedding a full prototype in every fixed
subprogram entry would waste memory.
Embed a scratch prototype in the verifier environment. Fill it by
walking BTF parameters and the cached slots with separate cursors,
mirroring the kfunc representation for parameters that occupy multiple
slots. Keep the compact cache as the source of truth. The verifier
environment is already heap allocated, so this avoids a separate
allocation, failure path, and cleanup.
Keep the existing validation loop, but make it consume the generated
prototype in preparation for moving BPF subprogram calls to the common
argument checker. No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-4-ameryhung@gmail.com
|
|
Prepare subprog calls to use the common check_func_args() path. That
checker distinguishes call kinds through bpf_call_arg_meta, but
btf_check_func_arg_match() leaves both BTF and the function ID zero even
though it validates a program-BTF signature.
Represent a subprog call with a non-NULL BTF and a zero function ID.
Define helpers as NULL BTF plus nonzero ID, and kfuncs as non-NULL BTF
plus nonzero ID. Requiring a nonzero kfunc ID also prevents unavailable
special-kfunc IDs from matching subprog metadata. The corresponding
subprog predicate is introduced later alongside its first use.
Setting BTF would expose checks that treat every BTF-backed call as a
kfunc. Restrict kfunc-only BTF parameter lookup, register admission,
release diagnostics, no-cast alias, and projection handling to actual
kfuncs.
The BTF is not modified here, but bpf_call_arg_meta::btf and several
downstream consumers use non-const pointers. Match those existing types
instead of broadening this series with a const-correctness cleanup.
No functional change is intended.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-3-ameryhung@gmail.com
|
|
Several kfunc argument checks index BTF parameters using the ABI slot
number. Parameter and slot indexes diverge after a by-value argument
wider than one eightbyte, so a later argument can select the wrong BTF
parameter or index past the end of the function prototype.
This affects the BTF lookups in check_func_arg_nullability() and
check_func_arg_release(), and the parameter passed to
btf_check_iter_arg() by process_iter_arg().
check_func_arg() already tracks the BTF parameter and ABI slot
separately. Pass the parameter index to all three consumers while
retaining argno for register and stack-slot diagnostics.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-2-ameryhung@gmail.com
|
|
Return ULONG_MAX for values that don't fit an unsigned long.
proc_int_u2k_conv_uop() now correctly rejects a result above INT_MAX.
This is the erroneous behaviour that is being fixed. The input has to
exceed ULONG_MAX / HZ for the product to wrap, so the value below is
specific to CONFIG_HZ=1000:
# echo 18446744073709552 > /proc/sys/net/ipv4/tcp_keepalive_time
# cat /proc/sys/net/ipv4/tcp_keepalive_time
0
That value is now rejected with an error.
The original bound ("*u_ptr > INT_MAX / HZ") was removed in commit
2dc164a48e6f ("sysctl: Create converter functions with two new macros").
Fixes: 2dc164a48e6f ("sysctl: Create converter functions with two new macros")
Cc: stable@vger.kernel.org
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Joel Granados <joel.granados@kernel.org>
|
|
proc_int_k2u_conv_kop() reports the sign through *negp and the magnitude
through *u_ptr, but for a negative value it hands the sign-extended int to
the converter and negates the result:
*u_ptr = k_ptr_op ? -k_ptr_op((ulong)val) : -(ulong)val;
With div_hz() and CONFIG_HZ=1000 a stored -1000 becomes
(ulong)-1000 / 1000 == 18446744073709550, and negating that wraps:
# echo -1 > /proc/sys/net/ipv4/tcp_fin_timeout
# cat /proc/sys/net/ipv4/tcp_fin_timeout
-18428297329635842066
The magnitude is HZ dependent but the wrap is not: the negation happens
after the division at every CONFIG_HZ.
Take the magnitude first and convert that, which is what the open-coded
version did before commit 2dc164a48e6f ("sysctl: Create converter
functions with two new macros") folded it into a macro. The
k_ptr_op == NULL branch was already correct.
All three int converters that pass a k_ptr_op are affected:
proc_dointvec_jiffies(), proc_dointvec_userhz_jiffies() and
proc_dointvec_ms_jiffies().
Fixes: 2dc164a48e6f ("sysctl: Create converter functions with two new macros")
Cc: stable@vger.kernel.org
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Joel Granados <joel.granados@kernel.org>
|
|
bpf_tramp_image_put() makes sure a trampoline image is not freed while
a task may still be running in it, but nothing similar is done for the
progs called by that image. Detach drops the last prog reference right
away and the prog is freed after grace periods, on the basis that a
task still in the traced function skips the fexit progs once the nop at
ip_after_call is patched to a jump.
That leaves out a task sleeping in a sleepable prog that runs before
the detached one, which no grace period waits for:
CPU 0 CPU 1
in image I, sleeping in prog S
detach P from I's trampoline
-> new image, bpf_tramp_image_put(I)
bpf_prog_put(P), last ref
grace periods, P freed
back from S
__bpf_prog_enter(P)
call P->bpf_func
If S and P are fexit progs the task is already past the patched jump,
and fentry only images don't have one. On x86 this is an int3 in
poisoned bpf_prog_pack memory:
Oops: int3: 0000 [#1] SMP NOPTI
CPU: 18 UID: 0 PID: 94573 Comm: x169 Not tainted 6.18.44 #1 PREEMPT(lazy)
RIP: 0010:0xffffffffc0601d8d
Call Trace:
<TASK>
? bpf_trampoline_6442515411+0x1a4/0x21b
bpf_lsm_bprm_committed_creds+0x5/0x10
security_bprm_committed_creds+0x5f/0x70
begin_new_exec+0x2d6/0x410
...
We hit this in production when progs attached through trampolines got
detached while their hooks were busy, and it was independently found
with a fuzzer and KASAN.
Have the JITs emit a patchable nop in front of each prog call sequence
and record it in the image. When a prog is detached, patch its nop to a
jump over the call sequence. Progs that stay attached keep running for
the tasks that are in the image, and ip_after_call isn't needed anymore.
The task can be in any image that isn't freed yet, not only in the
current one. It sleeps in image I1 that calls S, P and Q, then P is
detached and the trampoline moves to image I2, then Q is detached and
its call is still in I1. So the trampoline keeps a list of its images
until they are freed, and detaching a prog patches its nop in all of
them. Images hold a reference on the trampoline for that long.
On riscv and loongarch a jump of any range takes several instructions,
and a task preempted in the middle of them could resume into half of the
new sequence. The nop is a single instruction there, patched to a near
branch through arch_bpf_trampoline_skip().
With the extra nops, BPF_MAX_TRAMP_LINKS progs no longer fit in a page
on x86 and arm64 (and already didn't on powerpc), so lower the limit
there like s390 does.
Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")
Reported-by: Sechang Lim <rhkrqnwk98@gmail.com>
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260926135605.1217928-3-florent.revest@linux.dev
Closes: https://lore.kernel.org/bpf/20260815071927.147049-1-zirajs7@gmail.com/
|
|
When a prog is detached from a trampoline, it is freed after an RCU
grace period, or an RCU tasks trace one if it is sleepable. This covers
the tasks that are running the prog, since the prog's enter helper takes
the matching read lock before the prog is called. On a preemptible
kernel, it doesn't cover a task that was preempted in the trampoline
just before the enter helper. That task holds no lock yet, and it calls
the prog after it was freed:
BUG: KASAN: vmalloc-out-of-bounds in __bpf_prog_enter_recur+0x3a5/0x3f0
Read of size 8 at addr ffffc90000055040 by task candidate/110
CPU: 1 UID: 0 PID: 110 Comm: candidate Not tainted 7.3.0-rc2-00014-g15071f2a1263-dirty #2 PREEMPT(full)
Call Trace:
<TASK>
__bpf_prog_enter_recur+0x3a5/0x3f0
bpf_trampoline_6442509193+0x37/0xf1
__x64_sys_futex+0x9/0x410
do_syscall_64+0xb0/0x530
...
Wait for an RCU tasks grace period before the existing one when freeing
a prog that was linked to a trampoline. An RCU tasks grace period only
ends once the tasks that were preempted have run again, and
bpf_tramp_image_put() already relies on it to free the image. It doesn't
wait for tasks that sleep in the trampoline, the next commit takes care
of those.
Only progs that were linked to a trampoline can be called this way, so
bpf_trampoline_add_prog() marks them and other progs are still freed as
before.
Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")
Reported-by: Junseo Lim <zirajs7@gmail.com>
Signed-off-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260926135605.1217928-2-florent.revest@linux.dev
Closes: https://lore.kernel.org/bpf/aqdrwVpanH3WGurX@omen-arch/
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull MM fixes from Andrew Morton:
- Fix module loading incorrectly returning success after alloc_tag
codetag setup failed
- Restore MADV_COLLAPSE semantics for shmem so forced collapse ignores
the shmem THP/mTHP sysfs settings
- Make alloc_tag UAPI structure padding explicit
- Fix DAMON schemes unexpectedly stopping after quotas are disabled
through the online parameter update interface
- Fix a false SW_TAGS KASAN invalid-access report when freeing vmapped
task stacks
- Split the MEMORY MANAGEMENT - MEMORY POLICY AND MIGRATION MAINTAINERS
entry into separate MIGRATION and NUMA PLACEMENT entries
- Move memory tiering maintenance under NUMA PLACEMENT
- Add Gregory Price as a NUMA PLACEMENT co-maintainer
- Add Heming Zhao as an ocfs2 reviewer
- Fix mmap_prepare() state being copied onto a merged VMA rather than
only onto a newly allocated VMA
* tag 'mm-hotfixes-stable-2026-09-27-19-12' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm:
mm/vma: predicate setting mmap_prepare VMA fields on new vma alloc
MAINTAINERS: add Heming Zhao as ocfs2 reviewer
MAINTAINERS: make Gregory a co-maintainer of MEMORY MANAGEMENT - NUMA PLACEMENT
MAINTAINERS: move memory tiering under MEMORY MANAGEMENT - NUMA PLACEMENT
MAINTAINERS: split up MEMORY MANAGEMENT - MEMORY POLICY AND MIGRATION
kasan: unpoison task stack below watermark only in generic mode
mm/damon/core: don't skip damos_adjust_quota() while esz is not zero
alloc_tag: avoid implicit padding in uapi
mm: shmem: ignore sysfs configs for shmem forced collapse
module: fix lost error code from codetag_load_module()
|
|
File capabilities need permitted and inheritable sets, an effective flag,
and a root UID. Process capabilities need four full capability sets but
neither a file effective flag nor a root UID. Give file capabilities their
own type instead of storing both representations in audit_cap_data.
This reduces each preallocated audit_names entry by 16 bytes on the tested
64-bit configurations, saving 80 bytes per audit context. The BPRM
capability auxiliary object also shrinks from 144 to 112 bytes. Retain all
logged fields, including file root IDs and process ambient capabilities;
record formats and collection behavior are unchanged.
Signed-off-by: Christian Göttsche <cgzones@googlemail.com>
Signed-off-by: Paul Moore <paul@paul-moore.com>
|
|
Proxy execution keeps a mutex-blocked donor runnable so that its
scheduling context can execute the mutex owner.
Define SCX_OPS_ENQ_BLOCKED as the admission contract for retained
donors. Schedulers without the flag block EXT donors normally.
Schedulers with the flag continue to own blocked donors and receive them
through ops.enqueue() with SCX_ENQ_BLOCKED, allowing BPF to choose their
DSQ, CPU and ordering.
After BPF dispatches a donor, proxy execution may move its context to
the rq of the mutex owner. This move is distinct from BPF placement:
proxy_set_task_cpu() changes task_cpu() while preserving the donor's
pre-proxy CPU in wake_cpu as the starting point for normal wakeup
placement. A proxy-migrated donor returns through the full wakeup
activation path when the mutex is released.
Do not carry a retained proxy session across BPF scheduler ownership
changes. Before root activation, parent-to-child takeover, rehome or
punt, fully deactivate a retained donor unconditionally. The incoming
scheduler then starts with clean task state and applies its admission
policy the next time the task blocks.
A retained donor can wake through ttwu_runnable() without another
enqueue_task_scx() call. Use WF_TTWU_RQ to identify this path, where the
wakeup skipped activation and enqueueing. wakeup_preempt_scx() requests
dispatch reconsideration after is_blocked is cleared. A full wakeup
activation has already enqueued the task and does not need another
reschedule.
Blocked-donor enqueueing takes precedence over the exiting and
migration-disabled local-DSQ fallbacks, so that an opted-in scheduler
sees every eligible donor request.
Co-developed-by: John Stultz <jstultz@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_can_stop_tick() currently treats any blocked rq->donor as an active
proxy session. However, rq->donor can temporarily still identify an
outgoing blocked task while the scheduler selects an ordinary execution
context. A dependency update in that window can keep TICK_DEP_BIT_SCHED
set after the context switch.
Track whether proxy resolution actually selected a different execution
context. Keep the tick active while that state is set instead of
deriving it from a potentially stale donor.
When proxy execution ends, clear the state and queue a balance callback
to reevaluate the tick dependency after context_switch() updates
rq->curr. This lets sched_can_stop_tick() observe the complete donor and
execution selection without adding work outside the existing
proxy-execution branch.
Conservatively re-enable the periodic scheduler tick for the duration of
proxy execution. Allowing proxy execution itself to run tickless would
require making the remote NOHZ scheduler tick aware of the split between
rq->curr and rq->donor, which is left as a future improvement.
Perform the tick update from scx_proxy_reenqueue_retry(), which already
runs after proxy resolution, so scheduler core needs no additional hook.
Compile the additional runqueue state and callback out when
CONFIG_NO_HZ_FULL is disabled.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
With proxy execution, the task selected by the scheduler and the task
physically executing can differ. A blocked mutex waiter donates its
scheduling context to the lock owner:
D -----------------> M -------------> O ----------------> T
[donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context
where:
D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task
During a proxy execution switch, D supplies the scheduling class,
priority, and runtime budget, while O supplies the execution context: O
is the task whose code physically executes. T is a competing runnable
task which may preempt the D/O proxy execution.
Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR
can be replaced with a higher scheduling class such as RT or deadline
without changing the class interaction described here.
An active proxy pair can only have a donor in the same scheduling class
as the owner or in a higher-priority class. If the owner belongs to a
higher-priority class, pick_next_task() selects the owner directly
before considering the donor, so find_proxy_task() is not entered for
that donor. The blocked-on relationship may still exist, but it does
not result in proxy execution.
The relevant combinations are:
1. D is EXT, O is EXT, T is EXT
D can interrupt T according to BPF scheduling policy. O executes
with D's EXT priority and runtime budget, while T waits in EXT.
2. D is EXT, O is EXT, T is FAIR
D is visible to the BPF scheduler, but cannot preempt T because
EXT is below FAIR. Once T stops, BPF can dispatch D and O executes
with D's EXT priority and runtime budget. If T becomes runnable
again, it preempts the D/O proxy execution.
3. D is FAIR, O is EXT, T is EXT
D preempts T as the higher-class scheduling context. O executes
with D's FAIR priority and runtime budget, while T waits in EXT.
D is not visible to the BPF scheduler.
4. D is FAIR, O is EXT, T is FAIR
D competes with T according to its FAIR deadline. When D is
selected, O executes with D's FAIR priority and runtime budget.
D is not visible to the BPF scheduler.
5. D is FAIR, O is FAIR, T is EXT
The D/O proxy execution has a FAIR scheduling context, so T cannot
preempt it from the lower EXT class.
6. D is FAIR, O is FAIR, T is FAIR
O, T, and D all have FAIR scheduling contexts. D remains runnable
as a blocked proxy donor. When CFS selects D, O executes using D's
FAIR scheduling context. When CFS selects O, O executes using its
own FAIR context, and when CFS selects T, T executes normally. D
is not visible to the BPF scheduler.
Thus, sched_ext policy and accounting must generally use rq->donor, the
scheduler-selected task which supplies the scheduling context, rather
than rq->curr, the task whose code physically executes. Without proxy
execution they are the same task.
On nohz_full CPUs, keep the tick running whenever the selected donor is
blocked. A selected blocked donor is necessarily awaiting proxy
execution, so this avoids relying on the transient relationship between
rq->curr and rq->donor during context switches. It also leaves ordinary
sched_ext context switches and non-nohz_full CPUs untouched. Allowing
proxy execution to run tickless remains a future improvement.
Moreover, extend scx_dump_state() to report both contexts. Each CPU
record now includes a donor= line. If an EXT donor differs from
rq->curr, also emit its detailed task record. The existing '*' marker
continues to identify rq->curr, while the donor= line identifies the
otherwise unmarked donor record.
Note that at this point in the series, CONFIG_SCHED_PROXY_EXEC still
depends on !CONFIG_SCHED_CLASS_EXT, so proxy execution and sched_ext
cannot be enabled together. The scheduling changes are therefore
preparatory. A later patch removes this restriction.
Co-developed-by: John Stultz <jstultz@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Without proxy execution, the DSQ lock and holding_cpu handshake ensure
that a task cannot be dequeued or start running during an rq-lock
handoff without clearing holding_cpu.
Proxy execution is an exception: a task can start physically executing
as a lock owner while its scheduling context remains on a DSQ; its
on-CPU or migration-disabled state can therefore change without
clearing holding_cpu.
Recheck these states after acquiring the source rq lock. If the transfer
can no longer proceed, park the task on the source rq's reject DSQ and
reenqueue it through its owning scheduler. This preserves the BPF
scheduler's placement policy and keeps descendant tasks within their
sub-scheduler's cap grants.
Queue a deferred drain for every rejection. If a drain finds a
proxy-rejected task still running or donating, leave it parked and
record a retry request on the rq. scx_proxy_reenqueue_retry() consumes
the request and schedules another drain when proxy state changes.
Without this change and proxy execution enabled, stress-ng --pipeherd
can trigger this race and migrate an active execution context, leading
to sleeping-while-atomic warnings and subsequent lockdep corruption.
This is a preparatory change to support proxy execution with sched_ext.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The reject DSQ is currently specific to sub-scheduler cap failures. Its
storage and initialization depend on CONFIG_EXT_SUB_SCHED, and the drain
path assumes every rejected task was rejected for SCX_TASK_REENQ_CAP.
Other transient placement failures need the same ability to park a task
on its source rq and return it to the owning BPF scheduler. Make the
reject DSQ unconditional and carry the reenqueue reason directly in
p->scx.flags from the rejection site.
A reenqueue reason remains valid while ops.enqueue() runs so that BPF
can inspect it. Clear it immediately after the callback returns and
before resolving a direct dispatch. Paths which bypass ops.enqueue()
clear the reason before the kernel-selected placement. A rejection from
the new placement can therefore install its reason into a clear field.
Route all paths through a common cleanup point to enforce this ordering.
This is a preparatory change to support proxy execution with sched_ext.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The reject DSQ drain will be generalized for placement failures outside
sub-scheduler capability rejection. Move the drain into the core
sched_ext implementation first so the subsequent behavioral changes can
be reviewed separately. No functional change.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
With proxy execution, pick_next_task() can select a blocked task as the
scheduling context before find_proxy_task() resolves the execution
context.
From the BPF scheduler perspective, that donor is running while its
scheduling context drives the lock owner; ops.tick() and other
accounting must therefore remain enclosed by a matching
ops.running()/ops.stopping() session.
In this scenario, the session boundaries do not always match physical
task switches. Keep the "running" session open when the same donor
continues on the same CPU. When proxy execution migrates a donor's
scheduling context to another CPU, end its running session on the source
CPU and start a new session on the destination CPU only after proxy
resolution succeeds. This prevents a failed resolution from exposing a
provisional ops.running() event.
Track these sessions with a new SCX_TASK_RUN_TRACKED flag. The explicit
running-state tracking is also required by later donor-based accounting:
it prevents an EXT owner executing for a non-EXT donor from being
treated as the active EXT scheduling context when it is dequeued.
This is a preparatory change for enabling proxy execution together with
sched_ext.
Acked-by: John Stultz <jstultz@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Proxy execution retains mutex-blocked donors on the runqueue so their
scheduling context can execute a lock owner. sched_ext cannot safely
inherit such donors unless the BPF scheduler explicitly participates in
their admission and ordering.
Make sched_ext reject retained donors by default. Implement
scx_allow_proxy_exec() to force blocked EXT tasks through the regular
block path in schedule().
A donor retained by another scheduling class may already be queued when
sched_ext is enabled or when a task switches to SCHED_EXT. Fully block
the donor before sched_change_begin() records its queued state. This
prevents sched_ext from importing a proxy session which the BPF
scheduler did not admit, while leaving ordinary FAIR, RT and DL class
transitions untouched.
This is a preparatory change to support proxy execution with sched_ext.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into for-7.4
Pull to receive:
313b652837d0 ("sched/core: Drop mutex locks before proxy rescheduling")
8f8c0417e973 ("sched/core: Dequeue waking proxy donors before reset")
a49653d0abeb ("sched/core: Mark wakeups completed through ttwu_runnable()")
57c75e3ae38c ("sched: Add helper to block retained proxy donors")
be100c77178e ("sched: Add sched_ext hooks for proxy execution")
for the upcoming sched_ext side of proxy execution support.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The sched_ext CID topology gives each core, LLC and NUMA node a
contiguous CID range. Without cluster entries, schedulers cannot obtain
their CID range or global index.
Walk cores by cache-sharing cluster and add cluster_cid and cluster_idx
to scx_cid_topo. Treat the level inclusively: a core without a wider
cluster forms one of its own, while an LLC-wide cache-sharing group
forms one cluster. This gives every cluster a contiguous CID range and a
dense index.
The walk now takes cores in cluster order rather than LLC order. Shards
are still cut on core boundaries, so a shard doesn't need to contain a
whole cluster; only core, cluster, LLC and node are nested, in that
order.
The level follows whatever the arch reports through
topology_cluster_cpumask(), which on x86 is the L2 sharing mask and is
populated independently of CONFIG_SCHED_CLUSTER.
Tested on a 13th Gen Intel Core i7-13800H: six SMT P-cores occupy CPUs
0-11, eight E-cores occupy CPUs 12-19 in two four-core L2 modules, and
all 20 CPUs share one LLC; the P-core pairs form two-CID clusters, the
E-core modules form four-CID clusters, and cluster_idx is dense across
all eight clusters.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
|
|
current_wq_worker() only tells us that %current is a kworker. It does
not guarantee that the worker is currently executing a work item, as
worker->current_pwq is populated only while process_one_work() is running
the work function.
is_chained_work() can be reached while queueing on a draining or
destroying workqueue. If a kworker outside work-item execution reaches
that path, current_wq_worker() returns a worker but worker->current_pwq is
NULL, and the chained-work test can fault before it emits the intended
warning.
Guard the chained-work test with worker->current_pwq. A kworker without
current_pwq is not executing work on the target workqueue, so the helper
should return false and let the draining/destroying warning fire.
Fixes: 8d03ecfe4718 ("workqueue: reimplement is_chained_work() using current_wq_worker()")
Cc: stable@vger.kernel.org
Signed-off-by: Pavankumar Kondeti <pavan.kondeti@oss.qualcomm.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
