summaryrefslogtreecommitdiff
path: root/kernel/sched
AgeCommit message (Collapse)Author
30 hoursMerge tag 'sched_ext-for-7.4' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext updates from Tejun Heo: - Proxy execution can now be enabled together with sched_ext. A BPF scheduler can opt in to receive mutex-blocked tasks that donate their scheduling context to the lock owner and place them itself. - Lazy preemption: a BPF scheduler can preempt or kick a CPU lazily, deferring the switch to the next return to user space or tick, as the fair class can. - Sub-schedulers: a parent now learns which child runs on each CPU it delegated and when a cap change takes effect. A detaching child loses its caps before the parent restores the CPUs. - New kfuncs re-enqueue single tasks from a dispatch queue iteration and report a cgroup's CPU count and a CID's NUMA node. The CID topology now includes clusters. - Hot path optimizations: dispatch queue and task lookups compare keys inline, remote task moves no longer take a dispatch queue lock twice, and sub-scheduler hooks cost nothing without sub-schedulers. - Kfuncs callable from tracing programs are hardened against NMI and IRQ nesting, which could hard-lock the CPU or pick a wrong idle CPU. - Built-in dispatch queues no longer allocate per-CPU state that grew with the square of the CPU count. - Header syncs with the scx repo, a BTF workaround for pahole 1.32, and selftest fixes and coverage. * tag 'sched_ext-for-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (69 commits) sched_ext: Drop the locks periodically while walking a user DSQ for re-enqueue sched_ext: scx_qmap: Also re-enqueue stranded tasks with scx_bpf_dsq_reenq_task() sched_ext: Add scx_bpf_dsq_reenq_task() to re-enqueue single tasks from a DSQ iteration sched_ext: Factor out the rq lock switch of scx_dsq_move() sched_ext: Don't queue balance callbacks from a core-sched pick sched_ext: Don't leave balance callbacks queued across a pick retry sched_ext: Fix a bypassed task stalling on a cpu its scheduler has no caps on sched_ext: Fix the running task keeping its protection into bypass sched_ext: Fix use-after-free of a local reenq node left on an inactive cpu sched_ext: Keep proxy donors with slice left on the local DSQ selftests/sched_ext: Report reload_loop attach failures selftests/sched_ext: Bound exit waits selftests/sched_ext: Add SCX_ASSERT_ALIVE and adopt it selftests/sched_ext: Fail loudly when EXIT_KIND is missing from BTF sched_ext: Add SCX_ECODE_RSN_CGROUP_OFFLINE to user_exit_info.h selftests/sched_ext: Fix reload_loop error-path use-after-free selftests/sched_ext: Fix build error in dequeue_remote selftest sched_ext: scx_qmap: Reset a child's cpuperf targets once its PERF revoke takes effect sched_ext: Add ops.sub_child_ecaps_updated() to report a child's effective cap changes sched_ext: scx_qmap: Show actual cid use per participant ...
2 daysMerge tag 'pm-7.4-rc1' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm Pull power management updates from Rafael Wysocki: "This time, the most visible part are the ACPI CPPC updates from Christian Loehle, including CPPC cpufreq driver changes, even though updates of cpufreq drivers proper are still the largest topic group. However, the updates of cpufreq drivers depend on the ACPI CPPC updates, so they should really be counted together and that is well over 1/2 of the total commit count. Interestingly enough, almost all of the updates included here are fixes or cleanups, with some new feature support in the amd-pstate cpufreq driver and IPQ9650 support in the qcom-nvmem cpufreq driver. The runtime PM documentation improvements from Brian Norris can also be mostly regarded as fixes, but a new kunit test goes along with them. Specifics: - Fix multiple issues related to input validation, synchronization, access safety, memory layout and register field validation, and error handling in the ACPI CPPC library (Christian Loehle and Sumit Gupta) - Select the frequency-invariance callback per CPU in the CPPC cpufreq driver and create the FIE worker before enabling PCC callbacks in it to avoid queueing work through a missing worker and dereferencing a NULL pointer (Christian Loehle) - Update the CPPC cpufreq driver entry in MAINTAINERS and add a new entry for the ACPI CPPC library (Christian Loehle) - Minor fixes in various cpufreq drivers (Alexander Koskovich, Daniel Schwierzeck, Frank Zhang, Haotian Zhang, Kathiravan Thirumoorthy, Sumeet Pawnikar, and Xueqin Luo) - Fix wrong behaviors on multi-node servers in the Loongson cpufreq driver (Huacai Chen) - Reject NULL from cpufreq_cpu_get() in the Rust code (Mehmet Koseoglu) - Add suspend OPP to mediatek cpufreq driver and DT (Ryan Brue) - Fixups for a few bugs identified by Sashiko in the amd-pstate cpufreq driver (Mario Limoncielloi, Wentao Guan) - Initial support for maximum frequency capacity handling on Zen6 client in amd-pstate (Mario Limonciello) - Fix for amd-pstate-ut with 'Requested CPU Min frequency' BIOS option (K Prateek Nayak) - New Zen6 tuning values for EPP in amd-pstate (Mario Limonciello, Vishal Badole) - amd-pstate documentation fix (Karl Mehltretter) - Restrict deferred downscaling to low load for shared policies in the conservative cpufreq governor to avoid undesirable CPU frequency reduction in some cases (Shengming Hu) - Serialize sugov_start() against rate limit updates in the schedutil cpufreq governor to avoid using a stale rate limit value (Hui Su) - Prevent cpuidle governors menu and teo from returning idle state 0 after it has been disabled (Xueqin Luo) - Replace min_t() with the better alternatives in the intel_idle driver and make it use 2-argument strscpy() (Andy Shevchenko) - Fix two repeated phrase instances in cpuidle (Hemanth Selam, Syazwina Fazrul) - Add rugged Panther Lake support to the intel_rapl power capping driver and make it use %pe for printing error pointers (Srinivas Pandruvada and Sumeet Pawnikar) - Update and improve runtime PM documentation and add a kunit test for supplier idle/suspend (Brian Norris) - Add a pm_runtime_dont_use_autosuspend() call to pm_runtime_reinit() to ensure autosuspend disabling on driver teardown (Joshua Crofts) - Ignore page-zero RAM in E820 checksum during hibernation on x86 to avoid rejecting images created after kexec() (Matthias Goergens) - Add resume event mapping for PMSG_POWEROFF to the PM core to allow it to carry out device shutdown in the future (Mario Limonciello) - Add DPM watchdog to prepare/late/early/noirq/complete phases of device suspend and resume (Mayank Rungta) - Update swsusp.txt references to swsusp.rst in hibernation documentation (Li Jun) - Fix use-after-free of perf domain in netlink doit handlers for energy models (Fan Wu) - Add synchronize_rcu() back to cpu_pm_unregister_notifier() after it was removed inadvertently in previous development cycles to avoid use-after-free (Bradley Morgan) - Unregister CPU PM notifier outside the timer lock in the timer-ti-dm clocksource driver (Bradley Morgan) - Fix typo "sucessfully" in a comment in pm_clk_add_clk() kerneldoc comment and typo "hierachy" in a comment in pm-graph (Hemanth Selam) - Fix the return value in dev_pm_opp_get_of_node() kernel-doc (Karl Mehltretter)" * tag 'pm-7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm: (89 commits) cpufreq: loongson3: Replace IOCSR read/write with MMIO ones cpufreq: loongson3: Use global physical CPU ID in get/target callbacks cpufreq: loongson3: Replace per-package mutex with per-node cpufreq: loongson3: Adjust the width of id and val in smc_message cpufreq: loongson3: Make this driver depend on MACH_LOONGSON64 PM: runtime: Expand introduction with core concepts and structure cpufreq: schedutil: Serialize sugov_start() against rate limit updates MAINTAINERS: ACPI: Add a separate CPPC entry MAINTAINERS: cpufreq: Replace CPPC reviewer arm64: dts: mediatek: mt8173: Mark the lowest CPU OPPs for suspend cpufreq: mediatek: Enter system sleep at the suspend OPP cpufreq: qcom-cpufreq-hw: fix device node refcount leak in phandle parsing cpufreq: qcom-nvmem: Add IPQ9650 support cpufreq: Add Milos to cpufreq-dt-platdev blocklist cpufreq: Use %pe to print error pointers symbolically rust: rcpufreq_dt: add module alias cpufreq: sti: avoid NULL dereference in dev_err() cpufreq: tegra194: fix double-pointer error in get_cpu_ndiv cpufreq: sparc-us2e: fix frequency table index copy-paste error rust: cpufreq: reject NULL from cpufreq_cpu_get() ...
3 daysMerge tag 'sched-core-2026-10-18' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull scheduler updates from Ingo Molnar: "Kconfig updates: - Simplify PREEMPT_DYNAMIC (Mark Rutland) Fair scheduler updates: - Remove redundant RUN_TO_PARITY check (Elif Topuz) - Declare augmented callbacks per field, fix rb_add_augmented_cached() descent (Yiwei Lin, Peter Zijlstra) - Restart fair hrtick after same-task repicks (Shubhang Kaushik) - Improve the scheduling latency of short slice tasks (Vincent Guittot) Deadline/RT scheduler updates: - Check start_dl_timer expiry with ktime_before() (Liang Hao) - irq_work: CPU-hotplug improvements on PREEMPT_RT (Sebastian Andrzej Siewior) Energy-aware scheduling updates: - Implement the steal_governor, introduce preferred CPUs and steal-driven vCPU backoff (Shrikanth Hegde) Multi-processor load-balancing updates: - Enable preferred SMT siblings on NVIDIA Olympus (Andrea Righi) - Randomize equally shallow idle CPU picks (Christian Loehle) - Remove redundant core_sched_seq (Hui Su) Cache-aware scheduling updates: - Avoid publishing partial LLC utilization (Jemmy Wong) NUMA-balancing updates: - Reset NUMA fault locality after scan period update (Eric Kim) Proxy execution updates: - Ongoing work to implement sleeping owner handling for proxy execution (John Stultz, Vasily Gorbik) - Preparatory steps to make sched_ext compatible with proxy execution (Andrea Righi) Scheduler infrastructure updates & fixes: - Remove the RCU wait from PSI trigger destruction (Shakeel Butt) - Avoid modifying wait-entry on failed priority registration (Hui Su) - Use the new static key API for sched_feat (Hongyan Xia) - Convert paravirt_steal to new static key APIs (Hongyan Xia) - cputime: Don't account idle time twice after dyntick-idle (Stian Halseth) Scheduler debugging updates: - Set TIF_NEED_RESCHED before calling __trace_set_need_resched() (Sechang Lim) - Fix context analysis errors in non-preferred CPU push (Shrikanth Hegde) Misc cleanups and fixes by Christian Loehle, Shubhang Kaushik, Thomas Huth and Vladimir Zapolskiy" * tag 'sched-core-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (64 commits) sched: Replace nr_iowait_cpu() with a bool accessor Revert "sched/fair: Remove dead code on enqueue_task_fair()" sched/cache: Avoid publishing partial LLC utilization sched/cputime: Don't account idle time twice after dyntick-idle sched/psi: Remove the RCU wait from trigger destruction sched/eevdf: Remove redundant RUN_TO_PARITY check sched/wait: Avoid modifying entry on failed priority registration rbtree_test: Use rb_add_augmented_cached() for the cached augmented test rbtree: Update augmented data on the way down in rb_add_augmented_cached() rbtree: Declare augmented callbacks per field with RB_AUG() rbtree_test: Use rb_add() and rb_add_cached() for the basic tests smpboot: Don't park the thread if work is pending irq_work: Flush lazy work CPU down on PREEMPT_RT irq_work: Update a comment regarding CPU hotplug invocation sched/topology: Add asymmetric SMT packing override sched/fair: Honor asymmetric SMT priority in idle selection sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched() sched/core: Fix context analysis errors in non-preferred CPU push virt/steal_governor: Enable the driver virt/steal_governor: Implement steal_governor policy loop ...
5 daysMerge branch 'pm-cpufreq'Rafael J. Wysocki
Merge cpufreq updates for 7.4: - Minor fixes in various cpufreq drivers (Alexander Koskovich, Daniel Schwierzeck, Frank Zhang, Haotian Zhang, Kathiravan Thirumoorthy, Sumeet Pawnikar, and Xueqin Luo) - Fix wrong behaviors on multi-node servers in the Loongson cpufreq driver (Huacai Chen) - Reject NULL from cpufreq_cpu_get() in the Rust code (Mehmet Koseoglu) - Add suspend OPP to mediatek cpufreq driver and DT (Ryan Brue) - Fixups for a few bugs identified by Sashiko in the amd-pstate cpufreq driver (Mario Limoncielloi, Wentao Guan) - Initial support for maximum frequency capacity handling on Zen6 client in amd-pstate (Mario Limonciello) - Fix for amd-pstate-ut with 'Requested CPU Min frequency' BIOS option (K Prateek Nayak) - New Zen6 tuning values for EPP in amd-pstate (Mario Limonciello, Vishal Badole) - amd-pstate documentation fix (Karl Mehltretter) - Restrict deferred downscaling to low load for shared policies in the conservative cpufreq governor to avoid undesirable CPU frequency reduction in some cases (Shengming Hu) - Serialize sugov_start() against rate limit updates in the schedutil cpufreq governor to avoid using a stale rate limit value (Hui Su) * pm-cpufreq: (34 commits) cpufreq: loongson3: Replace IOCSR read/write with MMIO ones cpufreq: loongson3: Use global physical CPU ID in get/target callbacks cpufreq: loongson3: Replace per-package mutex with per-node cpufreq: loongson3: Adjust the width of id and val in smc_message cpufreq: loongson3: Make this driver depend on MACH_LOONGSON64 cpufreq: schedutil: Serialize sugov_start() against rate limit updates arm64: dts: mediatek: mt8173: Mark the lowest CPU OPPs for suspend cpufreq: mediatek: Enter system sleep at the suspend OPP cpufreq: qcom-cpufreq-hw: fix device node refcount leak in phandle parsing cpufreq: qcom-nvmem: Add IPQ9650 support cpufreq: Add Milos to cpufreq-dt-platdev blocklist cpufreq: Use %pe to print error pointers symbolically rust: rcpufreq_dt: add module alias cpufreq: sti: avoid NULL dereference in dev_err() cpufreq: tegra194: fix double-pointer error in get_cpu_ndiv cpufreq: sparc-us2e: fix frequency table index copy-paste error rust: cpufreq: reject NULL from cpufreq_cpu_get() cpufreq: amd-pstate-ut: Fix max_freq and EPP test failures on Zen6 cpufreq: amd-pstate: Propagate cppc_set_auto_sel() errors on mode change cpufreq: amd-pstate: Restore previous mode when changing driver mode fails ...
5 daysMerge tag 'sched-urgent-2026-10-18' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull scheduler fixes from Ingo Molnar: - Fix __sched_setscheduler() class change race causing lost reschedules, exposed by recent RCU changes (K Prateek Nayak) - Fix s64 multiplication overflow in place_entity()'s vlag scaling code that can corrupt an entity's placement (Hui Su) * tag 'sched-urgent-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: sched/fair: Avoid overflow in place_entity() sched/core: Call wakeup_preempt() for queued tasks during class change
6 dayssched_ext: Drop the locks periodically while walking a user DSQ for re-enqueueTejun Heo
reenq_user() drops dsq->lock around each re-enqueue and its rq lock every SCX_TASK_ITER_BATCH re-enqueues. A walk filtered by SCX_REENQ_MARKED skips unmarked tasks without dropping either, so a long run of unmarked tasks is walked in one hold of both locks. Count the visited tasks instead of the re-enqueued ones and drop both locks every SCX_TASK_ITER_BATCH visits. An unfiltered walk drops the locks as often as before. Suggested-by: Andrea Righi <arighi@nvidia.com> Link: https://lore.kernel.org/all/atP-LIKpxPhR-Nuc@gpd4/ Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
6 dayssched_ext: Add scx_bpf_dsq_reenq_task() to re-enqueue single tasks from a ↵Tejun Heo
DSQ iteration scx_bpf_dsq_reenq() re-enqueues a whole DSQ, but a scheduler that loses the cpus serving a DSQ, such as a sub-scheduler after a cap revoke, needs only the stranded tasks placed again. Add scx_bpf_dsq_reenq_task(), which marks a task from a DSQ iteration. When the iteration ends, the DSQ's deferred re-enqueue walk is scheduled once and re-enqueues only the marked tasks. The kfunc locks the task's rq and the DSQ like scx_bpf_dsq_move() and is callable from the same contexts. The mark is cleared when the task leaves the DSQ, so a task consumed before the walk is not re-enqueued. An iteration's DSQ can be destroyed before the iteration ends, so schedule_dsq_reenq() now skips a destroyed DSQ instead of aborting the scheduler. scx_bpf_dsq_reenq() racing scx_bpf_destroy_dsq() takes the same path. Declare the kfunc for the tools and add a compat probe. v2: Note in the kerneldoc that another iteration's re-enqueue may take a marked task early, and that a task on another scheduler returns false (Andrea). Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
6 dayssched_ext: Factor out the rq lock switch of scx_dsq_move()Tejun Heo
Factor the rq lock switch on entry to scx_dsq_move() and its undo on exit out into kf_switch_rq_lock() and kf_restore_rq_lock() so that other kfuncs operating on a task from a DSQ iteration can share them. No functional change. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
6 dayssched_ext: Merge branch 'for-7.3-fixes' into for-7.4Tejun Heo
Pull to receive 680edc7f23ad ("sched_ext: Don't queue balance callbacks from a core-sched pick") and resolve its conflict with the proxy execution work here so that for-next merges clean. Conflicts: - do_pick_task_scx(): 680edc7f23ad samples sched_core_enabled() into a new local next to the prev declaration, which 3d6577ea93c9 ("sched_ext: Split curr|donor references properly") changed to rq->donor. Kept rq->donor and added the local. Signed-off-by: Tejun Heo <tj@kernel.org>
6 dayssched_ext: Don't queue balance callbacks from a core-sched pickAndrea Righi
Commit ff0545e0c263 ("sched_ext: Don't leave balance callbacks queued across a pick retry") queues the deferred work as a balance callback only when do_pick_task_scx() returns a final pick. However, under core scheduling, a pick that returns a task isn't final either: pick_next_task() goes on to pick the SMT siblings, and their dispatch can release the shared rq lock with the callback still queued, triggering the rq_pin_lock() warning. Moreover, dispatch_core_pick() returns early when another dispatch is in flight on the rq, and do_pick_task_scx() then consumes %SCX_RQ_BAL_CB_PENDING set by that dispatch while it's still running. This can be reproduced running scx_qmap -I on SMT siblings with coresched-tagged workloads that frequently sleep, mixed with untagged CPU-bound tasks. Fix this by always deferring the pending work to the irq_work path from dispatch_core_pick() under core scheduling, for both the local and the sibling picks, and by leaving %SCX_RQ_BAL_CB_PENDING alone in do_pick_task_scx(), so that it's handled by the dispatch that set it. Sample the core scheduling state once at the start of the pick, since core scheduling can get enabled while a non-core dispatch has the rq lock released, and the follow-up work must match the dispatch path that ran. Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()") Cc: stable@vger.kernel.org # v6.19+ Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
6 dayssched_ext: Don't leave balance callbacks queued across a pick retryAndrea Righi
The dispatch path can request deferred work to be run from a balance callback and pick_task_scx() queues rq->scx.deferred_bal_cb right after dispatching. However, a balance callback must be consumed before the rq lock is released and pick_task_scx() is not necessarily the end of the pick: - it returns RETRY_TASK when a higher priority class task has been enqueued while the dispatch had the rq lock released; - the dl_server pick can return NULL, in which case the lower classes are picked next. In both cases the callback stays queued while the core keeps picking, and the rq lock can be released again, either by prev_balance() or by another dispatch. For example, with an RT task as prev: __schedule() __pick_next_task() pick_task_scx() dispatch_pick() maybe_queue_balance_callback() <- deferred_bal_cb queued return RETRY_TASK prev_balance() balance_rt() pull_rt_task() double_lock_balance() <- rq lock released Any CPU acquiring the rq lock in that window finds a pending balance callback and triggers the following warning in rq_pin_lock(): WARNING: kernel/sched/sched.h:1919 at rq_lock+0x7f/0xe0, CPU#10: stress-ng-sleep/533 CPU: 10 UID: 0 PID: 533 Comm: stress-ng-sleep Not tainted 7.2.0-rc6-virtme #61 PREEMPT(full) Sched_ext: qmap (enabled+all) Call Trace: <IRQ> try_to_wake_up+0x2c1/0x5c0 hrtimer_wakeup+0x22/0x30 __hrtimer_run_queues+0x1d5/0x350 hrtimer_interrupt+0x106/0x1e0 __sysvec_apic_timer_interrupt+0x6f/0x210 sysvec_apic_timer_interrupt+0x71/0x90 </IRQ> This can be reproduced running scx_qmap -I together with a SCHED_FIFO workload that frequently sleeps, e.g.: $ vng -v --cpus 16 --memory 4G -- \ 'tools/sched_ext/build/bin/scx_qmap -I >/dev/null & \ stress-ng --sleep 4 --sched fifo --sched-prio 1 -t 30 -q' Fix this by queueing the balance callbacks only when the pick is final. Otherwise, defer the pending work to the irq_work path, which doesn't depend on the current rq lock section, and leave the kick sync wait pending for the next pick. Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()") Cc: stable@vger.kernel.org # v6.19+ Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
6 dayssched_ext: Merge branch 'for-7.3-fixes' into for-7.4Tejun Heo
Pull to receive 30fb2b1e4d41 ("sched_ext: Fix the running task keeping its protection into bypass") and 5809f749fbdb ("sched_ext: Fix a bypassed task stalling on a cpu its scheduler has no caps on") and resolve their conflicts with the proxy execution work here. Conflicts: - scx_bypass(): 30fb2b1e4d41 drops the protection strip in the runnable_list walk, which never sees the running task, and strips rq->curr ahead of the walk instead. 3d6577ea93c9 ("sched_ext: Split curr|donor references properly") moved the walk's strip to task_current_donor(), as protection belongs to the scheduling context. set_next_task_scx() takes the donor off the runnable_list, so that strip is dead too. Dropped it, and the new strip takes rq->donor. - scx_bypass(), merged without a marker: 5809f749fbdb evicts the running task when bypass lifts if it can't stay on the cpu, on rq->curr. 3d6577ea93c9 moved the cap revoke eviction of the running task in reenq_local() to rq->donor. The lift eviction takes rq->donor too, and both comments in the block say the current donor. Signed-off-by: Tejun Heo <tj@kernel.org>
7 dayssched_ext: Fix a bypassed task stalling on a cpu its scheduler has no caps onTejun Heo
A bypassing sub-scheduler's tasks are run by its nearest ancestor that isn't bypassing. The ancestor puts them on the local DSQ under its own caps. Whether such a task may stay on the cpu and whether a cap revoke reenqueues it are still checked against the caps of the task's own scheduler. The bypass load balancer moves tasks between cpus without looking at caps, so a bypassed task can run on a cpu where its own scheduler holds none. Once its slice runs out, it can't stay. It goes to the ancestor's bypass DSQ without a kick, and the cpu can go idle with it queued until something else reschedules the cpu. Make both checks against the caps of the scheduler running the task. When bypass lifts, the running task keeps its slice without a cap check, so evict it then if it can't stay on the cpu. Fixes: 75a8c8202c91 ("sched_ext: Add reject DSQ for cap-rejected dispatches") Fixes: 46a85ae6fe5b ("sched_ext: Tie cpu occupancy to SCX_CAP_BASE through the task slice") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
7 dayssched_ext: Fix the running task keeping its protection into bypassTejun Heo
Protection ends when the scheduler enters bypass, so that bypass scheduling can cut every task's slice. scx_bypass() clears it for the queued tasks by cycling them, and strips the running task's in the same walk over rq->scx.runnable_list. The running task is never on that list. set_next_task_scx() takes it off and put_prev_task_scx() puts it back, so the strip never runs. A protected running task then keeps its slice into bypass, where the tick's attempts to cut it are refused. Strip the running task's protection outside the walk. Fixes: 9cfc6ab34a31 ("sched_ext: Add SCX_TASK_PROTECTED") Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
8 dayssched: Replace nr_iowait_cpu() with a bool accessorChristian Loehle
The menu governor was the last consumer of the per-CPU I/O-wait count that needed its magnitude, apart from nr_iowait() summing it globally. Commit 38f83090f515 ("cpuidle: menu: Remove iowait influence") removed that use. All remaining per-CPU consumers only test for zero/nonzero. Replace nr_iowait_cpu() with bool cpu_has_iowait() and update the generic and s390 idle accounting callers. Have nr_iowait() read the counters directly so that the global total and existing accounting are preserved. Replace the stale menu comment with a description of the predicate. Also convert the remaining open-coded idle accounting checks to use the predicate, as suggested by Prateek. The conversion was done with coccinelle: @ predicate depends on patch @ expression cpu; @@ - nr_iowait_cpu(cpu) > 0 + cpu_has_iowait(cpu) @ callers depends on patch @ expression cpu; @@ - nr_iowait_cpu(cpu) + cpu_has_iowait(cpu) Signed-off-by: Christian Loehle <christian.loehle@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com> Link: https://patch.msgid.link/20261016073341.575472-1-christian.loehle@arm.com
8 daysRevert "sched/fair: Remove dead code on enqueue_task_fair()"Kayra Cizmeci
This reverts commit 819224e506bc7c2d61ec6a58ec6e876b505abce9. The cfs_rq->curr == se case can actually happen with CFS_BANDWIDTH, so the removed code is needed. Document this. Fixes: 819224e506bc ("sched/fair: Remove dead code on enqueue_task_fair()") Reported-by: Christian Loehle <christian.loehle@arm.com> Signed-off-by: Kayra Cizmeci <kayracizmeci@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lore.kernel.org/lkml/fd3f9d83-cb5b-496b-b7c1-b40f31cf795e@arm.com/ Link: https://patch.msgid.link/20261015082919.4344-1-kayracizmeci@gmail.com
8 dayssched/cache: Avoid publishing partial LLC utilizationJemmy Wong
update_sg_lb_stats() sums utilization over the CPUs in both sched_group_span(group) and env->cpus, but takes capacity from group->sgc->capacity for the whole group. record_sg_llc_stats() publishes both. Rejecting an all-affinity-pinned source runqueue removes that CPU from env->cpus and retries group selection. The retry can publish the partial utilization with the full capacity. can_migrate_llc() then treats a busy preferred LLC as having spare capacity. More tasks can aggregate into it, migrations out can be blocked, and two busy LLCs may not fall back to generic balancing. This increases CPU contention and task latency. Found by inspection. Skip publication on affinity retries using LBF_ALL_PINNED, retaining the first-pass snapshots and avoiding repeated cpumask checks. On the first pass, publish either shared LLC statistic only when env->cpus covers the whole group. Load balancing still uses the affinity-filtered statistics. The same window exists during CPU hotplug. env->cpus drops a CPU at set_cpu_active(false), but the group span and sgc->capacity still include it until cpuset_cpu_inactive() rebuilds the domains. Keeping the previous snapshot avoids the mismatched publish. After the rebuild, the next periodic balance records both statistics again. Fixes: f025ef275388 ("sched/cache: Record per LLC utilization to guide cache aware scheduling decisions") Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com> Suggested-by: Tim Chen <tim.c.chen@linux.intel.com> Signed-off-by: Jemmy Wong <jemmywong512@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20261013084338.55100-1-jemmywong512@gmail.com
8 dayssched/cputime: Don't account idle time twice after dyntick-idleStian Halseth
On idle exit the dyntick-idle accounting accounts the time up to now, then the tick is restarted on its old period. The first tick accounts a whole TICK_NSEC to whatever runs, although the part of that period before the idle exit has just been accounted as idle time, or with IRQ_TIME_ACCOUNTING as IRQ time. That is up to a full tick per idle exit, and /proc/stat reports more idle time than wall time. Record how much of the current tick period dyntick-idle has accounted, and leave it out of the tick that ends the period. Fixes: cf6444c3e1bb7 ("tick/sched: Unify idle cputime accounting") Signed-off-by: Stian Halseth <stian@itx.no> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Frederic Weisbecker <frederic@kernel.org> Link: https://lore.kernel.org/all/20261004142724.3896396-1-stian@itx.no/ Link: https://patch.msgid.link/20261007142950.547604-1-stian@itx.no
8 dayssched/psi: Remove the RCU wait from trigger destructionShakeel Butt
In the Meta fleet, we found PSI trigger destruction waiting for an RCU grace period while holding kernfs node_mutex, blocking other users [1]. File close and node drain hold this mutex while calling the release callback. Cgroup removal and disabling cgroup.pressure also hold cgroup_mutex during the drain. The wait was added by commit 0e94682b73bf ("psi: introduce psi monitor") to protect worker and trigger lookups. Commit 461daba06bdc ("psi: eliminate kthread_worker from psi trigger scheduling mechanism") replaced the worker queue with a group timer. The scheduler now only checks whether rtpoll_task is NULL; it uses neither the task nor the trigger. Commit a06247c6804f ("psi: Fix uaf issue when psi trigger is destroyed while being polled") removed trigger replacement and polling's RCU lookup. Trigger lists are protected by mutexes. Proc poll/select entries are removed before their file references are dropped. eventpoll_release() removes epoll entries before the file release callback. Commit aff037078eca ("sched/psi: use kernfs polling functions for PSI trigger polling") moved cgroup polling to a kernfs waitqueue whose lifetime follows the file. A late timer firing only wakes the group waitqueue. The system group is static, and commit 5457025fa8ca ("sched/psi: Shut down rtpoll_timer in psi_cgroup_free()") shuts down the cgroup timer before freeing the group. Cgroup reclamation already waits for scheduler readers. Remove synchronize_rcu() from psi_trigger_destroy() and use rcu_access_pointer() for the worker NULL check. Keep kthread_stop() outside the trigger mutex. Tested trigger churn, notifications and cgroup removal in an 8-CPU VM with KASAN, lockdep, and full and lazy preemption. No warnings were found. Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Tejun Heo <tj@kernel.org> Link: https://github.com/bpftrace/user-tools/tree/master/runnablelockmonitor [1] Link: https://patch.msgid.link/20261010191500.3895736-1-shakeel.butt@linux.dev
8 dayssched/fair: Avoid overflow in place_entity()Hui Su
place_entity() inflates an entity's virtual lag with lag * (load + weight) / load Commit 4823725d9d1d ("sched/fair: Increase weight bits for avg_vruntime") replaced the scaled-down weights used in this calculation with avg_vruntime_weight(), removing the arithmetic headroom provided by scale_load_down(). The multiplication can overflow s64 before the division even when the final quotient is representable. A task's hierarchical effective weight can be as small as 2 after avg_vruntime_weight() scaling under group scheduling, while entity_lag() bounds |vlag| by calc_delta_fair(cfs_rq_max_slice(cfs_rq) + TICK_NSEC, se) For example, with HZ=1000, NICE_0_LOAD=1048576, and the minimum scaled hierarchical weight scale_load(2)=2048, this bound is 891289600. With 114 maximum-weight entities, the runqueue load is 10361604096. The intermediate product is: 891289600 * (10361604096 + 2) = 9235189971864780800 > S64_MAX The signed overflow corrupts the entity's placement. Rewrite lag * (load + weight) / load as lag + lag * weight / load The two expressions are equivalent, but the latter avoids multiplying the virtual lag by the total runqueue weight. entity_lag() bounds vlag against the entity's effective weight, so the remaining lag * weight product stays bounded. rescale_entity() preserves this scale across h_load changes and already relies on the same product when rescaling vlag. Keep the original zero-load warning semantics by multiplying lag by weight after replacing the divisor with one. Fixes: 4823725d9d1d ("sched/fair: Increase weight bits for avg_vruntime") Signed-off-by: Hui Su <sh_def@163.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260928124754.1692077-1-sh_def@163.com
8 dayssched_ext: Fix use-after-free of a local reenq node left on an inactive cpuTejun Heo
A local DSQ reenq is queued on the rq through a per-sched node and run from deferred work. When the reenq is requested from dispatch, the work is a balance callback queued once dispatch is done. A cpu going down installs balance push early in its teardown, and queue_balance_callback() refuses new callbacks from then on. ops.dispatch() keeps running on that cpu until the rq goes offline after an RCU grace period, so a reenq it requests in that window may never run. The IMMED handling in dispatch_one() requests one the same way. A reenq requested from elsewhere for a cpu that is already offline goes through irq_work_queue_on(), which warns, and its work doesn't run until the cpu comes back. Either way the node stays on the rq's list. When the scheduler is freed, scx_sched_free_rcu_work() warns and frees the pcpu with the node still linked, and the list is walked again once the cpu comes back. An inactive cpu's local DSQ holds only tasks that are bound to it or are being moved off, so the reenq is pointless. Ignore reenqs of an inactive cpu's local DSQ. The test is lockless. The hotplug thread of a cpu going down runs on that cpu, and it marks the cpu inactive before it installs balance push. A dispatch on the cpu runs to completion before that thread can run there, so a dispatch that saw the cpu active has queued its callback by then. Requests from outside dispatch run from the wakeup path or through irq_work, which balance push doesn't affect. The grace period after the cpu goes inactive waits for any requester that saw it active, so that irq_work is queued while the cpu is online. Fixes: 238eba8c210d ("sched_ext: Use schedule_deferred_locked() in schedule_dsq_reenq()") Fixes: 76edc2761ab8 ("sched_ext: Use irq_work_queue_on() in schedule_deferred()") Cc: stable@vger.kernel.org # v7.1+ Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> Tested-by: Andrea Righi <arighi@nvidia.com>
9 daysMerge tag 'sched_ext-for-7.3-rc7-fixes' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext fixes from Tejun Heo: - A task that skipped the BPF scheduler on its way back to its CPU's local queue could stall there until the watchdog fired, as the CPU had already decided to go idle. - Use-after-free of a destroyed user dispatch queue by a deferred re-enqueue request. - A task could be moved off a dispatch queue while the dispatching CPU was still running ops.dequeue() on it. - Bypass mode mishandled a sub-scheduler bypass overlapping an ancestor's, as across a system suspend. Descendants could leave bypass mid-teardown and tasks could be stranded on bypass queues. - The idle CPU selection helper could return a CPU outside the allowed CPUs of a pinned or migration-disabled task. * tag 'sched_ext-for-7.3-rc7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs sched_ext: Rename scx_reenq_wait_dispatching() to scx_wait_dispatching() sched_ext: idle: Respect task affinity in restricted CPU selection sched_ext: Fix the bypass dispatch claim for schedulers bypassed by an ancestor sched_ext: Propagate every bypass depth change to the descendants sched_ext: Fix use-after-free of a destroyed user DSQ by a deferred reenqueue sched_ext: Fix stall when a task enqueued with SCX_ENQ_LAST lands back on its CPU's local DSQ
9 dayssched_ext: Keep proxy donors with slice left on the local DSQAndrea Righi
Commit ee172227d0dc ("sched_ext: Delegate proxy donor admission to BPF schedulers") makes put_prev_task_scx() pass a retained proxy donor to ops.enqueue() with SCX_ENQ_BLOCKED. Some of these puts are only proxy bookkeeping; proxy_resched_idle() drops the rq's donor reference before switching to idle when: - find_proxy_task() cannot run the mutex owner yet and retries through idle (e.g., the owner is on a remote CPU), - proxy_migrate_task() detaches the donor before migration, - proxy_deactivate() blocks a task in the chain whose owner cannot run. In the first case, BPF has already selected a donor with slice left. Returning it to ops.enqueue() forces BPF to dispatch it again before proxy resolution can continue. In the other two cases, the donor is typically deactivated right after ops.enqueue(), undoing any placement BPF makes. As also discussed at the sched_ext microconference at Linux Plumbers 2026, these ops.enqueue/dequeue() invocations do not represent any meaningful scheduling events, so we should avoid triggering them. A more reasonable semantic is to keep a donor with slice left at the head of the local DSQ, so the next pick can resolve its owner, or deactivation can remove it without an unnecessary BPF handoff. An IMMED donor can stay there only for a bookkeeping put during proxy resolution. Mark a blocked pick as awaiting resolution with SCX_RQ_PROXY_PENDING to distinguish that put from a real preemption and re-insert the donor with SCX_ENQ_IMMED, so that it only needs SCX_CAP_ENQ_IMMED rather than SCX_CAP_ENQ. A preempted IMMED donor still returns to BPF. If a higher priority class takes the CPU once proxy resolution completes, schedule a local reenqueue, so that a parked IMMED donor returns to BPF instead of lingering on the local DSQ. Update the SCX_ENQ_IMMED and SCX_ENQ_BLOCKED documentation accordingly. Moreover, a donor that has run out of slice now follows the regular put path, so it can receive SCX_ENQ_LAST when the CPU is about to go idle and the scheduler needs to arrange a follow-up scheduling event. This does not apply to bookkeeping puts, where the switch to idle is only temporary. The blocked-donor specific sanity checks go away together with the special case. When a retained donor is removed ahead of a scheduler ownership or policy change, mark the proxy reset with SCX_RQ_PROXY_BLOCKING and skip reenqueueing the active donor: sched_proxy_block_task() will dequeue it next. Report it as not runnable to ops.stopping(), as a regular dequeue of the current donor does. This avoids a transient ops.enqueue/dequeue() pair and a false ENQ_LAST warning. With this, the sched_ext core absorbs the transient proxy-bookkeeping puts and the BPF scheduler only sees a blocked donor when there is an actual placement decision to make. Fixes: ee172227d0dc ("sched_ext: Delegate proxy donor admission to BPF schedulers") Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
9 dayscpufreq: schedutil: Serialize sugov_start() against rate limit updatesHui Su
The current value of sg_policy->tunables->rate_limit_us is used in sugov_start() without locking to update sg_policy->freq_update_delay_ns, so if rate_limit_us_store() runs in parallel with it, the rate limit update may go unnoticed and the new freq_update_delay_ns value may be stale until the rate limit is updated again. To avoid that, acquire the update_lock mutex of the rate limit sysfs attribute, that is held by rate_limit_us_store() during rate limit updates, around the freq_update_delay_ns in sugov_start(). Fixes: 9bdcb44e391d ("cpufreq: schedutil: New governor based on scheduler utilization data") Reviewed-by: Zhongqiu Han <zhongqiu.han@oss.qualcomm.com> Signed-off-by: Hui Su <sh_def@163.com> Cc: All applicable <stable@vger.kernel.org> [ rjw: Subject adjustment, new changelog, edits of the new comment ] [ rjw: Add empty code line after guard()() ] Link: https://patch.msgid.link/20261013034235.2678177-1-sh_def@163.com Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
10 dayssched/core: Call wakeup_preempt() for queued tasks during class changeK Prateek Nayak
Paul reported recent RCU changes exposed a lost resched where CPU idles with a rcutorture thread waiting while being completely runnable. Dumping the rq state at the time of idling with runnable tasks showed the following: rq->next_class = &fair_sched_class rq->nr_running = 1 rq->cfs.h_nr_queued = 0 rq->rt.rt_nr_running = 1 Although the waiting task was an RT task, "rq->next_class" did not point to rt_sched_class and was stuck at the lower priority fair_sched_class. Inspecting sched_change_end() revealed the following race with __sched_setscheduler() for queued, but non-running tasks: CPU0 CPU1 ==== ==== __schedule() /* * rq->donor is a fair task that blocks. * rq->next_class = &fair_sched_class */ rq_lock(rq0) pick_next_task() /* nr_running = 0 */ pick_task_fair() sched_balance_newidle() raw_spin_rq_unlock(rq0) rq_lock(rq0) activate_task(rq0, p) /* p is a fair task */ rq0->nr_running++; rq0->cfs.h_nr_queued++; wakeup_preempt(rq, p) rq->next_class = &fair_sched_class resched_curr() rq_unlock(rq0) __sched_setscheduler(p) /* p is moved to SCHED_RR */ task_rq_lock(p) sched_change_begin() rq0->cfs.h_nr_queued--; rq0->rt.rt_nr_running++; sched_change_end() /* * p->on_rq = 1 (ctx->queued) * but p is not rq->donor * (!ctx->running) hence * resched_curr() is and * rq->next_class upgrade is * skipped. */ task_rq_unlock() raw_spin_rq_lock(rq0) /* * rq->nr_running = 1 * rq->cfs.h_nr_queued = 0 * rq->rt.rt_nr_running = 1 * rq->next_class = &fair_sched_class */ if (rq->cfs.h_nr_queued && ...) /* Above is false */ if (rq_modified_above(this_rq, &fair_sched_class)) /* Also false since rq->next_class is still fair */ /* * sched_balance_newidle() is considered to * have failed to pull any fair tasks and no * other wakeups were detected. */ return NULL; /* pick_task_fair() */ pick_task_idle() return rq->idle; next = rq->idle; /* Preemption indicators are lost */ clear_tsk_need_resched(prev); clear_preempt_need_resched(); rq->curr = rq->idle; !!! rq->idle switches in with rq->running == 1 but !need_resched() !!! In simple words, enqueue, followed by a class change of the newly queued task goes unnoticed in rq->next_class indicator and sched_balance_newidle() sees no fair task, nor any higher priority task enqueued. The above sequence will lead to CPU idling with a runnable task on the CPU. Call wakeup_preempt() for ctx->queued cases in sched_change_end() to re-evaluate rq->next_class after class change (ENQUEUE_CLASS) and set the NEED_RESCHED indicator accordingly. Newidle balance will correctly see an enqueue of a higher priority task if a queued task was upgraded when the rq_lock() was dropped and will retry the pick. Closes: https://lore.kernel.org/lkml/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/ Fixes: 704069649b5b ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()") Reported-by: Paul E. McKenney <paulmck@kernel.org> Tested-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org> Reviewed-by: Phil Auld <pauld@redhat.com> Link: https://patch.msgid.link/20261013152455.9039-1-kprateek.nayak@amd.com
10 dayssched_ext: Merge branch 'for-7.3-fixes' into for-7.4Tejun Heo
Pull to receive 2935062be771 ("sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs") and resolve its conflicts with the proxy execution and reject DSQ work here so that for-next merges clean. Conflicts: - unlink_dsq_and_switch_rq_lock(): efa7bc7d7195 ("sched_ext: Avoid relocking DSQ during remote consumption") clears p->scx.dsq in the success branch and 2935062be771 adds the wait there. Kept both, the wait after the clear, so that the proxy reject in consume_remote_task() is covered too. - scx_reenq_reject(): 18f5c699d1d0 ("sched_ext: Rename scx_reenq_wait_dispatching() to scx_wait_dispatching()") renames the call in sub.c while cea8b60e4aee ("sched_ext: Move reject DSQ draining into core") moved the function into ext.c. Dropped the sub.c side and applied the rename at the new home. Signed-off-by: Tejun Heo <tj@kernel.org>
10 dayssched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQsKuba Piecuch
A task can be found on a non-local DSQ while the CPU which dispatched it is still running ops.dequeue() and hasn't cleared SCX_OPSS_DISPATCHING. The DSQ consume and move paths don't wait for the dispatch to finish and can operate on the task concurrently with the dispatcher. Call scx_wait_dispatching() in scx_move_local_task_to_local_dsq(), unlink_dsq_and_switch_rq_lock() and move_task_between_dsqs() before they operate on the task. In scx_move_local_task_to_local_dsq(), wait before scx_resolve_local_dsq() as it may update p->scx.flags. Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics") Fixes: cb86607ada73 ("sched_ext: Don't run ops.dequeue() with a DSQ lock held") Cc: stable@vger.kernel.org # v7.1+ Reviewed-by: Andrea Righi <arighi@nvidia.com> Assisted-by: Claude:claude-opus-5.5 Signed-off-by: Kuba Piecuch <jpiecuch@google.com> Signed-off-by: Tejun Heo <tj@kernel.org>
10 dayssched_ext: Rename scx_reenq_wait_dispatching() to scx_wait_dispatching()Kuba Piecuch
Waiting for SCX_OPSS_DISPATCHING to clear isn't specific to the reenqueue paths. Any path which takes a task off a DSQ can find it while the dispatch is still in flight, and the next patch calls the helper from the DSQ consume and move paths. Rename it to scx_wait_dispatching() and update its comment accordingly. No functional change. Reviewed-by: Andrea Righi <arighi@nvidia.com> Assisted-by: Claude:claude-opus-5.5 Signed-off-by: Kuba Piecuch <jpiecuch@google.com> Signed-off-by: Tejun Heo <tj@kernel.org>
11 dayssched/eevdf: Remove redundant RUN_TO_PARITY checkElif Topuz
update_protect_slice() is only called from wakeup_preempt_fair() when RUN_TO_PARITY is enabled. Remove RUN_TO_PARITY check inside update_protect_slice() and use cfs_rq_min_slice() directly. Signed-off-by: Elif Topuz <elif.topuz@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org> Link: https://patch.msgid.link/20261012105001.131311-1-elif.topuz@arm.com
11 dayssched/wait: Avoid modifying entry on failed priority registrationHui Su
add_wait_queue_priority_exclusive() sets WQ_FLAG_EXCLUSIVE and WQ_FLAG_PRIORITY before checking whether another priority waiter is already present. Thus, when registration fails with -EBUSY, the entry is left unqueued but with modified flags. There is no need to modify the entry before the failure check. Move the flag update after the check so that a rejected registration does not unnecessarily alter the entry. Signed-off-by: Hui Su <sh_def@163.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Sean Christopherson <seanjc@google.com> Link: https://patch.msgid.link/20261011015019.4056529-1-sh_def@163.com
11 daysrbtree: Declare augmented callbacks per field with RB_AUG()Peter Zijlstra (Intel)
RB_DECLARE_CALLBACKS_MULTI() asks its user for a function that copies the augmented fields and one that recomputes them from the children, with the early-exit protocol of ->propagate() hand-coded in the latter. sched/eevdf, its only user, needs five helpers for three fields. Describe each augmented field instead: RB_AUG(val, aug, fold) names the per-node value, the member holding the subtree aggregate and how two aggregates combine (min, max, a sum, ...). RB_AUG_FUNC() takes a function for the per-node value. The template then generates, per field, a recompute that works on a local and stores once, and a copy, and combines them into the callbacks; the early exit is the AND of the per-field results. RB_DECLARE_CALLBACKS() is the only template left. RB_DECLARE_CALLBACKS_MAX() becomes RB_AUG_FUNC(RBVALUE, RBAUGMENTED, max) in a wrapper, with its RBCOMPUTE argument renamed to RBVALUE, since it returns the per-node scalar and is a different thing from the RBCOMPUTE of the generic template it used to build on. RB_DECLARE_CALLBACKS_MULTI() goes away. sched/eevdf shrinks to a wrapping-safe min() for min_vruntime and three RB_AUG() lines. No functional change intended. [yiwei: take a fold(a, b) instead of a "replace?" compare so that sums and counts can be expressed too, which also lets min()/max() replace RB_MIN()/RB_MAX(); wrapped the lines over 100 columns] Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Yiwei Lin <s921975628@gmail.com> Assisted-by: LLM Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lore.kernel.org/r/20260928213736.GA2947991@noisy.programming.kicks-ass.net Link: https://patch.msgid.link/20261005093236.62702-3-s921975628@gmail.com
11 daysMerge branch 'tip/sched/urgent'Peter Zijlstra
Pickup dependencies; notably the RB-tree fix. Signed-off-by: Peter Zijlstra <peterz@infradead.org>
11 dayssched_ext: idle: Respect task affinity in restricted CPU selectionHui Su
The restricted select_cpu kfunc path handles single-CPU and migration-disabled tasks without going through scx_select_cpu_dfl(). It currently uses prev_cpu as the candidate, even though prev_cpu is only a hint and may no longer be usable after an affinity change. With a caller-provided @allowed mask, this can also return an idle CPU outside p->cpus_ptr. Derive the candidate from task state instead: use the sole CPU in p->cpus_ptr for single-CPU tasks and task_cpu(p) for migration-disabled tasks. Validate it against p->cpus_ptr and @allowed before claiming its idle state. Since the derived candidate may be on a different NUMA node from prev_cpu, also preserve SCX_PICK_IDLE_IN_NODE by rejecting cross-node candidates. Perform all checks before scx_idle_test_and_clear_cpu(), as a successful call consumes the CPU's idle bit. Fixes: 683d2d0faba1 ("sched_ext: idle: Introduce scx_bpf_select_cpu_and()") Cc: stable@vger.kernel.org Signed-off-by: Hui Su <sh_def@163.com> Reviewed-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
11 dayssched_ext: Fix the bypass dispatch claim for schedulers bypassed by an ancestorTejun Heo
scx_bypass() claims bypass dispatch when the scheduler's depth goes from zero to one and releases it when the depth returns to zero, but propagates only the depth to the descendants, not the claim. A descendant that starts bypassing with its ancestor never claims, and one whose own bypass ends under the ancestor's keeps its claim, as its depth does not return to zero. A sub-scheduler enable whose bypass ends under a PM suspend thus leaks its claim: the parent keeps hosting bypass dispatch and the sub-scheduler's next bypass trips the WARN_ON_ONCE() on the claim. A sub-scheduler disable that starts under a PM suspend has no claim, so once the resume lifts the root's bypass its tasks sit on the parent's bypass DSQs with nobody consuming them. Claim for every descendant whose depth goes from zero to one in the propagation and release every scheduler in the subtree whose depth returned to zero once the tasks have moved out of the bypass DSQs. A parent that is bypassing itself when a descendant claims consumes only its own bypass DSQs, empty until it leaves bypass and becomes the host. Fixes: aa2a0a19686c ("sched_ext: Separate bypass dispatch enabling from bypass depth tracking") Cc: stable@vger.kernel.org # v7.1+ Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
11 dayssched_ext: Propagate every bypass depth change to the descendantsTejun Heo
scx_bypass() propagates a depth change to the descendants only when the scheduler's own depth crosses zero, so an own bypass that starts under an ancestor's never reaches them. When the ancestor's bypass ends first, the descendants leave bypass while the scheduler is still bypassing: a sub-scheduler disabled under a PM suspend has its children re-queue their tasks through their own BPF schedulers at resume, mid-teardown. If the own bypass then ends, every descendant is decremented once more than incremented and dec_bypass_depth() warns, though no current caller gets there. Propagate on every call so that a descendant's depth is the sum of its own bypasses and its ancestors'. The early return follows the walk, as a descendant's depth is at least the scheduler's and nothing changes bypass state unless the scheduler's own does. Fixes: c7f0e467a27a ("sched_ext: Prepare bypass mode for hierarchical operation") Cc: stable@vger.kernel.org # v7.1+ Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
14 dayssched_ext: Merge branch 'for-7.3-fixes' into for-7.4Tejun Heo
Pull to receive 069403dff337 ("sched_ext: Fix use-after-free of a destroyed user DSQ by a deferred reenqueue") and point its new cancel sweep in free_dsq_irq_workfn() at dsq->pcpu_user, which 92344fad5b54 ("sched_ext: Skip per-CPU data allocation for built-in DSQs") renamed here, so that for-next builds. Signed-off-by: Tejun Heo <tj@kernel.org>
14 dayssched_ext: Fix use-after-free of a destroyed user DSQ by a deferred reenqueueTejun Heo
A deferred reenqueue can read a freed user DSQ. The request stays on its CPU's deferred list until run_deferred() takes it, and the RCU callback that frees the DSQ assumes that the grace period has flushed it, so exit_dsq() checks the per-CPU node without the deferred list lock. A grace period waits for readers in flight, not for a pending run_deferred(). A run_deferred() which starts after the grace period began can detach the request right before the callback looks and then read the freed DSQ's id. Cancel the pending requests before the grace period instead. The irq work that frees the DSQ unlinks every CPU's request under the per-CPU deferred list lock, and schedule_dsq_reenq() ignores a new request under the same lock once the id is invalid. Only a run_deferred() that started before the grace period can hold a request, and it runs with IRQs disabled, so the grace period waits for it and nothing touches the DSQ after the callback. Fixes: 84b1a0ea0b7c ("sched_ext: Implement scx_bpf_dsq_reenq() for user DSQs") Cc: stable@vger.kernel.org # v7.1+ Reported-and-tested-by: Hui Su <sh_def@163.com> Link: https://lore.kernel.org/all/20261008024753.4096008-1-sh_def@163.com/ Signed-off-by: Tejun Heo <tj@kernel.org>
14 dayssched_ext: Fix stall when a task enqueued with SCX_ENQ_LAST lands back on ↵Tejun Heo
its CPU's local DSQ With SCX_OPS_ENQ_LAST, the last runnable task on a CPU is passed to ops.enqueue() instead of being kept running. Exiting tasks without SCX_OPS_ENQ_EXITING, migration disabled tasks without SCX_OPS_ENQ_MIGRATION_DISABLED and tasks on an offline rq skip ops.enqueue() and go straight to the local DSQ. This happens in put_prev_task_scx() after the pick has settled on idle, so nothing reschedules the CPU, which idles with the task queued until an unrelated wakeup lands on it or the watchdog fires. scx_mitosis, which sets SCX_OPS_ENQ_LAST without SCX_OPS_ENQ_EXITING, hit this during mass cgroup teardown: exiting tasks stuck in exit_mmap() for over 40 seconds. A task put on the local DSQ runs, as anywhere else. If the SCX_ENQ_LAST enqueue left the task on this CPU's local DSQ, set need_resched on the task being switched in, as proxy_resched_idle() does. resched_curr() would flag the task being put instead, and __schedule() clears that right after the pick. Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class") Cc: stable@vger.kernel.org # v6.12+ Reported-by: Xiangyu Bu <xbu@meta.com> Link: https://github.com/sched-ext/scx/pull/3870 Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Dan Schatzberg <schatzberg.dan@gmail.com>
2026-10-09sched_ext: Add ops.sub_child_ecaps_updated() to report a child's effective ↵Tejun Heo
cap changes A grant or revoke only records the target caps. They take effect on a cid at its next dispatch, and nothing tells the parent when. Without that the parent cannot act on a revoke: a child that ran a cid at a low cpuperf target leaves the target there when PERF is revoked, as the kernel resets targets only at root enable. Nor can the parent tell when it may schedule on the cid again. Add ops.sub_child_ecaps_updated(), delivered to the direct parent right after the child's own ops.sub_ecaps_updated() with the child's cgroup id and the same before and after caps, in the same dispatch context. Both deliveries are suppressed while the child is bypassing and replayed together afterwards. A disabled child reports nothing: ops.sub_detach() is where the parent restores what it had delegated. A sub is now bypassed before it is linked, so that the grants queued for it from ops.sub_attach() are consumed while it is bypassed and replayed to it and the parent once it is enabled. Otherwise a sync consumed before the sub's ops are registered would reach the parent but never the sub. v2: Drop the disable report with its sleep lock and nested-dispatch gate, the parent restores from ops.sub_detach() instead. Signed-off-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andrea Righi <arighi@nvidia.com>
2026-10-06sched_ext: Add ops.sub_cid_sched_updated() to report the sched running on a cidTejun Heo
A parent sched that delegates cpus to sub-scheds has no way of telling how much each delegatee is using them. A parent that sizes its delegations by demand cannot measure it, and one that wants to hand a cid from one sub to another cannot see which sub it would take the cid from. Add ops.sub_cid_sched_updated(), invoked on a sched when what it sees running on a cid changes: NONE when no task of its subtree runs there, SELF for its own task, or a direct child's cgroup id. Changes inside a child's subtree are not reported. The op costs one pointer compare per context switch when the sched does not change, a notification reaches only the scheds that see a change, and the whole mechanism sits behind a static key that is on only while a loaded sched implements the op. After a sched's tasks are gone, its disable closes every rq session still pointing at it, so nothing refers to it once it is freed. The op is cid-form only like the other sub ops and runs with the rq lock held on the context switch path. Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-07Merge tag 'sched_ext-for-7.3-rc6-fixes' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext fixes from Tejun Heo: - Taking a CPU offline could hang, or stall until the watchdog ejected the BPF scheduler, when tasks on the dying CPU were still held by the scheduler or sitting on a user dispatch queue. Re-enqueue them onto the local queue when the runqueue goes offline so that the CPU pushes them off like the other sched classes. - The sequence number guarding against stale dispatches was per runqueue, so a task re-enqueued on another CPU could get the same number and a dispatch meant for its earlier instance was applied to the new one. Use a per-task counter. - A task dispatched to another CPU's local queue got its ops.dequeue() only when picked to run and flagged as a core-sched pick. Call it at insertion like for same-CPU dispatches. * tag 'sched_ext-for-7.3-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: sched_ext: Generate qseq from a per-task counter selftests/sched_ext: Add a test for ops.dequeue() on remote local DSQ moves sched_ext: Call ops.dequeue() when a task arrives on a remote local DSQ sched_ext: Fix CPU hotplug hang when a dying CPU's tasks sit in the BPF scheduler
2026-10-06sched_ext: Clear a sub-scheduler's caps before ops.sub_detach()Tejun Heo
A sub-scheduler can leave standing state behind on the cids delegated to it, cpuperf targets being the obvious case, and the kernel does not track who wrote it. Restoring it is the parent's job and ops.sub_detach() is the place for it. Today the parent can't rely on that: the dying sub-scheduler keeps every cap until it is marked dead after ops.exit(), so its ops.exit() or a late timer can write after the parent has cleaned up. Clear every cap of the sub-scheduler before ops.sub_detach(). The pshard caps go first as a pending sync recomputes ecaps from them, and ecaps are then zeroed directly. A grant racing the clear is refused under the child's pshard lock, so nothing hands the caps back. Fixes: 86094b95efcf ("sched_ext: Add per-shard cap delegation for sub-schedulers") Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-03sched_ext: Generate qseq from a per-task counterKuba Piecuch
finish_dispatch() uses the qseq embedded in p->scx.ops_state to tell whether the QUEUED instance of a task it's about to claim is the one scx_bpf_dsq_insert() saw. qseq is generated from rq->scx.ops_qseq, but the counters of different rqs are independent, so if a task is dequeued and re-enqueued on a different rq between scx_bpf_dsq_insert() and finish_dispatch(), the new QUEUED instance can end up with the same qseq as the old one: CPU X CPU Z ----- ----- enqueue p on rq A, qseq = N ops.dispatch() scx_bpf_dsq_insert(p) records qseq N sched_setaffinity(p) dequeue p from rq A enqueue p on rq B, qseq = N finish_dispatch(p, N) qseq matches, p is claimed The claim itself is still atomic so the core stays consistent, but an insert issued for a previous QUEUED instance gets applied to a new one which the BPF scheduler has just received through ops.enqueue(). This breaks the guarantee that dispatches targeting a stale instance are ignored. Generate qseq from a per-task counter, p->scx.ops_qseq, instead so that consecutive QUEUED instances of a task never share a qseq regardless of which rq they're on. The counter is only updated in scx_do_enqueue_task() with the task's rq locked, so no additional synchronization is needed, and it fits in an existing hole in struct sched_ext_entity on 64bit. Remove the now unused rq->scx.ops_qseq. Never generate qseq 0. NONE and DISPATCHING don't carry a qseq, so scx_bpf_dsq_insert() on a task in either state records 0. With a per-task counter, every task's first QUEUED instance would otherwise get qseq 0 and could be claimed by such an insert. Wrap the counter where the QSEQ field wraps so that it can't reach a value that shifts to 0 on 32bit either. Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class") Cc: stable@vger.kernel.org # v6.12+ Assisted-by: Claude:claude-opus-5.5 Signed-off-by: Kuba Piecuch <jpiecuch@google.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-10-01sched/topology: Add asymmetric SMT packing overrideAndrea Righi
Architectures can use SD_ASYM_PACKING to describe preferred CPU ordering at the SMT scheduling domain. Some systems benefit from this policy, but their firmware cannot describe the preference and inferring it from the CPU model would embed a platform-specific policy in the kernel. Add sched_smt_asym_packing=on boot option to force SD_ASYM_PACKING at the SMT level. Omitting the option preserves the topology provided by the architecture or firmware. Apply the override centrally to domains with SD_SHARE_CPUCAPACITY so it also covers architectures with custom SMT topology callbacks, including powerpc. When enabled, priority remains defined by arch_asym_cpu_priority(). The weak default prefers lower-numbered logical CPUs, while architecture overrides remain authoritative. Siblings with equal priorities remain unordered. In particular, x86 with CONFIG_SCHED_MC_PRIO normally gives both SMT siblings the same core priority, so enabling the option there does not prioritize a sibling. Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Tested-by: Breno Leitao <leitao@debian.org> Link: https://patch.msgid.link/20260929165538.726616-3-arighi@nvidia.com
2026-10-01sched/fair: Honor asymmetric SMT priority in idle selectionAndrea Righi
POWER7 uses SD_ASYM_PACKING at the shared-capacity SMT level to order hardware threads, and NVIDIA Olympus benefits from the same policy. Idle CPU selection does not consult that order, so a task can wake on an arbitrary sibling and remain there until load balancing corrects the placement. On these systems, that initial choice can prevent the core from entering its preferred lower-thread resource mode and cause a large and persistent performance loss. When idle selection finds an available CPU in an SMT core, choose the highest-priority available sibling. On SMT2 Olympus this only changes selection on fully idle cores. A partially idle core has only one available CPU. On wider SMT systems such as POWER7, it also fills available siblings in priority order while the core is partially busy. Apply the preference to idle-core and idle-CPU scans, asymmetric-capacity scans, target, previous, recently-used CPU fast paths and the slow path. Inspect the lowest scheduling domain directly, but require both CPUs to share its span because isolcpus can split hardware siblings across scheduling domains. Keep physical-core capacity selection independent from SMT sibling ordering. SD_ASYM_CPUCAPACITY first selects among cores with different maximum capacities, then SD_ASYM_PACKING selects the preferred available sibling inside the chosen core, whose siblings continue to share equal capacity. Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com> Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com> Tested-by: Kayra Cizmeci <kayracizmeci@gmail.com> Tested-by: Breno Leitao <leitao@debian.org> Link: https://patch.msgid.link/20260929165538.726616-2-arighi@nvidia.com
2026-10-01sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched()Sechang Lim
set_tsk_need_resched() tests TIF_NEED_RESCHED, calls __trace_set_need_resched() if the flag is clear, then sets it via set_tsk_thread_flag(). A BPF raw_tp program attached to sched_set_need_resched executes synchronously inside __bpf_trace_run(). On return, __bpf_trace_run() drops the RCU lock with rcu_read_unlock_migrate(), which on the preempt-or-BH-disabled path calls set_need_resched_current() -> set_tsk_need_resched() again. set_tsk_thread_flag() follows the tracepoint call, so every re-entrant frame sees TIF_NEED_RESCHED clear and calls __trace_set_need_resched() again: BUG: TASK stack guard page was hit at ffffc9001224ff98 Oops: stack guard page: 0000 [#1] SMP KASAN PTI RIP: 0010:__bpf_trace_sched_set_need_resched_tp+0x1c/0x190 Call Trace: trace_sched_set_need_resched_tp+0x110/0x130 set_tsk_need_resched include/linux/sched.h:2076 set_need_resched_current include/linux/sched.h:2094 rcu_read_unlock_special+0x43a/0x440 __rcu_read_unlock+0x9e/0x120 rcu_read_unlock_migrate+0xa9/0x240 __bpf_trace_run+0x131/0x180 bpf_trace_run3+0x333/0x430 __bpf_trace_sched_set_need_resched_tp+0x13a/0x190 trace_sched_set_need_resched_tp+0x110/0x130 set_tsk_need_resched include/linux/sched.h:2076 ... __resched_curr() has the same ordering, firing the tracepoint before setting the flag via set_ti_thread_flag() or set_nr_and_not_polling(). Fix it for consistency. Replace the separate test_tsk_thread_flag() + set_tsk_thread_flag() pair in set_tsk_need_resched() with test_and_set_tsk_thread_flag(). In __resched_curr(), move the tracepoint call after the flag is set in each path. Fixes: adcc3bfa8806 ("sched: Adapt sched tracepoints for RV task model") Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> Acked-by: Gabriele Monaco <gmonaco@redhat.com> Link: https://patch.msgid.link/20260630084750.2792851-1-rhkrqnwk98@gmail.com
2026-10-01sched/core: Fix context analysis errors in non-preferred CPU pushShrikanth Hegde
Nathan reported build failure with clang23 when CONFIG_PREFERRED_CPU=y. kernel/sched/core.c:11298:3: error: releasing raw_spinlock 'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis] 11298 | rq_unlock(rq, &rf); | ^ kernel/sched/core.c:11303:1: error: raw_spinlock 'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis] context_unsafe_alias(rq) was placed after rq_lock(), by which point it had already associated the lock with the original rq alias. Move the alias barrier before rq_lock() so that the changing rq alias is accounted for before acquiring the lock. Fixes: 74699f56ebcf ("sched/core: Push current task from non preferred CPU") Closes: https://lore.kernel.org/all/20260929121838.GA1814129@ax162/ Reported-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Tested-by: Nathan Chancellor <nathan@kernel.org> # build Link: https://patch.msgid.link/20260929164712.1054883-1-sshegde@linux.ibm.com
2026-09-30sched_ext: Introduce scx_bpf_cgroup_nr_cpus()Andrea Righi
Hierarchical BPF schedulers may need to know how many CPUs a cgroup can run on when distributing group-wide weights. For example, fair's group share calculation scales a group's weight by the smaller of its estimated runnable task count and the number of CPUs in its effective cpuset. Without that count, a BPF scheduler can only use the machine-wide CPU count and may give a group confined to a few CPUs excessive shares. A task's effective affinity can include per-task restrictions, so it cannot reliably represent the group's cpuset. Reading the cpuset directly from BPF would require CO-RE access to private cgroup and cpuset structures, including inherited cpuset resolution, making the approach fragile across kernel versions. Add scx_bpf_cgroup_nr_cpus(), which returns the same count fair uses through cpuset_num_cpus(). The count is independent of CPU or CID numbering, so the kfunc is available to both cpu-form and cid-form schedulers from any context. Also add a compat wrapper which falls back to nr_cpu_ids on older kernels. Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-09-30sched_ext: Merge branch 'for-7.3-fixes' into for-7.4Tejun Heo
Pull to receive cc07fae1de5d ("selftests/sched_ext: Add a test for ops.dequeue() on remote local DSQ moves") and resolve its conflict with f73ec6791424 ("sched_ext: Add selftest for blocked donor admission") here so that for-next merges clean. Conflicts: kept both auto-test-targets entries in the selftests Makefile, where each side added one after dequeue_iter. Signed-off-by: Tejun Heo <tj@kernel.org>
2026-09-30sched_ext: Call ops.dequeue() when a task arrives on a remote local DSQKuba Piecuch
When a task in the BPF scheduler's custody is moved to another CPU's local DSQ, ops.dequeue() is only called once the task is picked for execution, from set_next_task_scx() with SCX_DEQ_CORE_SCHED_EXEC, even though no core-sched pick took place. It should instead be called without flags when the task is inserted into the destination DSQ, as it is for same-rq dispatches. move_remote_task_to_local_dsq() sets p->scx.sticky_cpu across the migration so that deactivate_task() on the source rq doesn't end custody. Since commit b75aaea24c9f ("sched_ext: Properly mark SCX-internal migrations via sticky_cpu"), enqueue_task_scx() only clears it after inserting the task, so task_leave_custody() still sees the migration in progress and skips the custody exit. Clear p->scx.sticky_cpu as soon as enqueue_task_scx() has read it, as was done before that commit. The source side is unaffected. Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics") Cc: stable@vger.kernel.org # 7.1.x: 18d62044cda7: sched_ext: Preserve rq tracking across local DSQ dispatch Cc: stable@vger.kernel.org # 7.1.x Reviewed-by: Andrea Righi <arighi@nvidia.com> Assisted-by: Claude:claude-opus-5.5 Signed-off-by: Kuba Piecuch <jpiecuch@google.com> Signed-off-by: Tejun Heo <tj@kernel.org>