summaryrefslogtreecommitdiff
path: root/kernel/sched/core.c
AgeCommit message (Collapse)Author
4 dayssched/core: Account I/O waits independently of delayed dequeueChristian Loehle
Starting blkio delay accounting and incrementing rq->nr_iowait in __block_task() ties both to removal from the runqueue. With EEVDF delayed dequeue, this misses the initial part of an I/O sleep, or the entire sleep if the task wakes before it is dequeued. Such tasks are also missing from procs_blocked while their dequeue is delayed. Final dequeue can run in another task's context, causing delayacct_blkio_start() to timestamp the wrong task. Start both forms of accounting in try_to_block_task(), when the current task commits to blocking, before delayed dequeue or proxy execution can retain it on the runqueue. End both when clearing is_blocked, including queued wakeups and the proxy execution paths that let the current task continue. Checking is_blocked excludes wakeups that precede blocking. Save the CPU charged at the start of the wait. Delayed tasks can migrate while still queued, and proxy execution can also move blocked tasks. Decrement the saved CPU's counter on unblock, regardless of the task's current placement, and remove the old dequeue-based accounting. CPU iowait continues to classify idle time using the counter on the charged CPU; it does not accumulate the full per-task blocked duration. Closes: https://lore.kernel.org/r/20261014081731.1068794-1-zhanxusheng@xiaomi.com Fixes: 152e11f6df29 ("sched/fair: Implement delayed dequeue") Reported-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Signed-off-by: Christian Loehle <christian.loehle@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com> Link: https://patch.msgid.link/fd8578a1-e60b-4993-ab51-2b5d63abcf77@arm.com
2026-10-25Merge tag 'sched-urgent-2026-10-25' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull scheduler fixes from Ingo Molnar: - Fix MM CID affinity-triggered switch back to per-task mode (Hui Su) - Fix scheduling regression observed on AMD Ryzen AI HX 370 systems, caused by cache aware scheduling incorrectly overriding asym packing placement (Tim Chen) - Fix stale get_pref_llc() prototype for !CONFIG_SCHED_CACHE (Yafang Shao) * tag 'sched-urgent-2026-10-25' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: sched: fix stale get_pref_llc() stub for !CONFIG_SCHED_CACHE sched/cache: Honor asym packing over cache aware scheduling on hybrid systems sched/mmcid: Fix affinity-triggered switch back to per-task mode
2026-10-20Merge tag 'sched-core-2026-10-18' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull scheduler updates from Ingo Molnar: "Kconfig updates: - Simplify PREEMPT_DYNAMIC (Mark Rutland) Fair scheduler updates: - Remove redundant RUN_TO_PARITY check (Elif Topuz) - Declare augmented callbacks per field, fix rb_add_augmented_cached() descent (Yiwei Lin, Peter Zijlstra) - Restart fair hrtick after same-task repicks (Shubhang Kaushik) - Improve the scheduling latency of short slice tasks (Vincent Guittot) Deadline/RT scheduler updates: - Check start_dl_timer expiry with ktime_before() (Liang Hao) - irq_work: CPU-hotplug improvements on PREEMPT_RT (Sebastian Andrzej Siewior) Energy-aware scheduling updates: - Implement the steal_governor, introduce preferred CPUs and steal-driven vCPU backoff (Shrikanth Hegde) Multi-processor load-balancing updates: - Enable preferred SMT siblings on NVIDIA Olympus (Andrea Righi) - Randomize equally shallow idle CPU picks (Christian Loehle) - Remove redundant core_sched_seq (Hui Su) Cache-aware scheduling updates: - Avoid publishing partial LLC utilization (Jemmy Wong) NUMA-balancing updates: - Reset NUMA fault locality after scan period update (Eric Kim) Proxy execution updates: - Ongoing work to implement sleeping owner handling for proxy execution (John Stultz, Vasily Gorbik) - Preparatory steps to make sched_ext compatible with proxy execution (Andrea Righi) Scheduler infrastructure updates & fixes: - Remove the RCU wait from PSI trigger destruction (Shakeel Butt) - Avoid modifying wait-entry on failed priority registration (Hui Su) - Use the new static key API for sched_feat (Hongyan Xia) - Convert paravirt_steal to new static key APIs (Hongyan Xia) - cputime: Don't account idle time twice after dyntick-idle (Stian Halseth) Scheduler debugging updates: - Set TIF_NEED_RESCHED before calling __trace_set_need_resched() (Sechang Lim) - Fix context analysis errors in non-preferred CPU push (Shrikanth Hegde) Misc cleanups and fixes by Christian Loehle, Shubhang Kaushik, Thomas Huth and Vladimir Zapolskiy" * tag 'sched-core-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (64 commits) sched: Replace nr_iowait_cpu() with a bool accessor Revert "sched/fair: Remove dead code on enqueue_task_fair()" sched/cache: Avoid publishing partial LLC utilization sched/cputime: Don't account idle time twice after dyntick-idle sched/psi: Remove the RCU wait from trigger destruction sched/eevdf: Remove redundant RUN_TO_PARITY check sched/wait: Avoid modifying entry on failed priority registration rbtree_test: Use rb_add_augmented_cached() for the cached augmented test rbtree: Update augmented data on the way down in rb_add_augmented_cached() rbtree: Declare augmented callbacks per field with RB_AUG() rbtree_test: Use rb_add() and rb_add_cached() for the basic tests smpboot: Don't park the thread if work is pending irq_work: Flush lazy work CPU down on PREEMPT_RT irq_work: Update a comment regarding CPU hotplug invocation sched/topology: Add asymmetric SMT packing override sched/fair: Honor asymmetric SMT priority in idle selection sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched() sched/core: Fix context analysis errors in non-preferred CPU push virt/steal_governor: Enable the driver virt/steal_governor: Implement steal_governor policy loop ...
2026-10-20sched/mmcid: Fix affinity-triggered switch back to per-task modeHui Su
MM CID switches to per-CPU ownership when the number of users of an mm exceeds the maximum number of CIDs. When the affinity union grows, the per-CPU threshold is recalculated and the MM can switch back to per-task ownership through deferred work. mm_cid_calc_pcpu_thrs() incorrectly uses min(nr_cpus_allowed, users) to calculate that switch-back threshold. This makes pcpu_thrs no larger than users, so the users < pcpu_thrs check in mm_update_cpus_allowed() can never succeed. For example, with three users restricted to two CPUs, the MM enters per-CPU mode. On an eight-CPU system, widening the affinity union to all eight CPUs should produce a threshold of four, but the old calculation produces three and returns early. The MM therefore remains in per-CPU mode even though the widened CPU set makes per-task ownership preferable. Calculate pcpu_thrs from nr_cpus_allowed directly. The users bound in __mm_update_max_cids() still controls when per-CPU mode is entered, so this does not change the threshold calculated when switching from per-task to per-CPU ownership. Tested with an x86_64 QEMU reproducer using three users, an initial two-CPU affinity, and an eight-CPU affinity expansion. The baseline does not invoke mm_cid_work_fn() after the affinity expansion. The fixed kernel invokes the deferred work once and completes the switch back to per-task mode. Fixes: fbd0e71dc370 ("sched/mmcid: Provide CID ownership mode fixup functions") Signed-off-by: Hui Su <sh_def@163.com> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260915155953.2856524-1-sh_def@163.com
2026-10-16sched: Replace nr_iowait_cpu() with a bool accessorChristian Loehle
The menu governor was the last consumer of the per-CPU I/O-wait count that needed its magnitude, apart from nr_iowait() summing it globally. Commit 38f83090f515 ("cpuidle: menu: Remove iowait influence") removed that use. All remaining per-CPU consumers only test for zero/nonzero. Replace nr_iowait_cpu() with bool cpu_has_iowait() and update the generic and s390 idle accounting callers. Have nr_iowait() read the counters directly so that the global total and existing accounting are preserved. Replace the stale menu comment with a description of the predicate. Also convert the remaining open-coded idle accounting checks to use the predicate, as suggested by Prateek. The conversion was done with coccinelle: @ predicate depends on patch @ expression cpu; @@ - nr_iowait_cpu(cpu) > 0 + cpu_has_iowait(cpu) @ callers depends on patch @ expression cpu; @@ - nr_iowait_cpu(cpu) + cpu_has_iowait(cpu) Signed-off-by: Christian Loehle <christian.loehle@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com> Link: https://patch.msgid.link/20261016073341.575472-1-christian.loehle@arm.com
2026-10-14sched/core: Call wakeup_preempt() for queued tasks during class changeK Prateek Nayak
Paul reported recent RCU changes exposed a lost resched where CPU idles with a rcutorture thread waiting while being completely runnable. Dumping the rq state at the time of idling with runnable tasks showed the following: rq->next_class = &fair_sched_class rq->nr_running = 1 rq->cfs.h_nr_queued = 0 rq->rt.rt_nr_running = 1 Although the waiting task was an RT task, "rq->next_class" did not point to rt_sched_class and was stuck at the lower priority fair_sched_class. Inspecting sched_change_end() revealed the following race with __sched_setscheduler() for queued, but non-running tasks: CPU0 CPU1 ==== ==== __schedule() /* * rq->donor is a fair task that blocks. * rq->next_class = &fair_sched_class */ rq_lock(rq0) pick_next_task() /* nr_running = 0 */ pick_task_fair() sched_balance_newidle() raw_spin_rq_unlock(rq0) rq_lock(rq0) activate_task(rq0, p) /* p is a fair task */ rq0->nr_running++; rq0->cfs.h_nr_queued++; wakeup_preempt(rq, p) rq->next_class = &fair_sched_class resched_curr() rq_unlock(rq0) __sched_setscheduler(p) /* p is moved to SCHED_RR */ task_rq_lock(p) sched_change_begin() rq0->cfs.h_nr_queued--; rq0->rt.rt_nr_running++; sched_change_end() /* * p->on_rq = 1 (ctx->queued) * but p is not rq->donor * (!ctx->running) hence * resched_curr() is and * rq->next_class upgrade is * skipped. */ task_rq_unlock() raw_spin_rq_lock(rq0) /* * rq->nr_running = 1 * rq->cfs.h_nr_queued = 0 * rq->rt.rt_nr_running = 1 * rq->next_class = &fair_sched_class */ if (rq->cfs.h_nr_queued && ...) /* Above is false */ if (rq_modified_above(this_rq, &fair_sched_class)) /* Also false since rq->next_class is still fair */ /* * sched_balance_newidle() is considered to * have failed to pull any fair tasks and no * other wakeups were detected. */ return NULL; /* pick_task_fair() */ pick_task_idle() return rq->idle; next = rq->idle; /* Preemption indicators are lost */ clear_tsk_need_resched(prev); clear_preempt_need_resched(); rq->curr = rq->idle; !!! rq->idle switches in with rq->running == 1 but !need_resched() !!! In simple words, enqueue, followed by a class change of the newly queued task goes unnoticed in rq->next_class indicator and sched_balance_newidle() sees no fair task, nor any higher priority task enqueued. The above sequence will lead to CPU idling with a runnable task on the CPU. Call wakeup_preempt() for ctx->queued cases in sched_change_end() to re-evaluate rq->next_class after class change (ENQUEUE_CLASS) and set the NEED_RESCHED indicator accordingly. Newidle balance will correctly see an enqueue of a higher priority task if a queued task was upgraded when the rq_lock() was dropped and will retry the pick. Closes: https://lore.kernel.org/lkml/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/ Fixes: 704069649b5b ("sched/core: Rework sched_class::wakeup_preempt() and rq_modified_*()") Reported-by: Paul E. McKenney <paulmck@kernel.org> Tested-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org> Reviewed-by: Phil Auld <pauld@redhat.com> Link: https://patch.msgid.link/20261013152455.9039-1-kprateek.nayak@amd.com
2026-10-13Merge branch 'tip/sched/urgent'Peter Zijlstra
Pickup dependencies; notably the RB-tree fix. Signed-off-by: Peter Zijlstra <peterz@infradead.org>
2026-10-01sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched()Sechang Lim
set_tsk_need_resched() tests TIF_NEED_RESCHED, calls __trace_set_need_resched() if the flag is clear, then sets it via set_tsk_thread_flag(). A BPF raw_tp program attached to sched_set_need_resched executes synchronously inside __bpf_trace_run(). On return, __bpf_trace_run() drops the RCU lock with rcu_read_unlock_migrate(), which on the preempt-or-BH-disabled path calls set_need_resched_current() -> set_tsk_need_resched() again. set_tsk_thread_flag() follows the tracepoint call, so every re-entrant frame sees TIF_NEED_RESCHED clear and calls __trace_set_need_resched() again: BUG: TASK stack guard page was hit at ffffc9001224ff98 Oops: stack guard page: 0000 [#1] SMP KASAN PTI RIP: 0010:__bpf_trace_sched_set_need_resched_tp+0x1c/0x190 Call Trace: trace_sched_set_need_resched_tp+0x110/0x130 set_tsk_need_resched include/linux/sched.h:2076 set_need_resched_current include/linux/sched.h:2094 rcu_read_unlock_special+0x43a/0x440 __rcu_read_unlock+0x9e/0x120 rcu_read_unlock_migrate+0xa9/0x240 __bpf_trace_run+0x131/0x180 bpf_trace_run3+0x333/0x430 __bpf_trace_sched_set_need_resched_tp+0x13a/0x190 trace_sched_set_need_resched_tp+0x110/0x130 set_tsk_need_resched include/linux/sched.h:2076 ... __resched_curr() has the same ordering, firing the tracepoint before setting the flag via set_ti_thread_flag() or set_nr_and_not_polling(). Fix it for consistency. Replace the separate test_tsk_thread_flag() + set_tsk_thread_flag() pair in set_tsk_need_resched() with test_and_set_tsk_thread_flag(). In __resched_curr(), move the tracepoint call after the flag is set in each path. Fixes: adcc3bfa8806 ("sched: Adapt sched tracepoints for RV task model") Signed-off-by: Sechang Lim <rhkrqnwk98@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Andrea Righi <arighi@nvidia.com> Acked-by: Gabriele Monaco <gmonaco@redhat.com> Link: https://patch.msgid.link/20260630084750.2792851-1-rhkrqnwk98@gmail.com
2026-10-01sched/core: Fix context analysis errors in non-preferred CPU pushShrikanth Hegde
Nathan reported build failure with clang23 when CONFIG_PREFERRED_CPU=y. kernel/sched/core.c:11298:3: error: releasing raw_spinlock 'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis] 11298 | rq_unlock(rq, &rf); | ^ kernel/sched/core.c:11303:1: error: raw_spinlock 'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis] context_unsafe_alias(rq) was placed after rq_lock(), by which point it had already associated the lock with the original rq alias. Move the alias barrier before rq_lock() so that the changing rq alias is accounted for before acquiring the lock. Fixes: 74699f56ebcf ("sched/core: Push current task from non preferred CPU") Closes: https://lore.kernel.org/all/20260929121838.GA1814129@ax162/ Reported-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Tested-by: Nathan Chancellor <nathan@kernel.org> # build Link: https://patch.msgid.link/20260929164712.1054883-1-sshegde@linux.ibm.com
2026-09-28sched/debug: Add migration stats due to non preferred CPUsShrikanth Hegde
Add a new per-task stat, - nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred. Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update /proc/schedstat. Hence increasing the schedstat version is not necessary. Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260928053728.797539-10-sshegde@linux.ibm.com
2026-09-28sched/core: Push current task from non preferred CPUShrikanth Hegde
Actively push out the current running task on a non-preferred CPU (NPC). Since the task is currently running, a stopper thread must be queued to push the task out. However, if the task is pinned only to non-preferred CPUs, it will continue running there. This helps to maintain userspace affinities, unlike CPU hotplug or isolated cpusets. The implementation follows the structure of __balance_push_cpu_stop(), but is kept separate to handle the preferred-CPU specific conditions and pending-work state under CONFIG_PREFERRED_CPU. Add the npc_push_work_pending flag to protect the work buffer. For now, only the currently running task is pushed out. This keeps the code simpler. In the future, an optimization may be added to move all queued tasks on the runqueue. This works only for the FAIR scheduling class. Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260928053728.797539-9-sshegde@linux.ibm.com
2026-09-28sched/core: Try to use a preferred CPU in is_cpu_allowedShrikanth Hegde
When possible, try to choose a preferred CPU. This is essential to maintain user affinities when preferred CPUs change. A task pinned on a non-preferred CPU should continue to run there, since this is a non-user triggered event. If a CPU is non-preferred and the task can run on other CPUs which are currently preferred, then choose a preferred CPU instead. This is decided by checking if cpus_ptr, cpu_preferred_mask and task possible CPUs intersect or not. If yes, then the task has other preferred CPUs. This takes care of tasks with architecture-specific CPU masks (e.g., 32-bit tasks on arm64). The push task mechanism uses a stopper thread which calls select_fallback_rq() and uses this mechanism to pick a preferred CPU. This takes care of the wakeup path for FAIR tasks too. is_cpu_allowed() is called to ensure wakeups happen on preferred CPUs. With that, additional checks in available_idle_cpu() are not necessary. Ignore the preferred CPU state if a task's affinity is changing and its new mask no longer includes the CPU it is currently running on. This ensures migration_cpu_stop() does not abort, preventing the task from being stranded outside its allowed affinity. For the majority of cases, this would still keep select_fallback_rq() as O(N). cpumask_intersects_and(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomes O(N**2) only for tasks pinned solely to non-preferred CPUs. That is a rare case. Overhead is minimal when the CPU is preferred. Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260928053728.797539-7-sshegde@linux.ibm.com
2026-09-28cpumask: Introduce cpu_preferred_maskShrikanth Hegde
Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred. CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state. A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor. Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case. Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260928053728.797539-5-sshegde@linux.ibm.com
2026-09-28sched: Add sched_ext hooks for proxy executionAndrea Righi
Proxy execution splits the scheduling context (the donor) from the execution context (the lock owner). sched_ext needs to observe that split at three points in __schedule(): - whether a blocked EXT task can be retained on the runqueue as a donor, - when a donor's scheduling context starts driving a lock owner, - after proxy resolution, when deferred reenqueue work that was blocked by an active proxy relationship can be retried. Introduce scx_allow_proxy_exec(), scx_proxy_donor_start() and scx_proxy_reenqueue_retry(), and add their call sites in __schedule(). The implementations are empty here and are filled in by the sched_ext changes that follow, so that all the sched core changes needed by proxy execution stay together in the preparatory patches. SCHED_PROXY_EXEC still depends on !SCHED_CLASS_EXT, so the new hooks are inert: they are compiled out with CONFIG_SCHED_CLASS_EXT=n and unreachable otherwise. This is a preparatory change to support proxy execution with sched_ext. No functional change. Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260922165445.943315-6-arighi@nvidia.com
2026-09-28sched: Add helper to block retained proxy donorsAndrea Righi
Scheduler ownership changes may need to turn a mutex-blocked task retained on the runqueue for proxy execution back into a normally blocked task. Add sched_proxy_block_task() to perform that transition while holding p->pi_lock and the task's runqueue lock. Reset an active donor to the physical current task before blocking it. If DELAY_DEQUEUE keeps a FAIR donor queued, the immediately following sched_change_begin() completes the dequeue through switching_from_fair() before recording its queued state. This is a preparatory change to support proxy execution with sched_ext. Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: John Stultz <jstultz@google.com> Link: https://patch.msgid.link/20260922165445.943315-5-arighi@nvidia.com
2026-09-28sched/core: Mark wakeups completed through ttwu_runnable()Andrea Righi
ttwu_runnable() completes a wakeup for a task which remains queued without calling activate_task() or its scheduling-class enqueue callback. Scheduling classes currently cannot distinguish this path from wakeups which performed a full activation. Add WF_TTWU_RQ to identify wakeups completed through ttwu_runnable(). A later sched_ext change uses it to reconsider retained proxy donors which wake without another ops.enqueue() call. No functional change. Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260922165445.943315-4-arighi@nvidia.com
2026-09-28sched/core: Dequeue waking proxy donors before resetAndrea Righi
proxy_needs_return() resets an active donor while holding blocked_lock. proxy_reset_donor() invokes scheduling-class callbacks, adding an unnecessary raw-spinlock nesting. For an EXT donor, resetting rq->donor first calls put_prev_task_scx() while SCX_TASK_QUEUED and is_blocked are still set. That path reenqueues the retained donor through scx_do_enqueue_task(), only for the following block_task() to dequeue it again. Split block_task() so the waking donor is first dequeued from its scheduling class while it is still rq->donor. This lets dequeue_task_scx() end the donor's running session and clear SCX_TASK_QUEUED before proxy_reset_donor() invokes put_prev_task_scx(). Keep the generic on_rq state set until donor references are replaced, then complete the runqueue removal. This follows the normal sleep ordering, avoids transiently re-enqueuing the waking donor, and moves the scheduling-class callbacks outside blocked_lock. This is a preparatory change to support proxy execution with sched_ext. Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260922165445.943315-3-arighi@nvidia.com
2026-09-28sched/core: Drop mutex locks before proxy reschedulingAndrea Righi
find_proxy_task() can call proxy_resched_idle() from return expressions inside mutex wait_lock and blocked task lock guard scopes. Return expressions are evaluated before scoped guards are released, so proxy_resched_idle() invokes scheduling-class callbacks while both raw spinlocks are held. Route these cases through a common label outside the guard scopes. The common label is required because a direct return would evaluate proxy_resched_idle() before guard cleanup and run the class callbacks under the proxy-exec locks again. This is a preparatory change to support proxy execution with sched_ext. Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com> Link: https://patch.msgid.link/20260922165445.943315-2-arighi@nvidia.com
2026-09-25sched/core: Remove redundant core_sched_seqHui Su
core_sched_seq records whether an rq has consumed the pick made by the last core-wide selection. However, rq->core_pick already carries the same per-rq state. A core-wide selection leaves core_pick populated only for siblings which still need to consume their picks. It is cleared when the current CPU consumes its pick, when a sibling is already running the selected task, or when a pending pick is consumed through the fastpath. The CPU offline path clears it as well. core_task_seq and core_pick_seq serve a separate purpose. core_task_seq changes when the task set changes and for each new core-wide pick, while core_pick_seq records the task sequence on which the selection was made. Their equality therefore establishes that a pending core_pick is still valid. Consequently, once core_pick_seq == core_task_seq, a non-NULL core_pick is sufficient to tell that this rq still has a pick to consume. core_sched_seq duplicates that pending/consumed state. Remove core_sched_seq and use core_pick directly as the pending marker. Testing included an instrumented comparison of the old and new pending predicates under core-scheduling stress. Additional guest tests exercised repeated core-cookie lifecycles, forced idle, and CPU hotplug under load. No predicate divergence or kernel warning was observed. Signed-off-by: Hui Su <sh_def@163.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260915163138.2973969-1-sh_def@163.com
2026-09-25sched: Restart fair hrtick after same-task repicksShubhang Kaushik (Ampere)
Fair hrticks are one-shot timers. A same-task repick after hrtick expiry skips set_next_task_fair() and leaves no hrtick armed for the next fair preemption point: hrtick() -> task_tick_fair(..., queued=1) -> entity_tick() -> resched_curr() -> schedule() -> pick_task_fair() selects current -> put_prev_set_next_task(rq, prev, next) -> next == prev -> return Introduce SNT_REPICK for the next == prev path: put_prev_set_next_task(rq, prev, next) -> next == prev -> next->sched_class->set_next_task(rq, next, SNT_REPICK) -> set_next_task_fair() -> hrtick_start_fair() set_next_task_fair() skips task-transition work for SNT_REPICK and restarts the fair hrtick. hrtick_start() records the delay during schedule(), and hrtick_schedule_exit() rearms the timer. Do not restart the SCHED_DEADLINE hrtick for SNT_REPICK. Same-task repicks skip put_prev_task() for both fair and SCHED_DEADLINE, but pick_task_fair() refreshes the selected fair entity before selecting it again. hrtick_start_fair() reads that refreshed entity, including with group scheduling. The SCHED_DEADLINE picker has no corresponding current-runtime update. Restarting its hrtick on SNT_REPICK could therefore use stale dl_se->runtime and delay the next runtime enforcement point. Suggested-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Link: https://patch.msgid.link/20260917-sched-fair-hrtick-restart-v4-1-4dd1414da81a@gentwo.org
2026-09-24sched/core: Account PSI IRQ time to the execution context, not the ↵Zhan Xusheng
scheduling context psi_account_irqtime() has two callers which share rq->psi_irq_time, and they disagree about the context: __schedule() passes the outgoing rq->curr, sched_tick() passes rq->donor. Under proxy execution the donor is blocked on a mutex while rq->curr burns the CPU. The tick charges PSI_IRQ_FULL to the donor's cgroup and advances the timestamp, so the call from __schedule() then finds delta <= 0 and charges nothing. The delta is not counted twice, it lands on the wrong cgroup. Pass rq->curr, which is what the call read before commit af0c8b2bf67b ("sched: Split scheduler and execution contexts") renamed 'curr' to 'donor' across sched_tick(). Without CONFIG_SCHED_PROXY_EXEC the two rq members are a union, so this only changes anything where that option is set, and it depends on EXPERT. Fixes: af0c8b2bf67b ("sched: Split scheduler and execution contexts") Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Link: https://patch.msgid.link/20260918132915.1236312-1-zhanxusheng@xiaomi.com
2026-09-18sched/core: Avoid false migration warning for proxy donorsAndrea Righi
Proxy execution can move a blocked donor's scheduling context to the lock owner's CPU even when the donor is migration-disabled. The donor does not execute there, and its original execution CPU remains recorded in wake_cpu. set_task_cpu() warns unconditionally for migration-disabled tasks, so a subsequent proxy migration or the wakeup path returning the donor home triggers a false positive: moving a blocked scheduling context does not violate the migration-disabled execution context. For example, creating a mutex owner on CPU1 and a migration-disabled waiter on CPU0 can trigger the following warning: proxy_migrate_repro: donor blocking on CPU0 with migration disabled proxy_migrate_repro: donor moved from CPU0 to CPU1 WARNING: kernel/sched/core.c:3389 at set_task_cpu+0x1d3/0x280 ... Call Trace: try_to_wake_up+0x43f/0x780 __mutex_unlock_slowpath+0x330/0x540 owner_fn+0x9f/0xc0 [proxy_migrate_repro] ... proxy_migrate_repro: donor woke on CPU0, task_cpu=0 proxy_migrate_repro: completed Exclude blocked proxy donors from the warning. The proxy wakeup path restores an executable placement before clearing the blocked state. Fixes: b049b81bdff6 ("sched: Handle blocked-waiter migration (and return migration)") Signed-off-by: Andrea Righi <arighi@nvidia.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: John Stultz <jstultz@google.com> Link: https://patch.msgid.link/20260915184101.2621252-1-arighi@nvidia.com
2026-09-10sched/core: Call wq_worker_tick() for the execution contextHui Su
wq_worker_tick() accounts CPU time and detects CPU-intensive work for the kworker that is actually running. With proxy execution, rq->donor is the scheduling context while rq->curr is the execution context. Calling the hook with rq->donor can skip workqueue accounting when a kworker is executing on behalf of a donor task. It can also account a blocked kworker when the donor is a worker but rq->curr is the task actually executing. The former can delay WORKER_CPU_INTENSIVE handling and pool concurrency management, which can delay pending kernel work and userspace operations depending on it. Use rq->curr for the workqueue tick hook while retaining rq->donor for scheduler accounting. Fixes: af0c8b2bf67b ("sched: Split scheduler and execution contexts") Signed-off-by: Hui Su <sh_def@163.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Tejun Heo <tj@kernel.org> Link: https://patch.msgid.link/20260902150208.1209922-2-sh_def@163.com
2026-09-06Merge tag 'sched-urgent-2026-09-06' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull scheduler fixes from Ingo Molnar: - Fix a timestamping bug in pick_task_fair() and yield_task_fair() (Zhan Xusheng) - Skip migrate-disabled tasks when picking a push candidate in the RT and DL schedulers (Seiji Nishikawa) - Skip rq->avg_idle update without a valid idle_stamp (Shubhang Kaushik) - Fix throttling bug in throttle_cfs_rq(), caused by the recent single-runqueue conversion (Wanwu Li) - Fix bandwidth calculation bug in distribute_cfs_runtime(), caused by the single-runqueue conversion (Wanwu Li) - Don't make x86 ITMT enablement depend on debugfs (Mario Limonciello) - Avoid creating misfits during cache-aware load-balancing on hybrid systems (Tim Chen) * tag 'sched-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: sched/fair: Avoid creating misfits during cache-aware balancing x86/itmt: Don't make ITMT enablement depend on debugfs sched/fair: Use cfs_rq->h_curr in distribute_cfs_runtime() sched/fair: Use cfs_rq->h_curr in throttle_cfs_rq() sched/core: Skip rq->avg_idle update without a valid idle_stamp sched/rt,dl: Skip migrate-disabled tasks when picking a push candidate sched/fair: Use update_curr_eevdf() for the remaining root cfs_rq callers
2026-09-04futex: Provide rt_mutex_.*_schedule() equivalents for futex schedulingSebastian Andrzej Siewior
There is rt_mutex_{pre|post}_schedule() around rt_mutex_wait_proxy_lock() to ensure that sched_submit_work()/ sched_update_worker() is invoked before we schedule out and block on rt_mutex while waiting for it become available. The reason is that blocking on rt_mutex assigns a pi_waiter for the PI chain and sched_submit_work() will also assign a pi_waiter if it blocks on lock but a this point we already have a waiter assigned. We can't skip sched_submit_work() entirely because I/O relies on the fact that I/O queue is flushed while it blocks on a sleeping lock. Therefore sched_submit_work() is moved before we block on the lock. Sleeping lock in this context means mutex or rw_semaphore not spinlock_t on PREEMPT_RT. Because the mutex abstraction on PREEMPT_RT uses the same abstraction as the futex proxy lock, the futex code ended up using rt_mutex_{pre|post}_schedule(), too. Using it is/ was just to keep the task_struct::sched_rt_mutex assertion happy. Futex proxy lock is used only in the syscall context of a task. At this point it never got any I/O that needs to be flushed and it can't be a workqueue that needs to notify that it will be scheduled out. Therefore sched_submit_work() does nothing here. By mistake futex_wait_requeue_pi() -> rt_mutex_wait_proxy_lock() did not get the rt_mutex_{pre|post}_schedule() annotation. This was not noticed because in this callchain the lock is (usually) not contended and so rt_mutex_slowlock_block() does not schedule, triggering the assert. Adding rt_mutex_pre_schedule() here looks wrong (as noted by PeterZ) because at this point there is a pi_waiter recorded and invoking sched_submit_work() with a possible lock contention would be wrong. Add rt_mutex_futex_{pre|post}_schedule() which toggles the sched_rt_mutex assert and does not involve sched_submit_work(). Add asserts here to ensure that sched_submit_work() would do nothing. Use it only in futex proxy lock case which is rt_mutex_wait_proxy_lock(). Remove it from futex_lock_pi(). Fixes: d14f9e930b90 ("locking/rtmutex: Use rt_mutex specific scheduler helpers") Reported-by: Yao Kai <yaokai34@huawei.com> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260901135453.3121948-2-bigeasy@linutronix.de Closes: https://lore.kernel.org/all/20260717084922.4153317-2-yaokai34@huawei.com
2026-09-02sched: dynamic: Fix preemption model stringsMark Rutland
We recently removed the enum values 'preempt_dynamic_none' and 'preempt_dynamic_voluntary', but forgot to update preempt_modes[] accordingly. Due to this, preempt_model_str() and sched_dynamic_show() produce incorrect output, with the wrong preemption model string in backtraces, and empty output for /sys/kernel/debug/sched/preempt. Update preempt_modes[] to account for the removed enum values. Given that CONFIG_PREEMPT_DYNAMIC now depends on CONFIG_ARCH_HAS_PREEMPT_LAZY, and given that only 'preempt_dynamic_full' and 'preempt_dynamic_lazy' remain, sched_dynamic_show() no longer needs to skip any model strings. Fixes: 9650ce11f2e3 ("sched: dynamic: Simplify preempt model accessors") Reported-by: Jinjie Ruan <ruanjinjie@huawei.com> Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260902101637.232129-1-mark.rutland@arm.com
2026-09-02sched: Remove unneeded function type cast in do_balance_callbacks()Vladimir Zapolskiy
It's a trivial and non-functional change, the removed explicit function type cast does not convert the function type of struct balance_callback member (*func), and the cast can be removed. Signed-off-by: Vladimir Zapolskiy <vz@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260819080159.105433-1-vz@kernel.org
2026-09-02sched: Convert paravirt_steal to new static key APIsHongyan Xia
paravirt_steal_rq_enabled and paravirt_steal_enabled use raw static_key APIs which are now deprecated. Use the new API instead. No functional change. Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Juergen Gross <jgross@suse.com> Link: https://patch.msgid.link/20260819081207.12150-1-hongyan.xia@transsion.com
2026-09-02sched: dynamic: Simplify preempt model accessorsMark Rutland
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models. When CONFIG_PREEMPT_DYNAMIC=y, the NONE and VOLUNTARY models cannot be used, and neither CONFIG_PREEMPT_NONE nor CONFIG_PREEMPT_VOLUNTARY can be selected. Simplify the preempt model accessors accordingly: remove the out-of-line implementations of preempt_model_none() and preempt_model_voluntary(), and remove the preempt_dynamic_{none,voluntary} values. There are no current users of preempt_model_voluntary(), but I've kept it around for consistency, with every scheduler model having its own preempt_model_*() accessor. Suggested-by: Shrikanth Hegde <sshegde@linux.ibm.com> Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com> Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com> Tested-by: Mete Durlu <meted@linux.ibm.com> Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com> Tested-by: Jinjie Ruan <ruanjinjie@huawei.com> Link: https://patch.msgid.link/20260803191731.3244294-7-mark.rutland@arm.com
2026-09-02sched: dynamic: Remove HAVE_PREEMPT_DYNAMIC_{CALL,KEY}Mark Rutland
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models. Switching model only changes the behaviour of dynamic_preempt_lazy(), which uses a static key. There are no other static calls or static keys, and so there's no need for HAVE_PREEMPT_DYNAMIC_CALL or HAVE_PREEMPT_DYNAMIC_KEY. The static key used by dynamic_preempt_lazy() is entirely local to kernel/sched/core.c, and any architecture which selects ARCH_HAS_PREEMPT_LAZY implements the necessary support. Remove PREEMPT_DYNAMIC's dependencies on HAVE_PREEMPT_DYNAMIC_CALL and HAVE_PREEMPT_DYNAMIC_KEY entirely, leaving the dependency on ARCH_HAS_PREEMPT_LAZY. For architectures which previously selected HAVE_PREEMPT_DYNAMIC_KEY, PREEMPT_DYNAMIC will now be selected by default, matching x86. As the runtime impact is limited to dynamic_preempt_lazy(), this shouldn't be as concerning as previously (e.g. where calls to {cond,might}_resched() stubs could introduce a measurable penalty). As all of this can work without jump labels, JUMP_LABEL is not selected explicitly. It is obviously preferable to have JUMP_LABEL enabled, but this is not functionally necessary. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com> Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com> Tested-by: Mete Durlu <meted@linux.ibm.com> Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com> Tested-by: Jinjie Ruan <ruanjinjie@huawei.com> Link: https://patch.msgid.link/20260803191731.3244294-6-mark.rutland@arm.com
2026-09-02sched: dynamic: Simplify irqentry_exit_cond_resched()Mark Rutland
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models. In either model, irqentry_exit_cond_resched() is always called and never disabled. Remove the unnecessary code for this when PREEMPT_DYNAMIC is selected. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com> Tested-by: Mete Durlu <meted@linux.ibm.com> Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com> Tested-by: Jinjie Ruan <ruanjinjie@huawei.com> Link: https://patch.msgid.link/20260803191731.3244294-5-mark.rutland@arm.com
2026-09-02sched: dynamic: Simplify preempt_schedule{,_notrace}()Mark Rutland
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models. In either model, both preempt_schedule() and preempt_schedule_notrace() are always called and never disabled. Remove the unnecessary code for these when PREEMPT_DYNAMIC is selected. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com> Tested-by: Mete Durlu <meted@linux.ibm.com> Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com> Tested-by: Jinjie Ruan <ruanjinjie@huawei.com> Link: https://patch.msgid.link/20260803191731.3244294-4-mark.rutland@arm.com
2026-09-02sched: dynamic: Simplify {cond,might}_resched()Mark Rutland
PREEMPT_DYNAMIC is now limited to the FULL and LAZY preemption models. In either model, both cond_resched() and might_resched() are always disabled and do nothing. Remove the unnecessary code for these when PREEMPT_DYNAMIC is selected. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com> Tested-by: Mete Durlu <meted@linux.ibm.com> Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com> Tested-by: Jinjie Ruan <ruanjinjie@huawei.com> Link: https://patch.msgid.link/20260803191731.3244294-3-mark.rutland@arm.com
2026-09-02sched: dynamic: Make PREEMPT_DYNAMIC depend on ARCH_HAS_PREEMPT_LAZYMark Rutland
On architectures which select ARCH_HAS_PREEMPT_LAZY, it has not been possible to select the NONE and VOLUNTARY preemption models since commit: 7dadeaa6e851 ("sched: Further restrict the preemption modes") ... which was merged in v7.0. All architectures which currently support PREEMPT_DYNAMIC select ARCH_HAS_PREEMPT_LAZY: [mark@lakrids:~/src/linux]% git describe HEAD v7.2-rc1-1-g871a4586ea2e1 [mark@lakrids:~/src/linux]% git grep 'select HAVE_PREEMPT_DYNAMIC_' -- arch arch/arm64/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY arch/loongarch/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY arch/powerpc/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY arch/riscv/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY arch/s390/Kconfig: select HAVE_PREEMPT_DYNAMIC_KEY arch/x86/Kconfig: select HAVE_PREEMPT_DYNAMIC_CALL [mark@lakrids:~/src/linux]% git grep 'select ARCH_HAS_PREEMPT_LAZY' -- arch arch/arm64/Kconfig: select ARCH_HAS_PREEMPT_LAZY arch/loongarch/Kconfig: select ARCH_HAS_PREEMPT_LAZY arch/powerpc/Kconfig: select ARCH_HAS_PREEMPT_LAZY arch/riscv/Kconfig: select ARCH_HAS_PREEMPT_LAZY arch/s390/Kconfig: select ARCH_HAS_PREEMPT_LAZY arch/x86/Kconfig: select ARCH_HAS_PREEMPT_LAZY ... and hence in practice PREEMPT_DYNAMIC no longer supports the NONE or VOLUNTARY preemption models. Make this official: have PREEMPT_DYNAMIC depend on ARCH_HAS_PREEMPT_LAZY, and remove the trivially unreachable code. Further simplifications will be made in subsequent patches. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com> Tested-by: Mete Durlu <meted@linux.ibm.com> Tested-by: Shrikanth Hegde <sshegde@linux.ibm.com> Tested-by: Jinjie Ruan <ruanjinjie@huawei.com> Link: https://patch.msgid.link/20260803191731.3244294-2-mark.rutland@arm.com
2026-09-02sched: Migrate whole chain in proxy_migrate_task()John Stultz
Instead of migrating one task each time through find_proxy_task(), we can walk up the blocked_donor ptrs and migrate the entire current chain in one go. This was broken out of earlier patches and held back while the series was being stabilized, but I wanted to re-introduce it. Signed-off-by: John Stultz <jstultz@google.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260807035232.1881495-8-jstultz@google.com
2026-09-02sched: Switch rq->next_class in proxy_reset_donor()John Stultz
Similar to commit 04f80f8b12a0 ("sched: Switch rq->next_class on proxy_resched_idle()"), we should also set the rq->next_class when we call proxy_reset_donor(). Fixes: f13beb010e4a ("sched: Have try_to_wake_up() handle return-migration for PROXY_WAKING case") Signed-off-by: John Stultz <jstultz@google.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260807035232.1881495-6-jstultz@google.com
2026-09-02sched/core: Don't proxy-exec unmatched cookie lock ownersVasily Gorbik
Core scheduling chooses a core-wide cookie before __schedule() installs the next task. With proxy-exec enabled, that task becomes the donor/scheduling context, and find_proxy_task() may then replace the execution context with the runnable mutex owner. If its cookie differs from the selected core cookie, running it would bypass core scheduling's cookie selection. When the final mutex owner found by find_proxy_task() does not match the selected core cookie, stop proxying the donor. If the current execution context is already in the blocked chain, fall back to idle like the existing proxy-exec retry paths do. Otherwise deactivate the donor and let __schedule() pick again. The mutex owner can be picked later under its own cookie. Fixes: 7de9d4f94638 ("sched: Start blocked_on chain processing in find_proxy_task()") [jstultz: Added tweak to ensure we deactivate donor, not runnable owner] Reported-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com> Signed-off-by: John Stultz <jstultz@google.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lore.kernel.org/lkml/10282ce9-f4ae-498f-9b57-f4e1e61fffbc@amd.com/ Link: https://patch.msgid.link/20260807035232.1881495-5-jstultz@google.com
2026-09-02sched/core: Avoid migrating blocked_on tasksJohn Stultz
Its not useful to migrate blocked_on tasks from other runqueues as the proxy logic will just migrate it back to the owner's rq. So skip blocked_on tasks here. Signed-off-by: John Stultz <jstultz@google.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260807035232.1881495-4-jstultz@google.com
2026-09-02sched/core: Don't steal a proxy-exec donorVasily Gorbik
try_steal_cookie() avoids stealing src->core_pick and src->curr before moving a task with the same cookie via move_queued_task_locked(). With proxy-exec, src->donor is the current scheduling context and may differ from src->curr. Stealing it migrates a task that the source rq still treats as current, leaving src's scheduler state for that task stale. For CFS this means cfs_rq->curr points at the stolen entity, and the next pick on the source rq hits the WARN_ON_ONCE in put_prev_entity(). Commit 7de9d4f94638 ("sched: Start blocked_on chain processing in find_proxy_task()") tweaked the fair class logic so that the donor task isn't migrated away while we're running the proxy. Do it similarly for try_steal_cookie() and skip src->donor as well. Fixes: 7de9d4f94638 ("sched: Start blocked_on chain processing in find_proxy_task()") Signed-off-by: Vasily Gorbik <gor@linux.ibm.com> Signed-off-by: John Stultz <jstultz@google.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260807035232.1881495-3-jstultz@google.com
2026-09-02sched/core: Skip rq->avg_idle update without a valid idle_stampShubhang Kaushik (Ampere)
Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and into put_prev_task_idle(), so that the idle interval is consumed whenever the idle task is switched out. The wakeup-side accounting that it replaced only updated rq->avg_idle when rq->idle_stamp was non-zero. The new helper lost that validity check and unconditionally computes: rq_clock(rq) - rq->idle_stamp If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is not a valid idle duration and can immediately drive rq->avg_idle to its clamp. This can happen when sched_balance_newidle() returns before setting rq->idle_stamp, for example when this_rq->ttwu_pending is set. In that case the rq can switch to the idle task with idle_stamp still zero and leave idle again when the pending wakeup is processed. Other paths can also switch to the idle task without setting rq->idle_stamp via newidle_balance(), for example find_proxy_task() or force-idling. Restore the idle_stamp validity check in update_rq_avg_idle() and skip the rq->avg_idle update when there is no measured idle interval. Fixes: 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU") Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com> Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org> Acked-by: John Stultz <jstultz@google.com> Link: https://patch.msgid.link/20260807-master-v3-1-c328354efed3@gentwo.org
2026-08-20Merge tag 'for-7.3/block-20260819' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux Pull block updates from Jens Axboe: - NVMe updates via Keith: - Enable Clang context analysis for the nvme host driver, adding context annotations across core, fabrics, rdma, tcp and pci - nvmet reservation state exposed through a new namespace-level debugfs directory, plus ABI documentation for the host sysfs and target configfs interfaces - nvme-tcp host memory disclosure fixes on the read path: reject a read that transferred too few bytes, don't accept C2HData based on blk_rq_payload_bytes() alone, and fix the R2T case for a read command - Parallelize nvme-rdma I/O queue allocation and startup (Surabhi) - Apple nvme fixes and quirks: page aligned admin queue buffers, destroy the admin queue on removal, and various DMA/NVMMU correctness fixes - A large pile of nvmet and host fixes for out-of-bounds reads, refcount/resource leaks, and NULL derefs across auth, zns, passthru, pci-epf, rdma and configfs - Various other fixes and cleanups - MD updates via Yu Kuai: - llbitmap reshape support, the large series wiring exact bitmap mapping and reshape lifecycle through raid5 and raid10, growing the page cache in place, and remapping checkpointed bits as reshape progresses - raid5 fixes for lockless max_nr_stripes and recovery_offset accesses, a reshape deadlock with more failed devices than max degraded, and bitmap batch counter consistency - Atomic write handling for raid1/raid10, and removal of the REQ_NOWAIT support from raid1/10/456 - raid5-ppl use-after-free fix in ppl_do_flush() - A batch of smaller fixes across md core and the bitmap code - s390/dasd ESE full-track write support and the surrounding infrastructure, plus enabling CONTEXT_ANALYSIS for s390/block - RWF_DONTCACHE support for block devices, built on new task-context bio completion infrastructure, and wiring it up for the iomap and buffer dropbehind writeback paths - Async io_uring zone reset all, plus zone management command cleanups allowing REQ_NOWAIT and tightening conventional zone rejection - Block integrity refactoring: lift BIP_CHECK_FLAGS to the shared header, handle nogenerate/noverify properly in fs-integrity, and drop the blk-integrity.h include from bdev.c - Split out a new blk_plug.h header - ublk improvements: add UBLK_F_IO_DESC_SIZE, split request validation from io_desc init, reject non-power-of-2 zone sizes in SET_PARAMS, and a series of hardening fixes around map/unmap and auto buf reg - null_blk cleanups and configfs serialization fixes - nbd queue freeze removal on the setup paths, and a new pre_defined_connections module parameter for pre-created devices - blk-cgroup fixes for the race between policy activation and blkg destruction, and accounting per-cpu stats over possible CPUs across blk-stat, iolatency, iocost and kyber - Various dio fixes: leak on metadata mapping error, validate user space vectors during extraction, and set dma_alignment from the backing file for loop and zloop direct I/O - bio cleanups - Various other fixes and cleanups all over * tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (241 commits) nbd: add pre_defined_connections module parameter for pre-created devices nbd: remove queue freeze for newly created nbd from netlink path nbd: factor out a nbd_genl_foreach_sock nbd: skip queue freeze when setting size at device startup nbd: remove queue freeze in nbd_add_socket nbd: clear queue limits on disconnect nbd: disallow NBD_SET_SOCK on an active device nbd: simplify find_fallback() by removing redundant logic blk-mq: add missing call to srcu_barrier() in blk_mq_free_tag_set() block: mtip32xx: synchronize ioctls with device removal ublk: avoid teardown retry loop on xarray allocation failure null_blk: fix UBSAN shift-out-of-bounds when zone_size is 0 or overflows block: don't include blk-integrity.h in bdev.c xfs: avoid double deferrals for RWF_DONTCACHE writes loop: Fix recently introduced lock inversion block: set QUEUE_FLAG_DYING unconditionally in blk_mark_disk_dead() swim3: Add missing MODULE_DESCRIPTION selftests: ublk: add SET_PARAMS validation test selftests: ublk: add helper for SET_PARAMS ublk: reject non-power-of-2 zone sizes in SET_PARAMS ...
2026-08-20Merge tag 'sched_ext-for-7.3' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext Pull sched_ext updates from Tejun Heo: "Most of this cycle completes the enqueue-path support for hierarchical sub-scheduling, which makes sub-scheduler support feature complete: a root BPF scheduler can now hand a cgroup subtree over to a nested sub-scheduler together with revocable CPU grants, and the sub-scheduler owns all scheduling decisions for its tasks on those CPUs. Development volume was high and a number of changes plugging holes in the new support landed late in the cycle. Also included are core scheduling fixes that were completed too late for the v7.2 release and are routed through this pull request. Sub-scheduler CPU delegation: - Parent schedulers now grant and revoke per-CPU capabilities (enqueueing, preemption, CPU frequency control) on their children, enforced on every path a scheduler can reach a CPU through. Previously only dispatching could be delegated; this lets sub-schedulers fully schedule their CPUs. - Rescue execution: a task whose scheduler doesn't have access to the CPUs the task needs to run on starved until the watchdog ejected the whole scheduler. The kernel now runs such tasks directly on a small bandwidth budget, turning a scheduler-killing failure into bounded degradation. - Cgroup integration: tasks migrating across a sub-scheduler boundary weren't re-homed to the new owner, causing wrong-scheduler scheduling and a use-after-free. Sub-schedulers now take over their cgroup subtree and receive its cgroup callbacks. - Arena objects now cross the kernel/BPF boundary as typed pointer arguments, translated transparently by the BPF tree's new arena argument support, replacing untyped arguments with manual translation. - scx_qmap now demonstrates full hierarchical sub-scheduling. Other fixes and updates: - Robustness improvements: the abort path is now NMI-safe, fixing deadlocks when errors are raised from NMI context and making hardlockup recovery direct. Reenqueue loops that could monopolize a CPU ahead of the watchdog now eject the offending scheduler, and stalls are blamed on the scheduler actually responsible. - Hardening: BPF-writable arena memory is validated before kernel use, and task slice and vtime writes got explicit synchronization rules, closing corruption vectors open to buggy or malicious schedulers. - Core scheduling: sched_ext dispatching can drop the rq lock inside the core-wide pick, which let interleaving selections corrupt each other's state and hard-hang the machine. The selection now restarts when the lock was released. The task ordering callback was also invoked with its arguments swapped, and the default ordering is updated to work across sub-scheduler boundaries. The fixes are marked for stable. - Other fixes headed for stable: a task init leak on fork failure during enable, tooling compat macros that silently failed to detect newer kernels, and a crash on reenqueueing against a destroyed dispatch queue. - Tooling: scx_pair moves off deprecated callbacks, and the deprecated scx_bpf_cpu_rq() kfunc is removed" * tag 'sched_ext-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (144 commits) sched_ext: Drop the dead SCX_DEQ_CORE_SCHED_EXEC test in dequeue_task_scx() sched_ext: Make core-sched task ordering hierarchy-aware sched_ext: Use runnable_at for the default core-sched task ordering sched_ext: Fix inverted ops.core_sched_before() invocation sched_ext: Move the config-off sub-cap kfunc stubs into sub.c sched_ext: Rename balance-era identifiers to dispatch terms sched_ext: Drop the stale keep_prev fixup in dispatch_pick() sched_ext: Keep kick_sync waiting on the rq's own CPU sched_ext: Make SCHED_CLASS_EXT select GENERIC_ALLOCATOR sched_ext/scx_flatcg: Fix cvtime true-up on slice expiry sched_ext: Don't BUG_ON a destroyed DSQ in process_deferred_reenq_users sched_ext: Fix scx_bpf_dsq_move_to_local___v2 compat detection sched_ext: Make scx_bpf_events() read the calling scheduler's counters sched_ext: Drop unlocked scx_rq_clock_invalidate() from scx_root_disable() selftests/sched_ext: Fix flaky ddsp failure tests on busy systems selftests/sched_ext: Make numa idle validation race-free sched_ext: Fix scx_bpf_dsq_reenq___compat kfunc extern prototype sched_ext/scx_flatcg: expire cached hweights on weight changes sched_ext: Fix exit_task leak on fork failure during enable sched_ext: fix stale references in doc comments ...
2026-08-20Merge tag 'sysctl-7.03-rc1' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl Pull sysctl updates from Joel Granados: - Fix kernel-doc warnings by adjusting in file documentation - Consolidate do_proc_* function into do_proc_vec Consolidate three slightly different implementations of applying a converter on all elements of a vector. Fixes to this function now propagate to the three types. - Replace CONFIG_PROC_SYSCTL with CONFIG_SYSCTL (they were the same) and restrict cad_pid modifications to global root (GLOBAL_ROOT_UID) * tag 'sysctl-7.03-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl: sysctl: remove CONFIG_PROC_SYSCTL, it just mirrors CONFIG_SYSCTL sysctl: move the "cad_pid" entry from pid_table[] to kern_reboot_table[] sysctl: repair some kernel-doc comments sysctl: add Returns: kernel-doc for all functions sysctl: Update API function documentation sysctl: Rename proc_doulongvec_minmax_conv to proc_doulongvec_conv sysctl: Group proc_handler declarations and document sysctl: Replace do_proc_do{int,ulong,uint}vec with do_proc_vec sysctl: Add negp parameter to douintvec converter functions sysctl: Move default converter assignment out of do_proc_dointvec
2026-08-18Merge tag 'sched-core-2026-08-17' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull scheduler updates from Ingo Molnar: "Load-balancing updates: - 'flatten the pick': improve cgroup scheduling, which has always been problematic and painful, which has caused various scheduling misbehavior such as the mishandling of reniced tasks et al. Add various cgroup weight distribution methods via cgroup_mode: 'up', 'max', 'concur' and 'tasks' - with the default being 'concur' which is the most precise yet also most expensive version. Finally, change cgroup scheduling to a single runqueue (Peter Zijlstra) - Series to improve the scheduling latency of short slice tasks (Vincent Guittot) - Series to fix cluster scheduling in the presence of asymmetric capacity (Ricardo Neri) - Prefer fully idle cores for NOHZ balancing (Andrea Righi) - Don't trigger active load-balancing if src_rq->curr is not on_rq (Xin Zhao) PSI updates: - Skip irqtime accounting when no new irq time has elapsed (Usama Arif) Scheduler debugging updates: - Remove unused schedstats (Shrikanth Hegde) - Defer freeing of cpumask memblock memory to initcall (Waiman Long) Misc fixes and updates by Yu C Chen, K Prateek Nayak, Peter Zijlstra, Vincent Guittot, Xin Zhao, Yury Norov, Zhan Xusheng" * tag 'sched-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (33 commits) sched/fair: Fix flat hierarchy sched/isolation: Defer freeing of cpumask memblock memory to initcall sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity sched/fair: Allow load balancing between CPUs of identical capacity sched/fair: Skip misfit load accounting when the destination CPU cannot help sched/fair: Check CPU capacity before comparing group types during load balance sched/fair: Also gate overloaded status update for SD_ASYM_CPUCAPACITY sched/fair: Do not skip CPUs of similar capacity with busy SMT siblings sched/fair: Prefer fully idle cores for NOHZ balancing stop_machine: Make stop_one_cpu_nowait() return void sched/eevdf: Delayed dequeue task can't preempt sched/fair: Fix stale comments referring to removed CFS concepts sched/debug: Remove unused schedstats sched/psi: skip irqtime accounting when no new irq time has elapsed sched/fair: Reflow sched_balance_rq() sched/fair: Simplify balance_interval reset logic in sched_balance_rq() sched/fair: Don't trigger active lb if src_rq->curr is not on_rq sched/eevdf: Speedup short slice task scheduling sched/eevdf: Always update slice protection sched/eevdf: Cancel slice protection if short slice task is eligible ...
2026-08-12Merge branch 'for-7.3' into for-7.3-arena-argsTejun Heo
2026-08-11sched/core: Make core-sched flips wait for in-flight selectionsTejun Heo
Core scheduling's pick_next_task() operates on all sibling rqs under one acquisition of the shared core-wide lock. A ->pick_task() that releases the rq lock leaves every sibling __lock momentarily free, letting __sched_core_flip(false) complete mid-selection and rebind rq_lockp() under it. The selection resumes on the split locks, touching sibling state it no longer protects, and __schedule() finally releases a lock that was never taken while leaking the one that was. Count in-flight core-wide selections in the leader's rq->core_pick_in_flight and make __sched_core_flip() wait for the count to drain. The count only changes under the shared lock, which the flip holds while sampling, so no other ordering is needed. The wait can repeat while selections overlap, but the flip backs off between samples and flips are rare cookie-lifetime events. sched_core_cpu_deactivate() moves the count to the new leader - a stale copy left behind would bias it forever if that CPU later returns as its own leader. Fixes: 539f65125d20 ("sched: Add core wide task selection and scheduling") Cc: stable@vger.kernel.org # v5.14+ Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2026-08-11sched/core: Handle pick_task() releasing the rq lockTejun Heo
Core scheduling's pick_next_task() breaks when a ->pick_task() implementation can release the rq lock. The selection state derived on entry is only valid while the lock is held continuously. Once a pick can drop the lock, an interleaving selection can invalidate all of it: the single-CPU fast path can commit an uncookied pick although the core went cookied during the release, and forceidle committed by the interleaving selection skews the restarted pass's accounting. Fix it by restarting the whole selection when a pick returns RETRY_TASK after releasing the lock: a single restart point above the state derivation replaces the per-loop restart labels, so a retry picks up state committed by interleaving selections and accounts and resets forceidle like a fresh selection would. need_sync and fi_before latch across retries. Clock validity can't be re-derived - there is no program-ordered way to tell whether the own and core rq clocks are still updated after the lock was released, as other lockers' pin cycles may or may not have invalidated them. When restarting, clear core_clock_updated so that the sibling loop re-updates the core rq, and update the own rq clock if invalidated. Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()") Cc: stable@vger.kernel.org # v6.19+ Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2026-08-10preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITSBoqun Feng
With the changes that enable preempt count to track IRQ disabling nesting, we don't have enough bits in 32-bit preempt count implementation, as a result we move NMI nesting bits out of the 32-bit preempt count. However on the architectures that can support 64-bit preempt count implementation, we can keep the NMI nesting bits in the 32-bit preempt count and avoid maintaining NMI nesting bits outside of the same cache line. Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow architectures to select this. Note that under this Kconfig, preempt count is maintained in a 64-bit word however preempt_count() still remains as an int because all the effective bits still fit in (previously we mask out NEED_RESCHED bit in preempt_count()). This should make no functional changes for existing preempt_count() users. Enable this for x86_64 along with the introduction of the Kconfig. [boqun: Undo the __preempt_count_{add,sub}() optimization in 32-bit preempt count since it may introduce {over,under}flow] Originally-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Boqun Feng <boqun@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260804161447.84806-11-boqun@kernel.org
2026-08-10sched: Avoid signed comparison of preempt_count() in __cant_migrate()Boqun Feng
Currently preempt_count() is always a non-negative int on all archs (PREEMPT_NEED_RESCHED archs will mask out the MSB when returning preempt_count()), hence the checking in __cant_migrate() is in fact just checking whether preempt_count() is 0 or not. In a future change, we are going to use all the 32 bits of preempt_count(), which would make negative int values possible from preempt_count(). Therefore convert the "> 0" comparison into a zero check to prepare for the future change. No functional changes are intended. Signed-off-by: Boqun Feng <boqun@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260804161447.84806-10-boqun@kernel.org
2026-08-10sched: Remove the unused preempt_offset parameter of __cant_sleep()Boqun Feng
The preempt_offset is always 0 in all the callsites of __cant_sleep(), hence remove it. It also allows us to clear up the code a bit by no longer using a "preempt_count() > .." comparison. Signed-off-by: Boqun Feng <boqun@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260804161447.84806-9-boqun@kernel.org