| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext updates from Tejun Heo:
- Proxy execution can now be enabled together with sched_ext. A BPF
scheduler can opt in to receive mutex-blocked tasks that donate their
scheduling context to the lock owner and place them itself.
- Lazy preemption: a BPF scheduler can preempt or kick a CPU lazily,
deferring the switch to the next return to user space or tick, as the
fair class can.
- Sub-schedulers: a parent now learns which child runs on each CPU it
delegated and when a cap change takes effect. A detaching child loses
its caps before the parent restores the CPUs.
- New kfuncs re-enqueue single tasks from a dispatch queue iteration
and report a cgroup's CPU count and a CID's NUMA node. The CID
topology now includes clusters.
- Hot path optimizations: dispatch queue and task lookups compare keys
inline, remote task moves no longer take a dispatch queue lock twice,
and sub-scheduler hooks cost nothing without sub-schedulers.
- Kfuncs callable from tracing programs are hardened against NMI and
IRQ nesting, which could hard-lock the CPU or pick a wrong idle CPU.
- Built-in dispatch queues no longer allocate per-CPU state that grew
with the square of the CPU count.
- Header syncs with the scx repo, a BTF workaround for pahole 1.32, and
selftest fixes and coverage.
* tag 'sched_ext-for-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (69 commits)
sched_ext: Drop the locks periodically while walking a user DSQ for re-enqueue
sched_ext: scx_qmap: Also re-enqueue stranded tasks with scx_bpf_dsq_reenq_task()
sched_ext: Add scx_bpf_dsq_reenq_task() to re-enqueue single tasks from a DSQ iteration
sched_ext: Factor out the rq lock switch of scx_dsq_move()
sched_ext: Don't queue balance callbacks from a core-sched pick
sched_ext: Don't leave balance callbacks queued across a pick retry
sched_ext: Fix a bypassed task stalling on a cpu its scheduler has no caps on
sched_ext: Fix the running task keeping its protection into bypass
sched_ext: Fix use-after-free of a local reenq node left on an inactive cpu
sched_ext: Keep proxy donors with slice left on the local DSQ
selftests/sched_ext: Report reload_loop attach failures
selftests/sched_ext: Bound exit waits
selftests/sched_ext: Add SCX_ASSERT_ALIVE and adopt it
selftests/sched_ext: Fail loudly when EXIT_KIND is missing from BTF
sched_ext: Add SCX_ECODE_RSN_CGROUP_OFFLINE to user_exit_info.h
selftests/sched_ext: Fix reload_loop error-path use-after-free
selftests/sched_ext: Fix build error in dequeue_remote selftest
sched_ext: scx_qmap: Reset a child's cpuperf targets once its PERF revoke takes effect
sched_ext: Add ops.sub_child_ecaps_updated() to report a child's effective cap changes
sched_ext: scx_qmap: Show actual cid use per participant
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"Kconfig updates:
- Simplify PREEMPT_DYNAMIC (Mark Rutland)
Fair scheduler updates:
- Remove redundant RUN_TO_PARITY check (Elif Topuz)
- Declare augmented callbacks per field, fix
rb_add_augmented_cached() descent (Yiwei Lin, Peter Zijlstra)
- Restart fair hrtick after same-task repicks (Shubhang Kaushik)
- Improve the scheduling latency of short slice tasks (Vincent
Guittot)
Deadline/RT scheduler updates:
- Check start_dl_timer expiry with ktime_before() (Liang Hao)
- irq_work: CPU-hotplug improvements on PREEMPT_RT (Sebastian Andrzej
Siewior)
Energy-aware scheduling updates:
- Implement the steal_governor, introduce preferred CPUs and
steal-driven vCPU backoff (Shrikanth Hegde)
Multi-processor load-balancing updates:
- Enable preferred SMT siblings on NVIDIA Olympus (Andrea Righi)
- Randomize equally shallow idle CPU picks (Christian Loehle)
- Remove redundant core_sched_seq (Hui Su)
Cache-aware scheduling updates:
- Avoid publishing partial LLC utilization (Jemmy Wong)
NUMA-balancing updates:
- Reset NUMA fault locality after scan period update (Eric Kim)
Proxy execution updates:
- Ongoing work to implement sleeping owner handling for proxy
execution (John Stultz, Vasily Gorbik)
- Preparatory steps to make sched_ext compatible with proxy execution
(Andrea Righi)
Scheduler infrastructure updates & fixes:
- Remove the RCU wait from PSI trigger destruction (Shakeel Butt)
- Avoid modifying wait-entry on failed priority registration
(Hui Su)
- Use the new static key API for sched_feat (Hongyan Xia)
- Convert paravirt_steal to new static key APIs (Hongyan Xia)
- cputime: Don't account idle time twice after dyntick-idle
(Stian Halseth)
Scheduler debugging updates:
- Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
(Sechang Lim)
- Fix context analysis errors in non-preferred CPU push (Shrikanth
Hegde)
Misc cleanups and fixes by Christian Loehle, Shubhang Kaushik,
Thomas Huth and Vladimir Zapolskiy"
* tag 'sched-core-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (64 commits)
sched: Replace nr_iowait_cpu() with a bool accessor
Revert "sched/fair: Remove dead code on enqueue_task_fair()"
sched/cache: Avoid publishing partial LLC utilization
sched/cputime: Don't account idle time twice after dyntick-idle
sched/psi: Remove the RCU wait from trigger destruction
sched/eevdf: Remove redundant RUN_TO_PARITY check
sched/wait: Avoid modifying entry on failed priority registration
rbtree_test: Use rb_add_augmented_cached() for the cached augmented test
rbtree: Update augmented data on the way down in rb_add_augmented_cached()
rbtree: Declare augmented callbacks per field with RB_AUG()
rbtree_test: Use rb_add() and rb_add_cached() for the basic tests
smpboot: Don't park the thread if work is pending
irq_work: Flush lazy work CPU down on PREEMPT_RT
irq_work: Update a comment regarding CPU hotplug invocation
sched/topology: Add asymmetric SMT packing override
sched/fair: Honor asymmetric SMT priority in idle selection
sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
sched/core: Fix context analysis errors in non-preferred CPU push
virt/steal_governor: Enable the driver
virt/steal_governor: Implement steal_governor policy loop
...
|
|
reenq_user() drops dsq->lock around each re-enqueue and its rq lock every
SCX_TASK_ITER_BATCH re-enqueues. A walk filtered by SCX_REENQ_MARKED skips
unmarked tasks without dropping either, so a long run of unmarked tasks is
walked in one hold of both locks.
Count the visited tasks instead of the re-enqueued ones and drop both locks
every SCX_TASK_ITER_BATCH visits. An unfiltered walk drops the locks as
often as before.
Suggested-by: Andrea Righi <arighi@nvidia.com>
Link: https://lore.kernel.org/all/atP-LIKpxPhR-Nuc@gpd4/
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
DSQ iteration
scx_bpf_dsq_reenq() re-enqueues a whole DSQ, but a scheduler that loses the
cpus serving a DSQ, such as a sub-scheduler after a cap revoke, needs only
the stranded tasks placed again.
Add scx_bpf_dsq_reenq_task(), which marks a task from a DSQ iteration. When
the iteration ends, the DSQ's deferred re-enqueue walk is scheduled once and
re-enqueues only the marked tasks. The kfunc locks the task's rq and the DSQ
like scx_bpf_dsq_move() and is callable from the same contexts. The mark is
cleared when the task leaves the DSQ, so a task consumed before the walk is
not re-enqueued.
An iteration's DSQ can be destroyed before the iteration ends, so
schedule_dsq_reenq() now skips a destroyed DSQ instead of aborting the
scheduler. scx_bpf_dsq_reenq() racing scx_bpf_destroy_dsq() takes the same
path.
Declare the kfunc for the tools and add a compat probe.
v2: Note in the kerneldoc that another iteration's re-enqueue may take a
marked task early, and that a task on another scheduler returns false
(Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Factor the rq lock switch on entry to scx_dsq_move() and its undo on exit
out into kf_switch_rq_lock() and kf_restore_rq_lock() so that other kfuncs
operating on a task from a DSQ iteration can share them. No functional
change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Pull to receive 680edc7f23ad ("sched_ext: Don't queue balance callbacks from
a core-sched pick") and resolve its conflict with the proxy execution work
here so that for-next merges clean.
Conflicts:
- do_pick_task_scx(): 680edc7f23ad samples sched_core_enabled() into a new
local next to the prev declaration, which 3d6577ea93c9 ("sched_ext: Split
curr|donor references properly") changed to rq->donor. Kept rq->donor and
added the local.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Commit ff0545e0c263 ("sched_ext: Don't leave balance callbacks queued
across a pick retry") queues the deferred work as a balance callback
only when do_pick_task_scx() returns a final pick. However, under core
scheduling, a pick that returns a task isn't final either:
pick_next_task() goes on to pick the SMT siblings, and their dispatch
can release the shared rq lock with the callback still queued,
triggering the rq_pin_lock() warning.
Moreover, dispatch_core_pick() returns early when another dispatch is
in flight on the rq, and do_pick_task_scx() then consumes
%SCX_RQ_BAL_CB_PENDING set by that dispatch while it's still running.
This can be reproduced running scx_qmap -I on SMT siblings with
coresched-tagged workloads that frequently sleep, mixed with untagged
CPU-bound tasks.
Fix this by always deferring the pending work to the irq_work path from
dispatch_core_pick() under core scheduling, for both the local and the
sibling picks, and by leaving %SCX_RQ_BAL_CB_PENDING alone in
do_pick_task_scx(), so that it's handled by the dispatch that set it.
Sample the core scheduling state once at the start of the pick, since
core scheduling can get enabled while a non-core dispatch has the rq lock
released, and the follow-up work must match the dispatch path that ran.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The dispatch path can request deferred work to be run from a balance
callback and pick_task_scx() queues rq->scx.deferred_bal_cb right after
dispatching. However, a balance callback must be consumed before the rq
lock is released and pick_task_scx() is not necessarily the end of the
pick:
- it returns RETRY_TASK when a higher priority class task has been
enqueued while the dispatch had the rq lock released;
- the dl_server pick can return NULL, in which case the lower classes
are picked next.
In both cases the callback stays queued while the core keeps picking,
and the rq lock can be released again, either by prev_balance() or by
another dispatch. For example, with an RT task as prev:
__schedule()
__pick_next_task()
pick_task_scx()
dispatch_pick()
maybe_queue_balance_callback() <- deferred_bal_cb queued
return RETRY_TASK
prev_balance()
balance_rt()
pull_rt_task()
double_lock_balance() <- rq lock released
Any CPU acquiring the rq lock in that window finds a pending balance
callback and triggers the following warning in rq_pin_lock():
WARNING: kernel/sched/sched.h:1919 at rq_lock+0x7f/0xe0, CPU#10: stress-ng-sleep/533
CPU: 10 UID: 0 PID: 533 Comm: stress-ng-sleep Not tainted 7.2.0-rc6-virtme #61 PREEMPT(full)
Sched_ext: qmap (enabled+all)
Call Trace:
<IRQ>
try_to_wake_up+0x2c1/0x5c0
hrtimer_wakeup+0x22/0x30
__hrtimer_run_queues+0x1d5/0x350
hrtimer_interrupt+0x106/0x1e0
__sysvec_apic_timer_interrupt+0x6f/0x210
sysvec_apic_timer_interrupt+0x71/0x90
</IRQ>
This can be reproduced running scx_qmap -I together with a SCHED_FIFO
workload that frequently sleeps, e.g.:
$ vng -v --cpus 16 --memory 4G -- \
'tools/sched_ext/build/bin/scx_qmap -I >/dev/null & \
stress-ng --sleep 4 --sched fifo --sched-prio 1 -t 30 -q'
Fix this by queueing the balance callbacks only when the pick is final.
Otherwise, defer the pending work to the irq_work path, which doesn't
depend on the current rq lock section, and leave the kick sync wait
pending for the next pick.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive 30fb2b1e4d41 ("sched_ext: Fix the running task keeping its
protection into bypass") and 5809f749fbdb ("sched_ext: Fix a bypassed task
stalling on a cpu its scheduler has no caps on") and resolve their conflicts
with the proxy execution work here.
Conflicts:
- scx_bypass(): 30fb2b1e4d41 drops the protection strip in the runnable_list
walk, which never sees the running task, and strips rq->curr ahead of the
walk instead. 3d6577ea93c9 ("sched_ext: Split curr|donor references
properly") moved the walk's strip to task_current_donor(), as protection
belongs to the scheduling context. set_next_task_scx() takes the donor off
the runnable_list, so that strip is dead too. Dropped it, and the new
strip takes rq->donor.
- scx_bypass(), merged without a marker: 5809f749fbdb evicts the running
task when bypass lifts if it can't stay on the cpu, on rq->curr.
3d6577ea93c9 moved the cap revoke eviction of the running task in
reenq_local() to rq->donor. The lift eviction takes rq->donor too, and
both comments in the block say the current donor.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A bypassing sub-scheduler's tasks are run by its nearest ancestor that isn't
bypassing. The ancestor puts them on the local DSQ under its own caps.
Whether such a task may stay on the cpu and whether a cap revoke reenqueues
it are still checked against the caps of the task's own scheduler.
The bypass load balancer moves tasks between cpus without looking at caps,
so a bypassed task can run on a cpu where its own scheduler holds none. Once
its slice runs out, it can't stay. It goes to the ancestor's bypass DSQ
without a kick, and the cpu can go idle with it queued until something else
reschedules the cpu.
Make both checks against the caps of the scheduler running the task. When
bypass lifts, the running task keeps its slice without a cap check, so evict
it then if it can't stay on the cpu.
Fixes: 75a8c8202c91 ("sched_ext: Add reject DSQ for cap-rejected dispatches")
Fixes: 46a85ae6fe5b ("sched_ext: Tie cpu occupancy to SCX_CAP_BASE through the task slice")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Protection ends when the scheduler enters bypass, so that bypass scheduling
can cut every task's slice. scx_bypass() clears it for the queued tasks by
cycling them, and strips the running task's in the same walk over
rq->scx.runnable_list. The running task is never on that list.
set_next_task_scx() takes it off and put_prev_task_scx() puts it back, so
the strip never runs. A protected running task then keeps its slice into
bypass, where the tick's attempts to cut it are refused.
Strip the running task's protection outside the walk.
Fixes: 9cfc6ab34a31 ("sched_ext: Add SCX_TASK_PROTECTED")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A local DSQ reenq is queued on the rq through a per-sched node and run from
deferred work. When the reenq is requested from dispatch, the work is a
balance callback queued once dispatch is done. A cpu going down installs
balance push early in its teardown, and queue_balance_callback() refuses new
callbacks from then on. ops.dispatch() keeps running on that cpu until the
rq goes offline after an RCU grace period, so a reenq it requests in that
window may never run. The IMMED handling in dispatch_one() requests one the
same way.
A reenq requested from elsewhere for a cpu that is already offline goes
through irq_work_queue_on(), which warns, and its work doesn't run until the
cpu comes back.
Either way the node stays on the rq's list. When the scheduler is freed,
scx_sched_free_rcu_work() warns and frees the pcpu with the node still
linked, and the list is walked again once the cpu comes back.
An inactive cpu's local DSQ holds only tasks that are bound to it or are
being moved off, so the reenq is pointless. Ignore reenqs of an inactive
cpu's local DSQ.
The test is lockless. The hotplug thread of a cpu going down runs on that
cpu, and it marks the cpu inactive before it installs balance push. A
dispatch on the cpu runs to completion before that thread can run there, so
a dispatch that saw the cpu active has queued its callback by then. Requests
from outside dispatch run from the wakeup path or through irq_work, which
balance push doesn't affect. The grace period after the cpu goes inactive
waits for any requester that saw it active, so that irq_work is queued while
the cpu is online.
Fixes: 238eba8c210d ("sched_ext: Use schedule_deferred_locked() in schedule_dsq_reenq()")
Fixes: 76edc2761ab8 ("sched_ext: Use irq_work_queue_on() in schedule_deferred()")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- A task that skipped the BPF scheduler on its way back to its CPU's
local queue could stall there until the watchdog fired, as the CPU
had already decided to go idle.
- Use-after-free of a destroyed user dispatch queue by a deferred
re-enqueue request.
- A task could be moved off a dispatch queue while the dispatching CPU
was still running ops.dequeue() on it.
- Bypass mode mishandled a sub-scheduler bypass overlapping an
ancestor's, as across a system suspend. Descendants could leave
bypass mid-teardown and tasks could be stranded on bypass queues.
- The idle CPU selection helper could return a CPU outside the allowed
CPUs of a pinned or migration-disabled task.
* tag 'sched_ext-for-7.3-rc7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs
sched_ext: Rename scx_reenq_wait_dispatching() to scx_wait_dispatching()
sched_ext: idle: Respect task affinity in restricted CPU selection
sched_ext: Fix the bypass dispatch claim for schedulers bypassed by an ancestor
sched_ext: Propagate every bypass depth change to the descendants
sched_ext: Fix use-after-free of a destroyed user DSQ by a deferred reenqueue
sched_ext: Fix stall when a task enqueued with SCX_ENQ_LAST lands back on its CPU's local DSQ
|
|
Commit ee172227d0dc ("sched_ext: Delegate proxy donor admission to BPF
schedulers") makes put_prev_task_scx() pass a retained proxy donor to
ops.enqueue() with SCX_ENQ_BLOCKED.
Some of these puts are only proxy bookkeeping; proxy_resched_idle()
drops the rq's donor reference before switching to idle when:
- find_proxy_task() cannot run the mutex owner yet and retries through
idle (e.g., the owner is on a remote CPU),
- proxy_migrate_task() detaches the donor before migration,
- proxy_deactivate() blocks a task in the chain whose owner cannot run.
In the first case, BPF has already selected a donor with slice left.
Returning it to ops.enqueue() forces BPF to dispatch it again before
proxy resolution can continue. In the other two cases, the donor is
typically deactivated right after ops.enqueue(), undoing any placement
BPF makes.
As also discussed at the sched_ext microconference at Linux Plumbers
2026, these ops.enqueue/dequeue() invocations do not represent any
meaningful scheduling events, so we should avoid triggering them.
A more reasonable semantic is to keep a donor with slice left at the
head of the local DSQ, so the next pick can resolve its owner, or
deactivation can remove it without an unnecessary BPF handoff.
An IMMED donor can stay there only for a bookkeeping put during proxy
resolution. Mark a blocked pick as awaiting resolution with
SCX_RQ_PROXY_PENDING to distinguish that put from a real preemption and
re-insert the donor with SCX_ENQ_IMMED, so that it only needs
SCX_CAP_ENQ_IMMED rather than SCX_CAP_ENQ. A preempted IMMED donor still
returns to BPF. If a higher priority class takes the CPU once proxy
resolution completes, schedule a local reenqueue, so that a parked IMMED
donor returns to BPF instead of lingering on the local DSQ. Update the
SCX_ENQ_IMMED and SCX_ENQ_BLOCKED documentation accordingly.
Moreover, a donor that has run out of slice now follows the regular put
path, so it can receive SCX_ENQ_LAST when the CPU is about to go idle
and the scheduler needs to arrange a follow-up scheduling event. This
does not apply to bookkeeping puts, where the switch to idle is only
temporary. The blocked-donor specific sanity checks go away together
with the special case.
When a retained donor is removed ahead of a scheduler ownership or
policy change, mark the proxy reset with SCX_RQ_PROXY_BLOCKING and skip
reenqueueing the active donor: sched_proxy_block_task() will dequeue it
next. Report it as not runnable to ops.stopping(), as a regular dequeue
of the current donor does. This avoids a transient ops.enqueue/dequeue()
pair and a false ENQ_LAST warning.
With this, the sched_ext core absorbs the transient proxy-bookkeeping
puts and the BPF scheduler only sees a blocked donor when there is an
actual placement decision to make.
Fixes: ee172227d0dc ("sched_ext: Delegate proxy donor admission to BPF schedulers")
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive 2935062be771 ("sched_ext: Wait for SCX_OPSS_DISPATCHING
before moving tasks off DSQs") and resolve its conflicts with the proxy
execution and reject DSQ work here so that for-next merges clean.
Conflicts:
- unlink_dsq_and_switch_rq_lock(): efa7bc7d7195 ("sched_ext: Avoid relocking
DSQ during remote consumption") clears p->scx.dsq in the success branch
and 2935062be771 adds the wait there. Kept both, the wait after the clear,
so that the proxy reject in consume_remote_task() is covered too.
- scx_reenq_reject(): 18f5c699d1d0 ("sched_ext: Rename
scx_reenq_wait_dispatching() to scx_wait_dispatching()") renames the call
in sub.c while cea8b60e4aee ("sched_ext: Move reject DSQ draining into
core") moved the function into ext.c. Dropped the sub.c side and applied
the rename at the new home.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A task can be found on a non-local DSQ while the CPU which dispatched it
is still running ops.dequeue() and hasn't cleared SCX_OPSS_DISPATCHING.
The DSQ consume and move paths don't wait for the dispatch to finish and
can operate on the task concurrently with the dispatcher.
Call scx_wait_dispatching() in scx_move_local_task_to_local_dsq(),
unlink_dsq_and_switch_rq_lock() and move_task_between_dsqs() before they
operate on the task. In scx_move_local_task_to_local_dsq(), wait before
scx_resolve_local_dsq() as it may update p->scx.flags.
Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics")
Fixes: cb86607ada73 ("sched_ext: Don't run ops.dequeue() with a DSQ lock held")
Cc: stable@vger.kernel.org # v7.1+
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Waiting for SCX_OPSS_DISPATCHING to clear isn't specific to the reenqueue
paths. Any path which takes a task off a DSQ can find it while the
dispatch is still in flight, and the next patch calls the helper from the
DSQ consume and move paths. Rename it to scx_wait_dispatching() and
update its comment accordingly.
No functional change.
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pickup dependencies; notably the RB-tree fix.
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
|
|
The restricted select_cpu kfunc path handles single-CPU and
migration-disabled tasks without going through scx_select_cpu_dfl().
It currently uses prev_cpu as the candidate, even though prev_cpu is only a
hint and may no longer be usable after an affinity change. With a
caller-provided @allowed mask, this can also return an idle CPU outside
p->cpus_ptr.
Derive the candidate from task state instead: use the sole CPU in
p->cpus_ptr for single-CPU tasks and task_cpu(p) for migration-disabled
tasks. Validate it against p->cpus_ptr and @allowed before claiming its
idle state.
Since the derived candidate may be on a different NUMA node from
prev_cpu, also preserve SCX_PICK_IDLE_IN_NODE by rejecting cross-node
candidates.
Perform all checks before scx_idle_test_and_clear_cpu(), as a successful
call consumes the CPU's idle bit.
Fixes: 683d2d0faba1 ("sched_ext: idle: Introduce scx_bpf_select_cpu_and()")
Cc: stable@vger.kernel.org
Signed-off-by: Hui Su <sh_def@163.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_bypass() claims bypass dispatch when the scheduler's depth goes from
zero to one and releases it when the depth returns to zero, but propagates
only the depth to the descendants, not the claim. A descendant that starts
bypassing with its ancestor never claims, and one whose own bypass ends
under the ancestor's keeps its claim, as its depth does not return to zero.
A sub-scheduler enable whose bypass ends under a PM suspend thus leaks its
claim: the parent keeps hosting bypass dispatch and the sub-scheduler's next
bypass trips the WARN_ON_ONCE() on the claim. A sub-scheduler disable that
starts under a PM suspend has no claim, so once the resume lifts the root's
bypass its tasks sit on the parent's bypass DSQs with nobody consuming them.
Claim for every descendant whose depth goes from zero to one in the
propagation and release every scheduler in the subtree whose depth returned
to zero once the tasks have moved out of the bypass DSQs. A parent that is
bypassing itself when a descendant claims consumes only its own bypass DSQs,
empty until it leaves bypass and becomes the host.
Fixes: aa2a0a19686c ("sched_ext: Separate bypass dispatch enabling from bypass depth tracking")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_bypass() propagates a depth change to the descendants only when the
scheduler's own depth crosses zero, so an own bypass that starts under an
ancestor's never reaches them. When the ancestor's bypass ends first, the
descendants leave bypass while the scheduler is still bypassing: a
sub-scheduler disabled under a PM suspend has its children re-queue their
tasks through their own BPF schedulers at resume, mid-teardown. If the own
bypass then ends, every descendant is decremented once more than incremented
and dec_bypass_depth() warns, though no current caller gets there.
Propagate on every call so that a descendant's depth is the sum of its own
bypasses and its ancestors'. The early return follows the walk, as a
descendant's depth is at least the scheduler's and nothing changes bypass
state unless the scheduler's own does.
Fixes: c7f0e467a27a ("sched_ext: Prepare bypass mode for hierarchical operation")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Pull to receive 069403dff337 ("sched_ext: Fix use-after-free of a
destroyed user DSQ by a deferred reenqueue") and point its new cancel
sweep in free_dsq_irq_workfn() at dsq->pcpu_user, which 92344fad5b54
("sched_ext: Skip per-CPU data allocation for built-in DSQs") renamed
here, so that for-next builds.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A deferred reenqueue can read a freed user DSQ. The request stays on its
CPU's deferred list until run_deferred() takes it, and the RCU callback that
frees the DSQ assumes that the grace period has flushed it, so exit_dsq()
checks the per-CPU node without the deferred list lock. A grace period waits
for readers in flight, not for a pending run_deferred(). A run_deferred()
which starts after the grace period began can detach the request right
before the callback looks and then read the freed DSQ's id.
Cancel the pending requests before the grace period instead. The irq work
that frees the DSQ unlinks every CPU's request under the per-CPU deferred
list lock, and schedule_dsq_reenq() ignores a new request under the same
lock once the id is invalid. Only a run_deferred() that started before the
grace period can hold a request, and it runs with IRQs disabled, so the
grace period waits for it and nothing touches the DSQ after the callback.
Fixes: 84b1a0ea0b7c ("sched_ext: Implement scx_bpf_dsq_reenq() for user DSQs")
Cc: stable@vger.kernel.org # v7.1+
Reported-and-tested-by: Hui Su <sh_def@163.com>
Link: https://lore.kernel.org/all/20261008024753.4096008-1-sh_def@163.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
its CPU's local DSQ
With SCX_OPS_ENQ_LAST, the last runnable task on a CPU is passed to
ops.enqueue() instead of being kept running. Exiting tasks without
SCX_OPS_ENQ_EXITING, migration disabled tasks without
SCX_OPS_ENQ_MIGRATION_DISABLED and tasks on an offline rq skip ops.enqueue()
and go straight to the local DSQ. This happens in put_prev_task_scx() after
the pick has settled on idle, so nothing reschedules the CPU, which idles
with the task queued until an unrelated wakeup lands on it or the watchdog
fires.
scx_mitosis, which sets SCX_OPS_ENQ_LAST without SCX_OPS_ENQ_EXITING, hit
this during mass cgroup teardown: exiting tasks stuck in exit_mmap() for
over 40 seconds.
A task put on the local DSQ runs, as anywhere else. If the SCX_ENQ_LAST
enqueue left the task on this CPU's local DSQ, set need_resched on the task
being switched in, as proxy_resched_idle() does. resched_curr() would flag
the task being put instead, and __schedule() clears that right after the
pick.
Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class")
Cc: stable@vger.kernel.org # v6.12+
Reported-by: Xiangyu Bu <xbu@meta.com>
Link: https://github.com/sched-ext/scx/pull/3870
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Dan Schatzberg <schatzberg.dan@gmail.com>
|
|
cap changes
A grant or revoke only records the target caps. They take effect on a cid at
its next dispatch, and nothing tells the parent when. Without that the
parent cannot act on a revoke: a child that ran a cid at a low cpuperf
target leaves the target there when PERF is revoked, as the kernel resets
targets only at root enable. Nor can the parent tell when it may schedule on
the cid again.
Add ops.sub_child_ecaps_updated(), delivered to the direct parent right
after the child's own ops.sub_ecaps_updated() with the child's cgroup id and
the same before and after caps, in the same dispatch context. Both
deliveries are suppressed while the child is bypassing and replayed together
afterwards. A disabled child reports nothing: ops.sub_detach() is where the
parent restores what it had delegated.
A sub is now bypassed before it is linked, so that the grants queued for it
from ops.sub_attach() are consumed while it is bypassed and replayed to it
and the parent once it is enabled. Otherwise a sync consumed before the
sub's ops are registered would reach the parent but never the sub.
v2: Drop the disable report with its sleep lock and nested-dispatch gate,
the parent restores from ops.sub_detach() instead.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A parent sched that delegates cpus to sub-scheds has no way of telling how
much each delegatee is using them. A parent that sizes its delegations by
demand cannot measure it, and one that wants to hand a cid from one sub to
another cannot see which sub it would take the cid from.
Add ops.sub_cid_sched_updated(), invoked on a sched when what it sees
running on a cid changes: NONE when no task of its subtree runs there, SELF
for its own task, or a direct child's cgroup id. Changes inside a child's
subtree are not reported.
The op costs one pointer compare per context switch when the sched does not
change, a notification reaches only the scheds that see a change, and the
whole mechanism sits behind a static key that is on only while a loaded
sched implements the op. After a sched's tasks are gone, its disable closes
every rq session still pointing at it, so nothing refers to it once it is
freed.
The op is cid-form only like the other sub ops and runs with the rq lock
held on the context switch path.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- Taking a CPU offline could hang, or stall until the watchdog ejected
the BPF scheduler, when tasks on the dying CPU were still held by the
scheduler or sitting on a user dispatch queue. Re-enqueue them onto
the local queue when the runqueue goes offline so that the CPU pushes
them off like the other sched classes.
- The sequence number guarding against stale dispatches was per
runqueue, so a task re-enqueued on another CPU could get the same
number and a dispatch meant for its earlier instance was applied to
the new one. Use a per-task counter.
- A task dispatched to another CPU's local queue got its ops.dequeue()
only when picked to run and flagged as a core-sched pick. Call it at
insertion like for same-CPU dispatches.
* tag 'sched_ext-for-7.3-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Generate qseq from a per-task counter
selftests/sched_ext: Add a test for ops.dequeue() on remote local DSQ moves
sched_ext: Call ops.dequeue() when a task arrives on a remote local DSQ
sched_ext: Fix CPU hotplug hang when a dying CPU's tasks sit in the BPF scheduler
|
|
A sub-scheduler can leave standing state behind on the cids delegated to it,
cpuperf targets being the obvious case, and the kernel does not track who
wrote it. Restoring it is the parent's job and ops.sub_detach() is the place
for it. Today the parent can't rely on that: the dying sub-scheduler keeps
every cap until it is marked dead after ops.exit(), so its ops.exit() or a
late timer can write after the parent has cleaned up.
Clear every cap of the sub-scheduler before ops.sub_detach(). The pshard
caps go first as a pending sync recomputes ecaps from them, and ecaps are
then zeroed directly. A grant racing the clear is refused under the child's
pshard lock, so nothing hands the caps back.
Fixes: 86094b95efcf ("sched_ext: Add per-shard cap delegation for sub-schedulers")
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
finish_dispatch() uses the qseq embedded in p->scx.ops_state to tell
whether the QUEUED instance of a task it's about to claim is the one
scx_bpf_dsq_insert() saw. qseq is generated from rq->scx.ops_qseq, but
the counters of different rqs are independent, so if a task is dequeued
and re-enqueued on a different rq between scx_bpf_dsq_insert() and
finish_dispatch(), the new QUEUED instance can end up with the same
qseq as the old one:
CPU X CPU Z
----- -----
enqueue p on rq A, qseq = N
ops.dispatch()
scx_bpf_dsq_insert(p)
records qseq N
sched_setaffinity(p)
dequeue p from rq A
enqueue p on rq B, qseq = N
finish_dispatch(p, N)
qseq matches, p is claimed
The claim itself is still atomic so the core stays consistent, but an
insert issued for a previous QUEUED instance gets applied to a new one
which the BPF scheduler has just received through ops.enqueue(). This
breaks the guarantee that dispatches targeting a stale instance are
ignored.
Generate qseq from a per-task counter, p->scx.ops_qseq, instead so that
consecutive QUEUED instances of a task never share a qseq regardless of
which rq they're on. The counter is only updated in
scx_do_enqueue_task() with the task's rq locked, so no additional
synchronization is needed, and it fits in an existing hole in struct
sched_ext_entity on 64bit. Remove the now unused rq->scx.ops_qseq.
Never generate qseq 0. NONE and DISPATCHING don't carry a qseq, so
scx_bpf_dsq_insert() on a task in either state records 0. With a
per-task counter, every task's first QUEUED instance would otherwise get
qseq 0 and could be claimed by such an insert. Wrap the counter where
the QSEQ field wraps so that it can't reach a value that shifts to 0 on
32bit either.
Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class")
Cc: stable@vger.kernel.org # v6.12+
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Hierarchical BPF schedulers may need to know how many CPUs a cgroup can
run on when distributing group-wide weights.
For example, fair's group share calculation scales a group's weight by
the smaller of its estimated runnable task count and the number of CPUs
in its effective cpuset. Without that count, a BPF scheduler can only
use the machine-wide CPU count and may give a group confined to a few
CPUs excessive shares.
A task's effective affinity can include per-task restrictions, so it
cannot reliably represent the group's cpuset. Reading the cpuset
directly from BPF would require CO-RE access to private cgroup and
cpuset structures, including inherited cpuset resolution, making the
approach fragile across kernel versions.
Add scx_bpf_cgroup_nr_cpus(), which returns the same count fair uses
through cpuset_num_cpus(). The count is independent of CPU or CID
numbering, so the kfunc is available to both cpu-form and cid-form
schedulers from any context. Also add a compat wrapper which falls
back to nr_cpu_ids on older kernels.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive cc07fae1de5d ("selftests/sched_ext: Add a test for
ops.dequeue() on remote local DSQ moves") and resolve its conflict with
f73ec6791424 ("sched_ext: Add selftest for blocked donor admission") here
so that for-next merges clean.
Conflicts: kept both auto-test-targets entries in the selftests Makefile,
where each side added one after dequeue_iter.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
When a task in the BPF scheduler's custody is moved to another CPU's
local DSQ, ops.dequeue() is only called once the task is picked for
execution, from set_next_task_scx() with SCX_DEQ_CORE_SCHED_EXEC, even
though no core-sched pick took place. It should instead be called
without flags when the task is inserted into the destination DSQ, as it
is for same-rq dispatches.
move_remote_task_to_local_dsq() sets p->scx.sticky_cpu across the
migration so that deactivate_task() on the source rq doesn't end custody.
Since commit b75aaea24c9f ("sched_ext: Properly mark SCX-internal
migrations via sticky_cpu"), enqueue_task_scx() only clears it after
inserting the task, so task_leave_custody() still sees the migration in
progress and skips the custody exit.
Clear p->scx.sticky_cpu as soon as enqueue_task_scx() has read it, as
was done before that commit. The source side is unaffected.
Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics")
Cc: stable@vger.kernel.org # 7.1.x: 18d62044cda7: sched_ext: Preserve rq tracking across local DSQ dispatch
Cc: stable@vger.kernel.org # 7.1.x
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
from BTF
With clang and pahole 1.32, x86-64 vmlinux BTF lacks
scx_bpf_task_set_lazy_resched() and sched_ext fails to initialize:
sched_ext: Failed to register kfunc sets (-22)
clang describes the lazy argument only after the prologue moves it into a
callee-saved register, which pahole 1.32 rejects as not being the argument
register. Released pahole is affected whatever the eventual fix. Keep the
argument on the stack with barrier_data() so that its location names no
register. OPTIMIZER_HIDE_VAR() would fix x86-64 but break arm64.
Fixes: f8e5a4e3f3be ("sched_ext: Add lazy preemption support")
Link: https://lore.kernel.org/r/20260925213619.2187751-1-yonghong.song@linux.dev
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Proxy execution keeps a mutex-blocked donor runnable so that its
scheduling context can execute the mutex owner.
Define SCX_OPS_ENQ_BLOCKED as the admission contract for retained
donors. Schedulers without the flag block EXT donors normally.
Schedulers with the flag continue to own blocked donors and receive them
through ops.enqueue() with SCX_ENQ_BLOCKED, allowing BPF to choose their
DSQ, CPU and ordering.
After BPF dispatches a donor, proxy execution may move its context to
the rq of the mutex owner. This move is distinct from BPF placement:
proxy_set_task_cpu() changes task_cpu() while preserving the donor's
pre-proxy CPU in wake_cpu as the starting point for normal wakeup
placement. A proxy-migrated donor returns through the full wakeup
activation path when the mutex is released.
Do not carry a retained proxy session across BPF scheduler ownership
changes. Before root activation, parent-to-child takeover, rehome or
punt, fully deactivate a retained donor unconditionally. The incoming
scheduler then starts with clean task state and applies its admission
policy the next time the task blocks.
A retained donor can wake through ttwu_runnable() without another
enqueue_task_scx() call. Use WF_TTWU_RQ to identify this path, where the
wakeup skipped activation and enqueueing. wakeup_preempt_scx() requests
dispatch reconsideration after is_blocked is cleared. A full wakeup
activation has already enqueued the task and does not need another
reschedule.
Blocked-donor enqueueing takes precedence over the exiting and
migration-disabled local-DSQ fallbacks, so that an opted-in scheduler
sees every eligible donor request.
Co-developed-by: John Stultz <jstultz@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_can_stop_tick() currently treats any blocked rq->donor as an active
proxy session. However, rq->donor can temporarily still identify an
outgoing blocked task while the scheduler selects an ordinary execution
context. A dependency update in that window can keep TICK_DEP_BIT_SCHED
set after the context switch.
Track whether proxy resolution actually selected a different execution
context. Keep the tick active while that state is set instead of
deriving it from a potentially stale donor.
When proxy execution ends, clear the state and queue a balance callback
to reevaluate the tick dependency after context_switch() updates
rq->curr. This lets sched_can_stop_tick() observe the complete donor and
execution selection without adding work outside the existing
proxy-execution branch.
Conservatively re-enable the periodic scheduler tick for the duration of
proxy execution. Allowing proxy execution itself to run tickless would
require making the remote NOHZ scheduler tick aware of the split between
rq->curr and rq->donor, which is left as a future improvement.
Perform the tick update from scx_proxy_reenqueue_retry(), which already
runs after proxy resolution, so scheduler core needs no additional hook.
Compile the additional runqueue state and callback out when
CONFIG_NO_HZ_FULL is disabled.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
With proxy execution, the task selected by the scheduler and the task
physically executing can differ. A blocked mutex waiter donates its
scheduling context to the lock owner:
D -----------------> M -------------> O ----------------> T
[donor] blocked on [mutex] owned by [owner] preempted by [task]
\_________________________________^
donates scheduling context
where:
D = blocked donor
M = mutex
O = mutex owner
T = competing runnable task
During a proxy execution switch, D supplies the scheduling class,
priority, and runtime budget, while O supplies the execution context: O
is the task whose code physically executes. T is a competing runnable
task which may preempt the D/O proxy execution.
Consider FAIR and EXT tasks with sched_ext running in partial mode. FAIR
can be replaced with a higher scheduling class such as RT or deadline
without changing the class interaction described here.
An active proxy pair can only have a donor in the same scheduling class
as the owner or in a higher-priority class. If the owner belongs to a
higher-priority class, pick_next_task() selects the owner directly
before considering the donor, so find_proxy_task() is not entered for
that donor. The blocked-on relationship may still exist, but it does
not result in proxy execution.
The relevant combinations are:
1. D is EXT, O is EXT, T is EXT
D can interrupt T according to BPF scheduling policy. O executes
with D's EXT priority and runtime budget, while T waits in EXT.
2. D is EXT, O is EXT, T is FAIR
D is visible to the BPF scheduler, but cannot preempt T because
EXT is below FAIR. Once T stops, BPF can dispatch D and O executes
with D's EXT priority and runtime budget. If T becomes runnable
again, it preempts the D/O proxy execution.
3. D is FAIR, O is EXT, T is EXT
D preempts T as the higher-class scheduling context. O executes
with D's FAIR priority and runtime budget, while T waits in EXT.
D is not visible to the BPF scheduler.
4. D is FAIR, O is EXT, T is FAIR
D competes with T according to its FAIR deadline. When D is
selected, O executes with D's FAIR priority and runtime budget.
D is not visible to the BPF scheduler.
5. D is FAIR, O is FAIR, T is EXT
The D/O proxy execution has a FAIR scheduling context, so T cannot
preempt it from the lower EXT class.
6. D is FAIR, O is FAIR, T is FAIR
O, T, and D all have FAIR scheduling contexts. D remains runnable
as a blocked proxy donor. When CFS selects D, O executes using D's
FAIR scheduling context. When CFS selects O, O executes using its
own FAIR context, and when CFS selects T, T executes normally. D
is not visible to the BPF scheduler.
Thus, sched_ext policy and accounting must generally use rq->donor, the
scheduler-selected task which supplies the scheduling context, rather
than rq->curr, the task whose code physically executes. Without proxy
execution they are the same task.
On nohz_full CPUs, keep the tick running whenever the selected donor is
blocked. A selected blocked donor is necessarily awaiting proxy
execution, so this avoids relying on the transient relationship between
rq->curr and rq->donor during context switches. It also leaves ordinary
sched_ext context switches and non-nohz_full CPUs untouched. Allowing
proxy execution to run tickless remains a future improvement.
Moreover, extend scx_dump_state() to report both contexts. Each CPU
record now includes a donor= line. If an EXT donor differs from
rq->curr, also emit its detailed task record. The existing '*' marker
continues to identify rq->curr, while the donor= line identifies the
otherwise unmarked donor record.
Note that at this point in the series, CONFIG_SCHED_PROXY_EXEC still
depends on !CONFIG_SCHED_CLASS_EXT, so proxy execution and sched_ext
cannot be enabled together. The scheduling changes are therefore
preparatory. A later patch removes this restriction.
Co-developed-by: John Stultz <jstultz@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Without proxy execution, the DSQ lock and holding_cpu handshake ensure
that a task cannot be dequeued or start running during an rq-lock
handoff without clearing holding_cpu.
Proxy execution is an exception: a task can start physically executing
as a lock owner while its scheduling context remains on a DSQ; its
on-CPU or migration-disabled state can therefore change without
clearing holding_cpu.
Recheck these states after acquiring the source rq lock. If the transfer
can no longer proceed, park the task on the source rq's reject DSQ and
reenqueue it through its owning scheduler. This preserves the BPF
scheduler's placement policy and keeps descendant tasks within their
sub-scheduler's cap grants.
Queue a deferred drain for every rejection. If a drain finds a
proxy-rejected task still running or donating, leave it parked and
record a retry request on the rq. scx_proxy_reenqueue_retry() consumes
the request and schedules another drain when proxy state changes.
Without this change and proxy execution enabled, stress-ng --pipeherd
can trigger this race and migrate an active execution context, leading
to sleeping-while-atomic warnings and subsequent lockdep corruption.
This is a preparatory change to support proxy execution with sched_ext.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The reject DSQ is currently specific to sub-scheduler cap failures. Its
storage and initialization depend on CONFIG_EXT_SUB_SCHED, and the drain
path assumes every rejected task was rejected for SCX_TASK_REENQ_CAP.
Other transient placement failures need the same ability to park a task
on its source rq and return it to the owning BPF scheduler. Make the
reject DSQ unconditional and carry the reenqueue reason directly in
p->scx.flags from the rejection site.
A reenqueue reason remains valid while ops.enqueue() runs so that BPF
can inspect it. Clear it immediately after the callback returns and
before resolving a direct dispatch. Paths which bypass ops.enqueue()
clear the reason before the kernel-selected placement. A rejection from
the new placement can therefore install its reason into a clear field.
Route all paths through a common cleanup point to enforce this ordering.
This is a preparatory change to support proxy execution with sched_ext.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The reject DSQ drain will be generalized for placement failures outside
sub-scheduler capability rejection. Move the drain into the core
sched_ext implementation first so the subsequent behavioral changes can
be reviewed separately. No functional change.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
With proxy execution, pick_next_task() can select a blocked task as the
scheduling context before find_proxy_task() resolves the execution
context.
From the BPF scheduler perspective, that donor is running while its
scheduling context drives the lock owner; ops.tick() and other
accounting must therefore remain enclosed by a matching
ops.running()/ops.stopping() session.
In this scenario, the session boundaries do not always match physical
task switches. Keep the "running" session open when the same donor
continues on the same CPU. When proxy execution migrates a donor's
scheduling context to another CPU, end its running session on the source
CPU and start a new session on the destination CPU only after proxy
resolution succeeds. This prevents a failed resolution from exposing a
provisional ops.running() event.
Track these sessions with a new SCX_TASK_RUN_TRACKED flag. The explicit
running-state tracking is also required by later donor-based accounting:
it prevents an EXT owner executing for a non-EXT donor from being
treated as the active EXT scheduling context when it is dequeued.
This is a preparatory change for enabling proxy execution together with
sched_ext.
Acked-by: John Stultz <jstultz@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Proxy execution retains mutex-blocked donors on the runqueue so their
scheduling context can execute a lock owner. sched_ext cannot safely
inherit such donors unless the BPF scheduler explicitly participates in
their admission and ordering.
Make sched_ext reject retained donors by default. Implement
scx_allow_proxy_exec() to force blocked EXT tasks through the regular
block path in schedule().
A donor retained by another scheduling class may already be queued when
sched_ext is enabled or when a task switches to SCHED_EXT. Fully block
the donor before sched_change_begin() records its queued state. This
prevents sched_ext from importing a proxy session which the BPF
scheduler did not admit, while leaving ordinary FAIR, RT and DL class
transitions untouched.
This is a preparatory change to support proxy execution with sched_ext.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into for-7.4
Pull to receive:
313b652837d0 ("sched/core: Drop mutex locks before proxy rescheduling")
8f8c0417e973 ("sched/core: Dequeue waking proxy donors before reset")
a49653d0abeb ("sched/core: Mark wakeups completed through ttwu_runnable()")
57c75e3ae38c ("sched: Add helper to block retained proxy donors")
be100c77178e ("sched: Add sched_ext hooks for proxy execution")
for the upcoming sched_ext side of proxy execution support.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The sched_ext CID topology gives each core, LLC and NUMA node a
contiguous CID range. Without cluster entries, schedulers cannot obtain
their CID range or global index.
Walk cores by cache-sharing cluster and add cluster_cid and cluster_idx
to scx_cid_topo. Treat the level inclusively: a core without a wider
cluster forms one of its own, while an LLC-wide cache-sharing group
forms one cluster. This gives every cluster a contiguous CID range and a
dense index.
The walk now takes cores in cluster order rather than LLC order. Shards
are still cut on core boundaries, so a shard doesn't need to contain a
whole cluster; only core, cluster, LLC and node are nested, in that
order.
The level follows whatever the arch reports through
topology_cluster_cpumask(), which on x86 is the L2 sharing mask and is
populated independently of CONFIG_SCHED_CLUSTER.
Tested on a 13th Gen Intel Core i7-13800H: six SMT P-cores occupy CPUs
0-11, eight E-cores occupy CPUs 12-19 in two four-core L2 modules, and
all 20 CPUs share one LLC; the P-core pairs form two-CID clusters, the
E-core modules form four-CID clusters, and cluster_idx is dense across
all eight clusters.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scheduler
A CPU going down has to empty its own rq. Its hotplug thread waits in
sched_cpu_wait_empty() until nothing else is left, and the only thing that
wakes it is balance_push(), which runs from __schedule() on the dying CPU
and pushes off the migratable tasks the CPU picks.
sched_ext breaks this. A task sitting on a user DSQ or held by the BPF
scheduler is still counted on its rq, but an inactive CPU no longer calls
ops.dispatch() and can't pull the task back. The dying CPU goes idle with
the task still counted, and one of two things happens:
- Another CPU consumes the task. The rq empties without a __schedule() on
the dying CPU, so the hotplug thread is never woken and cpu_down() hangs
holding cpu_hotplug_lock.
- The task is affine only to the dying CPU. Nothing can move it, and the
offline stalls until the watchdog ejects the BPF scheduler.
When the rq goes offline, re-enqueue every task on it that isn't already on
the local DSQ. An enqueue on an offline rq lands on the local DSQ, so the
dying CPU itself picks the tasks and balance_push() pushes them off, the
same as for the other sched classes. From then on no sched_ext path on
another CPU can pull a task off the rq.
ops.dispatch() currently stops as soon as the CPU goes inactive, and CPU
hotplug then waits for an RCU grace period before taking the rq offline. A
BPF-held task affine only to the dying CPU and preempted inside an RCU
read-side critical section would block that grace period, and the rq would
never go offline. Test SCX_RQ_ONLINE directly so that ops.dispatch() keeps
running until the rq goes offline. Only the dying CPU's own hotplug thread
clears the flag during teardown, and task_can_run_on_remote_rq() still keeps
other rqs' tasks off the CPU.
Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class")
Fixes: 991ef53a4832 ("sched_ext: Make scx_rq_online() also test cpu_active() in addition to SCX_RQ_ONLINE")
Cc: stable@vger.kernel.org # v6.12+
Reported-by: Alap Mohan <mohaalap@meta.com>
Reported-by: Joonwoo Park <joonwoo@meta.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Proxy execution splits the scheduling context (the donor) from the
execution context (the lock owner). sched_ext needs to observe that
split at three points in __schedule():
- whether a blocked EXT task can be retained on the runqueue as a
donor,
- when a donor's scheduling context starts driving a lock owner,
- after proxy resolution, when deferred reenqueue work that was blocked
by an active proxy relationship can be retried.
Introduce scx_allow_proxy_exec(), scx_proxy_donor_start() and
scx_proxy_reenqueue_retry(), and add their call sites in __schedule().
The implementations are empty here and are filled in by the sched_ext
changes that follow, so that all the sched core changes needed by proxy
execution stay together in the preparatory patches.
SCHED_PROXY_EXEC still depends on !SCHED_CLASS_EXT, so the new hooks are
inert: they are compiled out with CONFIG_SCHED_CLASS_EXT=n and
unreachable otherwise.
This is a preparatory change to support proxy execution with sched_ext.
No functional change.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260922165445.943315-6-arighi@nvidia.com
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fix from Tejun Heo:
- The CPU topology helper for BPF schedulers took no buffer size, so
its structure couldn't grow without breaking schedulers built against
the older layout. Add a size argument.
* tag 'sched_ext-for-7.3-rc4-fixes-2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Add a size argument to scx_bpf_cid_topo() so struct scx_cid_topo can grow
|
|
Pull to receive 94480606a677 ("sched_ext: Add a size argument to
scx_bpf_cid_topo() so struct scx_cid_topo can grow") for the upcoming
cluster level.
Conflicts: kept both declarations where 5cadf7545cc6 ("sched_ext: Add a
CID NUMA node lookup kfunc") added one next to the changed
scx_bpf_cid_topo() line, and pointed 29871903f3a3 ("sched_ext: Don't
deliver duplicate ops.cgroup_set_idle() for same value") at the renamed
tg->scx.sched_idle.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
can grow
scx_bpf_cid_topo() copies struct scx_cid_topo into a buffer the BPF program
sized from its own vmlinux.h while the verifier sizes the write from the
running kernel's BTF. The struct may grow and each growth then breaks every
scheduler built against the older layout, rejected at load or written past
its buffer. This is the usual hole for a struct handed to BPF, closed
elsewhere with a size argument, and it was missed here.
Take the buffer size, copy the smaller of it and the kernel's struct and set
the rest to -1. Accesses to the copy are CO-RE relocated, so the struct can
grow by appending fields, which its comment now states. The kfunc changes in
place: the cid interface is still being finalized and no released scheduler
uses the current form.
Fixes: e9b55af47edf ("sched_ext: Add topological CPU IDs (cids)")
Cc: stable@vger.kernel.org # v7.2+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
CID-form schedulers may need the physical NUMA node for a CID, for
example to allocate arena memory on that node.
The CID topology's node index is not a physical NUMA node ID, and
scx_bpf_cpu_node() is restricted to CPU-form schedulers.
Add scx_bpf_cid_node() to return the physical NUMA node for a CID and
provide a compatibility helper that returns NUMA_NO_NODE on kernels
without this kfunc.
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Fair hrticks are one-shot timers. A same-task repick after hrtick expiry
skips set_next_task_fair() and leaves no hrtick armed for the next fair
preemption point:
hrtick()
-> task_tick_fair(..., queued=1)
-> entity_tick()
-> resched_curr()
-> schedule()
-> pick_task_fair() selects current
-> put_prev_set_next_task(rq, prev, next)
-> next == prev
-> return
Introduce SNT_REPICK for the next == prev path:
put_prev_set_next_task(rq, prev, next)
-> next == prev
-> next->sched_class->set_next_task(rq, next, SNT_REPICK)
-> set_next_task_fair()
-> hrtick_start_fair()
set_next_task_fair() skips task-transition work for SNT_REPICK and
restarts the fair hrtick. hrtick_start() records the delay during
schedule(), and hrtick_schedule_exit() rearms the timer.
Do not restart the SCHED_DEADLINE hrtick for SNT_REPICK. Same-task
repicks skip put_prev_task() for both fair and SCHED_DEADLINE, but
pick_task_fair() refreshes the selected fair entity before selecting it
again. hrtick_start_fair() reads that refreshed entity, including with
group scheduling.
The SCHED_DEADLINE picker has no corresponding current-runtime update.
Restarting its hrtick on SNT_REPICK could therefore use stale
dl_se->runtime and delay the next runtime enforcement point.
Suggested-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Shubhang Kaushik (Ampere) <sh@gentwo.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Link: https://patch.msgid.link/20260917-sched-fair-hrtick-restart-v4-1-4dd1414da81a@gentwo.org
|
