| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
"This is a big pull for fixes, but much of it was in the works before
the merge window.
Most of these fixes are in sub-scheduler support, which nests BPF
schedulers under a parent that hands them CPUs. It was heavily
reworked over recent cycles and is now functionally complete. Porting
real schedulers on it drove the latest additions and keeps finding
corner cases.
For example, once CPUs are shared and can be taken back, the kernel
must keep every task moving when its scheduler can't run it or is
failing, and those fallbacks interact subtly with each other and with
other new core features.
- The fallback paths meant to keep a task moving when its scheduler
can't run it or is failing could themselves leave it stuck
- The stall watchdog could blame a scheduler for time it wasn't
responsible for, even taking down all of sched_ext
- Sub-schedulers were told late, or not at all, which CPUs they could
use
- Deferred callbacks could stay queued or be dropped under core
scheduling and proxy execution, with a sched/core fix to the
duplicate check
- Hardening: tracing programs can no longer call kfuncs that could
deadlock the CPU"
* tag 'sched_ext-for-7.4-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (34 commits)
sched_ext: Reject scx_bpf_sub_grant/revoke() in tracing programs
sched_ext: Fix the rescue overload pick ejecting the wrong sub under bypass
sched_ext: Fix use-after-free of a NONE rescuee leaving its sched while running
sched_ext: Fix a lone IMMED task waiting behind a kept prev
sched_ext: Fix queued bypassed tasks stalling when the host loses SCX_CAP_BASE
sched_ext: Let the bypass load balancer move tasks homed on other cpus
sched_ext: Factor out bypass_lb_move() and insert with the donor DSQ unlocked
sched_ext: Fix the bypass load balancer stranding tasks on cpus where the host lacks SCX_CAP_BASE
sched_ext: Fix bypassed tasks stalling when the host lacks SCX_CAP_BASE
sched_ext: Expire prev's slice in dispatch_one() under bypass
sched_ext: Fix a cpu idling after put_prev_task_scx() queues its task on the local DSQ
sched_ext: Refuse extending a kernel slice
sched_ext: Mark the rescue grant as a kernel slice with SCX_TASK_KERNEL_SLICE
sched_ext: Fix a cpu idling with a runnable rescuee
sched_ext: Restore a rescuee's slice after ops.stopping()
sched_ext: Fix rescue served time disagreeing with the slice under proxy execution
sched_ext: Move refill_task_slice_dfl() below apply_slice_vtime()
sched_ext: Fix bypassed tasks looping on a host without SCX_CAP_ENQ
sched_ext: Fix bypassed tasks stranded by the reenqueue limit
sched_ext: Clear holding_cpu where the claim is confirmed
...
|
|
A tracing program can call scx_bpf_sub_grant() and scx_bpf_sub_revoke()
through scx_kfunc_ids_any, which is registered on the tracing kfunc hook.
Before returning, both invoke ops.sub_caps_updated() of the sub-schedulers
whose caps changed, so those ops run in the tracing program's context. No
SCX op expects to run from an arbitrary tracing context.
With an rq lock held, this leads to the same deadlock that b07a7450f6ef
("sched_ext: Reject rq-lock-taking kfuncs in tracing programs") fixed for
direct calls:
1. A tp_btf/sched_switch program fires from __schedule() with the rq lock
held. scx_locked_rq() only tracks rq locks held across SCX callbacks, so
it returns NULL there.
2. The program calls scx_bpf_sub_grant() on a child, which invokes the
child's ops.sub_caps_updated() before returning.
3. That op calls scx_bpf_cidperf_set() on this cpu, which sees no locked rq
and takes the rq lock the cpu already holds.
Add both kfuncs to scx_kfunc_ids_no_tracing and rewrite the set's comment to
cover both reasons for membership: taking an rq lock and running SCX ops.
Fixes: 5f2a9a4c2e6c ("sched_ext: Add coalescing sub_caps_updated() notifier for sub-schedulers")
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Since an earlier patch sends a bypassed task to rescue when its host lacks
SCX_CAP_BASE, a task scx_fail_parent() punted to its bypassing parent at
NONE can be a rescuee.
When a cgroup migration into a root-owned cgroup re-homes it while running,
nothing resets it: the DEQUEUE_SAVE dequeue and put_prev_task_scx() keep a
running task's slice, and __scx_disable_and_exit_task() skips a NONE task.
rq->scx.rescue.curr keeps pointing at the task, now on the root. An RT task,
for example, can keep the task from serving its grant until the last sub is
freed. From then on scx_rescuee() returns NULL and the rescue never ends.
scx_rescue_timerfn() keeps accessing the task after it is freed.
Move the reset out of scx_disable_task() into scx_reset_task_fields() and
call it on every path that takes a task out of its sched's control, the exit
of a NONE task included.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
While dispatch is in progress, rq_is_open() returns true and dsq_inc_nr()
schedules no reenqueue for an IMMED task inserted on an empty local DSQ. The
check at the end of dispatch_one() schedules the reenqueue only when more
than one task is queued, so a dispatch that keeps prev leaves a lone IMMED
task behind:
1. a parent's ops.dispatch() sub-dispatches into a child, which consumes a
task with SCX_ENQ_IMMED onto the empty local DSQ
2. the parent extends prev's slice and dispatch keeps prev
3. the local DSQ holds one task, so no reenqueue is scheduled
The IMMED task waits on the local DSQ behind prev.
Schedule the reenqueue when dispatch keeps prev and an IMMED task is queued,
as even a lone IMMED task can't run right away behind prev.
v2: The description and dispatch_one()'s comment are reworded.
Fixes: 98d709cba319 ("sched_ext: Implement SCX_ENQ_IMMED")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A task already queued on a bypass DSQ stalls when the host loses
SCX_CAP_BASE on that cpu, as the cpu no longer consumes the DSQ. Sending a
bypassed task to rescue at enqueue time covers only a task enqueued after
the loss, and the balancer no longer donates to a capless cpu.
Have the bypass load balancer drain such a DSQ: a cpu where the host lacks
BASE is a donor with a target of zero, and the donees take its tasks under
their ordinary fill.
A drained task no donee can take is reenqueued at its home: to rescue when
the host lacks BASE there and rescue is enabled, and to the home's bypass
DSQ otherwise.
A balancing donor leaves a task that can't move to its donee where it is.
Reenqueueing it at home would put a task homed on the donor cpu at the tail
of its own DSQ, and a blocked task that proxy execution migrated there,
which every remote cpu refuses, would go to the tail on every pass.
Without rescue, a task homed on the drained cpu goes back onto the drained
DSQ, once per pass, and waits for the host to regain BASE, the bypass to
end, or the stall watchdog to eject the host. Its reenqueues are not counted
against the reenqueue limit, as its scheduler is bypassing.
A BASE loss kicks the balancer so that the drain doesn't wait for the next
interval. With a zero interval the kick is the only pass, and a task that
lands on the drained DSQ after it waits the same way.
v2: The description drops an in-flight affinity change from the causes of a
refused balancer move.
v3:
- The description says that a stuck task's reenqueues are not counted
against the reenqueue limit (sashiko).
- The description names the stall watchdog as the way out for a task stuck
without rescue, and the tasks a zero interval leaves on a drained DSQ.
- bypass_lb_reenq_home()'s kerneldoc and the description give the placement
with rescue disabled and for a task that must run at home, and the other
two comments on that placement point to the kerneldoc.
v4:
- Skip a task the donee would refuse on a balancing donor instead of
reenqueueing it at home (Andrea).
- bypass_lb_reenq_home()'s kerneldoc is corrected on a migration-disabled
task homed on a donee and on the local DSQ admission.
- The description is condensed.
Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The following patch uses the bypass load balancer to drain a cpu's bypass
DSQ when the host loses SCX_CAP_BASE there. The host no longer consumes that
DSQ, so the drain has to move every task on it. The balancer skips the ones
homed on another cpu.
An earlier pass moved such a task in from its home cpu's DSQ, and the donor
cpu has not consumed it yet. A task's DSQ membership may only change under
the task's rq lock, which the balancer does not hold.
Take such a task off the donor DSQ with unlink_dsq_and_switch_rq_lock(), the
handshake a remote consumer uses. It drops the DSQ lock and switches to the
task's rq lock. A dequeue that gets in between takes the task, and the
balancer moves on. The insert then runs under the task's home rq lock, as it
already does for a donor-homed task.
A task whose donee is its own cpu can always run there.
task_can_run_on_remote_rq() warns when asked about a task's own cpu, so the
balancer skips the test for such a task.
The walk holds the donor rq lock, which keeps the donor cpu's dispatch and
its enqueuers away from the donor DSQ. A move of a task homed elsewhere
switches to the task's rq lock, and both can reach the DSQ meanwhile. The
parked cursor keeps the walk's place through a consume, and the sequence
bound keeps a task enqueued meanwhile out of the pass.
v2: The description says "the following patch", not "patches", uses the
balancer to drain a DSQ.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Prepare the bypass load balancer for moving tasks homed on other cpus. Such
a move takes the task's own rq lock through the remote consumer's handshake,
which drops the donor DSQ lock on the way, so its insert runs with the donor
DSQ unlocked and under the task's rq lock. The balancer's own move holds the
donor DSQ lock throughout, with the donee DSQ lock nested inside.
Switch the donor-homed move to the same scheme so that one body can serve
both: factor it out of bypass_lb_cpu() into bypass_lb_move(), which takes
the task off the donor DSQ and inserts with the DSQ unlocked. A task that
can't run on the donee goes back to the head of the donor DSQ instead of
being skipped in place.
The walk parks its cursor behind the task before each move and resumes from
it afterwards. It is bounded to the tasks queued when it began, by their DSQ
sequence numbers, so a task it put back is not visited again in the same
pass.
Before, the walk held both the donor rq lock and the donor DSQ lock across
every move. The donor DSQ can be very long and contended, a walk competing
with the cpu's dispatch and the enqueues for the DSQ lock would crawl, and
the walk is what relieves that contention by moving tasks off the DSQ. The
walk now cycles the DSQ lock around each insert but still holds the rq lock
throughout, and that alone keeps dispatch and enqueues away. The DSQ lock is
uncontended when retaken, so cycling it is cheap.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
host lacks SCX_CAP_BASE
The bypass load balancer spreads a host's bypassed tasks over the cpus of a
node by moving them between its per-cpu bypass DSQs. It counts every cpu of
the node as capacity and takes any cpu below the target as a donee. A cpu's
dispatch only descends into schedulers holding SCX_CAP_BASE on it, so a
bypass DSQ on a cpu where the host lacks BASE is never consumed. A task
moved there sits until the host regains the cap or the bypass window ends.
Count only cpus where the host holds BASE as capacity and pick donees from
them. The target rises to spread the node's tasks over the cpus that can run
them, and a capless cpu receives nothing. A capless cpu with tasks is still
an ordinary donor, offloaded down to the donor target when its queue grows
past the threshold; draining it is left to the following patches.
Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A bypassed task stalls on a cpu where its host lacks SCX_CAP_BASE. The host,
the nearest ancestor that is not bypassing, runs a bypassing sub-scheduler's
tasks through its per-cpu bypass DSQs. A cpu's dispatch only descends into
schedulers holding BASE on it, so the host's bypass DSQ on such a cpu is
never consumed until the host regains the cap.
Resolve a bypassed task against its host at enqueue time. If the host lacks
BASE on the cpu, send the task to rescue on that cpu, which runs it at
bounded bandwidth without the host's dispatch having to reach it.
The enqueue does not look above the host for an ancestor that holds BASE.
That ancestor would run the task on cpus it never granted the host.
With rescue disabled the task goes to the host's bypass DSQ as before and
stalls there until the host regains BASE or the bypass ends, or until the
stall watchdog ejects the host, which it blames for a task stuck on its DSQ.
A task that must run on the cpu, migration-disabled or with a migration
pending, is admitted to the local DSQ instead, as any insert of such a task
is.
Bypass also used to end a rescue, sending a preempted rescuee to the bypass
DSQ with every other preempted task. Now that rescue is how a bypassed task
runs while its host lacks BASE, keep a preempted rescuee on the local DSQ
under bypass too. dispatch_one() also keeps such a rescuee running, as an
earlier patch's bypass expiry leaves a kernel slice alone. The grant stays
bounded by served time.
A task scx_fail_parent() punted to its bypassing parent at NONE can be a
rescuee too. switched_from_scx() ends its slice as it does for an enabled
task, so that a NONE rescuee leaving sched_ext while running doesn't leave
rq->scx.rescue.curr pointing at it.
While at it, the rescue test in __scx_resolve_local_dsq() goes through the
new scx_rescue_enabled() helper.
v2:
- The description notes that dispatch_one() keeps a bypassed rescuee too,
through the new patch before this one (sashiko).
- scx_rescue_keep()'s kerneldoc drops dispatch_one() declining a rescuee
with slice left, which the new patch removes.
v3:
- With rescue disabled, a task that must run on the cpu is admitted to the
local DSQ instead of waiting on the host's bypass DSQ.
- The description names the stall watchdog as the way out for a task stuck
without rescue.
v4:
- switched_from_scx() ends the slice of a NONE task, which can now be a
rescuee (sashiko).
- scx_punt_task()'s kerneldoc no longer says that switched_from_scx() skips
a NONE task.
- scx_bypass()'s kerneldoc and the ops.sub_ecaps_updated() doc say where a
bypassed task can be besides the bypass DSQs.
Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The next patch keeps a bypassed rescuee on the local DSQ and needs
dispatch_one() to keep it running too. dispatch_one()'s keep test refuses
any prev whose scheduler is bypassing, so another local task could run ahead
of the rescuee, even an escalated SCX_TASK_PROTECTED one. The test exists
because a cpu-form scheduler can write p->scx.slice directly from BPF, so
the slice can't be trusted under bypass.
The refusal doesn't hold anyway. Since 025b1bd41965 ("sched_ext: Implement
hierarchical bypass mode"), the host's ops.dispatch() runs while prev's
scheduler bypasses, and the post-dispatch test in scx_dispatch_sched() keeps
a queued prev with slice left without checking bypass. A task dispatched in
between waits behind prev until the next bypass tick zeroes the slice, at
most a tick.
Expire the slice before the keep test instead when prev's scheduler is
bypassing, as the bypass tick would. Neither test keeps prev on a zeroed
slice, so prev rotates. A kernel slice survives the expiry and keeps prev.
Only a sub-scheduler's task can hold one, and sub-schedulers are cid-form,
which rejects direct slice writes, so it can be trusted.
Drop the stale pointer to scx_disable_workfn(), update scx_bypass()'s
kerneldoc, and add a TODO noting that the bypass slice handling can be
cleaned up once the cpu form is deprecated and the kfuncs refuse slice
raises from a bypassing scheduler.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
local DSQ
put_prev_task_scx() enqueues a still-runnable task when the pick has settled
on idle. The insert can land the task on the cpu's own local DSQ, and then
the cpu must not go idle over it. 7a9df7667963 ("sched_ext: Fix stall when a
task enqueued with SCX_ENQ_LAST lands back on its CPU's local DSQ")
reschedules the idle task in that case, but only after the SCX_ENQ_LAST
insert.
The plain insert, taken when the task can't stay on the cpu, can land on the
local DSQ too: rescue can admit a task there when its scheduler lacks
SCX_CAP_BASE on the cpu. Without the resched the cpu idles with the task
queued until an unrelated wakeup or the rescue timer's escalation.
Reschedule after either insert. In the following patches, the plain insert
becomes the common path for bypassed tasks under rescue.
v2: The description's forward reference to the plain insert becoming the
common path no longer names the following patch.
Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A rescuee can keep running after its grant is served, on a cpu its scheduler
has lost. The write rules deny a slice extension only where the scheduler
lacks SCX_CAP_BASE on the cpu, and a scheduler holding BASE but not ENQ can
have a task under rescue there:
1. the scheduler extends the rescuee's slice
2. BASE is revoked. The cap-revoke eviction skips the rescuee.
3. the grant is served and scx_rescue_charge() ends the rescue
4. the task runs out the extension on the cpu its scheduler lost, outside
the rescue's bandwidth
A scheduler shortens or zeroes a rescuee's slice to preempt it, and
scx_rescue_keep() resumes the grant afterwards. An extension serves no such
purpose.
Refuse extending a kernel slice in both scheduler write paths, the direct
write in scx_bpf_task_set_slice() and the out-of-band application, counted
in SCX_EV_SLICE_DENIED like an extension without BASE. Shortening stays
allowed.
scx_rescue_keep()'s restore of the remainder raises the slice from zero.
Like the kernel's other slice writes, it goes through scx_set_task_slice()
and not the scheduler write paths.
v2:
- scx_rescue_admit()'s kerneldoc no longer mentions a served-time snapshot.
- SCX_EV_SLICE_DENIED's comment lists every refusal it counts.
- scx_bpf_task_set_slice()'s kerneldoc says that shortening is refused on a
protected slice.
Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
In the following patches, bypassed tasks run under rescue when their host
lacks SCX_CAP_BASE on the cpu. Under bypass the kernel zeroes the running
task's slice at every tick, as it does not trust the scheduler's slices.
A rescue grant has to survive that, or it is served a tick at a time with
the rescuee sent to the tail of the local DSQ each time. The generic slice
handling can't tell a kernel grant from a scheduler's slice.
Mark the grant as a kernel slice with SCX_TASK_KERNEL_SLICE, set at rescue
admission and cleared when the slice ends. The generic slice handling leaves
it alone. The default refill does nothing for it, and the three zeroings go
through the new expire_task_slice(), which skips it:
- the bypass tick's zeroing of the running task
- the eviction of a running task that can't stay on the cpu when bypass
lifts
- the cap-revoke eviction of a running task
Schedulers keep their control. Preemption and scx_bpf_task_set_slice() still
apply, and the rescue stays bounded by served time.
The sites that recognized the rescuee by identity, the reenqueue scan and
put_prev_task_scx()'s reinsert, test the flag instead.
scx_task_has_kernel_slice() hides the test behind the scx_has_subs() static
key, as only a sub-scheduler's task can carry the flag. The flag's entries
join the enum tables in tools/sched_ext.
No kernel slice reaches a zeroing until bypassed tasks run under rescue. The
cap-revoke eviction skipped the rescuee already, and bypass entry ends an
open rescue, so the bypass tick and the eviction at bypass lift see none.
The refill is the one behavior change. The pick refills a zero slice only
for a task that can stay on the cpu, so it concerns a rescuee whose
scheduler holds BASE there. A scheduler may zero such a queued rescuee's
slice, and the next pick used to refill it to the default.
Now the rescuee runs on the zero slice until the next tick. There
put_prev_task_scx() restores the remainder when another task is runnable.
When nothing else is runnable, dispatch_one() keeps the rescuee as the last
task. The pick then ends its zero slice, which ends the rescue, and refills
the default. A scheduler with SCX_OPS_ENQ_LAST skips the keep-last test.
scx_rescue_keep() restores the remainder there instead and the rescue goes
on.
v2:
- scx_rescue_admit()'s kerneldoc no longer mentions a served-time snapshot.
- The description covers a scheduler with SCX_OPS_ENQ_LAST, whose rescuee
dispatch_one() keeps on the remainder.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A scheduler preempts a running rescuee like any other task by zeroing its
slice. The rescue grant survives the preemption. put_prev_task_scx()
restores the unserved remainder through scx_rescue_keep() and queues the
rescuee at the tail of the local DSQ, behind whatever the scheduler wanted
to run first.
That breaks down when the scheduler zeroes the slice but dispatches nothing.
dispatch_one() keeps the previous task running only with slice left or, when
nothing else is runnable, with its scheduler holding SCX_CAP_BASE on the cpu
and not taking the last task through SCX_OPS_ENQ_LAST. A rescuee fails both
once its slice is zeroed if its scheduler lacks BASE there or takes the last
task:
1. ops.tick() or scx_bpf_task_set_slice() zeroes the rescuee's slice
2. the pick finds nothing else to run and settles on idle
3. put_prev_task_scx() restores the remainder and queues the rescuee on the
local DSQ
4. nothing reschedules the cpu, as the pick has settled and __schedule()
clears need_resched right after it
The cpu idles with the rescuee queued until an unrelated wakeup or the
rescue timer's escalation.
Restore the remainder from dispatch_one() too. When nothing else is
runnable, its last keep now keeps the rescuee running on the remainder, so
the pick never settles on idle with a rescuee runnable. put_prev_task_scx()
keeps handling a displaced rescuee, where another task won the pick.
A rescuee that passes the existing keep-last test is kept there instead and
continues as an ordinary task, its rescue ended with the zeroed slice. That
is fine as its scheduler holds SCX_CAP_BASE on the cpu.
The new keep also covers bypassed tasks run under rescue in the following
patches.
v2: The dispatch_one() comment and description now say that a rescuee whose
scheduler holds SCX_CAP_BASE on the cpu passes the keep-last test and
continues as an ordinary task (sashiko).
Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
put_prev_task_scx() restores a preempted rescuee's unserved slice through
scx_rescue_keep() before it calls ops.stopping(). A rescuee whose slice the
scheduler zeroes there is left to the reinsert instead:
1. ops.stopping() zeroes the rescuee's slice
2. the reinsert condition special-cases the rescuee and keeps it on the
local DSQ with a zero slice
3. the pick's default refill supplies a slice, or the task runs on a zero
slice until the tick when its scheduler lacks BASE
Move the keep after the stopping call so that scx_rescue_keep() restores the
remainder whether a preemption or ops.stopping() zeroed the slice, and drop
the special case from the reinsert condition as a rescuee reaching it always
has a slice. A rescuee whose slice ops.stopping() zeroed now goes to the
tail of the local DSQ with its remainder restored, behind what the scheduler
wanted to run first, instead of staying at the head on a zero slice.
The slice end in the same block moves with it, which changes nothing for a
protected task. Its slice is the unserved remainder, and scx_rescue_charge()
ends the protection in the same update_curr_scx() call that zeroes the
slice, so the block never sees a protected task in either position.
v2: The description no longer says that a protected task's protection
outlasts ops.stopping().
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Move refill_task_slice_dfl() below apply_slice_vtime(), with the other slice
writers, to prepare for the kernel-slice handling the following patches add
to them. No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_do_enqueue_task() counts a task's reenqueues until the task runs. Past
SCX_REENQ_MAX_REPEAT it ejects the owning scheduler and leaves the task off
every DSQ for the scheduler's exit to requeue. The count comes before the
bypass test, so it also counts the reenqueues of a task whose scheduler is
bypassing.
Those reenqueues are not the scheduler's placements, and ejecting the
scheduler for them fixes nothing. Under bypass the kernel queues the
scheduler's tasks on the bypass DSQs. A host calling
scx_bpf_reenqueue_local() reenqueues a bypassed child's tasks on the cpu's
local DSQ along with its own, and the reject drain reenqueues a consumed
task that a cap check rejected.
In the following patches, the consume itself reenqueues a task it can't run
right away, and the bypass load balancer reenqueues a task at its home.
The count is reset only when the task runs or leaves sched_ext, so an
ejection can lead to a stranding the exit does not recover:
1. the task's reenqueues pass the limit, which ejects its scheduler and
strands the task
2. the exit's scx_bypass() cycles the task through dequeue and enqueue,
which requeues it on a bypass DSQ with the count still past the limit
3. the task's next reenqueue under bypass strands it again
4. scx_bypass() requeues stranded tasks only when the scheduler wasn't
bypassing yet, so the task stays off every DSQ until the disable walks
the tasks
5. the disable takes scx_enable_mutex, the cgroup lock and scx_fork_rwsem
before the walk, and a sub-scheduler's disable the cpu hotplug lock too.
It hangs if the stranded task holds one of them, forking, attaching to a
cgroup or doing a cpu hotplug
Test bypass before the count. A task reenqueued under bypass stays queued
until a later dispatch or balancer pass, so there is no tight loop to bound,
and the task's first run resets the count. The offline test moves up with it
to stay ahead of the bypass test. A reenqueue on an offline cpu goes to the
local DSQ and isn't counted either.
A host that keeps calling scx_bpf_reenqueue_local() could get a bypassed
child ejected, as the child's tasks on the local DSQ counted against the
child. The reenqueues are the host's, and its own tasks still count against
it.
v2: The description lists the cpu hotplug lock among the locks a
sub-scheduler's disable takes before the walk.
Fixes: 7706d6e4f2e3 ("sched_ext: Bound per-task reenqueues and eject the owning scheduler")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The bypass load balancer will take tasks off a remote cpu's bypass DSQ with
unlink_dsq_and_switch_rq_lock() in the following patches and insert them
into another bypass DSQ. Clearing holding_cpu after the claim is up to each
claimant, so the balancer would need a clear of its own.
unlink_dsq_and_switch_rq_lock() marks a task with holding_cpu for the gap
between dropping the DSQ lock and taking the task's rq lock, so that a
dequeue in that gap can tell the claimant it lost. Once the claimant holds
the rq lock and finds the marker intact, nothing can take the task away
under it and the marker has no reader left.
Each claimant clears it at its own site after that point:
- dispatch_to_local_dsq(), which runs the handshake inline, in its same-rq
and fallback branches
- scx_proxy_reject_task() for the proxy reject
- the dequeue inside deactivate_task() for both remote moves
Clear it where the claim is confirmed instead and drop the explicit clears.
No behavior change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_bpf_cpuperf_set() and scx_bpf_cidperf_set() are members of
scx_kfunc_ids_any, which is registered for BPF_PROG_TYPE_TRACING.
A tracing program can run in a context which already holds the target
rq lock, in which case scx_cpuperf_set() attempts to acquire the same
lock again:
__schedule()
rq_lock(rq); /* rq->__lock is held */
trace_sched_switch();
BPF tracing program
scx_bpf_cpuperf_set(smp_processor_id(), 1024);
scx_cpuperf_set();
locked_rq = scx_locked_rq(); /* NULL */
rq_lock_irqsave(rq, &rf); /* same lock */
scx_locked_rq() only tracks rq locks held during SCX callback
invocations, so it cannot detect an rq lock held by the surrounding
kernel code.
The deadlock can be reproduced with two BPF programs. The first is a
minimal SCX scheduler, which only needs to be registered and enabled:
SEC(".struct_ops.link")
struct sched_ext_ops minimal_ops = {
.name = "minimal",
};
The second is a tracing program attached to tp_btf/sched_switch,
which fires from __schedule() with rq->__lock already held:
SEC("tp_btf/sched_switch")
int BPF_PROG(on_sched_switch, bool preempt, struct task_struct *prev,
struct task_struct *next, unsigned int prev_state)
{
scx_bpf_cpuperf_set(bpf_get_smp_processor_id(), 1024);
return 0;
}
With both programs attached, the next context switch on any CPU
deadlocks that CPU. Lockdep reports:
WARNING: possible recursive locking detected
7.3.0-rc7 #3 Not tainted
swapper/3/0 is trying to acquire lock:
(&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x1e/0x30
but task is already holding lock:
(&rq->__lock){-.-.}-{2:2}, at: __schedule+0xeb/0x1430
Possible unsafe locking scenario:
CPU0
----
lock(&rq->__lock);
lock(&rq->__lock);
*** DEADLOCK ***
Call Trace:
<TASK>
print_deadlock_bug.cold+0xc0/0xd0
__lock_acquire+0x12fa/0x1b80
lock_acquire+0xd1/0x2f0
_raw_spin_lock_nested+0x32/0x40
raw_spin_rq_lock_nested+0x1e/0x30
scx_cpuperf_set+0x1e9/0x310
scx_bpf_cpuperf_set+0x9b/0x230
bpf_prog_..._on_sched_switch+0x3d/0x41
bpf_trace_run4+0x125/0x450
__traceiter_sched_switch+0x4b/0x70
__schedule+0x94a/0x1430
schedule_idle+0x26/0x50
cpu_startup_entry+0x2d/0x30
start_secondary+0xfc/0x100
</TASK>
The CPU remains locked up with interrupts disabled.
scx_bpf_cidperf_set() is similarly affected, as it resolves the CID
through scx_cid_to_cpu() before reaching the same rq-locking path.
There is no reliable way for these kfuncs to determine whether the
surrounding tracing context already holds an rq lock. Reject calls to
both kfuncs from tracing programs at verification time, while
preserving their availability to SCX struct_ops and syscall programs.
Fixes: a11d6784d731 ("sched_ext: Fix missing rq lock in scx_bpf_cpuperf_set()")
Signed-off-by: Junbin Zhang <sprooc69@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
bypass
While a sched is bypassing, its tasks are run by its host, the nearest
ancestor that isn't bypassing. check_rq_for_timeouts() tests a runnable task
against its owner's timeout. Since 2c91169377f4 ("sched_ext: Blame the DSQ's
owning scheduler for a runnable stall"), it ejects the sched owning the DSQ
the task waits on. The sched tested, the sched ejected and the sched running
the task can all differ.
- Timeouts are per sched. A bypassing sub-scheduler with a 2s timeout under
a root with the default 30s has its tasks on the root's bypass DSQ. The
root, not bypassing itself, consumes that DSQ before calling
ops.dispatch() only on every other pass, and on the others only after
ops.dispatch() yields nothing. A task that waits 2s there ejects the root,
disabling all of sched_ext.
- A bypassing owner's task on the local DSQ is blamed on the owner, which
places nothing while bypassing. If the owner is already exiting, e.g.
during its own disable, the exit is a no-op and nothing is ejected. If the
host isn't running the task, the owner's disable, which waits on it, may
never finish.
- The wait isn't restarted when bypass moves a task. The time it waited
under its owner counts against the host once it is on the host's bypass
DSQ, and its time there against the owner once bypass lifts.
The sched running a task is its owner, or the host while the owner is
bypassing. Test the task against that sched's timeout and blame it. A bypass
requeue can change the sched running a task, so restart the wait of every
task scx_bypass() requeues except the root's own, which the root runs
bypassing or not. Single-scheduler behavior is unchanged.
A task that migrates while scx_bypass() works through the cpus, leaving one
before its requeue and arriving on another after it, is requeued by neither
and keeps its wait, which then counts against the sched now running it. That
is what happened to every task before this patch. This might be worth
addressing in the future, but the race is rare and the real-world
implications should be limited.
Fixes: 2c91169377f4 ("sched_ext: Blame the DSQ's owning scheduler for a runnable stall")
Cc: stable@vger.kernel.org # v7.3+: 30fb2b1e4d41: sched_ext: Fix the running task keeping its protection into bypass
Cc: stable@vger.kernel.org # v7.3+: 5809f749fbdb: sched_ext: Fix a bypassed task stalling on a cpu its scheduler has no caps on
Cc: stable@vger.kernel.org # v7.3+
Reported-by: Andrea Righi <arighi@nvidia.com>
Link: https://lore.kernel.org/all/at4l0yXTk7T_ouk0@gpd4/
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext updates from Tejun Heo:
- Proxy execution can now be enabled together with sched_ext. A BPF
scheduler can opt in to receive mutex-blocked tasks that donate their
scheduling context to the lock owner and place them itself.
- Lazy preemption: a BPF scheduler can preempt or kick a CPU lazily,
deferring the switch to the next return to user space or tick, as the
fair class can.
- Sub-schedulers: a parent now learns which child runs on each CPU it
delegated and when a cap change takes effect. A detaching child loses
its caps before the parent restores the CPUs.
- New kfuncs re-enqueue single tasks from a dispatch queue iteration
and report a cgroup's CPU count and a CID's NUMA node. The CID
topology now includes clusters.
- Hot path optimizations: dispatch queue and task lookups compare keys
inline, remote task moves no longer take a dispatch queue lock twice,
and sub-scheduler hooks cost nothing without sub-schedulers.
- Kfuncs callable from tracing programs are hardened against NMI and
IRQ nesting, which could hard-lock the CPU or pick a wrong idle CPU.
- Built-in dispatch queues no longer allocate per-CPU state that grew
with the square of the CPU count.
- Header syncs with the scx repo, a BTF workaround for pahole 1.32, and
selftest fixes and coverage.
* tag 'sched_ext-for-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (69 commits)
sched_ext: Drop the locks periodically while walking a user DSQ for re-enqueue
sched_ext: scx_qmap: Also re-enqueue stranded tasks with scx_bpf_dsq_reenq_task()
sched_ext: Add scx_bpf_dsq_reenq_task() to re-enqueue single tasks from a DSQ iteration
sched_ext: Factor out the rq lock switch of scx_dsq_move()
sched_ext: Don't queue balance callbacks from a core-sched pick
sched_ext: Don't leave balance callbacks queued across a pick retry
sched_ext: Fix a bypassed task stalling on a cpu its scheduler has no caps on
sched_ext: Fix the running task keeping its protection into bypass
sched_ext: Fix use-after-free of a local reenq node left on an inactive cpu
sched_ext: Keep proxy donors with slice left on the local DSQ
selftests/sched_ext: Report reload_loop attach failures
selftests/sched_ext: Bound exit waits
selftests/sched_ext: Add SCX_ASSERT_ALIVE and adopt it
selftests/sched_ext: Fail loudly when EXIT_KIND is missing from BTF
sched_ext: Add SCX_ECODE_RSN_CGROUP_OFFLINE to user_exit_info.h
selftests/sched_ext: Fix reload_loop error-path use-after-free
selftests/sched_ext: Fix build error in dequeue_remote selftest
sched_ext: scx_qmap: Reset a child's cpuperf targets once its PERF revoke takes effect
sched_ext: Add ops.sub_child_ecaps_updated() to report a child's effective cap changes
sched_ext: scx_qmap: Show actual cid use per participant
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"Kconfig updates:
- Simplify PREEMPT_DYNAMIC (Mark Rutland)
Fair scheduler updates:
- Remove redundant RUN_TO_PARITY check (Elif Topuz)
- Declare augmented callbacks per field, fix
rb_add_augmented_cached() descent (Yiwei Lin, Peter Zijlstra)
- Restart fair hrtick after same-task repicks (Shubhang Kaushik)
- Improve the scheduling latency of short slice tasks (Vincent
Guittot)
Deadline/RT scheduler updates:
- Check start_dl_timer expiry with ktime_before() (Liang Hao)
- irq_work: CPU-hotplug improvements on PREEMPT_RT (Sebastian Andrzej
Siewior)
Energy-aware scheduling updates:
- Implement the steal_governor, introduce preferred CPUs and
steal-driven vCPU backoff (Shrikanth Hegde)
Multi-processor load-balancing updates:
- Enable preferred SMT siblings on NVIDIA Olympus (Andrea Righi)
- Randomize equally shallow idle CPU picks (Christian Loehle)
- Remove redundant core_sched_seq (Hui Su)
Cache-aware scheduling updates:
- Avoid publishing partial LLC utilization (Jemmy Wong)
NUMA-balancing updates:
- Reset NUMA fault locality after scan period update (Eric Kim)
Proxy execution updates:
- Ongoing work to implement sleeping owner handling for proxy
execution (John Stultz, Vasily Gorbik)
- Preparatory steps to make sched_ext compatible with proxy execution
(Andrea Righi)
Scheduler infrastructure updates & fixes:
- Remove the RCU wait from PSI trigger destruction (Shakeel Butt)
- Avoid modifying wait-entry on failed priority registration
(Hui Su)
- Use the new static key API for sched_feat (Hongyan Xia)
- Convert paravirt_steal to new static key APIs (Hongyan Xia)
- cputime: Don't account idle time twice after dyntick-idle
(Stian Halseth)
Scheduler debugging updates:
- Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
(Sechang Lim)
- Fix context analysis errors in non-preferred CPU push (Shrikanth
Hegde)
Misc cleanups and fixes by Christian Loehle, Shubhang Kaushik,
Thomas Huth and Vladimir Zapolskiy"
* tag 'sched-core-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (64 commits)
sched: Replace nr_iowait_cpu() with a bool accessor
Revert "sched/fair: Remove dead code on enqueue_task_fair()"
sched/cache: Avoid publishing partial LLC utilization
sched/cputime: Don't account idle time twice after dyntick-idle
sched/psi: Remove the RCU wait from trigger destruction
sched/eevdf: Remove redundant RUN_TO_PARITY check
sched/wait: Avoid modifying entry on failed priority registration
rbtree_test: Use rb_add_augmented_cached() for the cached augmented test
rbtree: Update augmented data on the way down in rb_add_augmented_cached()
rbtree: Declare augmented callbacks per field with RB_AUG()
rbtree_test: Use rb_add() and rb_add_cached() for the basic tests
smpboot: Don't park the thread if work is pending
irq_work: Flush lazy work CPU down on PREEMPT_RT
irq_work: Update a comment regarding CPU hotplug invocation
sched/topology: Add asymmetric SMT packing override
sched/fair: Honor asymmetric SMT priority in idle selection
sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
sched/core: Fix context analysis errors in non-preferred CPU push
virt/steal_governor: Enable the driver
virt/steal_governor: Implement steal_governor policy loop
...
|
|
maybe_queue_balance_callback() clears SCX_RQ_BAL_CB_PENDING when it
queues the deferred callback. Proxy resolution can then zap the callback
without running it, leaving the deferred actions unscheduled until a
later wakeup or deferred request.
Use the existing irq_work fallback for dispatch-time deferred requests
when proxy execution is enabled. This keeps the work scheduled even if
proxy resolution retries the pick or falls back to idle, without relying
on another SCX dispatch to restore a discarded callback.
Fixes: a8ad873113d3 ("sched_ext: defer queue_balance_callback() until after ops.dispatch")
Reported-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/08cac02c604dd73ef96fc92eb29d692b@kernel.org
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The SCX_KICK_WAIT wait is deferred to rq->scx.kick_sync_bal_cb, which a
final pick queues on the local rq. Under core scheduling, a pick that
returns a task isn't final: pick_next_task() goes on to pick the SMT
siblings, and their dispatch can release the shared rq lock with the
callback still queued. A CPU acquiring the rq lock in that window then
finds a pending balance callback and triggers the warning in
rq_pin_lock().
Fix this by queuing the callback from set_next_task_scx() on SNT_PICK
and SNT_REPICK. put_prev_set_next_task() calls it once the pick is
final, after the core-wide selection, so the rq lock can't be dropped
with the callback queued. If the CPU ends up running idle or a higher
class task, no SCX task runs there and there's nothing to wait for:
kick_sync_pending stays set and the next pick of an SCX task waits
before running it.
Also, proxy exec zaps the balance callbacks and picks again when
find_proxy_task() fails, so clear kick_sync_pending when the callback
runs rather than when it's queued.
Fixes: 415cb193bb97 ("sched_ext: Fix SCX_KICK_WAIT deadlock by deferring wait to balance callback")
Cc: stable@vger.kernel.org # v7.0+: c72945693b90: sched: Restart fair hrtick after same-task repicks
Cc: stable@vger.kernel.org # v7.0+
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The watchdog measures how long a task has been waiting from
p->scx.runnable_at, which set_task_runnable() resets when the task becomes
runnable after it last ran. ops_dequeue() never asks for a reset, as the
dequeue of a waiting task is temporary, e.g. for a migration or an
attribute change.
Leaving the scheduler isn't temporary. A task which is switched out of
SCHED_EXT while it's waiting keeps its runnable_at. When it's switched back
and becomes runnable, its wait appears to have started before it left, and
the time it spent in another class counts towards the timeout. If the
watchdog checks before the task runs, it aborts the scheduler:
sched_ext: class_switch_stall: runner[418] failed to run for 3.010s
Here, the task waited for a few milliseconds before it was switched to
SCHED_OTHER, slept outside of SCHED_EXT for three seconds, and was charged
for all of it as soon as it woke up on SCHED_EXT again.
The same happens when scx_sub_enable_workfn() hands a waiting task over to
a new sub-scheduler, which can then be blamed for time the task waited
under the parent. scx_rehome_task() isn't affected, as it moves the task
through SCX_TASK_INIT, which resets runnable_at. The stale runnable_at also
skews the default core-sched ordering in scx_prio_less().
Reset runnable_at in scx_disable_task(), when the task leaves the control
of its scheduler.
Fixes: 8a010b81b3a5 ("sched_ext: Implement runnable task stall watchdog")
Cc: stable@vger.kernel.org # v6.12+
Assisted-by: LLM
Signed-off-by: Michal Blaszczyk <michalblk@google.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Suspend and hibernation take down every non-boot cpu and bring them back. A
loss and a regain per cpu to every sub-scheduler would be noise. Leave them
out: a cpu they take down reports no loss and keeps the caps last delivered,
and the reseed after resume reports only the net change. Their downs are the
ones with cpuhp_tasks_frozen set between the PM notifications. A kexec jump
sets the flag too but sends no PM notification.
Two things need care. A cpu that suspend or hibernation takes down doesn't
wait for its syncs, and no ecaps_report() mark can be pending on it then.
The attach report runs while the attaching task waits uninterruptibly, which
holds off the freezer, and the detach report holds cpus_read_lock().
The detach report reaches only active cpus, and the loss on a cpu that
suspend or hibernation took down was never delivered. The detach waits for
those cpus to come back, so every loss is delivered. They come back before
the tasks thaw, so the wait doesn't hold up the freezer. If one fails to
come back, the hierarchy is aborted at the end of the PM cycle.
v2: Fix why no report mark is pending at a PM down, add a data_race().
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A sub-scheduler's ecaps ops run in dispatch context, and ops.cpu_online()
and ops.cpu_offline() document that the cpu doesn't call ops.dispatch()
outside them. The reseed of a cpu coming back is queued before the root's
ops.cpu_online(), so the ops of a sub-scheduler and its parent can run for a
cpu the root still considers offline.
Add SCX_RQ_OPS_ONLINE, set while the root considers the cpu online, from its
ops.cpu_online() or its enable until its ops.cpu_offline(). A sync outside
that span leaves the ecaps alone, and the reseed, now queued after
ops.cpu_online(), reports the net change. SCX_RQ_ONLINE can't be used, as
the core changes it ahead of the root's view on both edges.
A cpu going offline still drops its caps without a report.
While at it, fix the SCX_RQ_ONLINE comments, which claimed that the flag
tracks ops.cpu_online() and ops.cpu_offline().
v2: Fix the SCX_RQ_ONLINE comments.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A sub-scheduler learns which cids it holds through ops.sub_ecaps_updated(),
but the caps granted from ops.sub_attach() reach it only after its enable
bypass lifts, when it is already scheduling its tasks.
Report the caps at the end of the enable, before bypass lifts. On disable,
report their loss before any task goes back to the parent, so that the
parent hears of the loss of every cap it was told of on an active cpu.
ops.sub_detach() remains where the parent cleans up after the sub-scheduler.
Deliveries no longer depend on bypass, so ops.sub_ecaps_updated() and
ops.sub_child_ecaps_updated() also run while a scheduler bypasses. The
replay at unbypass goes, and the enable's bypass returns to after
ops.sub_attach().
A sub-scheduler's ecaps now change only with the deliveries that report
them, so that whenever its ops run under the rq lock, its caps are the ones
it was last told of. A sync before the attach report leaves them alone, and
the detach report zeroes them as it delivers the loss. A cpu going offline
still drops its caps without a report, and those are the only changes not
delivered. clear_all_caps() drops that unreported loss on inactive cpus.
The following things need care. A report reschedules only the cpus the
sub-scheduler was ever granted a cap on. Rescheduling every cpu would
disrupt a large machine split many ways, and which cpus still owe a delivery
can't be read off the caps.
The report and a queued sync reschedule the cpu from hard irq work on that
cpu, not with a kick or a direct resched. dispatch_one() processes the syncs
only at its start, and a kick only guarantees that a scheduling event
completes after it, which can be the pick in progress. That pick has already
processed the syncs and clears the resched, and a cpu that then idles or
runs a task with an infinite slice leaves the report waiting. A pick runs
with IRQs off, so the irq work runs after any pick in progress on that cpu.
Under core scheduling, a sibling's committed pick can switch the cpu without
dispatch_one(), and a sync queued after that pick is processed only at the
cpu's next scheduling event. A cpu running a task of a higher class isn't
rescheduled, as its pick would return that task again. While the root
bypasses, a queued sync doesn't reschedule the cpu, as a grant or revoke can
run in the PM syscore phase, where irq work can malfunction. The report
never runs there and reschedules the cpu itself.
The report waits on an rcuwait woken from hard irq work. Under core
scheduling, a cpu going down can process a sibling's marked sync after
hotplug has parked its irq_work thread. On PREEMPT_RT, a regular irq_work
queued there would be left behind, and a hard irq can't take a wait queue's
lock.
v2: Reschedule from hard irq work on the cpu instead of a kick (sashiko).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The free path removes a sub-scheduler's ecaps sync node from each cpu's list
and then cycles the rq lock until the node reads off-list, as a sync batch
can hold the node privately. Cycling the lock is correct only because the
sched is bypassing and a bypassing sched's sync calls no op, so the batch
touches the pcpu within one rq lock hold.
Tying the free path to what the sync does for a bypassing sched is unusual
and brittle, and it gets in the way of changing the sync. Delivering ecaps
to a bypassing sched or its parent would break the wait, as a delivery drops
the rq lock.
Wait for the batches with RCU instead, the usual way to wait for lockless
users before a free. Discard the nodes after the first grace period as
before, and free the sched after a second one. The first grace period waits
for the paths that found the sched before it was unlinked, such as the
sub-cap kfuncs and the cpu hotplug hooks. Until they finish, they can queue
its syncs, and a cpu can take a queued node into a sync batch that the first
grace period doesn't wait for. After it, nothing queues the syncs, and the
discards remove the nodes still queued.
A batch that took a node keeps irqs off from llist_del_all() to its last
pcpu write, even across the rq lock drops of a delivery. The second grace
period waits for it whether or not it called an op. The exit and disable
irq_works, the helper kthread and the bypass load balancing timer are
stopped ahead of the per-cpu checks as before, so they move into
scx_sched_quiesce_rcu_work() with the checks.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
reenq_user() drops dsq->lock around each re-enqueue and its rq lock every
SCX_TASK_ITER_BATCH re-enqueues. A walk filtered by SCX_REENQ_MARKED skips
unmarked tasks without dropping either, so a long run of unmarked tasks is
walked in one hold of both locks.
Count the visited tasks instead of the re-enqueued ones and drop both locks
every SCX_TASK_ITER_BATCH visits. An unfiltered walk drops the locks as
often as before.
Suggested-by: Andrea Righi <arighi@nvidia.com>
Link: https://lore.kernel.org/all/atP-LIKpxPhR-Nuc@gpd4/
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
DSQ iteration
scx_bpf_dsq_reenq() re-enqueues a whole DSQ, but a scheduler that loses the
cpus serving a DSQ, such as a sub-scheduler after a cap revoke, needs only
the stranded tasks placed again.
Add scx_bpf_dsq_reenq_task(), which marks a task from a DSQ iteration. When
the iteration ends, the DSQ's deferred re-enqueue walk is scheduled once and
re-enqueues only the marked tasks. The kfunc locks the task's rq and the DSQ
like scx_bpf_dsq_move() and is callable from the same contexts. The mark is
cleared when the task leaves the DSQ, so a task consumed before the walk is
not re-enqueued.
An iteration's DSQ can be destroyed before the iteration ends, so
schedule_dsq_reenq() now skips a destroyed DSQ instead of aborting the
scheduler. scx_bpf_dsq_reenq() racing scx_bpf_destroy_dsq() takes the same
path.
Declare the kfunc for the tools and add a compat probe.
v2: Note in the kerneldoc that another iteration's re-enqueue may take a
marked task early, and that a task on another scheduler returns false
(Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Factor the rq lock switch on entry to scx_dsq_move() and its undo on exit
out into kf_switch_rq_lock() and kf_restore_rq_lock() so that other kfuncs
operating on a task from a DSQ iteration can share them. No functional
change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Pull to receive 680edc7f23ad ("sched_ext: Don't queue balance callbacks from
a core-sched pick") and resolve its conflict with the proxy execution work
here so that for-next merges clean.
Conflicts:
- do_pick_task_scx(): 680edc7f23ad samples sched_core_enabled() into a new
local next to the prev declaration, which 3d6577ea93c9 ("sched_ext: Split
curr|donor references properly") changed to rq->donor. Kept rq->donor and
added the local.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Commit ff0545e0c263 ("sched_ext: Don't leave balance callbacks queued
across a pick retry") queues the deferred work as a balance callback
only when do_pick_task_scx() returns a final pick. However, under core
scheduling, a pick that returns a task isn't final either:
pick_next_task() goes on to pick the SMT siblings, and their dispatch
can release the shared rq lock with the callback still queued,
triggering the rq_pin_lock() warning.
Moreover, dispatch_core_pick() returns early when another dispatch is
in flight on the rq, and do_pick_task_scx() then consumes
%SCX_RQ_BAL_CB_PENDING set by that dispatch while it's still running.
This can be reproduced running scx_qmap -I on SMT siblings with
coresched-tagged workloads that frequently sleep, mixed with untagged
CPU-bound tasks.
Fix this by always deferring the pending work to the irq_work path from
dispatch_core_pick() under core scheduling, for both the local and the
sibling picks, and by leaving %SCX_RQ_BAL_CB_PENDING alone in
do_pick_task_scx(), so that it's handled by the dispatch that set it.
Sample the core scheduling state once at the start of the pick, since
core scheduling can get enabled while a non-core dispatch has the rq lock
released, and the follow-up work must match the dispatch path that ran.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The dispatch path can request deferred work to be run from a balance
callback and pick_task_scx() queues rq->scx.deferred_bal_cb right after
dispatching. However, a balance callback must be consumed before the rq
lock is released and pick_task_scx() is not necessarily the end of the
pick:
- it returns RETRY_TASK when a higher priority class task has been
enqueued while the dispatch had the rq lock released;
- the dl_server pick can return NULL, in which case the lower classes
are picked next.
In both cases the callback stays queued while the core keeps picking,
and the rq lock can be released again, either by prev_balance() or by
another dispatch. For example, with an RT task as prev:
__schedule()
__pick_next_task()
pick_task_scx()
dispatch_pick()
maybe_queue_balance_callback() <- deferred_bal_cb queued
return RETRY_TASK
prev_balance()
balance_rt()
pull_rt_task()
double_lock_balance() <- rq lock released
Any CPU acquiring the rq lock in that window finds a pending balance
callback and triggers the following warning in rq_pin_lock():
WARNING: kernel/sched/sched.h:1919 at rq_lock+0x7f/0xe0, CPU#10: stress-ng-sleep/533
CPU: 10 UID: 0 PID: 533 Comm: stress-ng-sleep Not tainted 7.2.0-rc6-virtme #61 PREEMPT(full)
Sched_ext: qmap (enabled+all)
Call Trace:
<IRQ>
try_to_wake_up+0x2c1/0x5c0
hrtimer_wakeup+0x22/0x30
__hrtimer_run_queues+0x1d5/0x350
hrtimer_interrupt+0x106/0x1e0
__sysvec_apic_timer_interrupt+0x6f/0x210
sysvec_apic_timer_interrupt+0x71/0x90
</IRQ>
This can be reproduced running scx_qmap -I together with a SCHED_FIFO
workload that frequently sleeps, e.g.:
$ vng -v --cpus 16 --memory 4G -- \
'tools/sched_ext/build/bin/scx_qmap -I >/dev/null & \
stress-ng --sleep 4 --sched fifo --sched-prio 1 -t 30 -q'
Fix this by queueing the balance callbacks only when the pick is final.
Otherwise, defer the pending work to the irq_work path, which doesn't
depend on the current rq lock section, and leave the kick sync wait
pending for the next pick.
Fixes: 4c95380701f5 ("sched/ext: Fold balance_scx() into pick_task_scx()")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive 30fb2b1e4d41 ("sched_ext: Fix the running task keeping its
protection into bypass") and 5809f749fbdb ("sched_ext: Fix a bypassed task
stalling on a cpu its scheduler has no caps on") and resolve their conflicts
with the proxy execution work here.
Conflicts:
- scx_bypass(): 30fb2b1e4d41 drops the protection strip in the runnable_list
walk, which never sees the running task, and strips rq->curr ahead of the
walk instead. 3d6577ea93c9 ("sched_ext: Split curr|donor references
properly") moved the walk's strip to task_current_donor(), as protection
belongs to the scheduling context. set_next_task_scx() takes the donor off
the runnable_list, so that strip is dead too. Dropped it, and the new
strip takes rq->donor.
- scx_bypass(), merged without a marker: 5809f749fbdb evicts the running
task when bypass lifts if it can't stay on the cpu, on rq->curr.
3d6577ea93c9 moved the cap revoke eviction of the running task in
reenq_local() to rq->donor. The lift eviction takes rq->donor too, and
both comments in the block say the current donor.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A bypassing sub-scheduler's tasks are run by its nearest ancestor that isn't
bypassing. The ancestor puts them on the local DSQ under its own caps.
Whether such a task may stay on the cpu and whether a cap revoke reenqueues
it are still checked against the caps of the task's own scheduler.
The bypass load balancer moves tasks between cpus without looking at caps,
so a bypassed task can run on a cpu where its own scheduler holds none. Once
its slice runs out, it can't stay. It goes to the ancestor's bypass DSQ
without a kick, and the cpu can go idle with it queued until something else
reschedules the cpu.
Make both checks against the caps of the scheduler running the task. When
bypass lifts, the running task keeps its slice without a cap check, so evict
it then if it can't stay on the cpu.
Fixes: 75a8c8202c91 ("sched_ext: Add reject DSQ for cap-rejected dispatches")
Fixes: 46a85ae6fe5b ("sched_ext: Tie cpu occupancy to SCX_CAP_BASE through the task slice")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Protection ends when the scheduler enters bypass, so that bypass scheduling
can cut every task's slice. scx_bypass() clears it for the queued tasks by
cycling them, and strips the running task's in the same walk over
rq->scx.runnable_list. The running task is never on that list.
set_next_task_scx() takes it off and put_prev_task_scx() puts it back, so
the strip never runs. A protected running task then keeps its slice into
bypass, where the tick's attempts to cut it are refused.
Strip the running task's protection outside the walk.
Fixes: 9cfc6ab34a31 ("sched_ext: Add SCX_TASK_PROTECTED")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A local DSQ reenq is queued on the rq through a per-sched node and run from
deferred work. When the reenq is requested from dispatch, the work is a
balance callback queued once dispatch is done. A cpu going down installs
balance push early in its teardown, and queue_balance_callback() refuses new
callbacks from then on. ops.dispatch() keeps running on that cpu until the
rq goes offline after an RCU grace period, so a reenq it requests in that
window may never run. The IMMED handling in dispatch_one() requests one the
same way.
A reenq requested from elsewhere for a cpu that is already offline goes
through irq_work_queue_on(), which warns, and its work doesn't run until the
cpu comes back.
Either way the node stays on the rq's list. When the scheduler is freed,
scx_sched_free_rcu_work() warns and frees the pcpu with the node still
linked, and the list is walked again once the cpu comes back.
An inactive cpu's local DSQ holds only tasks that are bound to it or are
being moved off, so the reenq is pointless. Ignore reenqs of an inactive
cpu's local DSQ.
The test is lockless. The hotplug thread of a cpu going down runs on that
cpu, and it marks the cpu inactive before it installs balance push. A
dispatch on the cpu runs to completion before that thread can run there, so
a dispatch that saw the cpu active has queued its callback by then. Requests
from outside dispatch run from the wakeup path or through irq_work, which
balance push doesn't affect. The grace period after the cpu goes inactive
waits for any requester that saw it active, so that irq_work is queued while
the cpu is online.
Fixes: 238eba8c210d ("sched_ext: Use schedule_deferred_locked() in schedule_dsq_reenq()")
Fixes: 76edc2761ab8 ("sched_ext: Use irq_work_queue_on() in schedule_deferred()")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- A task that skipped the BPF scheduler on its way back to its CPU's
local queue could stall there until the watchdog fired, as the CPU
had already decided to go idle.
- Use-after-free of a destroyed user dispatch queue by a deferred
re-enqueue request.
- A task could be moved off a dispatch queue while the dispatching CPU
was still running ops.dequeue() on it.
- Bypass mode mishandled a sub-scheduler bypass overlapping an
ancestor's, as across a system suspend. Descendants could leave
bypass mid-teardown and tasks could be stranded on bypass queues.
- The idle CPU selection helper could return a CPU outside the allowed
CPUs of a pinned or migration-disabled task.
* tag 'sched_ext-for-7.3-rc7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs
sched_ext: Rename scx_reenq_wait_dispatching() to scx_wait_dispatching()
sched_ext: idle: Respect task affinity in restricted CPU selection
sched_ext: Fix the bypass dispatch claim for schedulers bypassed by an ancestor
sched_ext: Propagate every bypass depth change to the descendants
sched_ext: Fix use-after-free of a destroyed user DSQ by a deferred reenqueue
sched_ext: Fix stall when a task enqueued with SCX_ENQ_LAST lands back on its CPU's local DSQ
|
|
Commit ee172227d0dc ("sched_ext: Delegate proxy donor admission to BPF
schedulers") makes put_prev_task_scx() pass a retained proxy donor to
ops.enqueue() with SCX_ENQ_BLOCKED.
Some of these puts are only proxy bookkeeping; proxy_resched_idle()
drops the rq's donor reference before switching to idle when:
- find_proxy_task() cannot run the mutex owner yet and retries through
idle (e.g., the owner is on a remote CPU),
- proxy_migrate_task() detaches the donor before migration,
- proxy_deactivate() blocks a task in the chain whose owner cannot run.
In the first case, BPF has already selected a donor with slice left.
Returning it to ops.enqueue() forces BPF to dispatch it again before
proxy resolution can continue. In the other two cases, the donor is
typically deactivated right after ops.enqueue(), undoing any placement
BPF makes.
As also discussed at the sched_ext microconference at Linux Plumbers
2026, these ops.enqueue/dequeue() invocations do not represent any
meaningful scheduling events, so we should avoid triggering them.
A more reasonable semantic is to keep a donor with slice left at the
head of the local DSQ, so the next pick can resolve its owner, or
deactivation can remove it without an unnecessary BPF handoff.
An IMMED donor can stay there only for a bookkeeping put during proxy
resolution. Mark a blocked pick as awaiting resolution with
SCX_RQ_PROXY_PENDING to distinguish that put from a real preemption and
re-insert the donor with SCX_ENQ_IMMED, so that it only needs
SCX_CAP_ENQ_IMMED rather than SCX_CAP_ENQ. A preempted IMMED donor still
returns to BPF. If a higher priority class takes the CPU once proxy
resolution completes, schedule a local reenqueue, so that a parked IMMED
donor returns to BPF instead of lingering on the local DSQ. Update the
SCX_ENQ_IMMED and SCX_ENQ_BLOCKED documentation accordingly.
Moreover, a donor that has run out of slice now follows the regular put
path, so it can receive SCX_ENQ_LAST when the CPU is about to go idle
and the scheduler needs to arrange a follow-up scheduling event. This
does not apply to bookkeeping puts, where the switch to idle is only
temporary. The blocked-donor specific sanity checks go away together
with the special case.
When a retained donor is removed ahead of a scheduler ownership or
policy change, mark the proxy reset with SCX_RQ_PROXY_BLOCKING and skip
reenqueueing the active donor: sched_proxy_block_task() will dequeue it
next. Report it as not runnable to ops.stopping(), as a regular dequeue
of the current donor does. This avoids a transient ops.enqueue/dequeue()
pair and a false ENQ_LAST warning.
With this, the sched_ext core absorbs the transient proxy-bookkeeping
puts and the BPF scheduler only sees a blocked donor when there is an
actual placement decision to make.
Fixes: ee172227d0dc ("sched_ext: Delegate proxy donor admission to BPF schedulers")
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive 2935062be771 ("sched_ext: Wait for SCX_OPSS_DISPATCHING
before moving tasks off DSQs") and resolve its conflicts with the proxy
execution and reject DSQ work here so that for-next merges clean.
Conflicts:
- unlink_dsq_and_switch_rq_lock(): efa7bc7d7195 ("sched_ext: Avoid relocking
DSQ during remote consumption") clears p->scx.dsq in the success branch
and 2935062be771 adds the wait there. Kept both, the wait after the clear,
so that the proxy reject in consume_remote_task() is covered too.
- scx_reenq_reject(): 18f5c699d1d0 ("sched_ext: Rename
scx_reenq_wait_dispatching() to scx_wait_dispatching()") renames the call
in sub.c while cea8b60e4aee ("sched_ext: Move reject DSQ draining into
core") moved the function into ext.c. Dropped the sub.c side and applied
the rename at the new home.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A task can be found on a non-local DSQ while the CPU which dispatched it
is still running ops.dequeue() and hasn't cleared SCX_OPSS_DISPATCHING.
The DSQ consume and move paths don't wait for the dispatch to finish and
can operate on the task concurrently with the dispatcher.
Call scx_wait_dispatching() in scx_move_local_task_to_local_dsq(),
unlink_dsq_and_switch_rq_lock() and move_task_between_dsqs() before they
operate on the task. In scx_move_local_task_to_local_dsq(), wait before
scx_resolve_local_dsq() as it may update p->scx.flags.
Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics")
Fixes: cb86607ada73 ("sched_ext: Don't run ops.dequeue() with a DSQ lock held")
Cc: stable@vger.kernel.org # v7.1+
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Waiting for SCX_OPSS_DISPATCHING to clear isn't specific to the reenqueue
paths. Any path which takes a task off a DSQ can find it while the
dispatch is still in flight, and the next patch calls the helper from the
DSQ consume and move paths. Rename it to scx_wait_dispatching() and
update its comment accordingly.
No functional change.
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pickup dependencies; notably the RB-tree fix.
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
|
|
scx_bypass() claims bypass dispatch when the scheduler's depth goes from
zero to one and releases it when the depth returns to zero, but propagates
only the depth to the descendants, not the claim. A descendant that starts
bypassing with its ancestor never claims, and one whose own bypass ends
under the ancestor's keeps its claim, as its depth does not return to zero.
A sub-scheduler enable whose bypass ends under a PM suspend thus leaks its
claim: the parent keeps hosting bypass dispatch and the sub-scheduler's next
bypass trips the WARN_ON_ONCE() on the claim. A sub-scheduler disable that
starts under a PM suspend has no claim, so once the resume lifts the root's
bypass its tasks sit on the parent's bypass DSQs with nobody consuming them.
Claim for every descendant whose depth goes from zero to one in the
propagation and release every scheduler in the subtree whose depth returned
to zero once the tasks have moved out of the bypass DSQs. A parent that is
bypassing itself when a descendant claims consumes only its own bypass DSQs,
empty until it leaves bypass and becomes the host.
Fixes: aa2a0a19686c ("sched_ext: Separate bypass dispatch enabling from bypass depth tracking")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_bypass() propagates a depth change to the descendants only when the
scheduler's own depth crosses zero, so an own bypass that starts under an
ancestor's never reaches them. When the ancestor's bypass ends first, the
descendants leave bypass while the scheduler is still bypassing: a
sub-scheduler disabled under a PM suspend has its children re-queue their
tasks through their own BPF schedulers at resume, mid-teardown. If the own
bypass then ends, every descendant is decremented once more than incremented
and dec_bypass_depth() warns, though no current caller gets there.
Propagate on every call so that a descendant's depth is the sum of its own
bypasses and its ancestors'. The early return follows the walk, as a
descendant's depth is at least the scheduler's and nothing changes bypass
state unless the scheduler's own does.
Fixes: c7f0e467a27a ("sched_ext: Prepare bypass mode for hierarchical operation")
Cc: stable@vger.kernel.org # v7.1+
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Pull to receive 069403dff337 ("sched_ext: Fix use-after-free of a
destroyed user DSQ by a deferred reenqueue") and point its new cancel
sweep in free_dsq_irq_workfn() at dsq->pcpu_user, which 92344fad5b54
("sched_ext: Skip per-CPU data allocation for built-in DSQs") renamed
here, so that for-next builds.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A deferred reenqueue can read a freed user DSQ. The request stays on its
CPU's deferred list until run_deferred() takes it, and the RCU callback that
frees the DSQ assumes that the grace period has flushed it, so exit_dsq()
checks the per-CPU node without the deferred list lock. A grace period waits
for readers in flight, not for a pending run_deferred(). A run_deferred()
which starts after the grace period began can detach the request right
before the callback looks and then read the freed DSQ's id.
Cancel the pending requests before the grace period instead. The irq work
that frees the DSQ unlinks every CPU's request under the per-CPU deferred
list lock, and schedule_dsq_reenq() ignores a new request under the same
lock once the id is invalid. Only a run_deferred() that started before the
grace period can hold a request, and it runs with IRQs disabled, so the
grace period waits for it and nothing touches the DSQ after the callback.
Fixes: 84b1a0ea0b7c ("sched_ext: Implement scx_bpf_dsq_reenq() for user DSQs")
Cc: stable@vger.kernel.org # v7.1+
Reported-and-tested-by: Hui Su <sh_def@163.com>
Link: https://lore.kernel.org/all/20261008024753.4096008-1-sh_def@163.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
its CPU's local DSQ
With SCX_OPS_ENQ_LAST, the last runnable task on a CPU is passed to
ops.enqueue() instead of being kept running. Exiting tasks without
SCX_OPS_ENQ_EXITING, migration disabled tasks without
SCX_OPS_ENQ_MIGRATION_DISABLED and tasks on an offline rq skip ops.enqueue()
and go straight to the local DSQ. This happens in put_prev_task_scx() after
the pick has settled on idle, so nothing reschedules the CPU, which idles
with the task queued until an unrelated wakeup lands on it or the watchdog
fires.
scx_mitosis, which sets SCX_OPS_ENQ_LAST without SCX_OPS_ENQ_EXITING, hit
this during mass cgroup teardown: exiting tasks stuck in exit_mmap() for
over 40 seconds.
A task put on the local DSQ runs, as anywhere else. If the SCX_ENQ_LAST
enqueue left the task on this CPU's local DSQ, set need_resched on the task
being switched in, as proxy_resched_idle() does. resched_curr() would flag
the task being put instead, and __schedule() clears that right after the
pick.
Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class")
Cc: stable@vger.kernel.org # v6.12+
Reported-by: Xiangyu Bu <xbu@meta.com>
Link: https://github.com/sched-ext/scx/pull/3870
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Dan Schatzberg <schatzberg.dan@gmail.com>
|
