| Age | Commit message (Collapse) | Author |
|---|
|
scheduler
A CPU going down has to empty its own rq. Its hotplug thread waits in
sched_cpu_wait_empty() until nothing else is left, and the only thing that
wakes it is balance_push(), which runs from __schedule() on the dying CPU
and pushes off the migratable tasks the CPU picks.
sched_ext breaks this. A task sitting on a user DSQ or held by the BPF
scheduler is still counted on its rq, but an inactive CPU no longer calls
ops.dispatch() and can't pull the task back. The dying CPU goes idle with
the task still counted, and one of two things happens:
- Another CPU consumes the task. The rq empties without a __schedule() on
the dying CPU, so the hotplug thread is never woken and cpu_down() hangs
holding cpu_hotplug_lock.
- The task is affine only to the dying CPU. Nothing can move it, and the
offline stalls until the watchdog ejects the BPF scheduler.
When the rq goes offline, re-enqueue every task on it that isn't already on
the local DSQ. An enqueue on an offline rq lands on the local DSQ, so the
dying CPU itself picks the tasks and balance_push() pushes them off, the
same as for the other sched classes. From then on no sched_ext path on
another CPU can pull a task off the rq.
ops.dispatch() currently stops as soon as the CPU goes inactive, and CPU
hotplug then waits for an RCU grace period before taking the rq offline. A
BPF-held task affine only to the dying CPU and preempted inside an RCU
read-side critical section would block that grace period, and the rq would
never go offline. Test SCX_RQ_ONLINE directly so that ops.dispatch() keeps
running until the rq goes offline. Only the dying CPU's own hotplug thread
clears the flag during teardown, and task_can_run_on_remote_rq() still keeps
other rqs' tasks off the CPU.
Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class")
Fixes: 991ef53a4832 ("sched_ext: Make scx_rq_online() also test cpu_active() in addition to SCX_RQ_ONLINE")
Cc: stable@vger.kernel.org # v6.12+
Reported-by: Alap Mohan <mohaalap@meta.com>
Reported-by: Joonwoo Park <joonwoo@meta.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
When a descendant scheduler enters bypass mode, its tasks are parked in
the bypass DSQs of the nearest non-bypassing ancestor, which is then
responsible for running them. On behalf of such a non-bypassing host,
scx_dispatch_sched() consumes those bypass DSQs from two places: the
attempt made every SCX_BYPASS_HOST_NTH dispatches, and the
end-of-dispatch fallback that keeps the CPU from going idle while
bypassed descendants still have tasks queued.
The former increments SCX_EV_SUB_BYPASS_DISPATCH but the latter does
not, even though both perform the same scx_consume_dispatch_q() on the
same bypass DSQ. The descendant bypass dispatches done by the fallback
are therefore missing from the counter exposed via sysfs,
scx_dump_state() and the scx_bpf_events() kfunc, which under-reports the
actual number of such dispatches.
Add the missing __scx_add_event() so the fallback counts them too. When
@sch itself is bypassing, scx_dispatch_sched() takes the earlier
self-bypass branch and returns before reaching these host paths; that
mode is accounted for by SCX_EV_BYPASS_DISPATCH at enqueue time and is
intentionally left unchanged.
Fixes: 025b1bd41965 ("sched_ext: Implement hierarchical bypass mode")
Signed-off-by: Liang Luo <luoliang@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
sched_class->balance() is gone from sched_ext and what balance_one() does is
run dispatch to produce something pickable. Update the balance-era names to
dispatch terms:
- balance_one() -> dispatch_one()
- SCX_RQ_IN_BALANCE -> SCX_RQ_IN_DISPATCH
No BPF scheduler reads the flag. The enum autogen headers gain the new name
with the old entry retained like other removed enumerators, zero-filling at
load time. No functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
- inlines.h: scx_bpf_dispatch() doesn't exist; the comment means
scx_bpf_sub_dispatch()
- internal.h: name %SCX_DEQ_SCHED_CHANGE instead of the never-defined
%SCX_DEQ_SAVE
- internal.h: @name shows up in the ops file in the scheduler's sysfs
directory, not a "kernel.sched_ext_ops" sysctl
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Pull to receive:
c10b216a072f ("sched/core: Handle pick_task() releasing the rq lock")
f3629c63a4af ("sched/core: Make core-sched flips wait for in-flight selections")
ffaab58d2175 ("sched_ext: Replace SCX_RQ_BAL_KEEP with a dispatch verdict return")
3dd52416e44a ("sched_ext: Fix this_rq() assumptions in dispatch kfuncs")
f2da9587118d ("sched_ext: Count rq lock releases in rq->scx.lock_drop_seq")
d954004205c1 ("sched_ext: Fix rq->core_pick corruption under core scheduling")
for the pending core scheduling follow-ups and to resolve the conflicts
with the code reorganization and cap gate work on for-7.3.
ffaab58d2175 converts scx_dispatch_sched() to a dispatch verdict return
which for-7.3 moved from ext.c into inlines.h. Resolved by applying the
conversion to the relocated copy and combining balance_one()'s verdict
returns with the scx_task_can_stay_on_cpu() gate from the cap work.
ffaab58d2175 and 3dd52416e44a update scx_bpf_sub_dispatch() which
for-7.3 moved into sub.c. Resolved by applying the scx_locked_rq()
switch and the verdict test to the sub.c copy.
f2da9587118d instruments the open-coded lock releases in
consume_remote_task() which for-7.3 folded into switch_rq_lock().
Resolved by keeping the accounting in switch_rq_lock() which covers all
its callers.
d954004205c1 widens the put_prev_task_scx() WARN suppression to all
core-sched rqs on the same condition that for-7.3 gated with
scx_task_can_stay_on_cpu(). Resolved by combining both.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
rq->scx.sub_dispatch_prev is sub-sched-only but was left unconditional. Move
it into the CONFIG_EXT_SUB_SCHED block next to ecaps_to_sync and gate its
updates.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_dispatch_sched() is common dispatch machinery and looks out of place in
sub.h, but it needs scx_cpu_arg() from cid.h and can't move into internal.h
without creating a circular include. Add inlines.h on top of internal.h and
cid.h, and move the function there. The function was sub.h's only cid.h
user, so drop that include. Pure code move, no functional change.
v2: Host the function in a new inlines.h instead of at internal.h's tail,
which formed a circular include with cid.h. Drop sub.h's now-unused
cid.h include. (sashiko AI)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
