| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
"This is a big pull for fixes, but much of it was in the works before
the merge window.
Most of these fixes are in sub-scheduler support, which nests BPF
schedulers under a parent that hands them CPUs. It was heavily
reworked over recent cycles and is now functionally complete. Porting
real schedulers on it drove the latest additions and keeps finding
corner cases.
For example, once CPUs are shared and can be taken back, the kernel
must keep every task moving when its scheduler can't run it or is
failing, and those fallbacks interact subtly with each other and with
other new core features.
- The fallback paths meant to keep a task moving when its scheduler
can't run it or is failing could themselves leave it stuck
- The stall watchdog could blame a scheduler for time it wasn't
responsible for, even taking down all of sched_ext
- Sub-schedulers were told late, or not at all, which CPUs they could
use
- Deferred callbacks could stay queued or be dropped under core
scheduling and proxy execution, with a sched/core fix to the
duplicate check
- Hardening: tracing programs can no longer call kfuncs that could
deadlock the CPU"
* tag 'sched_ext-for-7.4-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (34 commits)
sched_ext: Reject scx_bpf_sub_grant/revoke() in tracing programs
sched_ext: Fix the rescue overload pick ejecting the wrong sub under bypass
sched_ext: Fix use-after-free of a NONE rescuee leaving its sched while running
sched_ext: Fix a lone IMMED task waiting behind a kept prev
sched_ext: Fix queued bypassed tasks stalling when the host loses SCX_CAP_BASE
sched_ext: Let the bypass load balancer move tasks homed on other cpus
sched_ext: Factor out bypass_lb_move() and insert with the donor DSQ unlocked
sched_ext: Fix the bypass load balancer stranding tasks on cpus where the host lacks SCX_CAP_BASE
sched_ext: Fix bypassed tasks stalling when the host lacks SCX_CAP_BASE
sched_ext: Expire prev's slice in dispatch_one() under bypass
sched_ext: Fix a cpu idling after put_prev_task_scx() queues its task on the local DSQ
sched_ext: Refuse extending a kernel slice
sched_ext: Mark the rescue grant as a kernel slice with SCX_TASK_KERNEL_SLICE
sched_ext: Fix a cpu idling with a runnable rescuee
sched_ext: Restore a rescuee's slice after ops.stopping()
sched_ext: Fix rescue served time disagreeing with the slice under proxy execution
sched_ext: Move refill_task_slice_dfl() below apply_slice_vtime()
sched_ext: Fix bypassed tasks looping on a host without SCX_CAP_ENQ
sched_ext: Fix bypassed tasks stranded by the reenqueue limit
sched_ext: Clear holding_cpu where the claim is confirmed
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull more MM updates from Andrew Morton:
- "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL" (Lorenzo
Stoakes):
Replace VM_SPECIAL with helpers describing mapping ownership and
behavior. Enforce the mmap flag rules and convert affected drivers,
removing ambiguous tests shared by driver mappings and ordinary
memory.
- "support kselftest on nommu platform" (Hajime Tazaki):
Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner
to avoid its fork-dependent timeout command and document the build
and runtime requirements.
- "mm/collapse: separate a collapse from its callers" (Kiryl
Shutsemau):
Separate huge-page collapse from caller policy. Replace khugepaged
identity checks with explicit permissions, and separate scanning from
collapse with the mmap_lock drop in one place.
- "mm: implement and use vma_has_anon_rmap(), silence KCSAN" (Lorenzo
Stoakes):
Provide a common helper for testing whether a VMA has anonymous
reverse mappings. Document the locking rules, annotate intentional
concurrent reads for KCSAN, and convert the core MM callers.
- "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups"
(David Hildenbrand):
Remove SECTION_MARKED_PRESENT in favor of the early-section flag.
Simplify sparse memory initialization, section metadata handling and
related PFN validity checks.
- "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport):
Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have
been unusable since the NO_BOOTMEM conversion, leaving only the code
and data overhead.
- "mm: Unify device DAX and HugeTLB vmemmap population paths" (Muchun
Song):
Unify the generic device DAX and HugeTLB vmemmap population paths and
shared tail-page handling. Remove the duplicate DAX machinery while
retaining the powerpc radix implementation.
- "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum
Yun):
Fix huge-page split tests racing with khugepaged recollapse. Use
pagemap and page flags instead of VMA-boundary assumptions which fail
when glibc has already enabled THP.
- "memcg: move memcgid refcount to objcg to unpin dying memcgs"
(Bingfang Guo):
Let dying memcgs be freed while swapped-out shmem still holds their
IDs. Move ID references to reparentable objcgs, with the necessary
swap accounting and RCU lifetime changes.
- "mm/truncate: fix data loss when truncating straddling large folios"
(Zhang Yi):
Preserve out-of-range data when truncating straddling large folios.
Fix a lookup race and split-failure handling, and handle minimum
folio orders for the forthcoming ext4 iomap buffered I/O path.
- "mm/damon: introduce damos quota goal target metric complement flag"
(SJ Park):
Let DAMOS quota goals use a metric's complement, so utilization can
also express free-memory goals without adding separate metrics. Add
the sysfs interface, documentation and tests.
- "arm64/mm: Standardize printing for pgtable entries" (Anshuman
Khandual):
Use the generic formatter for arm64 page-table entries, ready for
future wider entries. Move the safe-PTE-update check out of the
header to avoid an include cycle.
- "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk):
Fix KMSAN metadata leaks on iounmap and failed ioremap allocations.
Free backing pages before removing their mappings, and share the
teardown with partial-allocation cleanup.
- "mm/damon/core: preserve quota state when constructing schemes" (Karl
Mehltretter):
Preserve running DAMOS quota state while validating parameter
updates. Even rejected updates could reset accounting and exceed the
quota window; add tests for both accepted and rejected updates.
- "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu):
Fix vmalloc_dump_obj() lookups for last-page addresses and
allocations spanning vmap zones. These allocations were being missed
in diagnostic output.
- "mm: various (urgent?) fixes" (David Hildenbrand):
Fix mmap charge leaks and hugetlb use-after-free, wrong-address
unmapping, stuck locks and endless faults. Also preserve reservations
on failed mremap and handle stale userfaultfd bits.
- "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris
Charisis):
Fix pointer and metadata confusion during RCU destruction of
maple_range_64 nodes, avoiding double frees and invalid accesses. Add
a regression test; no in-tree user currently uses this combination.
- "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu
Li):
Fix hugetlb migration and preallocation accounting so /proc/vmstat
retains the correct count and memory.numa_stat follows the folio to
its new node.
- "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park):
Fix 32-bit DAMON age overflow which made old regions look young.
Share hugetlb folio acquisition code and add a regression test for
the previously fixed last-region skipping bug.
- "mm/damon/core: cleanup code, reduce stack usage, and add kunit" (SJ
Park):
Reduce kdamond stack usage and remove unused enums. Add deterministic
PSI quota-goal tests which do not depend on the machine's current
memory pressure.
- "mm/damon: fix the temporal goal tuner's size quota conversion"
(Donggeun Yoo):
Bound the temporal tuner's quota conversion to prevent overflow and
zero effective quotas, which could stop schemes making progress. Add
boundary tests.
- "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He):
Remove redundant GFP_DMA flags from coherent DMA allocation calls.
The DMA API already selects the zone from device limits and ignores
the caller's flag.
- "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum
Yun):
Propagate child-process failures in the khugepaged tests and prevent
background collapse from racing their checks. This removes false
passes and intermittent failures.
- "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM"
(Enze Li):
Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds
the right maintainers. Also label the closing hugetlb_cgroup.h
include guard.
Summary of the 65 singletons in this merge:
The singletons include a few worthwhile performance fixes. Avoiding
unnecessary reverse-map tree updates gave a reported 15.9% improvement
in UnixBench's execl test on a 192-core x86 system. A reclaim retry
fix stops defrag_mode from spinning under memory.low protection; a
reported file-creation stress test ran 5.7 times faster, with far less
memory pressure. Narrower hugetlb invalidations also spare KVM guests
needless refaults: the reported Redis-in-VM P99 QPS penalty while
punching holes fell from 7.09% to 1.45%.
Several fixes prevent lost data or corrupted memory. Lazyfree
huge-page races with mremap()/munmap() could lose data or access freed
memory, and MADV_COLD could lose a concurrent write's dirty bit.
Another fix prevents stale migration entries from outliving large
folios. On filesystems with blocks smaller than a page, extending a
file by 4 GiB or more could expose bytes written past the old EOF
through mmap instead of zeroes. That is fixed, along with
use-after-free races in PAGEMAP_SCAN during process exit and in
memory-cgroup accounting on PREEMPT_RT kernels.
Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios,
which could wedge other memory operations, and stop mremap() leaking
commit charges until allocations fail under strict overcommit. A
further fix keeps userfaultfd tracking intact on neighboring mappings.
NUMA binding no longer prevents long-term pins when the chosen node
has only movable memory, fixing QEMU/VFIO startup and io_uring buffer
registration failures. MGLRU also gets a fix for early-boot crashes
with memory cgroups disabled at boot, notably in small crash kernels.
* tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (213 commits)
mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline
mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root
mm/memcontrol: publish null pointer before dropping css reference
test_maple_tree: test maple_range_64 node in RCU mode
maple_tree: fix maple_range_64 crashes in RCU mode
mm/vma: avoid broken uffd merge on mremap
tools/testing/shared: catch a failed posix_memalign()
mm/damon/core: remove unreachable damos charge_{target,addr}_from
mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache
riscv: mm: exclude invalid THP PMDs from page table check
mm/damon/core: filter out damos address filter-split failed region
fs/proc/task_mmu: pin mm_users in do_pagemap_scan()
mm/mempolicy: apply bind policies against the scoped allocation zone
mm/vmscan: reclaim every zone in per-node proactive reclaim
mm/mremap: correctly uncharge accountable VMAs in error path
mm/mremap: populate calculated fields in vrm state correctly
MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP
mm: bdi: reset min_ratio after the sysfs attributes are gone
mm/page_vma_mapped: continue past nonmatching huge PMDs
selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers
...
|
|
A tracing program can call scx_bpf_sub_grant() and scx_bpf_sub_revoke()
through scx_kfunc_ids_any, which is registered on the tracing kfunc hook.
Before returning, both invoke ops.sub_caps_updated() of the sub-schedulers
whose caps changed, so those ops run in the tracing program's context. No
SCX op expects to run from an arbitrary tracing context.
With an rq lock held, this leads to the same deadlock that b07a7450f6ef
("sched_ext: Reject rq-lock-taking kfuncs in tracing programs") fixed for
direct calls:
1. A tp_btf/sched_switch program fires from __schedule() with the rq lock
held. scx_locked_rq() only tracks rq locks held across SCX callbacks, so
it returns NULL there.
2. The program calls scx_bpf_sub_grant() on a child, which invokes the
child's ops.sub_caps_updated() before returning.
3. That op calls scx_bpf_cidperf_set() on this cpu, which sees no locked rq
and takes the rq lock the cpu already holds.
Add both kfuncs to scx_kfunc_ids_no_tracing and rewrite the set's comment to
cover both reasons for membership: taking an rq lock and running SCX ops.
Fixes: 5f2a9a4c2e6c ("sched_ext: Add coalescing sub_caps_updated() notifier for sub-schedulers")
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
An earlier patch sends a bypassed task to rescue when its host lacks
SCX_CAP_BASE on the cpu. The task's rescue time is charged to its owner, the
bypassing sub, which is usually exiting. The overload pick skips exiting
subs and ejects an unrelated sub or nobody. The host runs the task and lacks
BASE, but carries none of the charge.
Charge the sched running the rescuee, the host while the owner bypasses.
SCX_EV_SUB_RESCUE already counts on that sched and the stall watchdog blames
it.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Since an earlier patch sends a bypassed task to rescue when its host lacks
SCX_CAP_BASE, a task scx_fail_parent() punted to its bypassing parent at
NONE can be a rescuee.
When a cgroup migration into a root-owned cgroup re-homes it while running,
nothing resets it: the DEQUEUE_SAVE dequeue and put_prev_task_scx() keep a
running task's slice, and __scx_disable_and_exit_task() skips a NONE task.
rq->scx.rescue.curr keeps pointing at the task, now on the root. An RT task,
for example, can keep the task from serving its grant until the last sub is
freed. From then on scx_rescuee() returns NULL and the rescue never ends.
scx_rescue_timerfn() keeps accessing the task after it is freed.
Move the reset out of scx_disable_task() into scx_reset_task_fields() and
call it on every path that takes a task out of its sched's control, the exit
of a NONE task included.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
While dispatch is in progress, rq_is_open() returns true and dsq_inc_nr()
schedules no reenqueue for an IMMED task inserted on an empty local DSQ. The
check at the end of dispatch_one() schedules the reenqueue only when more
than one task is queued, so a dispatch that keeps prev leaves a lone IMMED
task behind:
1. a parent's ops.dispatch() sub-dispatches into a child, which consumes a
task with SCX_ENQ_IMMED onto the empty local DSQ
2. the parent extends prev's slice and dispatch keeps prev
3. the local DSQ holds one task, so no reenqueue is scheduled
The IMMED task waits on the local DSQ behind prev.
Schedule the reenqueue when dispatch keeps prev and an IMMED task is queued,
as even a lone IMMED task can't run right away behind prev.
v2: The description and dispatch_one()'s comment are reworded.
Fixes: 98d709cba319 ("sched_ext: Implement SCX_ENQ_IMMED")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A task already queued on a bypass DSQ stalls when the host loses
SCX_CAP_BASE on that cpu, as the cpu no longer consumes the DSQ. Sending a
bypassed task to rescue at enqueue time covers only a task enqueued after
the loss, and the balancer no longer donates to a capless cpu.
Have the bypass load balancer drain such a DSQ: a cpu where the host lacks
BASE is a donor with a target of zero, and the donees take its tasks under
their ordinary fill.
A drained task no donee can take is reenqueued at its home: to rescue when
the host lacks BASE there and rescue is enabled, and to the home's bypass
DSQ otherwise.
A balancing donor leaves a task that can't move to its donee where it is.
Reenqueueing it at home would put a task homed on the donor cpu at the tail
of its own DSQ, and a blocked task that proxy execution migrated there,
which every remote cpu refuses, would go to the tail on every pass.
Without rescue, a task homed on the drained cpu goes back onto the drained
DSQ, once per pass, and waits for the host to regain BASE, the bypass to
end, or the stall watchdog to eject the host. Its reenqueues are not counted
against the reenqueue limit, as its scheduler is bypassing.
A BASE loss kicks the balancer so that the drain doesn't wait for the next
interval. With a zero interval the kick is the only pass, and a task that
lands on the drained DSQ after it waits the same way.
v2: The description drops an in-flight affinity change from the causes of a
refused balancer move.
v3:
- The description says that a stuck task's reenqueues are not counted
against the reenqueue limit (sashiko).
- The description names the stall watchdog as the way out for a task stuck
without rescue, and the tasks a zero interval leaves on a drained DSQ.
- bypass_lb_reenq_home()'s kerneldoc and the description give the placement
with rescue disabled and for a task that must run at home, and the other
two comments on that placement point to the kerneldoc.
v4:
- Skip a task the donee would refuse on a balancing donor instead of
reenqueueing it at home (Andrea).
- bypass_lb_reenq_home()'s kerneldoc is corrected on a migration-disabled
task homed on a donee and on the local DSQ admission.
- The description is condensed.
Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The following patch uses the bypass load balancer to drain a cpu's bypass
DSQ when the host loses SCX_CAP_BASE there. The host no longer consumes that
DSQ, so the drain has to move every task on it. The balancer skips the ones
homed on another cpu.
An earlier pass moved such a task in from its home cpu's DSQ, and the donor
cpu has not consumed it yet. A task's DSQ membership may only change under
the task's rq lock, which the balancer does not hold.
Take such a task off the donor DSQ with unlink_dsq_and_switch_rq_lock(), the
handshake a remote consumer uses. It drops the DSQ lock and switches to the
task's rq lock. A dequeue that gets in between takes the task, and the
balancer moves on. The insert then runs under the task's home rq lock, as it
already does for a donor-homed task.
A task whose donee is its own cpu can always run there.
task_can_run_on_remote_rq() warns when asked about a task's own cpu, so the
balancer skips the test for such a task.
The walk holds the donor rq lock, which keeps the donor cpu's dispatch and
its enqueuers away from the donor DSQ. A move of a task homed elsewhere
switches to the task's rq lock, and both can reach the DSQ meanwhile. The
parked cursor keeps the walk's place through a consume, and the sequence
bound keeps a task enqueued meanwhile out of the pass.
v2: The description says "the following patch", not "patches", uses the
balancer to drain a DSQ.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Prepare the bypass load balancer for moving tasks homed on other cpus. Such
a move takes the task's own rq lock through the remote consumer's handshake,
which drops the donor DSQ lock on the way, so its insert runs with the donor
DSQ unlocked and under the task's rq lock. The balancer's own move holds the
donor DSQ lock throughout, with the donee DSQ lock nested inside.
Switch the donor-homed move to the same scheme so that one body can serve
both: factor it out of bypass_lb_cpu() into bypass_lb_move(), which takes
the task off the donor DSQ and inserts with the DSQ unlocked. A task that
can't run on the donee goes back to the head of the donor DSQ instead of
being skipped in place.
The walk parks its cursor behind the task before each move and resumes from
it afterwards. It is bounded to the tasks queued when it began, by their DSQ
sequence numbers, so a task it put back is not visited again in the same
pass.
Before, the walk held both the donor rq lock and the donor DSQ lock across
every move. The donor DSQ can be very long and contended, a walk competing
with the cpu's dispatch and the enqueues for the DSQ lock would crawl, and
the walk is what relieves that contention by moving tasks off the DSQ. The
walk now cycles the DSQ lock around each insert but still holds the rq lock
throughout, and that alone keeps dispatch and enqueues away. The DSQ lock is
uncontended when retaken, so cycling it is cheap.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
host lacks SCX_CAP_BASE
The bypass load balancer spreads a host's bypassed tasks over the cpus of a
node by moving them between its per-cpu bypass DSQs. It counts every cpu of
the node as capacity and takes any cpu below the target as a donee. A cpu's
dispatch only descends into schedulers holding SCX_CAP_BASE on it, so a
bypass DSQ on a cpu where the host lacks BASE is never consumed. A task
moved there sits until the host regains the cap or the bypass window ends.
Count only cpus where the host holds BASE as capacity and pick donees from
them. The target rises to spread the node's tasks over the cpus that can run
them, and a capless cpu receives nothing. A capless cpu with tasks is still
an ordinary donor, offloaded down to the donor target when its queue grows
past the threshold; draining it is left to the following patches.
Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A bypassed task stalls on a cpu where its host lacks SCX_CAP_BASE. The host,
the nearest ancestor that is not bypassing, runs a bypassing sub-scheduler's
tasks through its per-cpu bypass DSQs. A cpu's dispatch only descends into
schedulers holding BASE on it, so the host's bypass DSQ on such a cpu is
never consumed until the host regains the cap.
Resolve a bypassed task against its host at enqueue time. If the host lacks
BASE on the cpu, send the task to rescue on that cpu, which runs it at
bounded bandwidth without the host's dispatch having to reach it.
The enqueue does not look above the host for an ancestor that holds BASE.
That ancestor would run the task on cpus it never granted the host.
With rescue disabled the task goes to the host's bypass DSQ as before and
stalls there until the host regains BASE or the bypass ends, or until the
stall watchdog ejects the host, which it blames for a task stuck on its DSQ.
A task that must run on the cpu, migration-disabled or with a migration
pending, is admitted to the local DSQ instead, as any insert of such a task
is.
Bypass also used to end a rescue, sending a preempted rescuee to the bypass
DSQ with every other preempted task. Now that rescue is how a bypassed task
runs while its host lacks BASE, keep a preempted rescuee on the local DSQ
under bypass too. dispatch_one() also keeps such a rescuee running, as an
earlier patch's bypass expiry leaves a kernel slice alone. The grant stays
bounded by served time.
A task scx_fail_parent() punted to its bypassing parent at NONE can be a
rescuee too. switched_from_scx() ends its slice as it does for an enabled
task, so that a NONE rescuee leaving sched_ext while running doesn't leave
rq->scx.rescue.curr pointing at it.
While at it, the rescue test in __scx_resolve_local_dsq() goes through the
new scx_rescue_enabled() helper.
v2:
- The description notes that dispatch_one() keeps a bypassed rescuee too,
through the new patch before this one (sashiko).
- scx_rescue_keep()'s kerneldoc drops dispatch_one() declining a rescuee
with slice left, which the new patch removes.
v3:
- With rescue disabled, a task that must run on the cpu is admitted to the
local DSQ instead of waiting on the host's bypass DSQ.
- The description names the stall watchdog as the way out for a task stuck
without rescue.
v4:
- switched_from_scx() ends the slice of a NONE task, which can now be a
rescuee (sashiko).
- scx_punt_task()'s kerneldoc no longer says that switched_from_scx() skips
a NONE task.
- scx_bypass()'s kerneldoc and the ops.sub_ecaps_updated() doc say where a
bypassed task can be besides the bypass DSQs.
Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The next patch keeps a bypassed rescuee on the local DSQ and needs
dispatch_one() to keep it running too. dispatch_one()'s keep test refuses
any prev whose scheduler is bypassing, so another local task could run ahead
of the rescuee, even an escalated SCX_TASK_PROTECTED one. The test exists
because a cpu-form scheduler can write p->scx.slice directly from BPF, so
the slice can't be trusted under bypass.
The refusal doesn't hold anyway. Since 025b1bd41965 ("sched_ext: Implement
hierarchical bypass mode"), the host's ops.dispatch() runs while prev's
scheduler bypasses, and the post-dispatch test in scx_dispatch_sched() keeps
a queued prev with slice left without checking bypass. A task dispatched in
between waits behind prev until the next bypass tick zeroes the slice, at
most a tick.
Expire the slice before the keep test instead when prev's scheduler is
bypassing, as the bypass tick would. Neither test keeps prev on a zeroed
slice, so prev rotates. A kernel slice survives the expiry and keeps prev.
Only a sub-scheduler's task can hold one, and sub-schedulers are cid-form,
which rejects direct slice writes, so it can be trusted.
Drop the stale pointer to scx_disable_workfn(), update scx_bypass()'s
kerneldoc, and add a TODO noting that the bypass slice handling can be
cleaned up once the cpu form is deprecated and the kfuncs refuse slice
raises from a bypassing scheduler.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
local DSQ
put_prev_task_scx() enqueues a still-runnable task when the pick has settled
on idle. The insert can land the task on the cpu's own local DSQ, and then
the cpu must not go idle over it. 7a9df7667963 ("sched_ext: Fix stall when a
task enqueued with SCX_ENQ_LAST lands back on its CPU's local DSQ")
reschedules the idle task in that case, but only after the SCX_ENQ_LAST
insert.
The plain insert, taken when the task can't stay on the cpu, can land on the
local DSQ too: rescue can admit a task there when its scheduler lacks
SCX_CAP_BASE on the cpu. Without the resched the cpu idles with the task
queued until an unrelated wakeup or the rescue timer's escalation.
Reschedule after either insert. In the following patches, the plain insert
becomes the common path for bypassed tasks under rescue.
v2: The description's forward reference to the plain insert becoming the
common path no longer names the following patch.
Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A rescuee can keep running after its grant is served, on a cpu its scheduler
has lost. The write rules deny a slice extension only where the scheduler
lacks SCX_CAP_BASE on the cpu, and a scheduler holding BASE but not ENQ can
have a task under rescue there:
1. the scheduler extends the rescuee's slice
2. BASE is revoked. The cap-revoke eviction skips the rescuee.
3. the grant is served and scx_rescue_charge() ends the rescue
4. the task runs out the extension on the cpu its scheduler lost, outside
the rescue's bandwidth
A scheduler shortens or zeroes a rescuee's slice to preempt it, and
scx_rescue_keep() resumes the grant afterwards. An extension serves no such
purpose.
Refuse extending a kernel slice in both scheduler write paths, the direct
write in scx_bpf_task_set_slice() and the out-of-band application, counted
in SCX_EV_SLICE_DENIED like an extension without BASE. Shortening stays
allowed.
scx_rescue_keep()'s restore of the remainder raises the slice from zero.
Like the kernel's other slice writes, it goes through scx_set_task_slice()
and not the scheduler write paths.
v2:
- scx_rescue_admit()'s kerneldoc no longer mentions a served-time snapshot.
- SCX_EV_SLICE_DENIED's comment lists every refusal it counts.
- scx_bpf_task_set_slice()'s kerneldoc says that shortening is refused on a
protected slice.
Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
In the following patches, bypassed tasks run under rescue when their host
lacks SCX_CAP_BASE on the cpu. Under bypass the kernel zeroes the running
task's slice at every tick, as it does not trust the scheduler's slices.
A rescue grant has to survive that, or it is served a tick at a time with
the rescuee sent to the tail of the local DSQ each time. The generic slice
handling can't tell a kernel grant from a scheduler's slice.
Mark the grant as a kernel slice with SCX_TASK_KERNEL_SLICE, set at rescue
admission and cleared when the slice ends. The generic slice handling leaves
it alone. The default refill does nothing for it, and the three zeroings go
through the new expire_task_slice(), which skips it:
- the bypass tick's zeroing of the running task
- the eviction of a running task that can't stay on the cpu when bypass
lifts
- the cap-revoke eviction of a running task
Schedulers keep their control. Preemption and scx_bpf_task_set_slice() still
apply, and the rescue stays bounded by served time.
The sites that recognized the rescuee by identity, the reenqueue scan and
put_prev_task_scx()'s reinsert, test the flag instead.
scx_task_has_kernel_slice() hides the test behind the scx_has_subs() static
key, as only a sub-scheduler's task can carry the flag. The flag's entries
join the enum tables in tools/sched_ext.
No kernel slice reaches a zeroing until bypassed tasks run under rescue. The
cap-revoke eviction skipped the rescuee already, and bypass entry ends an
open rescue, so the bypass tick and the eviction at bypass lift see none.
The refill is the one behavior change. The pick refills a zero slice only
for a task that can stay on the cpu, so it concerns a rescuee whose
scheduler holds BASE there. A scheduler may zero such a queued rescuee's
slice, and the next pick used to refill it to the default.
Now the rescuee runs on the zero slice until the next tick. There
put_prev_task_scx() restores the remainder when another task is runnable.
When nothing else is runnable, dispatch_one() keeps the rescuee as the last
task. The pick then ends its zero slice, which ends the rescue, and refills
the default. A scheduler with SCX_OPS_ENQ_LAST skips the keep-last test.
scx_rescue_keep() restores the remainder there instead and the rescue goes
on.
v2:
- scx_rescue_admit()'s kerneldoc no longer mentions a served-time snapshot.
- The description covers a scheduler with SCX_OPS_ENQ_LAST, whose rescuee
dispatch_one() keeps on the remainder.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A scheduler preempts a running rescuee like any other task by zeroing its
slice. The rescue grant survives the preemption. put_prev_task_scx()
restores the unserved remainder through scx_rescue_keep() and queues the
rescuee at the tail of the local DSQ, behind whatever the scheduler wanted
to run first.
That breaks down when the scheduler zeroes the slice but dispatches nothing.
dispatch_one() keeps the previous task running only with slice left or, when
nothing else is runnable, with its scheduler holding SCX_CAP_BASE on the cpu
and not taking the last task through SCX_OPS_ENQ_LAST. A rescuee fails both
once its slice is zeroed if its scheduler lacks BASE there or takes the last
task:
1. ops.tick() or scx_bpf_task_set_slice() zeroes the rescuee's slice
2. the pick finds nothing else to run and settles on idle
3. put_prev_task_scx() restores the remainder and queues the rescuee on the
local DSQ
4. nothing reschedules the cpu, as the pick has settled and __schedule()
clears need_resched right after it
The cpu idles with the rescuee queued until an unrelated wakeup or the
rescue timer's escalation.
Restore the remainder from dispatch_one() too. When nothing else is
runnable, its last keep now keeps the rescuee running on the remainder, so
the pick never settles on idle with a rescuee runnable. put_prev_task_scx()
keeps handling a displaced rescuee, where another task won the pick.
A rescuee that passes the existing keep-last test is kept there instead and
continues as an ordinary task, its rescue ended with the zeroed slice. That
is fine as its scheduler holds SCX_CAP_BASE on the cpu.
The new keep also covers bypassed tasks run under rescue in the following
patches.
v2: The dispatch_one() comment and description now say that a rescuee whose
scheduler holds SCX_CAP_BASE on the cpu passes the keep-last test and
continues as an ordinary task (sashiko).
Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
put_prev_task_scx() restores a preempted rescuee's unserved slice through
scx_rescue_keep() before it calls ops.stopping(). A rescuee whose slice the
scheduler zeroes there is left to the reinsert instead:
1. ops.stopping() zeroes the rescuee's slice
2. the reinsert condition special-cases the rescuee and keeps it on the
local DSQ with a zero slice
3. the pick's default refill supplies a slice, or the task runs on a zero
slice until the tick when its scheduler lacks BASE
Move the keep after the stopping call so that scx_rescue_keep() restores the
remainder whether a preemption or ops.stopping() zeroed the slice, and drop
the special case from the reinsert condition as a rescuee reaching it always
has a slice. A rescuee whose slice ops.stopping() zeroed now goes to the
tail of the local DSQ with its remainder restored, behind what the scheduler
wanted to run first, instead of staying at the head on a zero slice.
The slice end in the same block moves with it, which changes nothing for a
protected task. Its slice is the unserved remainder, and scx_rescue_charge()
ends the protection in the same update_curr_scx() call that zeroes the
slice, so the block never sees a protected task in either position.
v2: The description no longer says that a protected task's protection
outlasts ops.stopping().
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
execution
Rescue measures served time as the rescuee's sum_exec_runtime minus a
snapshot taken at admission. Under proxy execution that measure follows the
wrong task. update_se() credits sum_exec_runtime to rq->curr, the mutex
owner that executes, while update_curr_scx() drains the slice of rq->donor
and charges its rescue. A rescuee can be either, so its served time and its
slice disagree in both directions.
As the owner running for another donor, the rescuee accrues served time that
drained no slice and charged no budget, and its rescue ends early. As the
donor, its rescue runs long:
1. the rescuee blocks on a mutex and the owner runs in its place
2. the owner's run drains the rescuee's slice and the budget, but not the
served time
3. the slice runs out with the rescue still open, and scx_rescue_keep()
restores a remainder that the owner's run didn't reduce
4. the owner keeps running on the rescue until it releases the mutex, past
the admitted slice and overdrawing the budget
Count served time in scx_rescue_charge() from the delta that drains the
slice. Charging the owner's run to the donor's rescue matches proxy
execution, which drains the donor's slice and accounts the cgroup time to
the donor for the owner's run.
scx_rescue_keep() also ignored the return of scx_set_task_slice(). The call
refuses the restore on a rescuee that escalation marked SCX_TASK_PROTECTED,
and the rescuee was kept with a zero slice and its protection set.
That refusal is now unreachable. Escalation sets the slice to the unserved
remainder before it protects the task. From then on every slice write a
scheduler can reach is refused. Only a sub-scheduler's task is rescued, and
sub-schedulers are cid-form, which rejects direct slice writes.
The slice then moves only by what update_curr_scx() drains, and that drain
charges the rescue by the same amount, so the slice runs out exactly when
the charge ends the rescue. WARN and end the rescue if the restore is
refused anyway.
Fixes: 5fd501744b10 ("sched_ext: Add bandwidth-limited rescue execution for stranded tasks")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Move refill_task_slice_dfl() below apply_slice_vtime(), with the other slice
writers, to prepare for the kernel-slice handling the following patches add
to them. No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Bypass exists so that runnable tasks make progress without trusting the BPF
scheduler. A host with SCX_CAP_ENQ_IMMED (BASE) but not SCX_CAP_ENQ on a cpu
cannot run a single bypassed task there. The task loops instead:
1. a bypassed task lands on the host's bypass DSQ for the cpu
2. the cpu's dispatch descends into the host, which holds BASE there
3. the host consumes the task into the local DSQ as a plain insert
4. the insert needs ENQ and the host lacks it, so the task is rejected
5. the reject drain reenqueues the task and it is back at step 1
A bypass consume puts a task on the cpu being dispatched for, to run right
away. That is the IMMED contract, and BASE is the cap it needs. Consume with
SCX_ENQ_IMMED. BASE is then the one cap a bypassed task ever needs. Dispatch
descends into a host only on a cpu where the host holds BASE, so a consume
never lacks it.
A consumed task is first on the local DSQ when dispatch enters the host with
the DSQ empty. That is always the case for the root and normally for a
parent dispatching into a child.
A consumed task that cannot run right away is reenqueued like any IMMED task
and goes back to the bypass DSQ. Waiting on the local DSQ would need ENQ.
The task cannot run right away when a higher class or a HEAD insert took the
cpu meanwhile, or when the parent queued its own task before dispatching
into the host.
Each such task waits on the bypass DSQ for the next dispatch. The reenqueue
is not counted against the reenqueue limit, as the task's scheduler is
bypassing.
v2: The description says that the reenqueue of a consumed task is not
counted against the reenqueue limit (sashiko).
Fixes: 701b7bcad868 ("sched_ext: Add the SCX_CAP_ENQ cap")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_do_enqueue_task() counts a task's reenqueues until the task runs. Past
SCX_REENQ_MAX_REPEAT it ejects the owning scheduler and leaves the task off
every DSQ for the scheduler's exit to requeue. The count comes before the
bypass test, so it also counts the reenqueues of a task whose scheduler is
bypassing.
Those reenqueues are not the scheduler's placements, and ejecting the
scheduler for them fixes nothing. Under bypass the kernel queues the
scheduler's tasks on the bypass DSQs. A host calling
scx_bpf_reenqueue_local() reenqueues a bypassed child's tasks on the cpu's
local DSQ along with its own, and the reject drain reenqueues a consumed
task that a cap check rejected.
In the following patches, the consume itself reenqueues a task it can't run
right away, and the bypass load balancer reenqueues a task at its home.
The count is reset only when the task runs or leaves sched_ext, so an
ejection can lead to a stranding the exit does not recover:
1. the task's reenqueues pass the limit, which ejects its scheduler and
strands the task
2. the exit's scx_bypass() cycles the task through dequeue and enqueue,
which requeues it on a bypass DSQ with the count still past the limit
3. the task's next reenqueue under bypass strands it again
4. scx_bypass() requeues stranded tasks only when the scheduler wasn't
bypassing yet, so the task stays off every DSQ until the disable walks
the tasks
5. the disable takes scx_enable_mutex, the cgroup lock and scx_fork_rwsem
before the walk, and a sub-scheduler's disable the cpu hotplug lock too.
It hangs if the stranded task holds one of them, forking, attaching to a
cgroup or doing a cpu hotplug
Test bypass before the count. A task reenqueued under bypass stays queued
until a later dispatch or balancer pass, so there is no tight loop to bound,
and the task's first run resets the count. The offline test moves up with it
to stay ahead of the bypass test. A reenqueue on an offline cpu goes to the
local DSQ and isn't counted either.
A host that keeps calling scx_bpf_reenqueue_local() could get a bypassed
child ejected, as the child's tasks on the local DSQ counted against the
child. The reenqueues are the host's, and its own tasks still count against
it.
v2: The description lists the cpu hotplug lock among the locks a
sub-scheduler's disable takes before the walk.
Fixes: 7706d6e4f2e3 ("sched_ext: Bound per-task reenqueues and eject the owning scheduler")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The bypass load balancer will take tasks off a remote cpu's bypass DSQ with
unlink_dsq_and_switch_rq_lock() in the following patches and insert them
into another bypass DSQ. Clearing holding_cpu after the claim is up to each
claimant, so the balancer would need a clear of its own.
unlink_dsq_and_switch_rq_lock() marks a task with holding_cpu for the gap
between dropping the DSQ lock and taking the task's rq lock, so that a
dequeue in that gap can tell the claimant it lost. Once the claimant holds
the rq lock and finds the marker intact, nothing can take the task away
under it and the marker has no reader left.
Each claimant clears it at its own site after that point:
- dispatch_to_local_dsq(), which runs the handshake inline, in its same-rq
and fallback branches
- scx_proxy_reject_task() for the proxy reject
- the dequeue inside deactivate_task() for both remote moves
Clear it where the claim is confirmed instead and drop the explicit clears.
No behavior change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
scx_bpf_cpuperf_set() and scx_bpf_cidperf_set() are members of
scx_kfunc_ids_any, which is registered for BPF_PROG_TYPE_TRACING.
A tracing program can run in a context which already holds the target
rq lock, in which case scx_cpuperf_set() attempts to acquire the same
lock again:
__schedule()
rq_lock(rq); /* rq->__lock is held */
trace_sched_switch();
BPF tracing program
scx_bpf_cpuperf_set(smp_processor_id(), 1024);
scx_cpuperf_set();
locked_rq = scx_locked_rq(); /* NULL */
rq_lock_irqsave(rq, &rf); /* same lock */
scx_locked_rq() only tracks rq locks held during SCX callback
invocations, so it cannot detect an rq lock held by the surrounding
kernel code.
The deadlock can be reproduced with two BPF programs. The first is a
minimal SCX scheduler, which only needs to be registered and enabled:
SEC(".struct_ops.link")
struct sched_ext_ops minimal_ops = {
.name = "minimal",
};
The second is a tracing program attached to tp_btf/sched_switch,
which fires from __schedule() with rq->__lock already held:
SEC("tp_btf/sched_switch")
int BPF_PROG(on_sched_switch, bool preempt, struct task_struct *prev,
struct task_struct *next, unsigned int prev_state)
{
scx_bpf_cpuperf_set(bpf_get_smp_processor_id(), 1024);
return 0;
}
With both programs attached, the next context switch on any CPU
deadlocks that CPU. Lockdep reports:
WARNING: possible recursive locking detected
7.3.0-rc7 #3 Not tainted
swapper/3/0 is trying to acquire lock:
(&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x1e/0x30
but task is already holding lock:
(&rq->__lock){-.-.}-{2:2}, at: __schedule+0xeb/0x1430
Possible unsafe locking scenario:
CPU0
----
lock(&rq->__lock);
lock(&rq->__lock);
*** DEADLOCK ***
Call Trace:
<TASK>
print_deadlock_bug.cold+0xc0/0xd0
__lock_acquire+0x12fa/0x1b80
lock_acquire+0xd1/0x2f0
_raw_spin_lock_nested+0x32/0x40
raw_spin_rq_lock_nested+0x1e/0x30
scx_cpuperf_set+0x1e9/0x310
scx_bpf_cpuperf_set+0x9b/0x230
bpf_prog_..._on_sched_switch+0x3d/0x41
bpf_trace_run4+0x125/0x450
__traceiter_sched_switch+0x4b/0x70
__schedule+0x94a/0x1430
schedule_idle+0x26/0x50
cpu_startup_entry+0x2d/0x30
start_secondary+0xfc/0x100
</TASK>
The CPU remains locked up with interrupts disabled.
scx_bpf_cidperf_set() is similarly affected, as it resolves the CID
through scx_cid_to_cpu() before reaching the same rq-locking path.
There is no reliable way for these kfuncs to determine whether the
surrounding tracing context already holds an rq lock. Reject calls to
both kfuncs from tracing programs at verification time, while
preserving their availability to SCX struct_ops and syscall programs.
Fixes: a11d6784d731 ("sched_ext: Fix missing rq lock in scx_bpf_cpuperf_set()")
Signed-off-by: Junbin Zhang <sprooc69@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
bypass
While a sched is bypassing, its tasks are run by its host, the nearest
ancestor that isn't bypassing. check_rq_for_timeouts() tests a runnable task
against its owner's timeout. Since 2c91169377f4 ("sched_ext: Blame the DSQ's
owning scheduler for a runnable stall"), it ejects the sched owning the DSQ
the task waits on. The sched tested, the sched ejected and the sched running
the task can all differ.
- Timeouts are per sched. A bypassing sub-scheduler with a 2s timeout under
a root with the default 30s has its tasks on the root's bypass DSQ. The
root, not bypassing itself, consumes that DSQ before calling
ops.dispatch() only on every other pass, and on the others only after
ops.dispatch() yields nothing. A task that waits 2s there ejects the root,
disabling all of sched_ext.
- A bypassing owner's task on the local DSQ is blamed on the owner, which
places nothing while bypassing. If the owner is already exiting, e.g.
during its own disable, the exit is a no-op and nothing is ejected. If the
host isn't running the task, the owner's disable, which waits on it, may
never finish.
- The wait isn't restarted when bypass moves a task. The time it waited
under its owner counts against the host once it is on the host's bypass
DSQ, and its time there against the owner once bypass lifts.
The sched running a task is its owner, or the host while the owner is
bypassing. Test the task against that sched's timeout and blame it. A bypass
requeue can change the sched running a task, so restart the wait of every
task scx_bypass() requeues except the root's own, which the root runs
bypassing or not. Single-scheduler behavior is unchanged.
A task that migrates while scx_bypass() works through the cpus, leaving one
before its requeue and arriving on another after it, is requeued by neither
and keeps its wait, which then counts against the sched now running it. That
is what happened to every task before this patch. This might be worth
addressing in the future, but the race is rare and the real-world
implications should be limited.
Fixes: 2c91169377f4 ("sched_ext: Blame the DSQ's owning scheduler for a runnable stall")
Cc: stable@vger.kernel.org # v7.3+: 30fb2b1e4d41: sched_ext: Fix the running task keeping its protection into bypass
Cc: stable@vger.kernel.org # v7.3+: 5809f749fbdb: sched_ext: Fix a bypassed task stalling on a cpu its scheduler has no caps on
Cc: stable@vger.kernel.org # v7.3+
Reported-by: Andrea Righi <arighi@nvidia.com>
Link: https://lore.kernel.org/all/at4l0yXTk7T_ouk0@gpd4/
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fixes from Ingo Molnar:
- Fix MM CID affinity-triggered switch back to per-task mode
(Hui Su)
- Fix scheduling regression observed on AMD Ryzen
AI HX 370 systems, caused by cache aware scheduling
incorrectly overriding asym packing placement (Tim Chen)
- Fix stale get_pref_llc() prototype for !CONFIG_SCHED_CACHE
(Yafang Shao)
* tag 'sched-urgent-2026-10-25' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched: fix stale get_pref_llc() stub for !CONFIG_SCHED_CACHE
sched/cache: Honor asym packing over cache aware scheduling on hybrid systems
sched/mmcid: Fix affinity-triggered switch back to per-task mode
|
|
The is_vm_hugetlb_page() predicate is badly named - the mapping can span
more than a page and it is inconsistent with other VMA predicates that
typically are prefixed by vma_.
Rename to vma_is_hugetlb() for consistency, and while we're here update
some VM_BUG_ON_VMA() to VM_WARN_ON_ONCE_VMA() as to avoid unnecessary
oopses.
No functional change intended.
Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-28-1aa3c9b139f9@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Marc Zyngier <maz@kernel.org>
Acked-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Anup Patel <anup@brainfault.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Paul Moore <paul@paul-moore.com>
Cc: Stephen Smalley <stephen.smalley.work@gmail.com>
Cc: Jaroslav Kysela <perex@perex.cz>
Cc: Takashi Iwai <tiwai@suse.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Eduard Zingerman <eddyz87@gmail.com>
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Doug Gilbert <dgilbert@interlog.com>
Cc: James Bottomley <james.bottomley@hansenpartnership.com>
Cc: Martin K. Petersen <mkp@kernel.org>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: Helge Deller <deller@gmx.de>
Cc: Sebastian Reichel <sre@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Harry Yoo <harry@kernel.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Will Deacon <will@kernel.org>
Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Oliver Upton <oupton@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Paul Walmsley <pjw@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Andreas Larsson <andreas@gaisler.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Youngjun Park <youngjun.park@lge.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Xu Xin <xu.xin@linux.dev>
Cc: Ihor Solodrai <ihor.solodrai@linux.dev>
Cc: Bert Karwatzki <spasswolf@web.de>
|
|
This header really makes little sense - every place it is included mm.h is
also included, and the header itself includes mm.h, so it does nothing to
reduce header size.
It also oddly does an #ifdef around checking VMA_HUGETLB_BIT, however
VMA_HUGETLB_BIT is unconditionally available, and will never be set if
hugetlb is not enabled.
Simply remove the header, eliminate the odd ifdeffery and place the
predicates in mm.h.
The naming of these predicates is odd, but to keep changes separate, we
will address this in a separate patch.
The file was never put into MAINTAINERS so there's no change required
there.
No functional change intended.
Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-27-1aa3c9b139f9@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Paul Moore <paul@paul-moore.com>
Cc: Stephen Smalley <stephen.smalley.work@gmail.com>
Cc: Jaroslav Kysela <perex@perex.cz>
Cc: Takashi Iwai <tiwai@suse.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Eduard Zingerman <eddyz87@gmail.com>
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Doug Gilbert <dgilbert@interlog.com>
Cc: James Bottomley <james.bottomley@hansenpartnership.com>
Cc: Martin K. Petersen <mkp@kernel.org>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: Helge Deller <deller@gmx.de>
Cc: Sebastian Reichel <sre@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Harry Yoo <harry@kernel.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Will Deacon <will@kernel.org>
Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Oliver Upton <oupton@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Anup Patel <anup@brainfault.org>
Cc: Paul Walmsley <pjw@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Andreas Larsson <andreas@gaisler.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Youngjun Park <youngjun.park@lge.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Xu Xin <xu.xin@linux.dev>
Cc: Ihor Solodrai <ihor.solodrai@linux.dev>
Cc: Bert Karwatzki <spasswolf@web.de>
|
|
We have now made it such that every driver which sets VMA_IO_BIT marks it
as not mm-managed.
However, vma_flags_is_mm_managed() still checks for VMA_IO_BIT and, if
set, treats the mapping as not mm-managed. This was previously required to
handle drivers which set VMA_IO_BIT but none of the other flags which
identify a mapping as not mm-managed.
We have now established that a driver which sets VMA_IO_BIT must also set
one of those flags, so the VMA_IO_BIT check is redundant.
So remove the VMA_IO_BIT check in vma_flags_is_mm_managed() and update
mmap_validate_vma_flags() to use vma_flags_is_mm_managed() rather than
open-coding the VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT check.
This change means that vma[_flags]_can_merge() doesn't check VMA_IO_BIT any
longer (which is now redundant) as it calls vma_flags_is_mm_managed().
Now that the predicate means precisely VMA_PFNMAP_BIT or VMA_MIXEDMAP_BIT,
also use it at the other sites which open-code that pair, so the intent is
stated rather than the flags, with no functional change:
zap_special_vma_range() only zaps mappings which are not mm-managed, as
drivers use it to tear down ranges they established themselves.
The mprotect() arch PFN modification check applies to mappings not managed
by core mm, which may map PFNs without struct pages.
NUMA balancing skips VM_MIXEDMAP mappings having already excluded VM_IO
and VM_PFNMAP mappings via vma_migratable(), so it skips exactly the
mappings not managed by core mm - say so.
Finally, update the VMA userland merge 'special' flag tests to no longer
assert that VMA_IO_BIT prevents merge as VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT
now suffices.
Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-26-1aa3c9b139f9@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Paul Moore <paul@paul-moore.com>
Cc: Stephen Smalley <stephen.smalley.work@gmail.com>
Cc: Jaroslav Kysela <perex@perex.cz>
Cc: Takashi Iwai <tiwai@suse.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Eduard Zingerman <eddyz87@gmail.com>
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Doug Gilbert <dgilbert@interlog.com>
Cc: James Bottomley <james.bottomley@hansenpartnership.com>
Cc: Martin K. Petersen <mkp@kernel.org>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: Helge Deller <deller@gmx.de>
Cc: Sebastian Reichel <sre@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Harry Yoo <harry@kernel.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Will Deacon <will@kernel.org>
Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Oliver Upton <oupton@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Anup Patel <anup@brainfault.org>
Cc: Paul Walmsley <pjw@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Andreas Larsson <andreas@gaisler.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Youngjun Park <youngjun.park@lge.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Xu Xin <xu.xin@linux.dev>
Cc: Ihor Solodrai <ihor.solodrai@linux.dev>
Cc: Bert Karwatzki <spasswolf@web.de>
|
|
Commit b636fef85bda ("sched/cache: Introduce task_struct->sched_cache_grp
to fix UAF") switched get_pref_llc() from taking a struct mm_struct * to
taking the task's struct sched_cache_group *, but only updated the
CONFIG_SCHED_CACHE definition and its caller: the stub in the
!CONFIG_SCHED_CACHE branch kept the old mm_struct parameter.
Fixes: b636fef85bda ("sched/cache: Introduce task_struct->sched_cache_grp to fix UAF")
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Cc: <stable@vger.kernel.org> #v7.2.x
Link: https://patch.msgid.link/20261023023105.28829-2-laoar.shao@gmail.com
|
|
A regression was reported on an AMD Ryzen AI HX 370 running a cache
intensive Clang full-LTO link. The little cores run at a much lower
frequency (3.3 GHz vs 5.1 GHz) and have only half of the L3 cache
(8 MB vs 16 MB), so pinning such a task to the little-core LLC
hurts twice, and full-LTO builds slow down dramatically compared to
pre-cache-aware-scheduling kernels.
Asym packing and cache aware scheduling express conflicting placement
strategies. Asym packing wants a task to run on the highest priority CPU,
whereas cache aware scheduling wants to co-locate the tasks of a process
on one LLC regardless of the priority of CPUs in that LLC.
When asym packing tries to migrate task to an idle core that has higher
priority than source cpu, let asym packing win. Moving tasks to a higher
performing idle core will buy more performance than cache co-location.
Prioritize asym packing over LLC balancing for regular and active load
balancing.
Fixes: 23b2b5ccc45c ("sched/cache: Introduce helper functions to enforce LLC migration policy")
Closes: https://lore.kernel.org/lkml/2180ea5a-eb28-4152-8d4d-cd00b0c24b2e@computerix.info/
Reported-by: Klaus Kusche <klaus.kusche@computerix.info>
Suggested-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Tested-by: Klaus Kusche <klaus.kusche@computerix.info>
Tested-by: Ricardo Neri <ricardo.neri@intel.com>
Tested-by: Chen Yu <yu.c.chen@intel.com>
Cc: stable@vger.kernel.org
Cc: stable@vger.kernel.org # 7.2.x
Link: https://patch.msgid.link/b29badede27bb6bfb4a759e06754cb19657f0778.1791308631.git.tim.c.chen@linux.intel.com
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext updates from Tejun Heo:
- Proxy execution can now be enabled together with sched_ext. A BPF
scheduler can opt in to receive mutex-blocked tasks that donate their
scheduling context to the lock owner and place them itself.
- Lazy preemption: a BPF scheduler can preempt or kick a CPU lazily,
deferring the switch to the next return to user space or tick, as the
fair class can.
- Sub-schedulers: a parent now learns which child runs on each CPU it
delegated and when a cap change takes effect. A detaching child loses
its caps before the parent restores the CPUs.
- New kfuncs re-enqueue single tasks from a dispatch queue iteration
and report a cgroup's CPU count and a CID's NUMA node. The CID
topology now includes clusters.
- Hot path optimizations: dispatch queue and task lookups compare keys
inline, remote task moves no longer take a dispatch queue lock twice,
and sub-scheduler hooks cost nothing without sub-schedulers.
- Kfuncs callable from tracing programs are hardened against NMI and
IRQ nesting, which could hard-lock the CPU or pick a wrong idle CPU.
- Built-in dispatch queues no longer allocate per-CPU state that grew
with the square of the CPU count.
- Header syncs with the scx repo, a BTF workaround for pahole 1.32, and
selftest fixes and coverage.
* tag 'sched_ext-for-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (69 commits)
sched_ext: Drop the locks periodically while walking a user DSQ for re-enqueue
sched_ext: scx_qmap: Also re-enqueue stranded tasks with scx_bpf_dsq_reenq_task()
sched_ext: Add scx_bpf_dsq_reenq_task() to re-enqueue single tasks from a DSQ iteration
sched_ext: Factor out the rq lock switch of scx_dsq_move()
sched_ext: Don't queue balance callbacks from a core-sched pick
sched_ext: Don't leave balance callbacks queued across a pick retry
sched_ext: Fix a bypassed task stalling on a cpu its scheduler has no caps on
sched_ext: Fix the running task keeping its protection into bypass
sched_ext: Fix use-after-free of a local reenq node left on an inactive cpu
sched_ext: Keep proxy donors with slice left on the local DSQ
selftests/sched_ext: Report reload_loop attach failures
selftests/sched_ext: Bound exit waits
selftests/sched_ext: Add SCX_ASSERT_ALIVE and adopt it
selftests/sched_ext: Fail loudly when EXIT_KIND is missing from BTF
sched_ext: Add SCX_ECODE_RSN_CGROUP_OFFLINE to user_exit_info.h
selftests/sched_ext: Fix reload_loop error-path use-after-free
selftests/sched_ext: Fix build error in dequeue_remote selftest
sched_ext: scx_qmap: Reset a child's cpuperf targets once its PERF revoke takes effect
sched_ext: Add ops.sub_child_ecaps_updated() to report a child's effective cap changes
sched_ext: scx_qmap: Show actual cid use per participant
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
Pull power management updates from Rafael Wysocki:
"This time, the most visible part are the ACPI CPPC updates from
Christian Loehle, including CPPC cpufreq driver changes, even though
updates of cpufreq drivers proper are still the largest topic group.
However, the updates of cpufreq drivers depend on the ACPI CPPC
updates, so they should really be counted together and that is well
over 1/2 of the total commit count.
Interestingly enough, almost all of the updates included here are
fixes or cleanups, with some new feature support in the amd-pstate
cpufreq driver and IPQ9650 support in the qcom-nvmem cpufreq driver.
The runtime PM documentation improvements from Brian Norris can also
be mostly regarded as fixes, but a new kunit test goes along with
them.
Specifics:
- Fix multiple issues related to input validation, synchronization,
access safety, memory layout and register field validation, and
error handling in the ACPI CPPC library (Christian Loehle and Sumit
Gupta)
- Select the frequency-invariance callback per CPU in the CPPC
cpufreq driver and create the FIE worker before enabling PCC
callbacks in it to avoid queueing work through a missing worker and
dereferencing a NULL pointer (Christian Loehle)
- Update the CPPC cpufreq driver entry in MAINTAINERS and add a new
entry for the ACPI CPPC library (Christian Loehle)
- Minor fixes in various cpufreq drivers (Alexander Koskovich, Daniel
Schwierzeck, Frank Zhang, Haotian Zhang, Kathiravan Thirumoorthy,
Sumeet Pawnikar, and Xueqin Luo)
- Fix wrong behaviors on multi-node servers in the Loongson cpufreq
driver (Huacai Chen)
- Reject NULL from cpufreq_cpu_get() in the Rust code (Mehmet
Koseoglu)
- Add suspend OPP to mediatek cpufreq driver and DT (Ryan Brue)
- Fixups for a few bugs identified by Sashiko in the amd-pstate
cpufreq driver (Mario Limoncielloi, Wentao Guan)
- Initial support for maximum frequency capacity handling on Zen6
client in amd-pstate (Mario Limonciello)
- Fix for amd-pstate-ut with 'Requested CPU Min frequency' BIOS
option (K Prateek Nayak)
- New Zen6 tuning values for EPP in amd-pstate (Mario Limonciello,
Vishal Badole)
- amd-pstate documentation fix (Karl Mehltretter)
- Restrict deferred downscaling to low load for shared policies in
the conservative cpufreq governor to avoid undesirable CPU
frequency reduction in some cases (Shengming Hu)
- Serialize sugov_start() against rate limit updates in the schedutil
cpufreq governor to avoid using a stale rate limit value (Hui Su)
- Prevent cpuidle governors menu and teo from returning idle state 0
after it has been disabled (Xueqin Luo)
- Replace min_t() with the better alternatives in the intel_idle
driver and make it use 2-argument strscpy() (Andy Shevchenko)
- Fix two repeated phrase instances in cpuidle (Hemanth Selam,
Syazwina Fazrul)
- Add rugged Panther Lake support to the intel_rapl power capping
driver and make it use %pe for printing error pointers (Srinivas
Pandruvada and Sumeet Pawnikar)
- Update and improve runtime PM documentation and add a kunit test
for supplier idle/suspend (Brian Norris)
- Add a pm_runtime_dont_use_autosuspend() call to pm_runtime_reinit()
to ensure autosuspend disabling on driver teardown (Joshua Crofts)
- Ignore page-zero RAM in E820 checksum during hibernation on x86 to
avoid rejecting images created after kexec() (Matthias Goergens)
- Add resume event mapping for PMSG_POWEROFF to the PM core to allow
it to carry out device shutdown in the future (Mario Limonciello)
- Add DPM watchdog to prepare/late/early/noirq/complete phases of
device suspend and resume (Mayank Rungta)
- Update swsusp.txt references to swsusp.rst in hibernation
documentation (Li Jun)
- Fix use-after-free of perf domain in netlink doit handlers for
energy models (Fan Wu)
- Add synchronize_rcu() back to cpu_pm_unregister_notifier() after it
was removed inadvertently in previous development cycles to avoid
use-after-free (Bradley Morgan)
- Unregister CPU PM notifier outside the timer lock in the
timer-ti-dm clocksource driver (Bradley Morgan)
- Fix typo "sucessfully" in a comment in pm_clk_add_clk() kerneldoc
comment and typo "hierachy" in a comment in pm-graph (Hemanth
Selam)
- Fix the return value in dev_pm_opp_get_of_node() kernel-doc (Karl
Mehltretter)"
* tag 'pm-7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm: (89 commits)
cpufreq: loongson3: Replace IOCSR read/write with MMIO ones
cpufreq: loongson3: Use global physical CPU ID in get/target callbacks
cpufreq: loongson3: Replace per-package mutex with per-node
cpufreq: loongson3: Adjust the width of id and val in smc_message
cpufreq: loongson3: Make this driver depend on MACH_LOONGSON64
PM: runtime: Expand introduction with core concepts and structure
cpufreq: schedutil: Serialize sugov_start() against rate limit updates
MAINTAINERS: ACPI: Add a separate CPPC entry
MAINTAINERS: cpufreq: Replace CPPC reviewer
arm64: dts: mediatek: mt8173: Mark the lowest CPU OPPs for suspend
cpufreq: mediatek: Enter system sleep at the suspend OPP
cpufreq: qcom-cpufreq-hw: fix device node refcount leak in phandle parsing
cpufreq: qcom-nvmem: Add IPQ9650 support
cpufreq: Add Milos to cpufreq-dt-platdev blocklist
cpufreq: Use %pe to print error pointers symbolically
rust: rcpufreq_dt: add module alias
cpufreq: sti: avoid NULL dereference in dev_err()
cpufreq: tegra194: fix double-pointer error in get_cpu_ndiv
cpufreq: sparc-us2e: fix frequency table index copy-paste error
rust: cpufreq: reject NULL from cpufreq_cpu_get()
...
|
|
Automatic topology packing starts a new shard before adding a core when
the current shard would exceed its target size. However, a first core is
always accepted so that cores remain indivisible. An indivisible core with
more than SCX_CID_SHARD_MAX_CPUS CPUs cannot be represented without
exceeding the hard shard limit.
Per-shard state uses fixed-size inline cmask storage sized for
SCX_CID_SHARD_MAX_CPUS. Passing an oversized shard to alloc_pshard() can
therefore make cmask initialization overwrite the following fields.
Reject an indivisible core larger than SCX_CID_SHARD_MAX_CPUS before adding
it to a shard. Existing non-empty shard splitting already keeps the total
within the hard cap, while an indivisible core may still exceed
cid_shard_size when it fits within the hard limit.
Report the oversized core and the hard limit with scx_error() so that the
rejection is diagnosable.
tj: The bug is only reproducible with artificial VM topologies. It can't
happen on real hardware.
Fixes: e2c841b91bf9 ("sched_ext: Add CID sharding")
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Hui Su <sh_def@163.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
With proxy execution, __schedule() can call set_next_task_scx() with
SNT_REPICK and then SNT_PICK for the same donor. Both calls queue the
kick_sync balance callback while kick_sync_pending remains set.
When the callback is the only list entry, its next pointer is NULL, so
the existing duplicate check misses it. Queueing it again links it to
itself and causes the wait to run twice.
Also check whether the callback is already the head of the list to
reject the second insertion.
This covers the kick_sync callback because nothing else is queued
between SNT_REPICK and SNT_PICK. Requeueing the tail of a longer list
remains possible and is outside the scope of this fix.
Fixes: e2859566ffbc ("sched_ext: Queue the kick_sync wait from set_next_task_scx()")
Reported-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/08cac02c604dd73ef96fc92eb29d692b@kernel.org
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"Kconfig updates:
- Simplify PREEMPT_DYNAMIC (Mark Rutland)
Fair scheduler updates:
- Remove redundant RUN_TO_PARITY check (Elif Topuz)
- Declare augmented callbacks per field, fix
rb_add_augmented_cached() descent (Yiwei Lin, Peter Zijlstra)
- Restart fair hrtick after same-task repicks (Shubhang Kaushik)
- Improve the scheduling latency of short slice tasks (Vincent
Guittot)
Deadline/RT scheduler updates:
- Check start_dl_timer expiry with ktime_before() (Liang Hao)
- irq_work: CPU-hotplug improvements on PREEMPT_RT (Sebastian Andrzej
Siewior)
Energy-aware scheduling updates:
- Implement the steal_governor, introduce preferred CPUs and
steal-driven vCPU backoff (Shrikanth Hegde)
Multi-processor load-balancing updates:
- Enable preferred SMT siblings on NVIDIA Olympus (Andrea Righi)
- Randomize equally shallow idle CPU picks (Christian Loehle)
- Remove redundant core_sched_seq (Hui Su)
Cache-aware scheduling updates:
- Avoid publishing partial LLC utilization (Jemmy Wong)
NUMA-balancing updates:
- Reset NUMA fault locality after scan period update (Eric Kim)
Proxy execution updates:
- Ongoing work to implement sleeping owner handling for proxy
execution (John Stultz, Vasily Gorbik)
- Preparatory steps to make sched_ext compatible with proxy execution
(Andrea Righi)
Scheduler infrastructure updates & fixes:
- Remove the RCU wait from PSI trigger destruction (Shakeel Butt)
- Avoid modifying wait-entry on failed priority registration
(Hui Su)
- Use the new static key API for sched_feat (Hongyan Xia)
- Convert paravirt_steal to new static key APIs (Hongyan Xia)
- cputime: Don't account idle time twice after dyntick-idle
(Stian Halseth)
Scheduler debugging updates:
- Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
(Sechang Lim)
- Fix context analysis errors in non-preferred CPU push (Shrikanth
Hegde)
Misc cleanups and fixes by Christian Loehle, Shubhang Kaushik,
Thomas Huth and Vladimir Zapolskiy"
* tag 'sched-core-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (64 commits)
sched: Replace nr_iowait_cpu() with a bool accessor
Revert "sched/fair: Remove dead code on enqueue_task_fair()"
sched/cache: Avoid publishing partial LLC utilization
sched/cputime: Don't account idle time twice after dyntick-idle
sched/psi: Remove the RCU wait from trigger destruction
sched/eevdf: Remove redundant RUN_TO_PARITY check
sched/wait: Avoid modifying entry on failed priority registration
rbtree_test: Use rb_add_augmented_cached() for the cached augmented test
rbtree: Update augmented data on the way down in rb_add_augmented_cached()
rbtree: Declare augmented callbacks per field with RB_AUG()
rbtree_test: Use rb_add() and rb_add_cached() for the basic tests
smpboot: Don't park the thread if work is pending
irq_work: Flush lazy work CPU down on PREEMPT_RT
irq_work: Update a comment regarding CPU hotplug invocation
sched/topology: Add asymmetric SMT packing override
sched/fair: Honor asymmetric SMT priority in idle selection
sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched()
sched/core: Fix context analysis errors in non-preferred CPU push
virt/steal_governor: Enable the driver
virt/steal_governor: Implement steal_governor policy loop
...
|
|
maybe_queue_balance_callback() clears SCX_RQ_BAL_CB_PENDING when it
queues the deferred callback. Proxy resolution can then zap the callback
without running it, leaving the deferred actions unscheduled until a
later wakeup or deferred request.
Use the existing irq_work fallback for dispatch-time deferred requests
when proxy execution is enabled. This keeps the work scheduled even if
proxy resolution retries the pick or falls back to idle, without relying
on another SCX dispatch to restore a discarded callback.
Fixes: a8ad873113d3 ("sched_ext: defer queue_balance_callback() until after ops.dispatch")
Reported-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/08cac02c604dd73ef96fc92eb29d692b@kernel.org
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
MM CID switches to per-CPU ownership when the number of users of an mm
exceeds the maximum number of CIDs. When the affinity union grows, the
per-CPU threshold is recalculated and the MM can switch back to per-task
ownership through deferred work.
mm_cid_calc_pcpu_thrs() incorrectly uses min(nr_cpus_allowed, users) to
calculate that switch-back threshold. This makes pcpu_thrs no larger than
users, so the users < pcpu_thrs check in mm_update_cpus_allowed() can never
succeed.
For example, with three users restricted to two CPUs, the MM enters per-CPU
mode. On an eight-CPU system, widening the affinity union to all eight CPUs
should produce a threshold of four, but the old calculation produces three
and returns early. The MM therefore remains in per-CPU mode even though the
widened CPU set makes per-task ownership preferable.
Calculate pcpu_thrs from nr_cpus_allowed directly. The users bound in
__mm_update_max_cids() still controls when per-CPU mode is entered, so this
does not change the threshold calculated when switching from per-task to
per-CPU ownership.
Tested with an x86_64 QEMU reproducer using three users, an initial two-CPU
affinity, and an eight-CPU affinity expansion. The baseline does not invoke
mm_cid_work_fn() after the affinity expansion. The fixed kernel invokes the
deferred work once and completes the switch back to per-task mode.
Fixes: fbd0e71dc370 ("sched/mmcid: Provide CID ownership mode fixup functions")
Signed-off-by: Hui Su <sh_def@163.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260915155953.2856524-1-sh_def@163.com
|
|
The SCX_KICK_WAIT wait is deferred to rq->scx.kick_sync_bal_cb, which a
final pick queues on the local rq. Under core scheduling, a pick that
returns a task isn't final: pick_next_task() goes on to pick the SMT
siblings, and their dispatch can release the shared rq lock with the
callback still queued. A CPU acquiring the rq lock in that window then
finds a pending balance callback and triggers the warning in
rq_pin_lock().
Fix this by queuing the callback from set_next_task_scx() on SNT_PICK
and SNT_REPICK. put_prev_set_next_task() calls it once the pick is
final, after the core-wide selection, so the rq lock can't be dropped
with the callback queued. If the CPU ends up running idle or a higher
class task, no SCX task runs there and there's nothing to wait for:
kick_sync_pending stays set and the next pick of an SCX task waits
before running it.
Also, proxy exec zaps the balance callbacks and picks again when
find_proxy_task() fails, so clear kick_sync_pending when the callback
runs rather than when it's queued.
Fixes: 415cb193bb97 ("sched_ext: Fix SCX_KICK_WAIT deadlock by deferring wait to balance callback")
Cc: stable@vger.kernel.org # v7.0+: c72945693b90: sched: Restart fair hrtick after same-task repicks
Cc: stable@vger.kernel.org # v7.0+
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The watchdog measures how long a task has been waiting from
p->scx.runnable_at, which set_task_runnable() resets when the task becomes
runnable after it last ran. ops_dequeue() never asks for a reset, as the
dequeue of a waiting task is temporary, e.g. for a migration or an
attribute change.
Leaving the scheduler isn't temporary. A task which is switched out of
SCHED_EXT while it's waiting keeps its runnable_at. When it's switched back
and becomes runnable, its wait appears to have started before it left, and
the time it spent in another class counts towards the timeout. If the
watchdog checks before the task runs, it aborts the scheduler:
sched_ext: class_switch_stall: runner[418] failed to run for 3.010s
Here, the task waited for a few milliseconds before it was switched to
SCHED_OTHER, slept outside of SCHED_EXT for three seconds, and was charged
for all of it as soon as it woke up on SCHED_EXT again.
The same happens when scx_sub_enable_workfn() hands a waiting task over to
a new sub-scheduler, which can then be blamed for time the task waited
under the parent. scx_rehome_task() isn't affected, as it moves the task
through SCX_TASK_INIT, which resets runnable_at. The stale runnable_at also
skews the default core-sched ordering in scx_prio_less().
Reset runnable_at in scx_disable_task(), when the task leaves the control
of its scheduler.
Fixes: 8a010b81b3a5 ("sched_ext: Implement runnable task stall watchdog")
Cc: stable@vger.kernel.org # v6.12+
Assisted-by: LLM
Signed-off-by: Michal Blaszczyk <michalblk@google.com>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Suspend and hibernation take down every non-boot cpu and bring them back. A
loss and a regain per cpu to every sub-scheduler would be noise. Leave them
out: a cpu they take down reports no loss and keeps the caps last delivered,
and the reseed after resume reports only the net change. Their downs are the
ones with cpuhp_tasks_frozen set between the PM notifications. A kexec jump
sets the flag too but sends no PM notification.
Two things need care. A cpu that suspend or hibernation takes down doesn't
wait for its syncs, and no ecaps_report() mark can be pending on it then.
The attach report runs while the attaching task waits uninterruptibly, which
holds off the freezer, and the detach report holds cpus_read_lock().
The detach report reaches only active cpus, and the loss on a cpu that
suspend or hibernation took down was never delivered. The detach waits for
those cpus to come back, so every loss is delivered. They come back before
the tasks thaw, so the wait doesn't hold up the freezer. If one fails to
come back, the hierarchy is aborted at the end of the PM cycle.
v2: Fix why no report mark is pending at a PM down, add a data_race().
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
When a cpu goes offline, a sub-scheduler's effective caps on its cid are
cleared, but neither the sub-scheduler nor its parent is notified. The
sub-scheduler keeps treating the cid as usable, and the parent can't act on
the loss, e.g. reset the cpuperf target.
Report the loss before the root's ops.cpu_offline() and the regain after its
ops.cpu_online(). The zeroing at offline was the last change to a sched's
ecaps outside a delivery, so they now change only with one. Syncs on an
inactive cpu are processed instead of discarded, and the reseed reschedules
the cpu itself, as they got no resched when queued. Suspend and hibernation
now report their cpu downs and ups as well. This is temporary, and a later
patch leaves them out.
Two things need care. A cpu going down marks a sync for each sub-scheduler,
and its hotplug thread waits until each marked sync has been processed. The
cpu's own ext picks process them, and the ext server is started in case a
fair task holds the cpu. Only the marked syncs count, so a flood of changes
on the cid can't hold up the hotplug.
The detach report reaches only active cpus. It holds off cpu hotplug, so a
cpu it skips has reported its loss as it went down, and no unreported loss
is left to drop.
Drop the cpuperf reset from scx_qmap's ops.sub_detach(), as its
ops.sub_child_ecaps_updated() now resets the targets on the loss reports.
v2: Reschedule from the irq work and document the regain order.
v3: Pack nr_ecaps_offline with kick_sync_pending to avoid a hole (Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A sub-scheduler's ecaps ops run in dispatch context, and ops.cpu_online()
and ops.cpu_offline() document that the cpu doesn't call ops.dispatch()
outside them. The reseed of a cpu coming back is queued before the root's
ops.cpu_online(), so the ops of a sub-scheduler and its parent can run for a
cpu the root still considers offline.
Add SCX_RQ_OPS_ONLINE, set while the root considers the cpu online, from its
ops.cpu_online() or its enable until its ops.cpu_offline(). A sync outside
that span leaves the ecaps alone, and the reseed, now queued after
ops.cpu_online(), reports the net change. SCX_RQ_ONLINE can't be used, as
the core changes it ahead of the root's view on both edges.
A cpu going offline still drops its caps without a report.
While at it, fix the SCX_RQ_ONLINE comments, which claimed that the flag
tracks ops.cpu_online() and ops.cpu_offline().
v2: Fix the SCX_RQ_ONLINE comments.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A sub-scheduler learns which cids it holds through ops.sub_ecaps_updated(),
but the caps granted from ops.sub_attach() reach it only after its enable
bypass lifts, when it is already scheduling its tasks.
Report the caps at the end of the enable, before bypass lifts. On disable,
report their loss before any task goes back to the parent, so that the
parent hears of the loss of every cap it was told of on an active cpu.
ops.sub_detach() remains where the parent cleans up after the sub-scheduler.
Deliveries no longer depend on bypass, so ops.sub_ecaps_updated() and
ops.sub_child_ecaps_updated() also run while a scheduler bypasses. The
replay at unbypass goes, and the enable's bypass returns to after
ops.sub_attach().
A sub-scheduler's ecaps now change only with the deliveries that report
them, so that whenever its ops run under the rq lock, its caps are the ones
it was last told of. A sync before the attach report leaves them alone, and
the detach report zeroes them as it delivers the loss. A cpu going offline
still drops its caps without a report, and those are the only changes not
delivered. clear_all_caps() drops that unreported loss on inactive cpus.
The following things need care. A report reschedules only the cpus the
sub-scheduler was ever granted a cap on. Rescheduling every cpu would
disrupt a large machine split many ways, and which cpus still owe a delivery
can't be read off the caps.
The report and a queued sync reschedule the cpu from hard irq work on that
cpu, not with a kick or a direct resched. dispatch_one() processes the syncs
only at its start, and a kick only guarantees that a scheduling event
completes after it, which can be the pick in progress. That pick has already
processed the syncs and clears the resched, and a cpu that then idles or
runs a task with an infinite slice leaves the report waiting. A pick runs
with IRQs off, so the irq work runs after any pick in progress on that cpu.
Under core scheduling, a sibling's committed pick can switch the cpu without
dispatch_one(), and a sync queued after that pick is processed only at the
cpu's next scheduling event. A cpu running a task of a higher class isn't
rescheduled, as its pick would return that task again. While the root
bypasses, a queued sync doesn't reschedule the cpu, as a grant or revoke can
run in the PM syscore phase, where irq work can malfunction. The report
never runs there and reschedules the cpu itself.
The report waits on an rcuwait woken from hard irq work. Under core
scheduling, a cpu going down can process a sibling's marked sync after
hotplug has parked its irq_work thread. On PREEMPT_RT, a regular irq_work
queued there would be left behind, and a hard irq can't take a wait queue's
lock.
v2: Reschedule from hard irq work on the cpu instead of a kick (sashiko).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
The free path removes a sub-scheduler's ecaps sync node from each cpu's list
and then cycles the rq lock until the node reads off-list, as a sync batch
can hold the node privately. Cycling the lock is correct only because the
sched is bypassing and a bypassing sched's sync calls no op, so the batch
touches the pcpu within one rq lock hold.
Tying the free path to what the sync does for a bypassing sched is unusual
and brittle, and it gets in the way of changing the sync. Delivering ecaps
to a bypassing sched or its parent would break the wait, as a delivery drops
the rq lock.
Wait for the batches with RCU instead, the usual way to wait for lockless
users before a free. Discard the nodes after the first grace period as
before, and free the sched after a second one. The first grace period waits
for the paths that found the sched before it was unlinked, such as the
sub-cap kfuncs and the cpu hotplug hooks. Until they finish, they can queue
its syncs, and a cpu can take a queued node into a sync batch that the first
grace period doesn't wait for. After it, nothing queues the syncs, and the
discards remove the nodes still queued.
A batch that took a node keeps irqs off from llist_del_all() to its last
pcpu write, even across the rq lock drops of a delivery. The second grace
period waits for it whether or not it called an op. The exit and disable
irq_works, the helper kthread and the bypass load balancing timer are
stopped ahead of the per-cpu checks as before, so they move into
scx_sched_quiesce_rcu_work() with the checks.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Factor the cid to pshard lookup out of the ecaps sync paths into
cid_pshard() so that new sync paths can share it. No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Merge cpufreq updates for 7.4:
- Minor fixes in various cpufreq drivers (Alexander Koskovich, Daniel
Schwierzeck, Frank Zhang, Haotian Zhang, Kathiravan Thirumoorthy,
Sumeet Pawnikar, and Xueqin Luo)
- Fix wrong behaviors on multi-node servers in the Loongson cpufreq
driver (Huacai Chen)
- Reject NULL from cpufreq_cpu_get() in the Rust code (Mehmet Koseoglu)
- Add suspend OPP to mediatek cpufreq driver and DT (Ryan Brue)
- Fixups for a few bugs identified by Sashiko in the amd-pstate cpufreq
driver (Mario Limoncielloi, Wentao Guan)
- Initial support for maximum frequency capacity handling on Zen6
client in amd-pstate (Mario Limonciello)
- Fix for amd-pstate-ut with 'Requested CPU Min frequency' BIOS option
(K Prateek Nayak)
- New Zen6 tuning values for EPP in amd-pstate (Mario Limonciello,
Vishal Badole)
- amd-pstate documentation fix (Karl Mehltretter)
- Restrict deferred downscaling to low load for shared policies in the
conservative cpufreq governor to avoid undesirable CPU frequency
reduction in some cases (Shengming Hu)
- Serialize sugov_start() against rate limit updates in the schedutil
cpufreq governor to avoid using a stale rate limit value (Hui Su)
* pm-cpufreq: (34 commits)
cpufreq: loongson3: Replace IOCSR read/write with MMIO ones
cpufreq: loongson3: Use global physical CPU ID in get/target callbacks
cpufreq: loongson3: Replace per-package mutex with per-node
cpufreq: loongson3: Adjust the width of id and val in smc_message
cpufreq: loongson3: Make this driver depend on MACH_LOONGSON64
cpufreq: schedutil: Serialize sugov_start() against rate limit updates
arm64: dts: mediatek: mt8173: Mark the lowest CPU OPPs for suspend
cpufreq: mediatek: Enter system sleep at the suspend OPP
cpufreq: qcom-cpufreq-hw: fix device node refcount leak in phandle parsing
cpufreq: qcom-nvmem: Add IPQ9650 support
cpufreq: Add Milos to cpufreq-dt-platdev blocklist
cpufreq: Use %pe to print error pointers symbolically
rust: rcpufreq_dt: add module alias
cpufreq: sti: avoid NULL dereference in dev_err()
cpufreq: tegra194: fix double-pointer error in get_cpu_ndiv
cpufreq: sparc-us2e: fix frequency table index copy-paste error
rust: cpufreq: reject NULL from cpufreq_cpu_get()
cpufreq: amd-pstate-ut: Fix max_freq and EPP test failures on Zen6
cpufreq: amd-pstate: Propagate cppc_set_auto_sel() errors on mode change
cpufreq: amd-pstate: Restore previous mode when changing driver mode fails
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler fixes from Ingo Molnar:
- Fix __sched_setscheduler() class change race causing
lost reschedules, exposed by recent RCU changes (K Prateek Nayak)
- Fix s64 multiplication overflow in place_entity()'s vlag
scaling code that can corrupt an entity's placement (Hui Su)
* tag 'sched-urgent-2026-10-18' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
sched/fair: Avoid overflow in place_entity()
sched/core: Call wakeup_preempt() for queued tasks during class change
|
|
reenq_user() drops dsq->lock around each re-enqueue and its rq lock every
SCX_TASK_ITER_BATCH re-enqueues. A walk filtered by SCX_REENQ_MARKED skips
unmarked tasks without dropping either, so a long run of unmarked tasks is
walked in one hold of both locks.
Count the visited tasks instead of the re-enqueued ones and drop both locks
every SCX_TASK_ITER_BATCH visits. An unfiltered walk drops the locks as
often as before.
Suggested-by: Andrea Righi <arighi@nvidia.com>
Link: https://lore.kernel.org/all/atP-LIKpxPhR-Nuc@gpd4/
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
DSQ iteration
scx_bpf_dsq_reenq() re-enqueues a whole DSQ, but a scheduler that loses the
cpus serving a DSQ, such as a sub-scheduler after a cap revoke, needs only
the stranded tasks placed again.
Add scx_bpf_dsq_reenq_task(), which marks a task from a DSQ iteration. When
the iteration ends, the DSQ's deferred re-enqueue walk is scheduled once and
re-enqueues only the marked tasks. The kfunc locks the task's rq and the DSQ
like scx_bpf_dsq_move() and is callable from the same contexts. The mark is
cleared when the task leaves the DSQ, so a task consumed before the walk is
not re-enqueued.
An iteration's DSQ can be destroyed before the iteration ends, so
schedule_dsq_reenq() now skips a destroyed DSQ instead of aborting the
scheduler. scx_bpf_dsq_reenq() racing scx_bpf_destroy_dsq() takes the same
path.
Declare the kfunc for the tools and add a compat probe.
v2: Note in the kerneldoc that another iteration's re-enqueue may take a
marked task early, and that a task on another scheduler returns false
(Andrea).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Factor the rq lock switch on entry to scx_dsq_move() and its undo on exit
out into kf_switch_rq_lock() and kf_restore_rq_lock() so that other kfuncs
operating on a task from a DSQ iteration can share them. No functional
change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
