| Age | Commit message (Collapse) | Author |
|---|
|
https://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
# Conflicts:
# Documentation/scheduler/index.rst
# arch/arm64/configs/defconfig
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
|
|
# New commits in timers/core:
348f54c435bf ("selftests/timers: clocksource-switch: Fix unchecked open()/read()")
b03638013add ("selftests: timers: Measure the CPU timers on the clock they count")
a252cb93e156 ("selftests: timers: Count what tick is worth in the drift estimate")
3945c4a3ea15 ("time/kunit: Add time64_to_tm() case beyond 32-bit day count")
2927f7ca7844 ("time: Prevent time64_to_tm() day truncation on 32-bit")
bc5b66c300b8 ("timekeeping: Use READ_ONCE/WRITE_ONCE() for ktime_sec to prevent tearing")
bb41ece16463 ("timers/migration: Mark racy updates to tmigr_event::ignore field")
1159ad0a6aaa ("timers: Mark racy updates to hlist_node::pprev field")
5dffe33bfdaf ("hrtimer: Apply READ_ONCE() to lockless base->running loads")
ad80926d7803 ("hrtimer: Mark the hrtimer_sleeper structure's ->task field __private")
15f84398330c ("hrtimer: Update hrtimer_resolution only if value changes")
7eed1771a6e8 ("futex: Use accessor for hrtimer_sleeper ->task field in requeue")
9cd2f4e304d9 ("rtmutex: Use accessor for hrtimer_sleeper ->task field")
3990d196954a ("net: pktgen: Use accessor for hrtimer_sleeper ->task field")
9b7bcd671e9c ("timers: Use accessor for hrtimer_sleeper ->task field in sleep_timeout.c")
fa3d486086d6 ("futex: Use accessor for hrtimer_sleeper ->task field in waitwake.c")
851c30277918 ("io-uring/rw: Use accessor for hrtimer_sleeper ->task field")
66b29025b676 ("wait: Use accessor for hrtimer_sleeper ->task field")
cefc1a24ace5 ("aio: Use accessor for hrtimer_sleeper ->task field")
d166a1cd9016 ("hrtimer: Mark data-racy accesses to hrtimer_sleeper ->task field")
7b07d15ed1d7 ("posix-timers: Handle exit in do_exit() completely")
54ad1e0ea42c ("posix-cpu-timers: Prevent enqueueing when PF_EXITING is set")
760ad335d600 ("posix-cpu-timers: Use PF_EXITING to indicate exit")
8e9ed3b66f40 ("posix-cpu-timers: Move inlines out of public header")
24adce0b86c9 ("posix-timers: Move POSIX timer group exit related code out of do_exit()")
7117334f022b ("posix-timers: Move posixtimer_exec_cleanup() out of exec.c")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
|
# New commits in sched/core:
4a3b51aab6e2 ("smpboot: Don't park the thread if work is pending")
40dcc9bdbef3 ("irq_work: Flush lazy work CPU down on PREEMPT_RT")
791b1760accd ("irq_work: Update a comment regarding CPU hotplug invocation")
648d44bda731 ("sched/topology: Add asymmetric SMT packing override")
c8fc4136fd3c ("sched/fair: Honor asymmetric SMT priority in idle selection")
53bc5c556b82 ("sched: Set TIF_NEED_RESCHED before calling __trace_set_need_resched()")
4b1f75be23c4 ("sched/core: Fix context analysis errors in non-preferred CPU push")
1fb28c664a19 ("virt/steal_governor: Enable the driver")
27d47ebce4d6 ("virt/steal_governor: Implement steal_governor policy loop")
4b9302d494ff ("virt/steal_governor: Add control knobs for handling steal values")
9a8e740ee9f6 ("virt: Introduce steal governor driver")
68957caaa9c0 ("sched/debug: Add migration stats due to non preferred CPUs")
74699f56ebcf ("sched/core: Push current task from non preferred CPU")
4ee29b029058 ("sched/fair: Load balance only among preferred CPUs")
d8a3da0de843 ("sched/core: Try to use a preferred CPU in is_cpu_allowed")
620824516557 ("sysfs: Add preferred CPU file")
518b32bd5bb3 ("cpumask: Introduce cpu_preferred_mask")
06a49ef784ac ("sched/docs: Document cpu_preferred_mask and Preferred CPU concept")
cfb463b7172d ("cpumask: Introduce cpumask_intersects_and")
a8d0854a76a8 ("sched/cputime: Add kcpustat_field_total helper")
be100c77178e ("sched: Add sched_ext hooks for proxy execution")
57c75e3ae38c ("sched: Add helper to block retained proxy donors")
a49653d0abeb ("sched/core: Mark wakeups completed through ttwu_runnable()")
8f8c0417e973 ("sched/core: Dequeue waking proxy donors before reset")
313b652837d0 ("sched/core: Drop mutex locks before proxy rescheduling")
627ea30aca3b ("sched/wait: Clarify WF_SYNC wakeup semantics")
d2e010082757 ("sched/eevdf: Handle more short slice waking cases")
4bf32ec3327d ("sched/eevdf: Align update_protect_slice to set_protect_slice")
aae2a33ea662 ("sched/eevdf: Ensure that vprot will never go above a min slice")
c9ce69fc43bd ("sched/fair: Randomize equally shallow slow-path candidates")
abe440b3770f ("sched/fair: Drop idle recency from slow-path CPU selection")
fbbc63fed0b0 ("sched/core: Remove redundant core_sched_seq")
819224e506bc ("sched/fair: Remove dead code on enqueue_task_fair()")
c72945693b90 ("sched: Restart fair hrtick after same-task repicks")
a9b3c7570564 ("sched/headers: Replace __ASSEMBLY__ with __ASSEMBLER__ in the <uapi/linux/sched.h> header")
e81ee0630837 ("sched/fair: Reset NUMA fault locality after scan period update")
ef9293b3b797 ("sched: dynamic: Fix preemption model strings")
879eaa76e608 ("sched: Remove unneeded function type cast in do_balance_callbacks()")
f549101187c8 ("sched/deadline: check start_dl_timer expiry with ktime_before()")
2a672daa4b27 ("sched/feat: Use the new static key API for sched_feat")
a5576ebce920 ("sched: Convert paravirt_steal to new static key APIs")
9650ce11f2e3 ("sched: dynamic: Simplify preempt model accessors")
5b9a28eeed37 ("sched: dynamic: Remove HAVE_PREEMPT_DYNAMIC_{CALL,KEY}")
aa4178f63847 ("sched: dynamic: Simplify irqentry_exit_cond_resched()")
b9d267b9d632 ("sched: dynamic: Simplify preempt_schedule{,_notrace}()")
88e0b3bb9930 ("sched: dynamic: Simplify {cond,might}_resched()")
d3d16750693b ("sched: dynamic: Make PREEMPT_DYNAMIC depend on ARCH_HAS_PREEMPT_LAZY")
772d9ffbfd26 ("sched: Migrate whole chain in proxy_migrate_task()")
6b73a09e943f ("sched: Break out core of attach_tasks() helper into sched.h")
1f8805138593 ("sched: Switch rq->next_class in proxy_reset_donor()")
09351db90a28 ("sched/core: Don't proxy-exec unmatched cookie lock owners")
9be817f991e2 ("sched/core: Avoid migrating blocked_on tasks")
3dd95f077371 ("sched/core: Don't steal a proxy-exec donor")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
|
is_child_subreaper, has_child_subreaper, autoreap and, with
CONFIG_POSIX_TIMERS=y, timer_create_restore_ids are 1-bit fields that
share one memory location in struct signal_struct. A store to any of
them is a read-modify-write of the whole word, and three of the stores
run without a common lock:
- PR_SET_CHILD_SUBREAPER writes me->signal->is_child_subreaper with
no lock held.
- propagate_has_child_subreaper(), run by walk_process_tree() from
the same prctl in an ancestor, writes p->signal->has_child_subreaper
of every descendant p under read_lock(&tasklist_lock), which does
not exclude the other writers.
- PR_TIMER_CREATE_RESTORE_IDS writes
current->signal->timer_create_restore_ids with no lock held.
When a process calls prctl(PR_SET_CHILD_SUBREAPER, 1) while an
ancestor's PR_SET_CHILD_SUBREAPER walk visits it, or while another
thread of the same process calls prctl(PR_TIMER_CREATE_RESTORE_IDS),
the read-modify-write sequences interleave and one of the bits is
silently lost:
- If has_child_subreaper is lost, the process's orphaned descendants
are reparented straight to init when it exits: find_new_reaper()
tests father->signal->has_child_subreaper first and never looks for
the ancestor subreaper. The service manager that registered as the
subreaper never gets SIGCHLD for them and cannot wait() on them.
- If is_child_subreaper is lost, the prctl returned 0 but the process
is not a subreaper: orphans of its descendants skip it and are
reparented to the next subreaper up or to init.
PR_GET_CHILD_SUBREAPER reads back 0.
- If timer_create_restore_ids is lost, the prctl returned 0 but
timer_create() does not restore the requested timer ids (or keeps
restoring them after _OFF), and _GET reads back the old value.
Documentation/memory-barriers.txt requires all fields of one bitfield
to be protected by one lock for exactly this reason. No lock is taken
or documented for these flags, and the prctls need no privilege, so any
process can hit the race against its own ancestors or descendants, or
between two of its own threads.
Make is_child_subreaper and has_child_subreaper separate bool members.
Each store then has its own memory location and no read-modify-write
of a neighbour. The readers and their locking are unchanged.
autoreap stays a bit-field and so shares a memory location only with
timer_create_restore_ids. That pair is race-free: autoreap is stored
exactly once, in copy_process() before the new signal_struct is
reachable by any other task, and timer_create_restore_ids is only
stored to the signal_struct of an already running process.
Fixes: 749860ce2427 ("prctl: propagate has_child_subreaper flag to every descendant")
Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Link: https://patch.msgid.link/20261011123440.768054-1-nickolai@csail.mit.edu
Acked-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- Taking a CPU offline could hang, or stall until the watchdog ejected
the BPF scheduler, when tasks on the dying CPU were still held by the
scheduler or sitting on a user dispatch queue. Re-enqueue them onto
the local queue when the runqueue goes offline so that the CPU pushes
them off like the other sched classes.
- The sequence number guarding against stale dispatches was per
runqueue, so a task re-enqueued on another CPU could get the same
number and a dispatch meant for its earlier instance was applied to
the new one. Use a per-task counter.
- A task dispatched to another CPU's local queue got its ops.dequeue()
only when picked to run and flagged as a core-sched pick. Call it at
insertion like for same-CPU dispatches.
* tag 'sched_ext-for-7.3-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Generate qseq from a per-task counter
selftests/sched_ext: Add a test for ops.dequeue() on remote local DSQ moves
sched_ext: Call ops.dequeue() when a task arrives on a remote local DSQ
sched_ext: Fix CPU hotplug hang when a dying CPU's tasks sit in the BPF scheduler
|
|
|
|
Signed-off-by: Christian Brauner <brauner@kernel.org>
|
|
Signed-off-by: Christian Brauner <brauner@kernel.org>
|
|
coredump_wait() sets core_state->nr_threads to the number of tasks
killed and waits for the last thread to enter coredump_task_exit() to
signal completion. Let's just wait on the count directly. The exiting
tasks can use atomic_dec_and_wake_up() and the dumping task sleeps in
wait_var_event_state().
The dumping task must remain freezable since commit f5d39b020809
("freezer,sched: Rewrite core freezer logic"). So keep the wait
TASK_UNINTERRUPTIBLE|TASK_FREEZABLE.
Drop the completion and rename nr_threads to threads_remaining.
No functional changes.
Suggested-by: NeilBrown <neilb@ownmail.net>
Link: https://lore.kernel.org/178899497961.207413.10554121774377911612@noble.neil.brown.name
Link: https://patch.msgid.link/20260910-work-coredump-unlock-self-v4-12-a5c1800dc930@kernel.org
Reviewed-by: NeilBrown <neil@brown.name>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
The core_state->dumper field isn't used anymore. Only its ->next pointer
is. The current task is always the dumping thread and the ->task pointer
is never read. Replace it with a plain pointer to the list of parked
threads.
Historically, core_state->dumper was used. Its ->task pointer was read.
by fill_note_info() started at &core_state->dumper to ensure that the
dumping thread came first in the ELF thread notes. That changed in
commit 4b0e21d64253 ("[elf][regset] simplify thread list handling in
fill_note_info()"). The first iteration was taken out of the loop. So
it's been unused ever since.
No functional changes.
Suggested-by: NeilBrown <neilb@ownmail.net>
Link: https://lore.kernel.org/178900159210.207413.8292125177519817528@noble.neil.brown.name
Link: https://patch.msgid.link/20260910-work-coredump-unlock-self-v4-10-a5c1800dc930@kernel.org
Reviewed-by: NeilBrown <neil@brown.name>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
|
|
finish_dispatch() uses the qseq embedded in p->scx.ops_state to tell
whether the QUEUED instance of a task it's about to claim is the one
scx_bpf_dsq_insert() saw. qseq is generated from rq->scx.ops_qseq, but
the counters of different rqs are independent, so if a task is dequeued
and re-enqueued on a different rq between scx_bpf_dsq_insert() and
finish_dispatch(), the new QUEUED instance can end up with the same
qseq as the old one:
CPU X CPU Z
----- -----
enqueue p on rq A, qseq = N
ops.dispatch()
scx_bpf_dsq_insert(p)
records qseq N
sched_setaffinity(p)
dequeue p from rq A
enqueue p on rq B, qseq = N
finish_dispatch(p, N)
qseq matches, p is claimed
The claim itself is still atomic so the core stays consistent, but an
insert issued for a previous QUEUED instance gets applied to a new one
which the BPF scheduler has just received through ops.enqueue(). This
breaks the guarantee that dispatches targeting a stale instance are
ignored.
Generate qseq from a per-task counter, p->scx.ops_qseq, instead so that
consecutive QUEUED instances of a task never share a qseq regardless of
which rq they're on. The counter is only updated in
scx_do_enqueue_task() with the task's rq locked, so no additional
synchronization is needed, and it fits in an existing hole in struct
sched_ext_entity on 64bit. Remove the now unused rq->scx.ops_qseq.
Never generate qseq 0. NONE and DISPATCHING don't carry a qseq, so
scx_bpf_dsq_insert() on a task in either state records 0. With a
per-task counter, every task's first QUEUED instance would otherwise get
qseq 0 and could be claimed by such an insert. Wrap the counter where
the QSEQ field wraps so that it can't reach a value that shifts to 0 on
32bit either.
Fixes: f0e1a0643a59 ("sched_ext: Implement BPF extensible scheduler class")
Cc: stable@vger.kernel.org # v6.12+
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Without proxy execution, the DSQ lock and holding_cpu handshake ensure
that a task cannot be dequeued or start running during an rq-lock
handoff without clearing holding_cpu.
Proxy execution is an exception: a task can start physically executing
as a lock owner while its scheduling context remains on a DSQ; its
on-CPU or migration-disabled state can therefore change without
clearing holding_cpu.
Recheck these states after acquiring the source rq lock. If the transfer
can no longer proceed, park the task on the source rq's reject DSQ and
reenqueue it through its owning scheduler. This preserves the BPF
scheduler's placement policy and keeps descendant tasks within their
sub-scheduler's cap grants.
Queue a deferred drain for every rejection. If a drain finds a
proxy-rejected task still running or donating, leave it parked and
record a retry request on the rq. scx_proxy_reenqueue_retry() consumes
the request and schedules another drain when proxy state changes.
Without this change and proxy execution enabled, stress-ng --pipeherd
can trigger this race and migrate an active execution context, leading
to sleeping-while-atomic warnings and subsequent lockdep corruption.
This is a preparatory change to support proxy execution with sched_ext.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
With proxy execution, pick_next_task() can select a blocked task as the
scheduling context before find_proxy_task() resolves the execution
context.
From the BPF scheduler perspective, that donor is running while its
scheduling context drives the lock owner; ops.tick() and other
accounting must therefore remain enclosed by a matching
ops.running()/ops.stopping() session.
In this scenario, the session boundaries do not always match physical
task switches. Keep the "running" session open when the same donor
continues on the same CPU. When proxy execution migrates a donor's
scheduling context to another CPU, end its running session on the source
CPU and start a new session on the destination CPU only after proxy
resolution succeeds. This prevents a failed resolution from exposing a
provisional ops.running() event.
Track these sessions with a new SCX_TASK_RUN_TRACKED flag. The explicit
running-state tracking is also required by later donor-based accounting:
it prevents an EXT owner executing for a non-EXT donor from being
treated as the active EXT scheduling context when it is dequeued.
This is a preparatory change for enabling proxy execution together with
sched_ext.
Acked-by: John Stultz <jstultz@google.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into for-7.4
Pull to receive:
313b652837d0 ("sched/core: Drop mutex locks before proxy rescheduling")
8f8c0417e973 ("sched/core: Dequeue waking proxy donors before reset")
a49653d0abeb ("sched/core: Mark wakeups completed through ttwu_runnable()")
57c75e3ae38c ("sched: Add helper to block retained proxy donors")
be100c77178e ("sched: Add sched_ext hooks for proxy execution")
for the upcoming sched_ext side of proxy execution support.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_dispatch_enqueue() calls rb_add() for a vtime-ordered DSQ while
holding dsq->lock. At each level of the descent, scx_dsq_priq_less()
loads the visited task's dsq_vtime, and that comparison selects rb_left
or rb_right from the same task's dsq_priq. In the x86-64 benchmark
configuration, these accesses fell on different 64-byte cache lines
before this change, so a cold visited task could require a second cache
line fill on the dependent descent path.
Move dsq_vtime immediately before dsq_priq, keeping dsq_seq and
dsq_flags next to dsq_list. In the two x86-64 layouts inspected, where
scx starts at offsets 776 and 840 in task_struct, the fields from
dsq_list through dsq_priq occupy one cache line. The exact cache line
placement depends on the containing task_struct layout and is not
guaranteed for every configuration or architecture. Where they share a
line, avoiding the dependent cache line fill helps reduce dsq->lock hold
time during vtime insertion and therefore contention on the lock.
In a 16-vCPU KVM guest, the instrumented enqueue interval was 4-6%
shorter with scx_lavd and scx_layered. scx_mitosis and end-to-end
workload time showed no consistent change.
This only reorders fields; no scheduling behavior change is intended.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
underestimation bug
The scheduler scales LLC capacity by the fraction of cache-sharing CPUs
covered by a domain:
llc_bytes = cache_size * span_weight / shared_weight
During CPU teardown, sched_cpu_deactivate() rebuilds scheduler domains
before cacheinfo_cpu_pre_down() removes the CPU from shared_cpu_map. The
new domains therefore use the old sharing weight. The later call to
sched_update_llc_bytes() looks up the departing CPU's sd_llc, which has
already been detached, and returns without correcting the surviving CPUs.
On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC,
offlining one SMT sibling left the remaining CPUs with:
llc_bytes = floor(16777216 * 11 / 12) = 15379114 bytes
The correct capacity is still 16777216 bytes. On systems with active
cache-aware scheduling, an underestimated capacity can cause
exceed_llc_capacity() to reject aggregation for a process whose footprint
would fit. Unchanged cpuset partitions sharing the physical cache can
also retain stale capacity when a CPU comes online in another partition.
Pass the cache-sharing mask already retained by cacheinfo to the
scheduler update. Refresh every surviving CPU using its own LLC domain
so that each partition receives the correct share. This also preserves
the correction needed as cache-sharing maps grow during boot.
Keep the existing CPU-hotplug and scheduler-domain synchronization. The
update remains on the hotplug path; no steady-state scheduling operation
or persistent allocation is added.
Fixes: 7030513a0877 ("sched/cache: Calculate the LLC size and store it in sched_domain")
Signed-off-by: Davi Chaves Azevedo <davichazbh@gmail.com>
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: Chen Yu <yu.c.chen@intel.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: <stable@kernel.org> # v7.2.x
Link: https://patch.msgid.link/6751d93e15889e624796c74db0bfe66603d60b1b.1790035273.git.tim.c.chen@linux.intel.com
|
|
The fair scheduling class can request lazy rescheduling, deferring an
in-kernel scheduling boundary until returning to user space or until the
next scheduler tick. sched_ext only exposes immediate preemption,
preventing BPF schedulers from making the same trade-off.
Add SCX_ENQ_PREEMPT_LAZY and SCX_KICK_PREEMPT_LAZY. Both expire the
current sched_ext task slice but request lazy rescheduling. Immediate
preemption, WAIT and plain kicks take precedence when requests are
combined, while a lazy enqueue to a non-local DSQ retains the
head-insertion semantics of SCX_ENQ_PREEMPT.
Add SCX_OPS_LAZY_RESCHED as the default slice-expiry policy for newly
enabled tasks and initialize it before ops.enable(). Add
scx_bpf_task_set_lazy_resched() so the owning scheduler can override the
persistent policy per task from any callback while preserving
sub-scheduler task ownership boundaries. Bypass continues to force
immediate rescheduling.
For an infinite-slice task whose tick is stopped, set the lazy
reschedule request before restoring the scheduler tick dependency. This
lets the dependency update's IPI serve the request on return to user
space and guarantees that the next tick promotes it otherwise. Use
pinned rq locking for remote kick delivery so RQCF_UPDATED reflects
clock updates since lock acquisition.
Accumulate kick requests independently and resolve precedence while
holding the target rq lock. Reject invalid SCX_KICK_IDLE combinations
and unknown kick flags, which were previously silently ignored.
Tested-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Move the POSIX timer group exit handling into the posix timer code and
provide a proper stub when POSIX timers are disabled in Kconfig.
No functional change.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Kijo Park <red993688@gmail.com>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260911090541.733966325@kernel.org
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs fixes from Christian Brauner:
- netfs:
- Fix an uninitialized return value in netfs_unbuffered_write()
when preparing the first subrequest fails
- For partial unbuffered/DIO writes return the amount transferred
rather than an error
- Update i_size with the amount actually written when a partial
transfer ends in an error
- Fix a subrequest reference leak when the io_iter ends up empty
- Handle netfs_alloc_subrequest() failure during unbuffered writes
- Load all readahead folios into the rolling buffer upfront and
drop the readahead references once the first subrequest is
dispatched
- Mark folios for copy-to-cache while issuing subrequests
- Fix read progress reporting
- afs:
- Add the missing kunmap in the error path of afs_dir_search_bucket()
- Fix a double kunmap in afs_edit_dir_remove()
- Don't free an existing server's endpoint state when cleaning up a
candidate server in afs_lookup_server()
- Unbind peers removed from a server's address list
- ufs:
- Load the cylinder group metadata before creating the root dentry
- Validate the cylinder group index and rotor positions before
caching them
- Treat an unreadable directory block as not empty
- exec:
- Close the close-on-exec files before taking exec_update_lock
Closing a file can block on the filesystem, so a hung filesystem
blocked everything that takes exec_update_lock and a FUSE server
inspecting the calling process could deadlock
- Drop the bprm loader before closing bprm->file in free_bprm()
- exit: Hold a reference to thread_pid across proc_flush_pid()
- reboot: Fix a use-after-free on cad_pid
- nsfs: Keep the namespace tree fields out of the rcu_head used by
kfree_rcu()
- nstree: Check listing permission before taking a namespace
reference in listns()
- super: Return 0 when a nested thaw drops its hold while other
freezers remain
- ext4: Don't set I_METADATA_WRITEBACK during fastcommit replay
- adfs: Free s_fs_info in ->kill_sb()
- autofs: Free the inode info allocated in autofs_fill_super() when
the root inode allocation fails
- ovl: Return EINVAL instead of EIO on a user namespace mismatch now
that it's a plain refusal and not an internal error
- cachefiles: Don't cast the variable-length coherency data to a
__be64 in the coherency tracepoint
* tag 'vfs-7.3-rc3.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (28 commits)
nstree: check listing permission before taking a namespace reference
exec: do_close_on_exec() before taking exec_update_lock
exit: hold a reference to thread_pid across proc_flush_pid
fs: autofs: fix memory leak in autofs_fill_super()
exec: Drop bprm loader before closing bprm->file
afs: Clear stale peer app data after address list changes
afs: Fix incorrect free in candidate cleanup in afs_lookup_server()
afs: Fix double-unmap of directory block
afs: Fix missing kunmap in afs_dir_search_bucket()
ovl: return EINVAL instead of EIO in case of mismatched user_ns
reboot: fix cad_pid use-after-free race
cachefiles: Fix potential UAF/KASAN warning
netfs: Fix read progress reporting
netfs: Mark folios with COPY_TO_CACHE whilst issuing subreqs
netfs: Fix readahead synchronisation issues by loading all folios upfront
netfs: break unbuffered write when netfs_alloc_subrequest() fails
netfs: Fix subreq ref leak
netfs: Fix i_size update for partial transfer
netfs: Fix error vs transferred passed to ->ki_complete()
netfs: Fix unbuffered/DIO write partial transfer error return
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking fixes from Ingo Molnar:
- Fix a softirq processing delay bug in local_interrupt_disable(),
which should mostly only affect the Rust runtime (Boqun Feng)
- Remove the hardirq_disable_count() function which caused the
previous bug and is now unused & unnecessary (Boqun Feng)
- lockdep: Invalidate stale class_cache entries for zapped classes
(Eric Dumazet)
- Fix rt_mutex specific futex scheduling helpers
(Sebastian Andrzej Siewior)
- Fix rcuwait use-after-free race during futex requeue PI (Yao Kai)
* tag 'locking-urgent-2026-09-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
futex: Prevent rcuwait use-after-free during requeue PI
futex: Provide rt_mutex_.*_schedule() equivalents for futex scheduling
locking/lockdep: Invalidate stale class_cache entries for zapped classes
preempt: Remove hardirq_disable_count()
interrupt: Disable interrupt before modifying hardirq_disable counter
|
|
There is rt_mutex_{pre|post}_schedule() around
rt_mutex_wait_proxy_lock() to ensure that sched_submit_work()/
sched_update_worker() is invoked before we schedule out and block on
rt_mutex while waiting for it become available.
The reason is that blocking on rt_mutex assigns a pi_waiter for the PI
chain and sched_submit_work() will also assign a pi_waiter if it blocks
on lock but a this point we already have a waiter assigned.
We can't skip sched_submit_work() entirely because I/O relies on the
fact that I/O queue is flushed while it blocks on a sleeping lock.
Therefore sched_submit_work() is moved before we block on the lock.
Sleeping lock in this context means mutex or rw_semaphore not spinlock_t
on PREEMPT_RT. Because the mutex abstraction on PREEMPT_RT uses the same
abstraction as the futex proxy lock, the futex code ended up using
rt_mutex_{pre|post}_schedule(), too.
Using it is/ was just to keep the task_struct::sched_rt_mutex assertion
happy. Futex proxy lock is used only in the syscall context of a task.
At this point it never got any I/O that needs to be flushed and it can't
be a workqueue that needs to notify that it will be scheduled out.
Therefore sched_submit_work() does nothing here.
By mistake futex_wait_requeue_pi() -> rt_mutex_wait_proxy_lock() did not
get the rt_mutex_{pre|post}_schedule() annotation. This was not noticed
because in this callchain the lock is (usually) not contended and so
rt_mutex_slowlock_block() does not schedule, triggering the assert.
Adding rt_mutex_pre_schedule() here looks wrong (as noted by PeterZ)
because at this point there is a pi_waiter recorded and invoking
sched_submit_work() with a possible lock contention would be wrong.
Add rt_mutex_futex_{pre|post}_schedule() which toggles the
sched_rt_mutex assert and does not involve sched_submit_work(). Add
asserts here to ensure that sched_submit_work() would do nothing. Use it
only in futex proxy lock case which is rt_mutex_wait_proxy_lock().
Remove it from futex_lock_pi().
Fixes: d14f9e930b90 ("locking/rtmutex: Use rt_mutex specific scheduler helpers")
Reported-by: Yao Kai <yaokai34@huawei.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260901135453.3121948-2-bigeasy@linutronix.de
Closes: https://lore.kernel.org/all/20260717084922.4153317-2-yaokai34@huawei.com
|
|
secretmem accounts folios by treating memory as if it were mlock()'d and
thus limited by the RLIMIT_MEMLOCK limit.
However the folios are unevictable and remain so until the inode is
evicted, eliminating usual mlock() semantics - mapping folios then
unmapping them does not clear their unevictable state, since it depends on
AS_UNEVICTABLE, not PG_mlocked.
A user can therefore easily work around the RLIMIT_MEMLOCK limit - simply
map then unmap and VmLck no longer counts the secretmem range. Worse,
folios are not accounted in the process's RSS, meaning the OOM killer
won't know to kill the process.
Repeatedly mapping/unmapping (or forking) can then result in the
consumption of all available system memory with unevictable folios and
cause system instability.
A secretmem fd can be passed between processes and over fork so a
per-process limit simply does not make sense, so follow the precedent set
by io_uring, perf, skbuff, iommufd and xdp by tracking the number of
locked pages in user_struct->locked_vm.
Since the scope tracked is actually inode lifetime, the RLIMIT_MEMLOCK
applies per-user not per-process, so it doesn't make sense to bypass for
users with CAP_IPC_LOCK, therefore remove this bypass.
There is simply no reason to carry on marking the mapping as mlock()'d
since it's misleading and the lifecycle is now correctly handled, so
remove this too.
Note that secretmem does not support any form of truncation (including
hole punching) and the folios are unreclaimable, so the folios need only
be accounted on fault and unaccounted on inode destruction.
__secretmem_account_pages() is more or less a duplicate of the code that
io_uring etc. use, but since this is a bug fix that needs backporting,
defer any de-duplication efforts to a follow-up.
test_mlock_limit() asserts mlock_future_ok() on mmap(), however this has
been removed, so remove the test altogether for the fix. A new test will
be sent separately for upstream.
Link: https://lore.kernel.org/20260826-secretmem-accounting-v3-1-94cb04399510@kernel.org
Fixes: 1507f51255c9 ("mm: introduce memfd_secret system call to create "secret" memory areas")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reported-by: Daehyeon Ko <4ncienth@gmail.com>
Closes: https://lore.kernel.org/linux-mm/20260813225328.2010303-1-4ncienth@gmail.com/
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Daehyeon Ko <4ncienth@gmail.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David S. Miller <davem@davemloft.net>
Cc: Hagen Paul Pfeifer <hagen@jauu.net>
Cc: Jakub Kacinski <kuba@kernel.org>
Cc: James Bottomley <james.bottomley@HansenPartnership.com>
Cc: Jesper Dangaard Brouer <hawk@kernel.org>
Cc: John Fastabend <john.fastabend@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Stanislav Fomichev <sdf@fomichev.me>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
paravirt_steal_rq_enabled and paravirt_steal_enabled use raw static_key
APIs which are now deprecated. Use the new API instead.
No functional change.
Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/20260819081207.12150-1-hongyan.xia@transsion.com
|
|
Only user DSQs use the per-DSQ deferred reenqueue state:
schedule_dsq_reenq() serves the local DSQ through
sch->pcpu->deferred_reenq_local and rejects all other built-in DSQ ids.
Every DSQ, built-in or not, still allocates a struct scx_dsq_pcpu per
possible CPU. As the local, reject, rescue and bypass DSQs are themselves
per-CPU, the wasted memory grows with the square of the CPU count, most of
it at boot before any scheduler is loaded.
Skip the allocation for built-in DSQ ids. While at it, rename ->pcpu to
->pcpu_user so that the per-CPU data reads as user-DSQ only.
Signed-off-by: fangqiurong <fangqiurong@kylinos.cn>
Reviewed-by: Zhan Xusheng <zhanxusheng@xiaomi.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
A bare "idle" reads as CPU idle state in sched_ext (ops.update_idle(),
idle cpumasks, scx_bpf_pick_idle_cpu()). cpu.idle is the cgroup analog
of the SCHED_IDLE policy, so name the task_group field sched_idle to
match scx_cgroup_init_args.sched_idle.
Pure rename, no behavior change.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
cad_pid is a single kernel-wide struct pid pointer. proc_do_cad_pid()
reads it and passes it to pid_vnr() without protecting the lifetime of
the referenced struct pid. A concurrent writer can replace cad_pid and
drop the final reference to the old struct pid after the reader has
loaded the pointer but before pid_vnr() has finished dereferencing it,
causing a use-after-free.
kill_cad_pid() has the same lifetime race when it passes cad_pid to
kill_pid().
At the time this issue was reported, an unprivileged user could reach the
sysctl through user and PID namespaces because cad_pid was registered in
pid_table[]. Moving cad_pid back to the global reboot sysctl table
corrected that namespace and permission mismatch, but did not fix the
underlying lifetime race.
Fix this by treating cad_pid as an RCU-protected pointer at both read
sites and by waiting for a grace period before dropping the old reference
on the write side.
call_rcu(&old_pid->rcu, ...) cannot be used here because free_pid()
also queues pid->rcu; queueing the same rcu_head twice can corrupt the
RCU callback list.
Original KASAN crash stack:
kernel/pid.c:545 pid_nr_ns() # reads freed pid->level
kernel/pid.c:556 pid_vnr() # calls pid_nr_ns()
kernel/pid.c:775 proc_do_cad_pid() # calls pid_vnr(cad_pid)
Fixes: 9ec52099e4b8 ("[PATCH] replace cad_pid by a struct pid")
Reported-by: AutonomousCodeSecurity@microsoft.com
Closes: https://lore.kernel.org/all/20260717210143.4734-1-blbllhy@gmail.com/
Link: https://lore.kernel.org/all/alz5ZYLE4kaq_v2P@redhat.com/
Link: https://lore.kernel.org/all/al4ICz9biJKtdZc4@redhat.com/
Suggested-by: Mateusz Guzik <mjguzik@gmail.com>
Suggested-by: Bradley Morgan <include@grrlz.net>
Suggested-by: Oleg Nesterov <oleg@redhat.com>
Suggested-by: Eric W. Biederman <ebiederm@xmission.com>
Suggested-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Cc: stable@vger.kernel.org
Signed-off-by: Cen Zhang (Microsoft) <blbllhy@gmail.com>
Link: https://patch.msgid.link/20260814040944.16561-1-blbllhy@gmail.com
Reviewed-by: Bradley Morgan <include@grrlz.net>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
TIF_NOTIFY_SIGNAL is used to kick a task in uninterruptible sleep to
return to userspace and run task work and then go back to sleep. This
mechanism works well but breaks e.g., coredumps. dump_interrupted() only
allows fatal signals to interrupt a coredump and the whole regular write
path going to actual filesystems is impervious to TIF_NOTIFY_SIGNAL as
well.
Add PF_NO_NOTIFY_SIGNAL and helpers to raise and restore it. This is the
same approach as memalloc_nofs_save(). signal_pending() will not report
a fake pending signal via TIF_NOTIFY_SIGNAL if inside a
PF_NO_NOTIFY_SIGNAL section.
No functional changes.
Link: https://patch.msgid.link/20260824-work-tif_notify_signal-v2-1-6609e42b3157@kernel.org
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull futex fixes from Ingo Molnar:
- Enforce that the private futex owner shares the mm when attaching
(Kyle Zeng, Thomas Gleixner)
- Fix race on the initial mm->futex.phash.ref allocation (Hyunwoo Kim)
- Fix might_sleep() warning in futex_pivot_pending() (Peter Zijlstra)
* tag 'locking-urgent-2026-08-22' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
futex: Fix might_sleep() warning in futex_pivot_pending()
futex: Fix race on the initial mm->futex.phash.ref allocation
futex: Clean up the redundant exit/exec functions
futex/pi: Plug private futex exec() race
futex: Sanitize and document task_struct::futex::state transitions
futex/pi: Reject cross-mm private futex owners
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull MM updates from Andrew Morton:
- "mm: drop "sub" prefix from various places" (Dev Jain)
page->folio conversion and a naming cleanup
- "mm/kasan: remove redundant initialization for kasan_flag_write_only"
(Igor Putko)
KASAN cleanup work
- "mm/filemap: reduce unnecessary xarray lookups" (Chi Zhiling)
Small speedup in the pagecaache read code
- "mm/percpu: Fix possible NOFS/NOIO reclaim recursion" (Kaitao Cheng)
Improve the vmalloc code - mainly the avoidance of GFP_KERNEL
allocations when the caller asked for GFP_NOFS or GFP_NOIO
- "mm/kmemleak: avoid soft lockup when scanning task stacks" (Breno
Leitao)
Avoid a soft lockup watchdog trigger from the kmemleak scanning code
in extreme situations
- "mm/page_owner: misc cleanups" (Ye Liu)
Cleanups to the page_owner code. For some reason lots of people have
been working on the page_owner code this cycle.
- "mm: convert to walk_page_range_vma() to eliminate find_vma()"
(Kefeng Wang)
Simplify and accelerate the page walking library function
- "mm/migrate: preparatory cleanups for batch copy and offload"
(Shivank Garg)
Cleanups in the migration code
- "mm/page_owner: add per-fd filter infrastructure for print_mode and
NUMA filtering" (Zhen Ni)
Per-fd filtering to page_owner in order to reduce the sometimes vast
amount of output it can produce
- "mm: Refactor bootmem gigantic hugepage allocation" (Muchun Song)
Fixes and preparatory cleanups around bootmem HugeTLB handling,
sparse initialization ordering, and related vmemmap setup
- "mm/zsmalloc: reduce lock contention in zs_free()" (Wenchao Hao)
Reduce lock contention in zs_free(), which dominates the unmap path
under memory pressure on Android (LMK kills) and on x86 servers
running zswap-heavy workloads.
Up to 1.83x improvement in microbenchmarking.
- "move alloc_tag.c file under mm/" (Suren Baghdasaryan)
- "samples/damon: handle damon_{start,stop}() failures" (SJ Park)
Fix improper handling of damon_start(), damon_stop(), and
damon_call() failures across DAMON sample modules to prevent
potential memory leaks, operation disruptions and use-after-free
bugs
- "mm/damon/sysfs: kobject_del() directories that users can
create/remove" (SJ Park)
Fix delayed sysfs directory removal under DEBUG_KOBJECT_RELEASE
causeing creation failures due to duplicate directory names by adding
missing kobject_del() calls before creating new directories
- "mm: cleanup clear_not_present_full_ptes()" (David Hildenbrand)
Clean up the core pte handling code
- "selftests/damon: misc fixes for test bugs" (Kunwu Chan)
Fix several bugs in the DAMON selftests
- "selftests/damon: fix memcg_path staging handling" (Cheng Nie)
Fix a bug in _damon_sysfs.py for damos_filter memcg_path setup, and
add a test case for it in sysfs.py.
- "selftests/damon: test kdamond refresh_ms" (Ruslan Valiyev)
Selftest coverage for DAMON's refresh_ms sysfs feature by updating
the test control module and verifying that scheme stats update
automatically without manual intervention
- "mm/damon: five misc fixups" (Akinobu Mita)
Miscellaneous DAMON fixups.
- "mm/damon/core: detect internal variation above max_nr_regions/2"
(Jiayuan Chen)
Fix DAMON's region splitting behavior when region counts exceed half
the maximum budget by dynamically scaling down the split fraction as
the limit approaches, preventing large regions from staying un-split,
and add corresponding KUnit test coverage
- "mm: preparatory patches for PMD level swap entries" (Usama Arif)
Refactor and clean up PMD softleaf helpers, call sites, and
architecture flags to lay the groundwork for a follow-up series that
introduces PMD page table swap entries
- "mm/damon: update, optimize, and clean up doc, tests, and code" (SJ
Park)
Update DAMON design and ABI documentation, expands unit and selftest
coverage, optimize damon_commit_target_regions(), and clean up
recently added sysfs interface code for better readability
- "mm/vmpressure: reduce CPU, memory and code overhead on cgroup v2"
(Usama Arif)
Optimize vmpressure() by skipping unnecessary work on cgroup v2 for
userspace event notifications and refactor v1-only eventfd handling
into mm/memcontrol-v1.c to reduce memory overhead and code complexity
- "selftests/mm: refactor pkey helpers and fix mmap error handling"
(Hongfu Li)
Refactor pkeys shared tracing and assertion helpers into a common
file, unify protection key selftests to use consistent diagnostic
logging and assertions, and enforce standardized MAP_FAILED return
checks for mmap() calls across the tests
- "mm/damon: optimize out nr_accesses_bp" (SJ Park)
Replace the error-prone, continuously updated nr_accesses_bp field in
damon_region with an on-demand moving sum function, reducing
structure memory overhead and avoiding state corruption bugs
- "Open HugeTLB allocation routine for more generic use" (Ackerley Tng)
Decouple HugeTLB folio allocation from VMA dependencies by
introducing hugetlb_alloc_folio(), enabling subsystems like
guest_memfd to allocate HugeTLB folios without standard VMA
reservations or pseudo-VMAs
- "mm/damon: provide pseudo moving sum probe_hits" (SJ Park)
Integrate DAMON's probe_hits attribute counter into the pseudo moving
sum infrastructure, enabling real-time, online monitoring without
waiting for full aggregation intervals
- "mm: Some cleanups for page allocator APIs" (Brendan Jackman)
Simplify and refactor the page allocator entry points and flags by
unifying allocation paths, adding internal alloc_flags arguments, and
eliminating redundant __ prefixed alloc_pages variants.
- "Fix incorrect access of hugetlb pte entries" (Dev Jain)
Enforce the consistent use of huge_ptep_get() instead of ptep_get()
for HugeTLB entries and fixes an unaligned address issue in arm64's
huge_ptep_get() implementation
- "mm/damon: validate all parameters in the core" (SJ Park)
Consolidate parameter validation into the DAMON core specifically
within damon_start() and damon_commit_ctx() to centralize error
checking, eliminate caller-side redundant checks and to improve
maintenance efficiency
- "tools/mm/page_owner_sort: fix filtering and cleanup issues" (Yichong
Chen)
Rename is_need() to filter_record() for clearer return semantics, fix
per-record allocation memory leaks and bound output copies in
search_pattern() to address an existing buffer issue
- "memcg: bail out reclaim when memcg is dying" (Jiayuan Chen)
Mitigate a system-wide stall which occurs when a cgroup is removed
while one of its memory control files is doing synchronous reclaim
- "mm/memory-failure: add panic option for unrecoverable pages" (Breno
Leitao)
Introduce an opt-in vm.panic_on_unrecoverable_memory_failure sysctl
that immediately panics the kernel on unrecoverable memory errors in
kernel-owned pages to preserve error context and prevent delayed,
silent data corruption
- "mm/damon: refactor damon_{start,stop,commit}() for simple error
handling" (SJ Park)
Refactor the DAMON core API functions to guarantee that all contexts
are fully stopped when damon_start(), damon_stop(), or damon_commit()
fail, eliminating the need for complex and error-prone caller-side
cleanup code
- "Keep tail page private zero at free and folio split" (Zi Yan)
Add checks to ensure tail_page->private is zero when freeing compound
or high-order pages and when promoting tail pages during large folio
splits. By validating these fields at free and split time, it allows
the removal of redundant private field clearing inside
prep_compound_tail()
- "mm: drop redundant lru_add_drain in anon folio reuse paths" (Barry
Song)
Eliminate redundant lru_add_drain() calls in
wp_can_reuse_anon_folio() and do_swap_page() to reduce LRU lock
contention and system overhead
By validating folio refcounts against the LRU cache before draining
and removing unnecessary drains in the swap path, it achieves up to a
30.5% reduction in drain calls during heavy swap workloads
- "mm: clean up folio LRU and swap declarations" (Jianyue Wu)
Reorganize folio LRU and swap code by relocating page-cluster state
to mm/swap_state.c, renaming mm/swap.c to mm/folio.c, and moving
MM-internal reclaim declarations into mm/internal.h.
- "userfaultfd: working set tracking for VM guest memory" (Kiryl
Shutsemau)
Add userfaultfd support for tracking the working set of VM guest
memory, so a VMM can identify hot pages and reclaim cold ones to
tiered or remote storage
- "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 2)" (David
Hildenbrand)
Remove the remaining pieces of CONFIG_HAVE_BOOTMEM_INFO_NODE,
performing some smaller cleanups around freeing of reserved vmemmap
pages on the way.
- "mm/damon: update probe hits for runtime parameter commits" (SJ Park)
Ensure that DAMON's probe_hits attribute counter is properly updated
when monitoring intervals are changed at runtime, matching the
behavior of nr_accesses. To achieve this, it refactors and renames
existing helper functions for shared use, applies the updates to
probe_hits, and handles edge cases in damon_probe_hits_mvsum() to
maintain measurement accuracy.
- "KSM: performance optimizations for rmap_walk_ksm" (xu xin)
Resolve a severe KSM reverse-mapping performance bottleneck where
thousands of split VMAs sharing a single anon_vma cause extended lock
contention.
By adding an interval-filtering check during the rmap walk, it
reduces worst-case anon_vma lock hold times from over 500ms down to
under 2ms, preventing application freezes and latency spikes under
memory pressure.
- "mm: split a couple of headers from internal.h" (Mike Rapoport)
Split declarations related to mm_init, memblock, vmalloc and sparse
into new headers
- "KSM: use linear_page_index in collect_procs_ksm()" (xu xin)
Apply the interval tree optimization from rmap_walk_ksm() to
collect_procs_ksm() to avoid iterating over non-matching VMAs during
KSM memory error handling.
It hoists loop-invariant address initialization and restricts the
anon_vma_interval_tree_foreach walk to a targeted page offset range,
reducing redundant checks and improving lookup efficiency.
- "selftests/mm: avoid false failures in hugetlb and KSM tests" (Sayali
Patil)
Fix issues in the hugetlb and KSM MM selftest categories that can
report failures when the prerequisites for the tests are not
satisfied
- "mm/damon: introduce data attributes only monitoring" (SJ Park)
Introduce attribute-weighted region management in DAMON, allowing
users to prioritize specific data attributes (such as page sizes or
cgroups) over or instead of access monitoring.
By assigning weights to attribute probes, DAMON can completely
disable access tracking and adjust monitoring regions based on
weighted probe-hit counters to optimize monitoring quality for
attribute-focused workloads.
- "mm/hmm: Add mmap lock-drop support for userfaultfd-backed mappings"
(Stanislav Kinsburskii)
Extend hmm_range_fault() to support userfaultfd-backed regions by
allowing the mmap lock to be dropped during fault handling via a new
hmm_range_fault_locked() helper.
By accepting a locked pointer and signaling retry status when lock
release occurs, it enables page fault resolution in userfaultfd
regions while preserving backward compatibility for existing callers.
- "mm: make VMA page offset handling more consistent" (Lorenzo Stoakes)
Clean up and standardize how vma->vm_pgoff is accessed and
manipulated across file-backed and anonymous mappings in the kernel
It introduces dedicated helper functions such as vma_start_pgoff(),
vma_end_pgoff(), vma_set_pgoff() and linear_page_delta() while
renaming rmap interval tree helpers to better reflect their
functionality.
These changes establish a cleaner foundation for future work that
will unify virtual page offset indexing for all anonymous and CoW'd
folios.
- "mm: handle device-private PMDs in walk callbacks" (Usama Arif)
Address kernel panics and state corruption caused by MM walk
callbacks reaching non-present device-private PMD swap entries
created during HMM migrations
It ensures that functions which acquire pmd_trans_huge_lock()
properly recognize device-private PMDs instead of assuming a present
THP or a standard migration entry.
- "mm/rmap: Refactor try_to_unmap_one" (Dev Jain)
Refactor try_to_unmap_one by modularizing Hugetlb,
anonymous-lazyfree, and anonymous-swapbacked logic into dedicated
functions, laying the structural groundwork for batched anonymous
large folio unmapping.
- "Docs/ABI/damon: sysfs ABI document fixes and additions" (Song Hu)
Fix typos and fills in missing entries in the DAMON sysfs ABI
document
- "dax/kmem: atomic whole-device hotplug via sysfs" (Gregory Price)
Introduce an atomic sysfs state attribute and supporting DAX/MM
infrastructure to prevent userland races when offlining and removing
entire memory regions
By adding an unplugged state alongside standard online modes, it
enables whole-device atomic hotplug control while preserving backward
compatibility.
- "mm: convert more vm_flags_t users to vma_flags_t" (Lorenzo Stoakes)
Continue transitioning the kernel from the deprecated vm_flags_t type
to vma_flags_t across core memory management infrastructure.
It replaces legacy type usage in core functions such as do_mmap(),
unmapped area allocation, mm->def_vma_flags, and VMA operations like
mlock, mprotect, and mremap.
- "Two small patches to clean up mm/mm_slot.h" (xu xin)
Refactor mm_slot.h by introducing mm_slot_remove() to unify duplicate
slot deletion sequences in khugepaged and KSM. It also adds code
documentation explaining why mm_slot_lookup and mm_slot_insert must
remain as preprocessor macros rather than static inline functions.
- "mm/damon/core: hide core-private struct fields" (SJ Park)
Clean up DAMON core structures by consistently marking internal-only
fields with private: comment tags to prevent improper direct access
from outer layers.
It enforces encapsulation across core structures including
damon_region, damon_target, and damon_ctx and updates DAMON_SYSFS to
interact through approved access APIs instead of exposing raw struct
members.
- "mm/damon: unurgent fixes for infinite loop, NULL de-ref and races"
(SJ Park)
Address potential infinite loops, NULL dereferences, and race
conditions identified in DAMON
It fixes an infinite loop triggered by extreme user configurations, a
NULL pointer dereference within unit tests and minor monitoring
accuracy degradation caused by subtle runtime races.
- "mm/page_alloc: fixes for free_pages_nolock() on RT/UP" (Brendan
Jackman)
Fix an NMI safety flaw in __free_frozen_pages() where freeing pages
on non-SMP or PREEMPT_RT kernels can bypass can_spin_trylock() checks
via non-PCP or isolated migration paths.
It also resolves potential kernel crashes and privilege escalation
risks triggered when BPF tracing runs in NMI context alongside memory
hotplug or large allocation frees.
- "mm/page_alloc: couple of followups for recent cleanups" (Brendan
Jackman)
Clean up and update page allocator nomenclature, documentation, and
debug assertions.
It aligns internal FPI_ flags with the public "nolock" naming
convention, removes outdated internal implementation details from
high-level page allocator comments, and eliminates obsolete
VM_BUG_ON() assertions in allocation paths.
- "mm/mseal: further cleanups" (Lorenzo Stoakes)
Refactor and simplify the mseal implementation by clarifying API
boundaries and removing unnecessary code complexity.
It replaces generic do_mseal() usage outside the syscall with a
dedicated mseal_mmap_page_zero() helper for MMAP_PAGE_ZERO,
eliminates mm_struct parameters to enforce that sealing applies only
to current->mm, and streamlines overall logic and comments with no
functional changes intended.
- "mm/vmscan: fix swappiness=max and clean up per-node proactive
reclaim" (Ridong Chen)
Resolve reclaim behavior bugs and clean up function parameters across
memory reclaim paths
It fixes swappiness=max in both standard reclaim and MGLRU so
unswappable anonymous memory no longer falls back to evicting page
cache, ensures reclaim_store() returns accurate error codes instead
of collapsing all failures into -EAGAIN, and removes the obsolete
gfp_mask parameter from __node_reclaim().
- "mm: mincore: misc cleanups" (Kefeng Wang)
Clean up and simplifies the mincore code. Most importantly, it
removes the historical special behavior that always reports VM_PFNMAP
pages as non-resident.
- "mm/huge_memory: drop dead split helper variants" (Kiryl Shutsemau)
Two trivial cleanups in the folio split API
- "mm/damon: fix uninitialized DAMOS field and kunit exec expectation
bugs" (SJ Park)
Resolve minor operational and testing bugs in DAMON identified by
Sashiko. It initializes the damos->last_applied field to prevent
occasional efficiency degradation and fixes invalid memory accesses
in DAMON KUnit tests during test failure handling.
- "cleanup for stable_page_flags()" (Jinjiang Tu)
Clean up and refactor stable_page_flags() used by /proc/kpageflags
without altering functionality.
It uses BIT_ULL() to prevent shift-overflow warnings on 64-bit flag
bits, converts folio-specific flag checks to standard folio_test_*()
helpers, and removes redundant CONFIG_PAGE_IDLE_FLAG handling.
- "Batch unmap of uffd-wp file folios" (Dev Jain)
Extend batched folio unmapping support to file folios within
userfaultfd write-protect (uffd-wp) VMAs by adding batching
capabilities to pte_install_uffd_wp_if_needed().
This removes special-case restrictions on uffd-wp VMAs in
try_to_unmap_one(), significantly simplifying the function's control
flow and complexity.
- "mm/early_ioremap: clarify and clean up early_ioremap_reset()"
(Sang-Heon Jeon)
Clarify and clean up the architecture-specific usage of
__late_set_fixmap() and __late_clear_fixmap() after
early_ioremap_reset()
It adds explicit documentation regarding when early_ioremap_reset()
must be called and removes redundant macro definitions and reset
calls in the RISC-V and ARM64 architectures.
- "mm: fix reclaim storms in defrag_mode" (Johannes Weiner)
Address severe performance regressions, swap storms, and spurious
OOMs caused by vm.defrag_mode=1 under high memory pressure in Meta
production
It updates the page allocator slowpath so non-movable allocation
requests actively trigger direct reclaim and direct compaction at
pageblock_order scale, allowing them to claim whole pageblocks rather
than spinning unproductively.
- "zram: lockmap tweaks" (Sebastian Siewior)
Optimize and fix lockdep tracking for zram devices by consolidating
per-entry lockmaps and isolate lock classes across multiple instances
This reduces memory overhead by replacing per-entry lockdep_map
instances with a single map per struct zram, and assigns a dynamic
lock_class_key to each instance to prevent false deadlock reports
when different zram devices are backed by distinct filesystems.
* tag 'mm-stable-2026-08-18-18-39' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (501 commits)
selftests/mm: thuge-gen: fix test_shmget() for PAGE_SIZE check
selftests/mm: unpoison pages in memory-failure teardown
mm/shmem: downgrade final i_blocks check in shmem_evict_inode() to pr_warn()
mm/khugepaged: replace mutex_lock/mutex_unlock usage with guard macro
mm/zsmalloc: fix release order of locks in zs_page_migrate()
Documentation: zram: remove sections numbering
ksm: stop iterating VMAs when ksm_test_exit returns true
mm: fold userfaultfd_rwp() to false without CONFIG_ARCH_HAS_PTE_PROTNONE
mm/migrate: report RCU-tasks quiescent states in migrate_pages_batch()
zram: use a custom key for each zram object
zram: move lockmap to be per-zram instead per table
selftests/mm: fix gup_longterm EINVAL error message
mm: page_alloc: fix non-movable reclaim storm in defrag_mode
mm: page_alloc: move capture_control to the page allocator
mm: compaction: support non-movable compaction for pageblock requests
mm: page_alloc: __GFP_FS lockdep annotation for direct compaction
hugetlb: evaluate subpool free state while locked
mm/damon: remove trailing semicolons after function definitions
mm/damon/ops-common: prevent migration fallback to non-target nodes
mm/damon: update outdated comment about DAMOS filter handling
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext updates from Tejun Heo:
"Most of this cycle completes the enqueue-path support for hierarchical
sub-scheduling, which makes sub-scheduler support feature complete: a
root BPF scheduler can now hand a cgroup subtree over to a nested
sub-scheduler together with revocable CPU grants, and the
sub-scheduler owns all scheduling decisions for its tasks on those
CPUs.
Development volume was high and a number of changes plugging holes in
the new support landed late in the cycle. Also included are core
scheduling fixes that were completed too late for the v7.2 release and
are routed through this pull request.
Sub-scheduler CPU delegation:
- Parent schedulers now grant and revoke per-CPU capabilities
(enqueueing, preemption, CPU frequency control) on their children,
enforced on every path a scheduler can reach a CPU through.
Previously only dispatching could be delegated; this lets
sub-schedulers fully schedule their CPUs.
- Rescue execution: a task whose scheduler doesn't have access to the
CPUs the task needs to run on starved until the watchdog ejected
the whole scheduler. The kernel now runs such tasks directly on a
small bandwidth budget, turning a scheduler-killing failure into
bounded degradation.
- Cgroup integration: tasks migrating across a sub-scheduler boundary
weren't re-homed to the new owner, causing wrong-scheduler
scheduling and a use-after-free. Sub-schedulers now take over their
cgroup subtree and receive its cgroup callbacks.
- Arena objects now cross the kernel/BPF boundary as typed pointer
arguments, translated transparently by the BPF tree's new arena
argument support, replacing untyped arguments with manual
translation.
- scx_qmap now demonstrates full hierarchical sub-scheduling.
Other fixes and updates:
- Robustness improvements: the abort path is now NMI-safe, fixing
deadlocks when errors are raised from NMI context and making
hardlockup recovery direct. Reenqueue loops that could monopolize a
CPU ahead of the watchdog now eject the offending scheduler, and
stalls are blamed on the scheduler actually responsible.
- Hardening: BPF-writable arena memory is validated before kernel
use, and task slice and vtime writes got explicit synchronization
rules, closing corruption vectors open to buggy or malicious
schedulers.
- Core scheduling: sched_ext dispatching can drop the rq lock inside
the core-wide pick, which let interleaving selections corrupt each
other's state and hard-hang the machine. The selection now restarts
when the lock was released. The task ordering callback was also
invoked with its arguments swapped, and the default ordering is
updated to work across sub-scheduler boundaries. The fixes are
marked for stable.
- Other fixes headed for stable: a task init leak on fork failure
during enable, tooling compat macros that silently failed to detect
newer kernels, and a crash on reenqueueing against a destroyed
dispatch queue.
- Tooling: scx_pair moves off deprecated callbacks, and the
deprecated scx_bpf_cpu_rq() kfunc is removed"
* tag 'sched_ext-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext: (144 commits)
sched_ext: Drop the dead SCX_DEQ_CORE_SCHED_EXEC test in dequeue_task_scx()
sched_ext: Make core-sched task ordering hierarchy-aware
sched_ext: Use runnable_at for the default core-sched task ordering
sched_ext: Fix inverted ops.core_sched_before() invocation
sched_ext: Move the config-off sub-cap kfunc stubs into sub.c
sched_ext: Rename balance-era identifiers to dispatch terms
sched_ext: Drop the stale keep_prev fixup in dispatch_pick()
sched_ext: Keep kick_sync waiting on the rq's own CPU
sched_ext: Make SCHED_CLASS_EXT select GENERIC_ALLOCATOR
sched_ext/scx_flatcg: Fix cvtime true-up on slice expiry
sched_ext: Don't BUG_ON a destroyed DSQ in process_deferred_reenq_users
sched_ext: Fix scx_bpf_dsq_move_to_local___v2 compat detection
sched_ext: Make scx_bpf_events() read the calling scheduler's counters
sched_ext: Drop unlocked scx_rq_clock_invalidate() from scx_root_disable()
selftests/sched_ext: Fix flaky ddsp failure tests on busy systems
selftests/sched_ext: Make numa idle validation race-free
sched_ext: Fix scx_bpf_dsq_reenq___compat kfunc extern prototype
sched_ext/scx_flatcg: expire cached hweights on weight changes
sched_ext: Fix exit_task leak on fork failure during enable
sched_ext: fix stale references in doc comments
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull scheduler updates from Ingo Molnar:
"Load-balancing updates:
- 'flatten the pick': improve cgroup scheduling, which has always
been problematic and painful, which has caused various scheduling
misbehavior such as the mishandling of reniced tasks et al.
Add various cgroup weight distribution methods via cgroup_mode:
'up', 'max', 'concur' and 'tasks' - with the default being 'concur'
which is the most precise yet also most expensive version.
Finally, change cgroup scheduling to a single runqueue (Peter
Zijlstra)
- Series to improve the scheduling latency of short slice tasks
(Vincent Guittot)
- Series to fix cluster scheduling in the presence of asymmetric
capacity (Ricardo Neri)
- Prefer fully idle cores for NOHZ balancing (Andrea Righi)
- Don't trigger active load-balancing if src_rq->curr is not on_rq
(Xin Zhao)
PSI updates:
- Skip irqtime accounting when no new irq time has elapsed (Usama
Arif)
Scheduler debugging updates:
- Remove unused schedstats (Shrikanth Hegde)
- Defer freeing of cpumask memblock memory to initcall (Waiman Long)
Misc fixes and updates by Yu C Chen, K Prateek Nayak, Peter Zijlstra,
Vincent Guittot, Xin Zhao, Yury Norov, Zhan Xusheng"
* tag 'sched-core-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (33 commits)
sched/fair: Fix flat hierarchy
sched/isolation: Defer freeing of cpumask memblock memory to initcall
sched/topology: Restore SD_PREFER_SIBLING in domains with asymmetric capacity
sched/fair: Allow load balancing between CPUs of identical capacity
sched/fair: Skip misfit load accounting when the destination CPU cannot help
sched/fair: Check CPU capacity before comparing group types during load balance
sched/fair: Also gate overloaded status update for SD_ASYM_CPUCAPACITY
sched/fair: Do not skip CPUs of similar capacity with busy SMT siblings
sched/fair: Prefer fully idle cores for NOHZ balancing
stop_machine: Make stop_one_cpu_nowait() return void
sched/eevdf: Delayed dequeue task can't preempt
sched/fair: Fix stale comments referring to removed CFS concepts
sched/debug: Remove unused schedstats
sched/psi: skip irqtime accounting when no new irq time has elapsed
sched/fair: Reflow sched_balance_rq()
sched/fair: Simplify balance_interval reset logic in sched_balance_rq()
sched/fair: Don't trigger active lb if src_rq->curr is not on_rq
sched/eevdf: Speedup short slice task scheduling
sched/eevdf: Always update slice protection
sched/eevdf: Cancel slice protection if short slice task is eligible
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull kthread vfs updates from Christian Brauner:
"This stops kernel threads from sharing filesystem state with
userspace. This work is about 3 cycles old and has been in -next
for about that time.
When the kernel boots init_task creates PID 1 and then kthreadd. From
that point every kthread and PID 1 share the same fs_struct. That is
why pivot_root() has to rewrite the fs_struct of all kthreads. The
rewriting exists so that kthreads can use init's filesystem state when
they want to. It also means userspace can move the ground out from
under the kernel.
PID 1 now gets a completely separate fs_struct. All kthreads are
anchored in a private SB_KERNMOUNT instance of nullfs that cannot be
mounted on and cannot be used to follow other mounts. Userspace init
can no longer affect kthread filesystem state and kthreads can no
longer affect userspace fs state without explicit opting in to that.
Path lookup from a kthread now fails by default. It makes it
deliberately hard to offload security sensitive operations into init's
filesystem state from a kthread.
Places that legitimately need to look something up there opt in
through the new scoped_with_init_fs() which temporarily overrides the
caller's fs_struct with init's. usermodehelpers remain the only kernel
tasks that genuinely share init's filesystem state, since they execute
random binaries in the root filesystem (excellent...).
The visible result is that /proc/2/root is a nullfs with an empty
mountinfo while /proc/1/root is the real root"
* tag 'vfs-7.3-rc1.kthread' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (26 commits)
initramfs_test: use test init/exit hooks to override init fs
fs: stop rewriting paths for PF_EXITING | PF_DUMPCORE
fs: stop rewriting kthread fs structs
fs: start all kthreads in nullfs
nullfs: make nullfs multi-instance
devtmpfs: create private mount namespace
fs: add umh argument to struct kernel_clone_args
fs: stop sharing fs_struct between init_task and pid 1
af_unix: use scoped_with_init_fs() for coredump socket lookup
initramfs: use scoped_with_init_fs() for rootfs unpacking
pnfs/blocklayout: use scoped_with_init_fs() for SCSI device lookup
ksmbd: use scoped_with_init_fs() for VFS path operations
ksmbd: use scoped_with_init_fs() for filesystem info path lookup
ksmbd: use scoped_with_init_fs() for share path resolution
fs: use scoped_with_init_fs() for kernel_read_file_from_path_initns()
coredump: use scoped_with_init_fs() for coredump path resolution
btrfs: use scoped_with_init_fs() for update_dev_time()
scsi: target: use scoped_with_init_fs() for APTPL metadata
scsi: target: use scoped_with_init_fs() for ALUA metadata
crypto: ccp: use scoped_with_init_fs() for SEV file access
...
|
|
The default core-sched ordering runs the longest waiting task first by
comparing p->scx.core_sched_at stamps. The stamp is maintained under two
rules. touch_core_sched() stamps when a task starts waiting for a CPU and
when its slice runs out. If the scheduler implements
ops.core_sched_before(), touch_core_sched_dispatch() re-stamps on every
dispatch.
A comparison can see one stamp taken under each rule, which isn't a
meaningful ordering. The dispatch rule also buys little - it only aligns
bypass-mode comparisons with the local DSQ order. Multiple schedulers make
the mixed comparisons more common.
Wait time is what p->scx.runnable_at already tracks for the stall watchdog.
Delete core_sched_at with both touch functions and compare runnable_at in
the scx_prio_less() fallback.
runnable_at is refreshed only on enqueue and goes stale while a task keeps
occupying its CPU. Instead of re-stamping, order a running task after every
waiting task as it is the most recently serviced.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
futex_exit_release() and futex_exec_release() are identical now. That means
also exit_mm_release() and exec_mm_release() are identical.
Consolidate the whole lot and remove the redundant copies.
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Kyle Zeng <kylebot@openai.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
|
|
Pull in dependents, the flat hierarchy fix depends on this.
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
|
|
Commit 9c63e84db29b ("sched/core: Disable SD_PREFER_SIBLING on asymmetric
CPU capacity domains") removed the SD_PREFER_SIBLING from the domains with
asymmetric capacity. This was done to avoid spreading tasks to sibling
scheduling groups with less capacity, but this does not happen: checks for
capacity in update_sd_pick_busiest(), sched_balance_find_src_group(), and
sched_balance_find_src_rq() prevent migrations from high- to low-capacity
CPUs if the busiest group is not overloaded.
The cluster topology is a notable example: some systems have scheduling
domains spanning CPUs of asymmetric capacity, grouped into two or more
equal-capacity clusters sharing an L2 cache. When CONFIG_SCHED_CLUSTER is
enabled, SD_PREFER_SIBLING is needed in the domain to spread load across
these clusters.
CPUs with spare capacity, big or small, have always helped overloaded
groups. Once the overloading condition disappears, misfit load will still
be used to move high-utilization tasks to bigger CPUs if they have spare
capacity.
Adding the SD_PREFER_SIBLING flag shifts load balancing in shared-LLC
domains from equalizing the number of idle CPUs to equalizing the number
of running tasks. This enables migrations among clusters from newly-idle
load balance, where the outgoing task is already dequeued but the CPU
has not yet transitioned to idle.
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: Christian Loehle <christian.loehle@arm.com>
Tested-by: Andrea Righi <arighi@nvidia.com>
Link: https://patch.msgid.link/20260720-rneri-fix-cas-clusters-v6-6-bb500bf4afd4@linux.intel.com
|
|
As part of the changes converting VMA flags from a system word size to a
bitmap, extend this change to generic_get_unmapped_area() and
generic_get_unmapped_area_topdown(), which also allows us to convert
stack_guard_placement() as well.
We retain arch_get_unmapped_area() and arch_get_unmapped_area_topdown()
as-is for now, using legacy_to_vma_flags() as necessary to do so.
No functional change intended.
Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-4-0fa2357d5431@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Dave Airlie <airlied@gmail.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nico Pache <npache@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Update __get_unmapped_area() to be parameterised by vma_flags_t rather
than vm_flags_t as part of the effort to move VMA flags from a system word
to a bitmap.
We cascade the changes up to arch_get_unmapped_area_topdown() and
arch_get_unmapped_area(), where, for now, we use vma_flags_to_legacy() in
order to propagate the VMA flags.
No functional change intended.
Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-3-0fa2357d5431@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Dave Airlie <airlied@gmail.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nico Pache <npache@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
When rescue demand on a cpu persistently exceeds the configured bandwidth,
tasks age on that cpu's rescue DSQ until the stall watchdog fires. The
watchdog blames the waiting task's owner, but the misbehaving party is
whoever floods the queue, not whoever happens to time out.
Track each sched's recent rescue consumption per cpu as a decaying average.
Once the oldest waiter on a cpu's rescue DSQ has been queued past a
threshold derived from the rescue knobs (4s at the defaults), the rescue
timer ejects the sub with the highest recent consumption on that cpu with
SCX_EXIT_ERROR_RESCUE. With no recent consumer there is no victim and
nothing is ejected - the generic stall watchdog eventually blames the
waiter's owner instead. Ejections on a cpu are spaced one threshold apart so
the freed bandwidth can drain the backlog before another sub is judged.
The overload check only wins the race against the stall watchdog when the
watchdog timeout clears the threshold, and a single in-budget wait must not
cross the trigger on its own. Warn on a scheduler whose timeout doesn't fit
and on knobs whose funding period exceeds half the threshold.
v2: - Track kill_at in jiffies_64 - on 32-bit, the time_before() grace check
wraps 2^31 ticks after the last ejection and suppresses ejections.
(sashiko AI)
- Track rescue_avg_at in jiffies_64 likewise - the unsigned long decay
delta truncates mod 2^32 on 32-bit and can revive a weeks-old usage
average in the victim pick.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A local DSQ insert lacking the needed caps is diverted to the reject DSQ and
bounced back through ops.enqueue() so the scheduler can re-decide. That
recovery assumes the scheduler has somewhere legal to send the task. When it
doesn't, e.g. when the task's affinity is restricted to cids delegated away,
the task starves until the stall watchdog ejects the scheduler. An exiting
task is worse - it skips ops.enqueue() and the rejection becomes a
self-requeuing cycle that burns the CPU until the watchdog fires.
Add SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the
insert would be rejected for missing caps, the kernel takes over and runs
the task on the target CPU without consulting the owning scheduler. The
kernel sets the flag itself when enqueueing an exiting task.
Rescue is a last-resort forward-progress backstop with a persistent
disadvantage, not a way around cap enforcement. A per-CPU token bucket
accrues rescue_bandwidth_ppt (default 2%) of CPU time and rescues run one at
a time in arrival order. Each is granted a slice of the rescue_quantum_us
(default 5ms) quantum divided across the waiters, waits at the tail of the
local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival
once the slice is served.
The schedulers keep their normal control over an admitted rescuee and may
preempt or reslice it. Service is measured on CPU time actually received, so
neither shortens the rescue. Prolonged denial escalates - the remaining
slice turns into protected execution (SCX_TASK_PROTECTED) and the rescuee
preempts the current task. Escalation is paced by the same bucket, and
delivered service converges on the configured bandwidth no matter how
aggressively the schedulers dispatch.
Both knobs are root-only and SCX_RESCUE_DISABLE turns rescue off, making
SCX_ENQ_RESCUE inserts reject as usual.
v2: - Add SCX_OPS_OPEN() fix-ups for the new ops fields so cpu-form
schedulers setting them still load on older kernels. (Andrea)
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
A BPF scheduler can displace any of its tasks at will - cut a running one's
slice with an SCX_ENQ_PREEMPT dispatch, an SCX_KICK_PREEMPT kick or a direct
shortening, and jump a queued one with HEAD insertions. Sometimes the kernel
needs a slice and a DSQ position to stick regardless.
Add SCX_TASK_PROTECTED, guarding both:
- The slice becomes immutable. Every scheduler-reachable write is refused
and counted as SCX_EV_SLICE_DENIED. Higher scheduling classes are
unaffected. PREEMPT|IMMED can't preempt a running protected task and gets
reenqueued.
- A protected task that reached the head of its DSQ keeps it - HEAD
insertions land behind the leading run of protected tasks and reenqueue
sweeps skip them. Only rq-owned DSQs can hold protected tasks, so the walk
runs only for them.
The bit lives in p->scx.flags so that both the refusal and the head walk
read it under the rq lock that protects it.
Protection ends when the slice is consumed, when the task leaves the rq
except for a save/restore on the running task, on a yield, when the
scheduler enters bypass, and when the task leaves scx. The flag is
kernel-internal and not used yet.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
p->scx.slice and p->scx.dsq_vtime writes have no synchronization rules. The
dsq insert kfuncs write both fields synchronously from whatever context
they're called in - a direct dispatch from ops.select_cpu() writes with only
pi_lock held - and, as the kfuncs are safe to call spuriously with the
invalid dispatch discarded later, a scheduler can modify any task's slice by
spuriously calling them. The latter stands in the way of an upcoming patch
which adds kernel-granted slices that the schedulers must not be able to
modify.
Give both fields explicit rules. While the task is running, sleeping or
queued on an rq-owned DSQ, the rq lock protects them - these are the states
where the kernel consumes the slice. While queued on a user DSQ or on the
BPF side, the kernel neither consumes nor decides on the fields and every
writer acts for the BPF scheduler - synchronizing the writers is the
scheduler's responsibility and whichever write lands last wins.
To conform, an insert kfunc no longer writes the fields when called. The
values travel with the dispatch and take effect when the task is inserted. A
discarded dispatch has no side effects. The rq lock rule is asserted at the
slice store.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Unlike local reenqueues, cap rejections have no repeat limit. A
malfunctioning scheduler can keep re-inserting a task to a cid it lacks caps
on, cycling the task through reject and reenqueue. This was assumed safe
because a task that never runs trips the stall watchdog. However, the
reenqueue irq_work re-arms itself and outranks the timer vector, blocking
everything else on the CPU including stall detection and recovery, until the
NMI hardlockup detector fires.
Local reenqueues already have a repeat cap, SCX_REENQ_LOCAL_MAX_REPEAT,
which needs generalizing to cover all reenqueues. It also has an attribution
problem. Counted per-cpu on root, it tears down the whole hierarchy even
when a sub-scheduler caused the repeated reenqueues.
Generalize by bounding every reenqueue with one per-task counter. reenq_cnt
is bumped in scx_do_enqueue_task() on each SCX_ENQ_REENQ, the single path
every reenqueue producer passes through, and cleared in clr_task_runnable()
when the task is picked to run and in scx_disable_task() when it leaves the
scheduler's control. Past SCX_REENQ_MAX_REPEAT the task's owning scheduler
is ejected with a new SCX_EXIT_ERROR_REENQ and the task is left stranded to
be picked up during sched exit.
The SCX_EV_REENQ_LOCAL_REPEAT event becomes SCX_EV_REENQ_REPEAT, counting
repeat reenqueues from all sources.
v2: Count SCX_EV_REENQ_REPEAT only when a reenqueue leads to another
reenqueue, not on every reenqueue.
v3: - Also clear reenq_cnt in scx_disable_task() so that the count doesn't
carry over to the next owner across sched class switches, scheduler
replacement or sub-scheduler rehoming (Andrea Righi).
- Update the stale SCX_EV_REENQ_LOCAL_REPEAT references in sched-ext.rst
(Andrea Righi).
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Sub-schedulers don't get cgroups yet: every task_group is inited on the root
sched and the routing added by the previous patches always resolves to it.
Add the handover: an enabling sub-scheduler takes over the cgroups in its
subtree and a disabling one returns them to its parent.
scx_cgroup_claim_subtree() runs while the sub enables, after the subtree's
cgrp->scx_sched's are set and before any task is claimed. It inits each
subtree task_group on the sub, exits it from the parent and updates
tg->scx.sched. A failed ops.cgroup_init() unwinds the sub-side inits and
aborts the enable with the parent untouched.
Disabling reverses it with scx_cgroup_return_subtree(): exit each cgroup
from the sub, then re-init it on the parent with the current tg->scx.*
values, resyncing weight and bandwidth changes made while the sub had it.
When a re-init fails, the parent is failed and the remaining task_groups
still transfer uninited and get no cgroup ops - the same punting done for
tasks. The dying parent's own disable moves them onward.
The handover walks include dying but not yet offlined task_groups, the same
as root's bulk walks: a removed cgroup keeps hosting scheduling events until
its dying tasks finish their final context switches, and its
ops.cgroup_exit() must follow the last of them. tg on/offlining is excluded
through cgroup_lock(), so either ordering against an rmdir of a subtree
cgroup delivers balanced init/exit pairs.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
With sub-schedulers claiming cgroup subtrees, cgroup ops must be delivered
to each task_group's sched rather than always to root. Add tg->scx.sched to
track which sched initialized the task_group. It is set and cleared together
with SCX_TG_INITED.
Deliver the ops accordingly:
- ops.cgroup_exit() goes to the sched whose ops.cgroup_init() it pairs with.
- ops.cgroup_prep_move/move/cancel_move() go to the task's sched, and only
for moves that don't re-home the task. A re-homing move is reported
through the ops.exit_task/init_task() pair instead. The cgroups passed to
the move ops can be outside the sched's inited set as the cpu controller
can be coarser than the sub-scheduler topology.
- Knobs of a cgroup belong to the parent, so ops.set_weight/idle/bandwidth()
go to the parent task_group's sched.
All task_groups currently resolve to the root sched, so no behavior changes
until sub-schedulers start claiming cgroups.
While at it, scx_cgroup_init() is restructured so both paths share the
recording.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
Pull to receive:
477869bfafea ("sched_ext: Reject setting disallow from init_task outside the enable path")
5f8b69642d18 ("sched_ext: Take cgroup_lock() first in scx_cgroup_lock()")
8c13364db9c9 ("sched_ext: Skip sub-disable teardown for never-linked sub-schedulers")
5cdc92859809 ("sched_ext: Don't enable non-ext tasks in the sub-sched task loops")
as dependencies for the upcoming cgroup migration patchset and to
resolve the conflicts with the ext.c/sub.c split on for-7.3.
5f8b69642d18 comments scx_cgroup_lock() which for-7.3 exported for
sub.c. Resolved by keeping the exported version with the comment.
8c13364db9c9 and 5cdc92859809 patch the pre-split sub-sched enable and
disable paths in ext.c which for-7.3 moved to sub.c. Resolved by
applying the never-linked teardown skip and the class gates to sub.c.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The p->scx.disallow revert assumes the root enable path, where the switching
loop reads the reverted policy right afterwards and leaves the task off SCX.
The sub-scheduler disable path also reaches it when re-initializing the
returned tasks on a root parent. Nothing reads the policy there: the task is
enabled on root anyway and keeps running on the ext class with a silently
rewritten policy.
Kill the sched instead, matching the fork and non-root branches, and update
the disallow documentation, which equated !fork with the load path and
pointed at a stale debugfs path for nr_rejected.
Fixes: 337ec00b1d9c ("sched_ext: Implement cgroup sub-sched enabling and disabling")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
