| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull more MM updates from Andrew Morton:
- "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL" (Lorenzo
Stoakes):
Replace VM_SPECIAL with helpers describing mapping ownership and
behavior. Enforce the mmap flag rules and convert affected drivers,
removing ambiguous tests shared by driver mappings and ordinary
memory.
- "support kselftest on nommu platform" (Hajime Tazaki):
Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner
to avoid its fork-dependent timeout command and document the build
and runtime requirements.
- "mm/collapse: separate a collapse from its callers" (Kiryl
Shutsemau):
Separate huge-page collapse from caller policy. Replace khugepaged
identity checks with explicit permissions, and separate scanning from
collapse with the mmap_lock drop in one place.
- "mm: implement and use vma_has_anon_rmap(), silence KCSAN" (Lorenzo
Stoakes):
Provide a common helper for testing whether a VMA has anonymous
reverse mappings. Document the locking rules, annotate intentional
concurrent reads for KCSAN, and convert the core MM callers.
- "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups"
(David Hildenbrand):
Remove SECTION_MARKED_PRESENT in favor of the early-section flag.
Simplify sparse memory initialization, section metadata handling and
related PFN validity checks.
- "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport):
Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have
been unusable since the NO_BOOTMEM conversion, leaving only the code
and data overhead.
- "mm: Unify device DAX and HugeTLB vmemmap population paths" (Muchun
Song):
Unify the generic device DAX and HugeTLB vmemmap population paths and
shared tail-page handling. Remove the duplicate DAX machinery while
retaining the powerpc radix implementation.
- "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum
Yun):
Fix huge-page split tests racing with khugepaged recollapse. Use
pagemap and page flags instead of VMA-boundary assumptions which fail
when glibc has already enabled THP.
- "memcg: move memcgid refcount to objcg to unpin dying memcgs"
(Bingfang Guo):
Let dying memcgs be freed while swapped-out shmem still holds their
IDs. Move ID references to reparentable objcgs, with the necessary
swap accounting and RCU lifetime changes.
- "mm/truncate: fix data loss when truncating straddling large folios"
(Zhang Yi):
Preserve out-of-range data when truncating straddling large folios.
Fix a lookup race and split-failure handling, and handle minimum
folio orders for the forthcoming ext4 iomap buffered I/O path.
- "mm/damon: introduce damos quota goal target metric complement flag"
(SJ Park):
Let DAMOS quota goals use a metric's complement, so utilization can
also express free-memory goals without adding separate metrics. Add
the sysfs interface, documentation and tests.
- "arm64/mm: Standardize printing for pgtable entries" (Anshuman
Khandual):
Use the generic formatter for arm64 page-table entries, ready for
future wider entries. Move the safe-PTE-update check out of the
header to avoid an include cycle.
- "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk):
Fix KMSAN metadata leaks on iounmap and failed ioremap allocations.
Free backing pages before removing their mappings, and share the
teardown with partial-allocation cleanup.
- "mm/damon/core: preserve quota state when constructing schemes" (Karl
Mehltretter):
Preserve running DAMOS quota state while validating parameter
updates. Even rejected updates could reset accounting and exceed the
quota window; add tests for both accepted and rejected updates.
- "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu):
Fix vmalloc_dump_obj() lookups for last-page addresses and
allocations spanning vmap zones. These allocations were being missed
in diagnostic output.
- "mm: various (urgent?) fixes" (David Hildenbrand):
Fix mmap charge leaks and hugetlb use-after-free, wrong-address
unmapping, stuck locks and endless faults. Also preserve reservations
on failed mremap and handle stale userfaultfd bits.
- "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris
Charisis):
Fix pointer and metadata confusion during RCU destruction of
maple_range_64 nodes, avoiding double frees and invalid accesses. Add
a regression test; no in-tree user currently uses this combination.
- "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu
Li):
Fix hugetlb migration and preallocation accounting so /proc/vmstat
retains the correct count and memory.numa_stat follows the folio to
its new node.
- "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park):
Fix 32-bit DAMON age overflow which made old regions look young.
Share hugetlb folio acquisition code and add a regression test for
the previously fixed last-region skipping bug.
- "mm/damon/core: cleanup code, reduce stack usage, and add kunit" (SJ
Park):
Reduce kdamond stack usage and remove unused enums. Add deterministic
PSI quota-goal tests which do not depend on the machine's current
memory pressure.
- "mm/damon: fix the temporal goal tuner's size quota conversion"
(Donggeun Yoo):
Bound the temporal tuner's quota conversion to prevent overflow and
zero effective quotas, which could stop schemes making progress. Add
boundary tests.
- "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He):
Remove redundant GFP_DMA flags from coherent DMA allocation calls.
The DMA API already selects the zone from device limits and ignores
the caller's flag.
- "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum
Yun):
Propagate child-process failures in the khugepaged tests and prevent
background collapse from racing their checks. This removes false
passes and intermittent failures.
- "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM"
(Enze Li):
Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds
the right maintainers. Also label the closing hugetlb_cgroup.h
include guard.
Summary of the 65 singletons in this merge:
The singletons include a few worthwhile performance fixes. Avoiding
unnecessary reverse-map tree updates gave a reported 15.9% improvement
in UnixBench's execl test on a 192-core x86 system. A reclaim retry
fix stops defrag_mode from spinning under memory.low protection; a
reported file-creation stress test ran 5.7 times faster, with far less
memory pressure. Narrower hugetlb invalidations also spare KVM guests
needless refaults: the reported Redis-in-VM P99 QPS penalty while
punching holes fell from 7.09% to 1.45%.
Several fixes prevent lost data or corrupted memory. Lazyfree
huge-page races with mremap()/munmap() could lose data or access freed
memory, and MADV_COLD could lose a concurrent write's dirty bit.
Another fix prevents stale migration entries from outliving large
folios. On filesystems with blocks smaller than a page, extending a
file by 4 GiB or more could expose bytes written past the old EOF
through mmap instead of zeroes. That is fixed, along with
use-after-free races in PAGEMAP_SCAN during process exit and in
memory-cgroup accounting on PREEMPT_RT kernels.
Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios,
which could wedge other memory operations, and stop mremap() leaking
commit charges until allocations fail under strict overcommit. A
further fix keeps userfaultfd tracking intact on neighboring mappings.
NUMA binding no longer prevents long-term pins when the chosen node
has only movable memory, fixing QEMU/VFIO startup and io_uring buffer
registration failures. MGLRU also gets a fix for early-boot crashes
with memory cgroups disabled at boot, notably in small crash kernels.
* tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (213 commits)
mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline
mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root
mm/memcontrol: publish null pointer before dropping css reference
test_maple_tree: test maple_range_64 node in RCU mode
maple_tree: fix maple_range_64 crashes in RCU mode
mm/vma: avoid broken uffd merge on mremap
tools/testing/shared: catch a failed posix_memalign()
mm/damon/core: remove unreachable damos charge_{target,addr}_from
mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache
riscv: mm: exclude invalid THP PMDs from page table check
mm/damon/core: filter out damos address filter-split failed region
fs/proc/task_mmu: pin mm_users in do_pagemap_scan()
mm/mempolicy: apply bind policies against the scoped allocation zone
mm/vmscan: reclaim every zone in per-node proactive reclaim
mm/mremap: correctly uncharge accountable VMAs in error path
mm/mremap: populate calculated fields in vrm state correctly
MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP
mm: bdi: reset min_ratio after the sysfs attributes are gone
mm/page_vma_mapped: continue past nonmatching huge PMDs
selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull non-MM updates from Andrew Morton:
- "ocfs2: exit recovery thread on mount error path" (Joseph Qi):
Stop OCFS2 recovery before freeing failed-mount state and replay
slots, closing use-after-free and double-free races. Defer
suballocator reclaim to break a transaction-lock deadlock with
dismount.
- "ocfs2: validate suballoc slot and bit of metadata blocks" (Joseph
Qi):
Validate OCFS2 metadata suballocator slots, bits and directory index
block locations. Reject corrupt metadata with filesystem errors
instead of BUGs or out-of-bounds accesses.
- "ocfs2: deal with legacy signed name hash values" (Joseph Qi):
Restore OCFS2 lookups for non-ASCII names stored with legacy signed
hashes. Use stored xattr hashes during bucket splits to avoid
duplicate entries and spurious ENOSPC failures.
- "ocfs2: stabilize dentry names in file and setattr traces" (Cen
Zhang):
Snapshot OCFS2 names during file and setattr tracing to prevent
rename races using freed storage or mismatched names and lengths.
Preserve trace formats and the locked release open-count sample.
- "panic: fix panic_force_cpu= redirect races and NMI bypass" (Bradley
Morgan):
Fix panic_force_cpu races and NMI redirection so crash-kernel CPU
selection is honored. Preserve variable arguments on failed redirects
and use a static message buffer.
- "fat: allocate multiple clusters in fallocate" (Matheus Alves de
Almeida):
Allocate FAT clusters in batches for FALLOC_FL_KEEP_SIZE and batch
FAT updates. This removes repeated single-cluster allocation
overhead, particularly on synchronous mounts.
- "scripts/gdb: fix radix tree lookup and stack depot diagnostic"
(Ting-Han Hou):
Fix GDB radix-tree lookup and its handling of XArray sibling, retry
and zero entries. Also repair the stack-depot diagnostic for an
out-of-range pool index.
- "hung_task: Improve warning budget handling and task reporting"
(Aaron Tomlin):
Reset the hung-task warning budget after recovery so later hangs
remain visible. Keep the configured limit intact and emit one summary
per scan when detailed reports exhaust the budget.
- "bootconfig: Reject unexpected data after null character and
cleanups" (Masami Hiramatsu):
Reject bootconfig data after an embedded null while accepting
trailing padding. Validate the full input in the userspace tool, and
retain internal tree sanity checks there rather than in the kernel.
- "kbuild: GCOV cleanups following the prefer-atomic fix" (Konstantin
Khorenko):
Use the actual GCOV build flags in the prefer-atomic compiler probe.
Move flag setup into scripts/Makefile.gcov and skip the probe when
GCOV is disabled.
- "ocfs2: xattr bucket validation fixes" (Joseph Qi):
Accept valid OCFS2 xattr buckets spanning 512-byte blocks, avoiding
false read-only remounts. Reject misaligned buckets and return errors
instead of panicking on corrupt defrag metadata.
- "ocfs2: fix metadata reservation for xattr value trees" (Joseph Qi):
Reserve enough OCFS2 metadata for new and relocated xattr value
trees. Fix shortfalls which could cause ENOSPC, dropped entries and
leaked clusters on fragmented filesystems; remove duplicate
reservations.
- "Introduce arch_do_panic" (Mete Durlu):
Add arch_do_panic() and move s390 and sparc handling out of generic
panic code. The architecture-specific actions now follow the
end-of-panic marker.
- "checkpatch: userspace improvements" (Petr Vorel):
Improve checkpatch's userspace detection and suppress kernel-only
checks. Recognize standard inttypes macros and add explicit
--userspace and --no-userspace overrides.
- "ocfs2: validate dr_fs_generation of dir index root blocks" (Joseph
Qi):
Check OCFS2 directory index generations and leaf block numbers.
Reject stale or misplaced blocks instead of using metadata left by a
previously formatted filesystem.
- "selftests/membarrier: Skip an unregistered memory barrier test on
Musl" (Chris Gellermann):
Skip the unregistered membarrier test when libc has already
registered the command. Musl does this during thread creation,
causing false failures in the multithreaded test.
- "init: fix array boundary bugs in boot parameter parsing"
(Wilson Felipe Pereira):
Fix boot argument boundary cases which leave stale arguments in
init's command line or panic when replacing the final environment
variable. Clear the last slot and check duplicates before the limit.
Summary of the 87 singletons in this merge:
The OCFS2 singletons fix deadlocks between xattr updates and inline
truncation, and during inode reclaim. Corrupt quota and extent
metadata could also crash the kernel or corrupt memory; these paths
now check the layout before using it. Other fixes close use-after-free
and publication races around cluster reconfiguration, node failure,
quota remount errors and shared refcount trees. New files now retain
security labels from all stacked LSMs instead of silently dropping all
but one. Valid xattr insertions also stop tripping FORTIFY on builds
with counted arrays.
Elsewhere, a lost wakeup could leave hardware-monitoring reads stuck
in unkillable sleep and block further I2C traffic. That is fixed, as
are kernel memory leaks in POSIX message-queue notifications and tmpfs
casefold setup which could grow until memory is exhausted. A driver
removal race which could crash the kernel is fixed too. The
hibernation command through reboot() also stops taking the same mutex
twice, avoiding a deadlock before suspension.
On the boot side, bzip2 decoding now accepts valid full blocks emitted
by lbzip2, fixing failures to boot those compressed kernels. Malformed
bzip2 streams can no longer overflow the decode buffer, and LZ4
initrds are checked before the decoder can read past their end. FAT
rejects geometry which overflows the data-area calculation on 32-bit
systems, and rolls back entries when inode setup fails, preventing
failed creations from reappearing after remount. The x86 XOR and RAID6
AVX paths also avoid imposing a slowdown on subsequent SSE code.
* tag 'mm-nonmm-stable-2026-10-25-14-57' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (134 commits)
scripts/gdb: fix stack depot out-of-bounds diagnostic
scripts/gdb: fix radix tree lookup pointer handling
ocfs2/cluster: remove zero-address nodes from IP tree on teardown
ocfs2: snapshot the dentry name while tracing file release
ocfs2: snapshot dentry names in file and setattr traces
ocfs2/dlm: drain join handlers before releasing module text
MAINTAINERS: update Markus's e-mail address
mailmap: add entries for Andrew Jones
stackdepot: stop preallocating after the final pool
ocfs2: drain quorum work before removing the cluster
watchdog: force console verbose when reporting lockups
profiling: fix sampling step on 64-bit big-endian systems
kernel/reboot: avoid potential deadlocks during suspension
compat: avoid double rounding in times()
ocfs2: pin owning modules for open debugfs files
ocfs2/cluster: initialize heartbeat regions before publishing them
selftests/proc: proc-pid-vm: fix unused variable warning on non-x86
ocfs2: pin refcount trees before dropping the lookup lock
ocfs2: validate the xattr tree root of an indexed xattr block
ocfs2/dlm: hold the lookup reference across convert AST enqueue
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux
Pull liveupdate updates from Mike Rapoport:
- Several small fixes to KHO and LiveUpdate selftest
- A series that renames "KHO scratch" to "KHO bootmem" because this
memory is way more important in kexec'ed kernels rather than in the
original cold boot kernel and the new name reflects it
* tag 'liveupdate-v7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux:
kho: do not reserve empty per-node bootmem areas
selftests/liveupdate: Prefer KVM and HVF over TCG in vmtest.sh
kho: rename kho_scratch= commandline parameter to kho_bootmem=
kho: rename KHO scratch to KHO bootmem
memblock: rename KHO_SCRATCH to KHO_NOPRSRV
memblock: get rid of CONFIG_MEMBLOCK_KHO_SCRATCH
kho: docs: fix scratch_phys debugfs cross-reference
kho: do not reserve scratch memory in the kdump kernel
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux
Pull kexec updates from Mike Rapoport:
- Fix probe error codes and error propagation:
- make architecture functions that probe loaded image consistenly
return -ENOEXEC when they do not recognize the image format
- make kexec_image_probe_default() distinguish image format
mismatches (-ENOEXEC) from real errors (e.g. -ENOMEM),
propagating real errors immediately and only continuing to the
next loader on -ENOEXEC
- Make sure the next kernel does not land on hardware-poisoned pages
- Remove unused arch_kexec_locate_mem_hole hook
* tag 'kexec-v7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux:
kexec_file: remove unused arch_kexec_locate_mem_hole hook
kexec: fix error propagation in kexec_image_probe_default()
kexec: return -ENOEXEC from image probe functions on mismatch
kexec: keep the next kernel off hardware-poisoned pages
kexec_file: stop the top-down search before it underflows
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/mm/memblock
Pull memblock updates from Mike Rapoport:
"Performance improvement:
- Use binary search in memblock_add_range() and
memblock_isolate_range()
Cleanups:
- Several simplifications in memblock and mm_init that remove
redundant code
- After rework of sparsemem buffer preallocation, memmap_alloc() does
not require exact_nid parameter. Remove that parameter, drop now
unused memblock_alloc_exact_nid_raw() and adjust memblock tests
accordingly
memblock test updates:
- add coverage for lowmem allocations
- fix 32-bit build warnings"
* tag 'memblock-v7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/mm/memblock:
memblock: use binary search to locate candidate regions
memblock: drop for_each_memblock_type() and open code its users
memblock: Remove unused memblock_alloc_exact_nid_raw()
memblock tests: Move exact nid tests to memblock_alloc_range_nid()
mm: Remove unused exact_nid parameter from memmap_alloc()
memblock: initialize the region merge end index
tools: use __pa() in virt_to_phys()
memblock tests: zero-extend simulated memory addresses
memblock tests: use physical addresses for region removal
memblock: reject non-page-aligned reserve_mem regions
memblock tests: cover allocations below the low address limit
memblock tests: model the low allocation limit within dummy memory
mm/mm_init: remove zone_absent_pages_in_node()
memblock: remove conditional return with no effect
memblock: simplify reset_all_zones_managed_pages()
|
|
When CONFIG_MEMCG=y, lruvec_pgdat() returns lruvec->pgdat, which for the
per-node pgdat->__lruvec is left NULL by lruvec_init() and only filled in
lazily by mem_cgroup_lruvec(). Normally that happens early, e.g. on the
first folio_add_lru().
With cgroup_disable=memory, lru_gen_shrink_node() passes &pgdat->__lruvec
directly to shrink_one(). If direct reclaim runs before anything has
called mem_cgroup_lruvec() for the node, which can happen during early
boot under memory pressure (e.g. a kdump kernel), get_swappiness() ends
up dereferencing a NULL pgdat:
Unable to handle kernel NULL pointer dereference at virtual address 0000000000005040
pc : get_swappiness+0x68/0xb0
lr : try_to_shrink_lruvec+0x54/0x2c0
Call trace:
get_swappiness+0x68/0xb0 (P)
try_to_shrink_lruvec+0x54/0x2c0
shrink_one+0x8c/0x240
shrink_node+0xc6c/0xe08
do_try_to_free_pages+0xdc/0x570
try_to_free_pages+0x124/0x278
__alloc_frozen_pages_noprof+0x6e8/0xf78
alloc_pages_mpol+0xc4/0x1c0
alloc_frozen_pages_noprof+0x54/0xe8
alloc_pages_noprof+0x1c/0x40
atomic_pool_expand+0x98/0x300
__dma_atomic_pool_init+0x58/0xd8
dma_atomic_pool_init+0x154/0x170
do_one_initcall+0x64/0x350
kernel_init_freeable+0x214/0x430
kernel_init+0x2c/0x208
ret_from_fork+0x10/0x20
The classic LRU path and MGLRU before per-node lru_gen_folio lists
obtained the lruvec through mem_cgroup_lruvec(), which sets
lruvec->pgdat.
The above was hit on an arm64 kdump kernel (6.18-based) booted with
cgroup_disable=memory nr_cpus=4. It also reproduces on mainline on an
arm64 VM with CONFIG_ARM64_64K_PAGES=y, CONFIG_MEMCG=y,
CONFIG_LRU_GEN_ENABLED=y and:
mem=160M nr_cpus=4 coherent_pool=512M cgroup_disable=memory
These options only serve to force direct reclaim early in boot; with
160M the boot eventually ends in OOM regardless. On mainline the first
LRU reclaim happens while unpacking the initramfs, and faults the same
way:
Unable to handle kernel NULL pointer dereference at virtual address 0000000000005040
pc : get_swappiness+0xd4/0x150
Call trace:
get_swappiness+0xd4/0x150 (P)
try_to_shrink_lruvec+0x44/0x2b8
shrink_one+0x80/0x1a0
shrink_node+0xd98/0xea8
do_try_to_free_pages+0xf4/0x4b0
try_to_free_pages+0x110/0x360
__alloc_frozen_pages_noprof+0x8b0/0x1718
...
__filemap_get_folio_mpol+0x2ac/0x548
simple_write_begin+0x44/0xf8
page_symlink+0x178/0x198
...
do_populate_rootfs+0x13c/0x200
Link: https://lore.kernel.org/20261017050928.128582-1-sidchintamaneni@gmail.com
Fixes: e4dde56cd208 ("mm: multi-gen LRU: per-node lru_gen_folio lists")
Signed-off-by: Siddharth Chintamaneni <sidchintamaneni@gmail.com>
Signed-off-by: Hardik Garg <hargar@linux.microsoft.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Assisted-by: LLM
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Yu Zhao <yuzhao@google.com>
Cc: <apais@microsoft.com>
Cc: <rlmenge@gmail.com>
Cc: <stable@vger.kernel.org> # 6.3+
|
|
page_counter_calculate_protection() runs without locks. Several
reclaimers can update the same counter's emin and elow at the same time,
and readers such as mem_cgroup_below_min(), mem_cgroup_protection() and
the dmem eviction code load them with READ_ONCE().
Commit b3a7822e5e75 ("mm, memcg: prevent mem_cgroup_protected store
tearing") made the main path store emin and elow with WRITE_ONCE(), but it
missed the early return for a counter whose parent is the reclaim root.
Those two stores are the only plain accesses to emin and elow left.
With CONFIG_KCSAN_STRICT=y, a few tasks writing to the same cgroup's
memory.reclaim trigger reports like this within seconds:
BUG: KCSAN: data-race in page_counter_calculate_protection / shrink_node
read (marked) to 0xff1f91b146d40e40 of 8 bytes by task 163 on cpu 4:
shrink_node+0x57a/0x1210
do_try_to_free_pages+0x194/0x930
try_to_free_mem_cgroup_pages+0x204/0x3f0
user_proactive_reclaim+0x28a/0x4d0
memory_reclaim+0x35/0x50
...
write to 0xff1f91b146d40e40 of 8 bytes by task 167 on cpu 6:
page_counter_calculate_protection+0x1fa/0x360
mem_cgroup_calculate_protection+0x4e/0x70
shrink_node+0x53e/0x1210
...
The default KCSAN config does not report it, because it treats aligned
word-sized plain writes as atomic.
Use WRITE_ONCE() for these two stores as well.
There is no functional change. Readers already accept stale values, and
compilers emit a single word-sized store either way: with GCC 14 the
x86-64 object code is the same before and after this change.
Link: https://lore.kernel.org/20261021204818.3431099-1-shakeel.butt@linux.dev
Fixes: b3a7822e5e75 ("mm, memcg: prevent mem_cgroup_protected store tearing")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: SJ Park <sj@kernel.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Chris Down <chris@chrisdown.name>
|
|
struct memcg_stock_pcp hold references to cgroups that it tracks to make
sure that the memcg never gets freed with unaccounted charges still
living in the stock. When the cached stock goes to 0 for a memcg, its
reference is dropped and the cached memcg pointer is set to NULL.
If that was the last reference, the css_put would free the associated
cgroup before publishing the NULL pointer, which can be racy on
PREEMPT_RT. To summarize Shakeel Butt's analysis in [2],
CPU 1 CPU 2
local_trylock() takes a per-CPU
spinlock, which is an rcu read
lock along with migrate_disable
drain_stock
css_put drops the last reference
release work is queued
kworker runs css_release_work_fn()
call_rcu() starts grace period G
which waits for the refill task.
<preempt>
drain_all_stock
is_memcg_drain_needed()
enters rcu_read_lock(), but
grace period G does not wait
since it started after G did
<prempt>
stock->cached[i] is cleared and
unlocks. Grace period G ends,
old is freed
mem_cgroup_is_descendant() reads
old->css.cgroup which has just
been freed.
To close this gap on PREEMPT_RT, switch the order of the referene drop
and the publishing so that we publish the NULL pointer before dropping
the reference. Note that this can only happen from drain_stock who can
hold the last reference, but not on consume_stock since another
reference will always be held. But fix this in both sites to maintain
consistency.
This is based on a report from Sashiko, who found this pre-existing
issue in my page_counter_stock series [1].
Link: https://lore.kernel.org/20261021010051.1529261-1-joshua.hahnjy@gmail.com
Link: https://sashiko.dev/#/patchset/20260928192349.3432886-1-joshua.hahnjy%40gmail.com [1]
Link: https://lore.kernel.org/all/atfMngq7nyZpt4u-@linux.dev/ [2]
Fixes: e1a366be5cb4 ("mm: memcontrol: switch to rcu protection in drain_all_stock()")
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Muchun Song <muchun.song@linux.dev>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
Userfaultfd often causes unexpected interactions within mm, and does so at
the intersection of uffd, mremap() and VMA merging.
When an mremap of a VMA occurs, copy_vma_and_data() calls
mremap_userfaultfd_prep() to notify uffd of this.
However, mremap_userfaultfd_prep() simply drops the UFFD context for the
VMA if the user did not register it for remap, i.e. didn't specify the
UFFD_FEATURE_EVENT_REMAP flag.
This is OK for the VMA itself, as upon page table move, move_ptes() clears
UFFD page table flags, using vma_has_uffd_without_event_remap() to
determine whether to do so.
However, this doesn't account for merging with adjacent VMAs, whose page
tables won't have received this treatment, and thus latent UFFD markers
will be left around.
Worse, it'll break UFFD for any VMAs it is merged with. This is very
surprising behaviour for a user, who will then simply not receive UFFD
events for VMAs they ought to.
Resolve the issue by explicitly checking for this case via
vma_has_uffd_without_event_remap() in vma_merge_copied_range(), the merge
function explicitly designated for mremap() operations.
Also update the userland VMA tests accordingly.
Link: https://lore.kernel.org/20261020-uffd-remap-fix-v1-1-0f34fc181f0a@kernel.org
Fixes: 3cfd22be0ad6 ("userfaultfd: clear flag if remap event not enabled")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: shaos <shawnq1@163.com>
Closes: https://lore.kernel.org/all/20261020110748.959870-1-shawnq1@163.com/
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Mike Rapoport <rppt@linux.vnet.ibm.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: <stable@vger.kernel.org>
|
|
damos_quota->charge_{target,addr}_from saves the target and region the
scheme's quota limit is reached. damos_skip_charged_region() skips the
target and regions before the point. The target that was assigned to
charge_target_from could be dynamically removed from the context by the
runtime parameters update (aka commit) feature. In this case,
damos_skip_charged_region() cannot find if the point is passed or not,
so the scheme is effectively disabled.
Similarly, the user could also remove all the regions of the target
using the runtime parameters update feature. It would be uncommon to do
that, but it is a supported feature. Because the skip pointer detection
logic (damos_skip_charged_region()) works for each region of targets,
this result in the logic is entirely skipped and the scheme is
effectively disabled again.
The user impact is that the DAMOS scheme could suddenly stop working
under certain uses of the runtime parameters update feature. It could
surprise users. No crash or leak happens though. Users could also
simply work around by re-installing the scheme.
Catch the corner cases in damos_adjust_quota() and reset the pointers.
Link: https://lore.kernel.org/20261020023958.108276-1-sj@kernel.org
Fixes: da87878010e5 ("mm/damon/sysfs: support online inputs update")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Closes: https://sashiko.dev/#/bug/linux-cd0c55fc-0cc8-45d8-9637-3334811c49b8
Cc: <stable@vger.kernel.org> # 5.19.x
|
|
shrinker_info_unit holds BITS_PER_LONG atomic_long_t of deferred work plus
a one-word bitmap, 520 bytes on 64-bit. kzalloc_node() takes it from
kmalloc-1k, so almost half of every unit is padding.
Allocate the units from a cache of their own, sized to the unit. The
cache is not cache-line aligned: the counters of a unit already share
cache lines between shrinkers, and aligning the unit would only keep
neighbouring units off each other's boundary line, at the cost of part of
the saving.
Link: https://lore.kernel.org/20261019-vstor-144715-shrinker_info_unit-kmem_cache-v3-1-0fb061e09bf4@virtuozzo.com
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Harry Yoo (Meta) <harry@kernel.org>
Reviewed-by: Qi Zheng <qi.zheng@linux.dev>
Acked-by: Muchun Song <muchun.song@linux.dev>
Assisted-by: LLM
Cc: Dave Chinner <david@fromorbit.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
|
|
When an address type DAMOS filter is configured, DAMOS tries to split the
region for the filter range, and apply DAMOS action to the filter-in
region. The split operation (damon_split_region_at()) could fail from
allocation failure under extreme memory pressure. In the case, DAMOS
action could be applied more or less than expected, ignoring the address
filter.
The issue only rarely happens under extreme memory pressure. No crash or
leak happens. Occasionally not applying the action to a few regions
shouldn't be a real problem. But applying the action to a region that the
user has excluded via address filter could be a surprise depending on the
use case. It is only theoretical. No such real case is found yet.
Filter out the entire region if the split is failed. Update
damos_filter_match() signature to receive the error from split. Also
update kunit tests for the updated signature.
The issue was discovered [1] by sashiko.
Link: https://lore.kernel.org/20261020004119.106690-1-sj@kernel.org
Link: https://lore.kernel.org/20260801203121.117449-1-sj@kernel.org [1]
Fixes: ab9bda001b68 ("mm/damon/core: introduce address range type damos filter")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: <stable@vger.kernel.org> # 6.6.x
|
|
QEMU fails to start a guest with a VFIO device when guest RAM comes
from a memory backend bound to a movable-only node:
vfio_container_dma_map(0x55b1b1317800, 0x0, 0x10000000,
0x7f39a2200000)
= -12 (Cannot allocate memory)
VFIO pins guest RAM when the device is realized, and the pin faults
the pages in. io_uring fixed buffers fail the same way. Faulting the
range in before pinning lets the same pin succeed, since resident
pages are migrated off the node instead.
A long-term pin faults under memalloc_pin_save(). PF_MEMALLOC_PIN makes
current_gfp_context() clear __GFP_MOVABLE, but policy_nodemask() tests
apply_policy_zone() against the caller's gfp.
The bind keeps its movable-only nodemask, and the allocation finds no
usable zone:
page allocation failure:
order:0,
mode:0x140dc2(GFP_HIGHUSER|__GFP_ZERO|__GFP_COMP),
nodemask=1,cpuset=/,mems_allowed=0-1
Call Trace:
warn_alloc+0x136/0x150
__alloc_frozen_pages_noprof+0xed8/0x1720
alloc_pages_mpol+0xaa/0x170
vma_alloc_folio_noprof+0x65/0xd0
__handle_mm_fault+0x137b/0x1ac0
handle_mm_fault+0xa0/0x270
__get_user_pages+0x250/0xfd0
__gup_longterm_locked+0x116/0x870
gup_fast_fallback+0xcfc/0xdf0
pin_user_pages_fast+0x47/0x60
io_pin_pages+0x9a/0x110
Base page faults fail with -ENOMEM.
hugetlb faults fail with -EFAULT.
Check the policy zone against current_gfp_context(gfp), so a pin-scoped
fault falls back off the movable-only nodemask, like any other
non-movable allocation under the same bind.
Tested in qemu with a dax/kmem movable-only node 1. io_uring and VFIO
pins of unfaulted base pages and 2MB hugetlb pages now succeed and land
on node 0 (correct). QEMU now starts with the same backend binding.
Link: https://lore.kernel.org/20261018023349.1697434-1-gourry@gourry.net
Fixes: 8e3560d963d2 ("mm: honor PF_MEMALLOC_PIN for all movable pages")
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: <stable@vger.kernel.org>
|
|
user_proactive_reclaim() limits per-node reclaim to
gfp_zone(GFP_KERNEL), which is ZONE_NORMAL. Proactive reclaim serves no
allocation, but that limit makes shrink_node() skip every ZONE_MOVABLE
folio, so writing to /sys/devices/system/node/nodeN/reclaim never
reclaims a node's movable memory. A movable-only node, such as dax/kmem
or hotplugged memory onlined movable, reclaims nothing and fails with
-EAGAIN.
To reproduce, fill a movable-only node with clean page cache and write
200M to its nodeN/reclaim: the write fails with -EAGAIN and the page
cache stays. The same write to a ZONE_NORMAL node frees 200MB.
Scan every zone, as memory.reclaim does.
Link: https://lore.kernel.org/20261019185002.3428594-1-gourry@gourry.net
Fixes: b980077899ea ("mm: introduce per-node proactive reclaim interface")
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Harry Yoo (Meta) <harry@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Reviewed-by: Balbir Singh <balbirs@nvidia.com>
Acked-by: Davidlohr Bueso <dave@stgolabs.net>
Acked-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: Vishal Moola (Fractile) <vishal.moola@gmail.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: <stable@vger.kernel.org>
|
|
The mremap error path is a fragile and unwieldy beast - it reverses the
page table move then updates vma_remap_struct state to swap VMA, length and
address between old and new VMAs in copy_vma_and_data().
If the mremap() operation didn't specify MREMAP_DONTUNMAP then its caller,
move_vma(), invokes unmap_source_vma(), which unmaps the new VMA rather
than the old one.
For accountable VMAs when MREMAP_DONTUNMAP is not specified,
unmap_source_vma() unsets VMA_ACCOUNT_BIT during this unmap in order that
only the delta of a VMA expansion is charged (performed by
vrm_calc_charge() earlier).
Therefore nothing uncharges the expanded VMA size, and there is a permanent
leak in global Committed_AS.
Resolve the issue by uncharging when MREMAP_DONTUNMAP is not specified.
Link: https://lore.kernel.org/20261019-other-mremap-charge-leak-v1-1-d94477fe6545@kernel.org
Fixes: fdbef6149135 ("mm/mremap: don't account pages in vma_to_resize()")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Brian Geffon <bgeffon@google.com>
Cc: Dmitry Safonov <0x7f454c46@gmail.com>
Cc: <stable@vger.kernel.org>
|
|
Commit d5c8aec0542e ("mm/mremap: initial refactor of move_vma()")
introduced the vma_remap_struct helper to pass state sanely through
mremap() logic.
Part of this state is vrm->delta - equal to the absolute difference between
vrm->old_len and vrm->new_len (the old and new lengths of the VMA,
respectively).
This value is used to correctly charge accounted VMAs.
When shrinking a VMA, mremap_to() sets vrm->old_len equal to vrm->new_len
having performed the shrink of the VMA first, prior to any potential move
(e.g. MREMAP_FIXED | MREMAP_MAYMOVE specified).
However it does not recalculate vrm->delta, which results in incorrectly
overcharging the VMA when both moving and shrinking a VMA.
The charge leaks to Committed_AS globally, meaning a mischievous user can
cause allocations to fail across the system in strict overcommit mode.
Resolve the issue by abstracting the update of vrm->delta to
vrm_populate_calculated() and invoking this from mremap_to() when
vrm->old_len is changed.
Additionally, update vrm->remap_type whose value depends on vrm->delta.
The field is not read again after the shrink, but it is the logically
correct thing to do and protects against future changes which break this.
Link: https://lore.kernel.org/20261019-mremap-fix-shrink-charge-v1-1-a3ce209e5360@kernel.org
Fixes: d5c8aec0542e ("mm/mremap: initial refactor of move_vma()")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Closes: https://lore.kernel.org/all/20261017233527.797191-1-nickolai@csail.mit.edu/
Reviewed-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: <stable@vger.kernel.org>
|
|
A write to /sys/class/bdi/<dev>/min_ratio (or min_ratio_fine or min_bytes)
that races with the removal of the device leaves the global sum of minimum
dirty shares charged for a bdi that no longer exists. Until the next
reboot, that share of the dirty threshold is withheld from every other bdi
(__wb_calc_thresh() scales by 100 % minus the sum), and a later min_ratio
store on any bdi that would push the sum to 100 % fails with EINVAL
although every bdi reads 0. Root privilege is needed for the store; the
removal is any del_gendisk() (a disk or loop device going away) or the
unmount of a filesystem with its own bdi. On an x86_64 defconfig kernel
(v7.3-rc7), 2000 rounds of LOOP_CTL_ADD/LOOP_CTL_REMOVE on one loop device
with a second thread writing 50 to its min_ratio leak the share on the
first round the store lands in the window; afterwards `echo 99 >
min_ratio` on another loop device fails with EINVAL. With the fix the
same run keeps the sum at 0 and the store succeeds. Affected since v5.16,
when the reset in bdi_unregister() was added.
bdi_unregister() gives the bdi's share back with bdi_set_min_ratio(bdi, 0)
and only then calls device_unregister(), which removes the sysfs
attributes. A min_ratio_store() that gets its kernfs active reference
before the removal runs to completion after the reset:
__bdi_set_min_ratio() takes bdi_lock, sees bdi->min_ratio == 0, adds the
new value to bdi_min_ratio and stores it in the bdi, which bdi_put() then
frees. Nothing gives that share back.
Move the reset after device_unregister(): once the attributes are gone no
store can run, and bdi_set_min_ratio() needs nothing from the device.
Found while formally verifying bdi_unregister() against the kernel binary.
Link: https://lore.kernel.org/20261017030620.793108-1-nickolai@csail.mit.edu
Fixes: 3c376dfafbf7 ("mm: bdi: initialize bdi_min_ratio when bdi is unregistered")
Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Assisted-by: LLM
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Manjong Lee <mj0123.lee@samsung.com>
|
|
A PTE-mapped large folio can straddle PMD boundaries after an unaligned
mremap(). If the first part is discarded and replaced with an unrelated
huge PMD, page_vma_mapped_walk() stops instead of looking for the rest of
the folio in subsequent PMDs.
This also affects migration entries. A walk that stops at an unrelated
huge PMD can leave migration PTEs behind when migration or an anonymous
folio split restores the mappings. Those entries do not hold mapping
references and can outlive the source folio.
This issue can lead to memory corruption and I have a reproducer for
it.
Skip the current PMD rather than ending the walk on a flag or PFN
mismatch. Do the same if the PMD became nonpresent before the locked
recheck. Release the PMD lock, clear the per-PMD state and advance with
step_forward(), so the existing range limit controls termination. This
covers present, migration and device-private huge PMDs while preserving
the matching-PMD and split-to-PTE paths.
Link: https://lore.kernel.org/20261016011549.61146-1-kylebot@openai.com
Fixes: ace71a19cec5 ("mm: introduce page_vma_mapped_walk()")
Signed-off-by: Kyle Zeng <kylebot@openai.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: <stable@vger.kernel.org>
|
|
A compound page of order N must start at a PFN aligned to 2^N pages.
Therefore, a range used to initialize compound pages must start on a
compound-page boundary and contain an integral number of compound pages.
Two recent Device DAX bugs were caused by ranges that violated this
invariant, but neither was detected by memmap_init_zone_device().
Warn if the adjusted zone-device range violates this invariant so future
violations are reported when the compound pages are initialized.
Link: https://lore.kernel.org/20261016131334.3419393-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/r/20261016053944.2579541-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/r/20261016054914.2601510-1-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Assisted-by: LLM
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: Alison Schofield <alison.schofield@intel.com>
Cc: Joao Martins <joao.m.martins@oracle.com>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
|
|
A process that extends a file by 4 GiB or more past a dirty folio, on a
file system whose block size is smaller than the page size (ext4 with 1k
blocks), reads back the bytes it wrote through a shared mapping beyond the
old end of file instead of zeroes, and a later write into that block
carries them to the disk. With the example of commit 52aecaee1c26 ("mm:
zero range of eof folio exposed by inode size extension") and the truncate
taken 4 GiB further (pwrite 2k at 0, mmap the first page and write 'X'
over [2k, 4k), truncate(2) to 4 GiB + 2k), the read of [2k, 4k) returns
the 2048 'X' bytes, and a pwrite of 16 bytes at 3k followed by fsync() and
a remount leaves 1008 'X' bytes on the disk in [3k+16, 4k). No privilege
is needed, and every size extension of 4 GiB or more reaches it:
truncate(2)/ftruncate(2) through truncate_setsize(), writes beyond the end
of file through generic_write_end(), ext4, iomap, btrfs, exfat, ntfs, afs
and orangefs. Affected since v6.13.
pagecache_isize_extended() zeroes the part of the dirty folio straddling
the old end of file that the size extension exposes, from the old size to
the new size or the end of the folio, whichever comes first:
end = min_t(unsigned int, to - folio_pos(folio), folio_size(folio));
min_t() converts both operands to unsigned int before comparing, so the
loff_t difference to - folio_pos(folio) is reduced modulo 2^32 first.
When the file grows by 4 GiB or more past the folio, end is that residue
instead of folio_size(folio): below offset nothing is zeroed, above it
only [offset, end) is. The bytes beyond the old end of file stay in the
dirty folio and, now inside i_size, are file data.
Compare in loff_t; the result is at most folio_size(folio) and fits end.
Found while formally verifying pagecache_isize_extended() against the
kernel binary.
Link: https://lore.kernel.org/20261016222721.791785-1-nickolai@csail.mit.edu
Fixes: 52aecaee1c26 ("mm: zero range of eof folio exposed by inode size extension")
Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>
Assisted-by: LLM
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Brian Foster <bfoster@redhat.com>
Cc: Theodore Ts'o <tytso@mit.edu>
Cc: <stable@vger.kernel.org>
|
|
With debug_guardpage_minorder set, expand() turns the unused halves of a
split into guard pages, which page_del_and_expand() leaves out of the free
counts. Freeing the allocated half credits only that half, and
__free_one_page() then merges the guard buddy with clear_page_guard(),
which no longer touches the counters.
The merged block goes on a free list with the guard pages never counted,
so NR_FREE_PAGES falls behind the free lists by every guard that merges.
Before commit e0932b6c1f94 ("mm: page_alloc: consolidate free page
accounting"), __set_page_guard() and __clear_page_guard() adjusted the
counts themselves; that commit dropped both adjustments but kept the
subtraction implicit in expand().
Count the guard pages under the merged block's migratetype when the buddy
is cleared. account_freepages() skips an isolated block as it does for
every other free page, and moving the block off the isolated list counts
it then.
Booting a 16GB VM with debug_pagealloc=on debug_guardpage_minorder=1,
check the difference in free pages reported between /proc/vmstat and
/proc/buddyinfo.
At three points (idle, after a read and file-creation load, after a second
load), check the difference between nr_free_pages and the buddyinfo
numbers in the normal zone, as a number of pages:
idle after load after second load
unpatched -130 79205 99121
patched -16 45 0
The small remaining difference seems to be due to the numbers not being
read at exactly the same time, and is also seen without guard pages.
Link: https://lore.kernel.org/20261016122304.5b1eba8e@fangorn
Fixes: e0932b6c1f94 ("mm: page_alloc: consolidate free page accounting")
Signed-off-by: Rik van Riel <riel@surriel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Dev Jain <dev.jain@arm.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Assisted-by: LLM
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Kairui Song <kasong@tencent.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Frank van der Linden <fvdl@google.com>
Cc: <stable@vger.kernel.org>
|
|
For vmemmap-optimized sections, tail struct pages may be backed by shared
vmemmap pages. Those shared pages must carry the same page zone ID as the
struct pages initialized for the section.
Warn in __init_single_page() if the shared tail page has a different
page_zone_id(), which would indicate inconsistent initialization.
[muchun.song@linux.dev: document ordering of shared and retained struct page initialization]
Link: https://lore.kernel.org/20261010115523.4005071-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20261010075755.83339-7-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
vmemmap_populate_address() no longer has any callers that need the
returned PTE. Its only remaining user, vmemmap_populate_range(), only
checks whether population succeeded.
Open-code vmemmap_populate_address() directly in
vmemmap_populate_basepages(), remove the now-redundant range helper, and
return -ENOMEM directly on failure.
Link: https://lore.kernel.org/20261010075755.83339-6-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
vmemmap_pte_populate() no longer uses ptpfn as an input. Drop the
argument to simplify the code.
Link: https://lore.kernel.org/20261010075755.83339-5-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
The common vmemmap path selects the shared page for optimized mappings
itself, so Device DAX no longer needs vmemmap_populate_compound_pages() to
find a shared tail page and pass its backing PFN through the generic
population helpers.
Remove the Device DAX-specific population path and let section memmap
population always use vmemmap_populate(). The powerpc retains an
architecture-specific compound-page implementation, so select it directly
from radix__vmemmap_populate() for optimizable sections.
Link: https://lore.kernel.org/20261010075755.83339-4-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
The common vmemmap population path cannot yet handle optimized Device DAX
mappings on its own. It uses pfn_to_zone() to find the shared tail page,
but Device DAX populates its vmemmap at runtime before the ZONE_DEVICE
span is initialized.
Teach the common path to use device_zone() for runtime optimized vmemmap
population while retaining pfn_to_zone() for early boot. This allows the
same path to support both early boot mappings and Device DAX.
The backing PFN supplied by the Device DAX-specific population path is no
longer used, allowing the redundant lookup and population code to be
removed later.
Link: https://lore.kernel.org/20261010075755.83339-3-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
Patch series "mm: Unify device DAX and HugeTLB vmemmap population paths",
v4.
This series is split out from the earlier, larger series "mm: Generalize
HVO for HugeTLB and device DAX" [2]. While the parent series generalizes
vmemmap optimization across HugeTLB and device DAX, this subset addresses
a single, self-contained step: unifying their vmemmap population paths.
After the preceding Device DAX conversion, both HugeTLB and Device DAX
describe optimized vmemmap mappings through memory-section metadata and
use per-zone shared tail vmemmap pages. The generic code, however, still
carries a Device DAX-specific population flag and compound-page population
path, along with arguments and helpers needed only by that path.
This series first removes VMEMMAP_POPULATE_DAX and moves selection and
reference handling for the shared tail page into the common vmemmap
population path. It then removes the generic Device DAX-specific
compound-page population path and routes section vmemmap population
through vmemmap_populate(). The powerpc radix path continues to use its
architecture-specific compound-page population implementation for
optimizable sections.
The remaining patches remove the unused ptpfn argument, open-code
vmemmap_populate_address() now that no caller needs its returned PTE, and
add a warning for inconsistent zone initialization of shared tail vmemmap
pages.
This is the fourth smaller step toward the broader HVO generalization.
After this series, HugeTLB and Device DAX use the same population model
instead of parallel generic paths, while powerpc keeps its
architecture-specific implementation.
This patch (of 6):
VMEMMAP_POPULATE_DAX currently distinguishes DAX vmemmap population in two
places: it keeps allocations on the normal path and takes a reference when
a backing page is supplied for reuse.
After Device DAX switched to the common per-zone shared tail page, both
conditions can be determined locally. DAX supplies ptpfn for every shared
tail mapping and requests an allocation only for compound head mappings,
whose PFNs are not optimizable. Therefore, vmemmap_optimizable_pfn()
alone selects the correct allocation path.
When ptpfn is supplied, the caller is reusing an existing backing page.
Once the slab allocator is available, take a reference for each reused
mapping to balance the release performed by vmemmap_free(). Although the
buddy allocator is available before slab, no vmemmap population occurs in
that interval. Earlier mappings are backed by memblock/reserved memory
and do not need page reference accounting.
Remove VMEMMAP_POPULATE_DAX and the flags argument from the vmemmap
population helpers.
Link: https://lore.kernel.org/20261010075755.83339-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20261010075755.83339-2-songmuchun@bytedance.com
Link: https://lore.kernel.org/20260930140627.57431-1-songmuchun@bytedance.com/ [1]
Link: https://lore.kernel.org/20260513130542.35604-1-songmuchun@bytedance.com/ [2]
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
swap_cache_get_folio() reads the folio from the swap table and takes a
reference with folio_try_get(), without any lock. If the folio is removed
from the swap cache and freed in between, and its memory gets reused,
folio_try_get() still succeeds and we return a page that has nothing to do
with the swap entry:
CPU 0 CPU 1
do_swap_page()
swap_cache_get_folio()
swp_tb = swap_table_get()
folio = swp_tb_to_folio(swp_tb)
madvise(MADV_DONTNEED)
zap_nonpresent_ptes()
swap_put_entries_direct()
swap_put_entries_cluster()
__try_to_reclaim_swap()
swap_cache_del_folio()
folio_put() <- freed
page reused, e.g. as an anon
folio of another task
folio_try_get(folio) <- succeeds
return folio
swap_update_readahead() <- may clear PG_readahead on it
folio_lock_or_retry() <- locks it
folio_matches_swap_entry() fails, unlock and put
filemap_get_entry() does the same lockless lookup for the page cache, and
after folio_try_get() it rechecks the xarray, because the folio may have
been freed and reused in between. swap_cache_get_folio() lost that
recheck when the swap cache moved to the swap table.
So CPU 0 writes to a folio it doesn't own: it may clear PG_readahead and
takes the folio lock of someone else's page, before
folio_matches_swap_entry() finds out the folio is not ours.
When the race occurs, a swap fault can acquire a reference to a
physical page that has been freed and reused for an unrelated purpose,
potentially by another process. The fault handler may then clear that
page's PG_readahead/PG_reclaim flag, wait on or acquire its folio lock,
and potentially attempt to remove it from an unrelated swap cache.
These operations can interfere with unrelated workloads and may cause
unexpected stalls and impaired reclaim. The subsequent swap-entry
validation normally prevents the wrong folio from being mapped into the
faulting process.
It's documented that caller always double checks the folio so this part
is harmless from the aspect of data corruption or crash.
Recheck the slot after folio_try_get() and retry if it no longer holds
this folio.
[akpm@linux-foundation.org: add LLM-generated description of userspace impact]
Link: https://lore.kernel.org/20261015060528.302634-1-jiayuan.chen@linux.dev
Fixes: 8578e0c00dcf ("mm, swap: use the swap table for the swap cache and switch API")
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Kairui Song <kasong@tencent.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Youngjun Park <youngjun.park@lge.com>
Cc: <stable@vger.kernel.org>
|
|
min_free_kbytes is recalculated on memory hotplug and on THP sysfs writes.
The new value comes from calculate_min_free_kbytes() or, while THP is
enabled, from set_recommended_min_free_kbytes(). A value written to the
vm.min_free_kbytes sysctl is replaced whenever the new value is higher, so
a lower value does not stick. This matters most when THP is enabled on
configurations with large pageblocks, where the new value can be excessive
[1].
For example, with default arm64 pageblock sizes, the THP recommendation
can reach 5.5 GiB per eligible zone with 64 KiB pages vs 22 MiB with 4 KiB
pages. This keeps a significant amount of memory free, reducing what
applications can use and triggering reclaim much earlier. Users can lower
vm.min_free_kbytes to avoid this, but the lower value only sticks until
the next recalculation.
Stop recalculating min_free_kbytes once a value has been written to the
sysctl. Systems where the sysctl is never written keep the current
behavior.
Link: https://lore.kernel.org/20261014140859.1006967-1-noren@nvidia.com
Link: https://lore.kernel.org/r/20260716173504.760369-1-noren@nvidia.com/ [1]
Signed-off-by: Nimrod Oren <noren@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Link: https://lore.kernel.org/r/d202dcc3-d7d7-4e98-a9a5-3137cfbfd041@kernel.org/
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Shuah Khan <skhan@linuxfoundation.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: David Laight <david.laight.linux@gmail.com>
|
|
So far the assumption has been that mm->mm_lock_seq overflow is benign,
resulting only in readers spuriously falling back to the mmap lock, however
it turns out that this was an unsafe assumption.
If a VMA can be spuriously considered write locked that means read lock
exclusion does not take place, and thus a VMA can end up write-locked with
concurrent readers.
Since overflow only occurs every 2^31 write lock/unlock cycles, resolve the
issue by resetting every VMA's vma->vm_lock_seq each time such an overflow
occurs.
This is done in vma_end_write_all(), which is called when the mmap write
lock is released, i.e. in mmap_write_unlock() and mmap_write_downgrade().
Care must be taken, as resetting the sequence count at overflow means
changing the seqcount of VMAs which are already attached and observable.
The mm's seqcount is guaranteed to be odd when a write lock is held, and
thus any write locked VMA will have an odd seqcount assigned.
Therefore resetting to an even value prevents spurious write locks, since
these can only occur with the mmap write lock held, at which point the mmap
seqcount will be odd.
However, vma_start_read() also checks vma->vm_lock_seq == mm->mm_lock_seq,
bailing if so (indicating a write lock).
This would cause any even reset value chosen to prevent VMA read locks from
being taken - avoid this by also checking whether the VMA seqcount is odd
before bailing on the read lock.
So far, vma->vm_lock_seq has defaulted to UINT_MAX, but this has been
unnecessary since commit e5e7fb278e59 ("mm: convert mm_lock_seq to a proper
seqcount"), since this is assigned when the VMA is still detached, and
cannot be observed.
Therefore perform both reset and initialisation of vma->vm_lock_seq by
simply setting vma->vm_lock_seq to zero.
The reset, performed in __mm_reset_vma_seqcounts(), only needs to
WRITE_ONCE() the value.
However the read of vma->vm_lock_seq in vma_start_read() needs to have
acquire semantics to pair with the smp_wmb() in do_raw_write_seqcount_end()
so a reader reading a reset value also observes VMA modifications made
under the write lock.
Since all VMA write locks were just released and the mmap write lock is
still held, there are no concurrent VMA writers.
Concurrent VMA readers are unaffected. New VMA readers observe either zero,
never a write lock, or the prior seqcount, behaving as before the reset.
Since this only occurs every 2^31 write lock/unlock cycles, the time taken
to update all VMAs is considered to be acceptable.
Also update comments, documentation and VMA userland tests to reflect the
change.
User facing effects are - dangerous races that could cause memory
corruption or NULL pointer derefences (especially on page table traversal)
are avoided.
Link: https://lore.kernel.org/20261016-fix-vma-lock-overflow-v2-1-9eb778b79b58@kernel.org
Fixes: 5e31275cc997 ("mm: add per-VMA lock and helper functions to control it")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Daehyeon Ko <4ncienth@gmail.com>
Closes: https://lore.kernel.org/all/20261015143735.1630957-1-4ncienth@gmail.com/
Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Tested-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: <stable@vger.kernel.org> # 7.3.x
|
|
When the cgroup has no memory pressure at all, writing to
/sys/devices/system/nodeX/reclaim triggers proactive reclaim on
NUMA node, causing increase in the writer cgroup's memory PSI.
Due to this reclaim is performed in the context of the write(),
accounted as memory pressure on the writer, like
commit e22c6ed90aa9 ("mm: memcontrol: don't count limit-setting reclaim
as memory pressure"). This is unexpected, the similar scenarios (e.g.
senpai) will continue to occur, only this time it is for the
/sys/devices/system/nodeX/reclaim knob.
The Documentation/ABI/stable/sysfs-devices-node documentation also
notes that "This interface is equivalent to the memcg variant."
This patch unifies the semantics of the memcg and node interfaces:
per-node proactive reclaim is no longer counted as memory pressure,
and the per-node proactive reclaim interface no longer produces
phantom pressure.
I ran demo[1] that performs per-node proactive reclaim 10000 times
in qemu, writer cgroup memory.pressure as follows:
without patch:
some avg10=31.53 avg60=13.42 avg300=3.31 total=10602985
full avg10=31.53 avg60=13.42 avg300=3.31 total=10602985
with patch:
some avg10=9.59 avg60=3.41 avg300=0.81 total=2686221
full avg10=9.59 avg60=3.41 avg300=0.81 total=2686221
Link: https://lore.kernel.org/20261012022917.1102467-1-vernon2gm@gmail.com
Link: https://github.com/vernon2gh/app_and_module/tree/main/reclaim_node_psi [1]
Fixes: b980077899ea ("mm: introduce per-node proactive reclaim interface")
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Acked-by: Davidlohr Bueso <dave@stgolabs.net>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: <stable@vger.kernel.org>
|
|
MGLRU evict_folios() isolates folios from the LRU without updating the
NR_ISOLATED_ANON/FILE counters, unlike the legacy shrink_inactive_list()
path. This causes compaction's too_many_isolated() check to under-count
isolated pages when MGLRU reclaim is active.
Add NR_ISOLATED counter updates in evict_folios(): increment after
isolate_folios() and decrement after all retry passes complete, using the
existing nr_isolated which holds the original isolated count.
Link: https://lore.kernel.org/141901355a7d9e4bbca5c4840ae32ea5c65fe3b6.1791968227.git.zhuhui@kylinos.cn
Link: https://lore.kernel.org/cover.1791968227.git.zhuhui@kylinos.cn
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
|
|
As-is, madvise_cold_or_pageout_pte_range() discards the hardware young bit
by invalidating the PMD, doing pmd_mkold() and subsequently writing to the
pmd again. However, it does so in a racy manner, looking at a
possibly-stale snapshot. It's entirely possible that a write elsewhere
set the hardware D bit, which this logic now loses.
Fix it by using pmdp_test_and_clear_young(). This is an atomic
modification of the PMD entry, and thus safe against any possible racing
writes. As a (nice) side-effect, this elides one non-required TLB flush
in pmdp_invalidate().
This bug can interact with the swapping system in complicated ways,
leading to data loss. For instance, if a folio is marked lazyfree, but
was later read from, and then MADV_COLD'd while racing a write to that
same THP, dropping the dirty bit may lead to data loss as
reclaim/MADV_PAGEOUT drops the "clean" anon folio:
T0 T1
================================================================
MADV_FREE(thp) |
folio marked lazyfree |
read(thp) |
PMD marked young by hardware |
| MADV_COLD(thp)
write(thp) | if (pmd_young(pmd)) {
PMD marked dirty by hardware | set(pmdp, pmd_mkold(pmd));
MADV_PAGEOUT/reclaim |
dirty bit may or may not be |
set |
reclaim/pageout may drop |
the folio |
Link: https://lore.kernel.org/20261013191719.95502-1-pfalcato@suse.de
Fixes: 9c276cc65a58 ("mm: introduce MADV_COLD")
Signed-off-by: Pedro Falcato <pfalcato@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Samarth Narang <samarth.colleges@gmail.com>
Closes: https://lore.kernel.org/linux-mm/20261011132227.413217-1-samarth.colleges@gmail.com/
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: <stable@vger.kernel.org>
|
|
When memory allocation profiling is enabled, the pages backing percpu
memory chunks are attributed to func:pcpu_alloc_pages. However, the
accounting of individual percpu memory allocations via alloc_percpu()
and friends record only the size of a single object rather than its
full size across all CPUs.
As a result, it is often not straightforward to identify which kernel code
is actually responsible for percpu memory, because the size is
under-accounted by a factor of num_possible_cpus().
Account full object size with pcpu_obj_full_size(size) instead, which
multiplies the object size by num_possible_cpus() and adds the objcg
pointer overhead. This is consistent with what memcg already does for
charging percpu allocations.
Verified with 64-CPU VM by spawning 400 processes in a memory cgroup and
measuring mm_init()'s rss_stat counters.
Delta of kernel/fork.c:1142 func:mm_init in /proc/allocinfo:
bytes calls per allocation
unpatched 6320 395 16
patched 418176 396 1056
'percpu' field in memory.stat: 422400 (1056 bytes per process)
Memory allocation profiling now accounts the same amount of memory as
what memcg charges.
Link: https://lore.kernel.org/20261013-fix-alloc-percpu-accounting-v1-1-03e788a1e41f@kernel.org
Fixes: 60fa4a9e2323 ("mm: percpu: add codetag reference into pcpuobj_ext")
Signed-off-by: Harry Yoo (Meta) <harry@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Assisted-by: LLM # Verification
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Hao Ge <hao.ge@linux.dev>
Cc: <stable@vger.kernel.org>
|
|
To move one page out of a large folio, move_pages_ptes() first has to
split the folio. __folio_split() refuses if the folio has any reference
beyond its page table mappings and the caller's own, and it reports the
refusal as -EAGAIN. move_pages() treats returned -EAGAIN as a transient
race and keeps retrying the same address, holding the VMA read locks the
whole time. This is incorrect since in some cases userspace can hold such
a reference indefinitely, such as by registering the page as an io_uring
buffer.
An unprivileged user can trigger this with three threads: first holds such
a reference on a page of a THP, second calls UFFDIO_MOVE on another page
of the same THP and spins in the kernel forever holding the VMA read
locks, and third calls fork() or mprotect(), which takes mmap_lock for
writing and then waits forever for those read locks. From then on every
mmap_lock user of that mm hangs, including other processes inspecting it
through /proc/<pid>/, until the process is SIGKILLed. The hung task
detector reports the blocked tasks (third one), and with
kernel.hung_task_panic=1 the machine panics:
INFO: task wedge:95 blocked for more than 122 seconds.
task:wedge state:D stack:12528 pid:95 tgid:95 ppid:94
Call Trace:
__schedule+0x596/0x1110
schedule+0x36/0xf0
__vma_start_exclude_readers+0x100/0x170
__vma_start_write+0x51/0xd0
__split_vma+0x208/0x3d0
vma_modify+0x6a8/0x840
vma_modify_flags+0xce/0x110
mprotect_fixup+0xbe/0x480
do_mprotect_pkey+0x2d9/0x580
__x64_sys_mprotect+0x1a/0x30
Fix by returning -EBUSY instead of -EAGAIN in move_pages_ptes(), the
caller of split_folio(), when the split fails. Thus, move_pages() does
not retry -EBUSY and the ioctl drops its locks and returns to userspace
instead of spinning.
Link: https://lore.kernel.org/20261012201514.97015-1-cenzhang@linux.microsoft.com
Fixes: adef440691ba ("userfaultfd: UFFDIO_MOVE uABI")
Signed-off-by: Cen Zhang (Microsoft) <cenzhang@linux.microsoft.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: <AutonomousCodeSecurity@microsoft.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Assisted-by: LLM
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Peter Xu <peterx@redhat.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: <tgopinath@linux.microsoft.com>
Cc: <stable@vger.kernel.org>
|
|
mlock_fixup() adjusts mm->locked_vm, an unsigned long, by the number of
pages whose VM_LOCKED state it changes, but computes that number in an
int:
int nr_pages;
...
nr_pages = (end - start) >> PAGE_SHIFT;
...
mm->locked_vm += nr_pages;
When a single VMA range of 2^31 pages (8 TiB with 4 KiB pages) or more
is locked or unlocked, the count is truncated to 32 bits and then
sign-extended, so locked_vm is changed by the wrong amount, while
munmap(), mremap() and the other paths that maintain locked_vm use the
full unsigned long page count.
Such a range is reachable on 64-bit architectures with a 47-bit or
larger user address space (x86_64, arm64 with 48-bit VA, riscv64 Sv48)
by a task with CAP_IPC_LOCK or a raised RLIMIT_MEMLOCK, which do_mlock()
checks against the full untruncated length. It costs nothing with
MAP_NORESERVE and MLOCK_ONFAULT/MCL_ONFAULT, since then no page is
populated. On x86_64, after
p = mmap(NULL, 8UL << 40, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
mlock2(p, 8UL << 40, MLOCK_ONFAULT);
VmLck in /proc/self/status reads 18446744065119617024 kB instead of
8589934592 kB; munmap() of the region then subtracts the true size and
leaves VmLck at 18446744056529682432 kB instead of 0. With 16 TiB,
locked_vm is not changed at all, VmLck stays 0 kB, and after the task
drops CAP_IPC_LOCK or lowers RLIMIT_MEMLOCK, mlock(), mmap(MAP_LOCKED)
and the other mlock_future_ok() and account_locked_vm() users enforce
RLIMIT_MEMLOCK against that count, so they let the task lock memory
beyond its limit (or, in the 8 TiB case, refuse every further lock).
The error persists for the life of the mm; unlocking the range in pieces
skews it further. The wrong count is per-mm accounting only: it does
not change which pages are locked.
The int has been there since before git history (the variable was named
"pages" until it was renamed in 2.6.28); every other updater of
locked_vm uses an unsigned long or long count. Make nr_pages a long,
which holds the page count of any user range and its negation, so the
existing sign handling is unchanged.
Found by formal verification of mm/mlock.c against the kernel binary.
Link: https://lore.kernel.org/20261014131358.780240-1-nickolai@csail.mit.edu
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Assisted-by: LLM
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: <stable@vger.kernel.org>
|
|
A collapse is two jobs. One reads a PTE table under mmap_lock and decides
whether the range is worth collapsing. The other allocates, isolates,
copies and flushes, and wants the lock given up first.
collapse_single_pmd() did both, so the boundary between them was somewhere
in the middle of a function.
Give each half its own function:
- collapse_scan_pmd() scans one table. It picks the side:
collapse_scan_anon(), the anonymous scan that used to carry the
collapse_scan_pmd() name, or collapse_scan_file().
- collapse_pmd() does the collapse the scan asked for, and is handed
what the scan returned. SCAN_SUCCEED means there is something to
collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page cache already
holds the PMD folio and only the PTE table is left to retract.
Anything else is why there is nothing to do. It calls
collapse_anon(), formerly mthp_collapse(), or collapse_file().
For now collapse_single_pmd() stays, as the two of them with the
mmap_lock drop in between, so its callers see what they saw before.
collapse_control_init() sets a control up before its first scan.
What the scan found and the collapse needs travels in collapse_control.
For an anonymous table that is the orders and the referenced and
swapped-out counts. collapse_anon() and collapse_huge_page() now read
them from there instead of taking them as arguments.
For a file it is the file itself and the offset in it. A file collapse
works on the page cache and never sees a VMA, so the scan takes the
reference while it still has one and the collapse gives it back.
The file scan moves under mmap_lock with the anonymous one, where before
the lock was given up first. The lock is now held over the page cache
walk, an RCU walk over one table's worth of slots with no PTL, and taken
fewer times. collapse_scan_mm_slot() ends its walk whenever the lock was
dropped, so a refused file table used to cost khugepaged an unlock, a
trip back through khugepaged_do_scan(), a relock and a VMA lookup. Now
only a table that goes on to be collapsed does.
Link: https://lore.kernel.org/20261014171145.3292736-10-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
collapse_scan_file() reads the page cache to decide whether a table is
worth collapsing and, when it is, calls collapse_file() itself. The
caller cannot get between the decision and the collapse.
Move the collapse_file() call up into collapse_single_pmd(), so the scan
stops at the decision.
Two things change with it. The writeback retry re-runs collapse_file()
alone instead of rescanning first; collapse_file() repeats the scan's
checks under the page cache lock anyway. And mm_khugepaged_scan_file
fires before the collapse, so for an accepted table its status reads
SCAN_SUCCEED; what the collapse made of the table is for
mm_khugepaged_collapse_file to report.
Preparation for splitting a collapse into a scan under mmap_lock and a
run without it. The file scan has to stop where the anonymous one will.
Link: https://lore.kernel.org/20261014171145.3292736-9-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
collapse_control_init_scan() resets what one scan accumulates: the node
load, the allocation nodemask and the eligible-PTE bitmap. It runs before
every PTE table a scan is given, not once per control, so its name points
at the wrong thing.
Call it collapse_scan_reset().
Preparation for giving a control a real init, run once for a whole series
of scans. Two names a word apart would then stand for two different jobs.
No functional change.
Link: https://lore.kernel.org/20261014171145.3292736-8-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
collapse_possible() only forwards to collapse_possible_orders() and turns
its mask into a bool. Its three callers can test the mask themselves.
No functional change.
Link: https://lore.kernel.org/20261014171145.3292736-7-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
Tests scattered through the collapse path decide what a collapse is
allowed to do by asking whether khugepaged started it. Between them they
settle:
- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, and
whether a sub-PMD window is held to a stricter rule than a PMD;
- whether a range has to look used, and whether a MADV_FREE'd page is
left alone;
- whether the PMD is mapped as part of the request, and whether dirty
pages are worth writing back and retrying.
None of those is a fact about khugepaged. Each is something the caller
decided before asking, and the collapse code should not have to look up
who called to find out.
Add struct collapse_policy for the caller to fill: khugepaged from its
own settings, MADV_COLLAPSE from the fact that a user asked explicitly.
Every test becomes a read of a field, and cc->is_khugepaged goes, having
no reader left.
The PTE limits come as two sets, one for a PMD-sized window and one for
anything smaller, so that the helpers pick a set for the order and read
it. khugepaged takes no swapped-out or shared PTE into a sub-PMD window,
and empty PTEs only when the knob says all or nothing; the warning for a
knob value in between moves to where khugepaged fills its policy. The
fields only one side reads say which: anon_ or file_. David Hildenbrand
asked for both.
khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per
call. That is the one change in behaviour. The max_ptes_* limits and the
defrag setting behind the allocation mask are sampled once per pass rather
than on every table. A table scanned early in a pass and one scanned late
are then treated alike.
collapse_file() also drops a NULL check on the collapse_control. It has
one call site, reached only from collapse_single_pmd(), which dereferences
cc unconditionally, so the check was already dead.
Link: https://lore.kernel.org/20261014171145.3292736-6-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
khugepaged.c holds both the users of collapse and the machinery that
performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's
entry point and the collapse itself all sit in one file and reach into
each other freely. Nothing marks where a user ends and the engine
begins.
Start drawing that line.
Add mm/collapse.h for what the two sides have to agree on:
- enum scan_result - what the engine hands back;
- struct collapse_control - the state a request carries.
And two constants move with them:
- KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT;
- KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER.
Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix.
No functional change.
Link: https://lore.kernel.org/20261014171145.3292736-5-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
The name says less than the bit means. A set bit means not only that the
PTE is present, but also that it passed the other checks: uffd,
lazyfree, anonymity, sharing. The PTE can be considered a collapse
source.
mthp_collapse() then reads the bitmap starting at the PMD order, so the
bitmap is not specific to mTHP either.
Name it for what a set bit means, and update the comments that named it.
No functional change.
Link: https://lore.kernel.org/20261014171145.3292736-4-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and
tests cc->is_khugepaged to know whether it should: the counter belongs to
the daemon, and MADV_COLLAPSE must not touch it.
The daemon sees every result of every collapse it asks for, so it can keep
its own counter without the shared path testing who called.
Link: https://lore.kernel.org/20261014171145.3292736-3-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
Patch series "mm/collapse: separate a collapse from its callers", v5.
There is no line between the collapse engine and the callers that ask for
a collapse. khugepaged.c holds both, and they reach into each other.
- Sixteen tests through the collapse path read cc->is_khugepaged to work
out what they are allowed to do, when every one of those decisions was
made by the caller before it asked.
- collapse_single_pmd() does both halves of a collapse behind one call.
Where it gives up mmap_lock between them is buried in the anonymous
scan on one side and ahead of the file scan on the other.
Start drawing the line. The caller states what it allows in a policy,
and the collapse code reads the policy instead of asking who called. The
scan of a PTE table and the collapse that follows get a function each,
with the mmap_lock drop between them in one place.
This patch (of 9):
madvise_collapse() holds an mmgrab() reference across its work. It is
redundant. Every caller already holds mm_users:
- madvise(2) works on current->mm, which lives as long as the task is in
the syscall;
- process_madvise(2) reaches a remote mm through mm_access(), which takes
an mm_users reference and holds it until the syscall returns;
- io_uring passes current->mm;
- DAMON takes one with get_task_mm() and drops it after the call.
Drop the mmgrab()/mmdrop() pair.
Link: https://lore.kernel.org/20261014171145.3292736-1-kas@kernel.org
Link: https://lore.kernel.org/20261014171145.3292736-2-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
find_vma_and_prepare_anon() can end up allocating an anon_vma in VMAs that
are not CoW mappings, which is wrong.
If we have a MAP_SHARED R/O file mapped, we lack VM_SHARED (but do have
VM_MAYSHARE) and lack VM_MAYWRITE.
While uffd later rejects such mappings (uffd registration requires
VM_MAYWRITE, see comment in validate_dst_vma) find_vma_and_prepare_anon()
still wrongly allocates an anon vma before it validates the vma.
This in turn later confuses copy_page_range()->vma_needs_copy() such
that we end up calling copy_hugetlb_page_range() with ranges that
actually do not contain *any* per-pagetable information that would
require copying, which is the basis for PMD-table sharing in the first
place.
copy_hugetlb_page_range() should not have to worry about shared
PMD tables at all: it doesn't make any sense to copy page tables that are
shared, as they can be fully recreated later. Ending up in the function
with shareable page tables is the real issue.
If a destination PMD table is shared in copy_hugetlb_page_range(), it
might get concurrently unshared as we are not holding relevant
locks, resulting in us using a page table that is no longer mapped and
possibly freed after calling:
dst_pte = huge_pte_alloc(dst, dst_vma, addr, sz);
To fix it, do not allocate an anon_vma in non-CoW mappings and add a
safety guard to copy_hugetlb_page_range(), making sure that we never
end up messing with shared PMD tables in there.
Before commit 867a43a34ff8a ("userfaultfd: use per-vma locks in userfaultfd
operations") we would always validate the VMA before allocating an
anon_vma, and consequently never wrongly allocate an anon_vma.
As uffd_lock_vma() is part of the same lookup machinery and
also contains a misleading VM_SHARED check, change that one as well
directly for consistency.
A follow-up cleanup will stop taking the hugetlb_vma_lock_read(src_vma)
lock in copy_hugetlb_page_range() entirely. Further, the usage of
VM_SHARED in userfaultfd code should be cleaned up later to focus on
CoW vs. non-CoW mappings.
Link: https://lore.kernel.org/20261014-copy_hugetlb_page_range-v1-1-5b68e03fe7fe@kernel.org
Fixes: 867a43a34ff8 ("userfaultfd: use per-vma locks in userfaultfd operations")
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Kyle Zeng <kylebot@openai.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Kyle Zeng <kylebot@openai.com>
Tested-by: Kyle Zeng <kylebot@openai.com>
Acked-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Tested-by: Lance Yang <lance.yang@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Lokesh Gidra <lokeshgidra@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: <stable@vger.kernel.org>
|
|
Replace the hardcoded 32 with sizeof(buf) at the remaining four call
sites, for consistency with the rest of the file.
Link: https://lore.kernel.org/20261014031147.444733-1-lienze@kylinos.cn
Signed-off-by: Enze Li <lienze@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: David Hildenbrand <david@kernel.org>
|
|
Since commit 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm
in defrag_mode"), direct reclaim and compaction for non-movable requests
under defrag_mode run at pageblock_order, to produce the whole blocks that
ALLOC_NOFRAGMENT needs. The retry decisions that follow still use the
request order. An order-0 request can therefore retry indefinitely
without ever reaching the ALLOC_NOFRAGMENT fallback:
- Reclaim at pageblock_order gives up after one pass as soon as a zone
looks compaction_ready(), and do_try_to_free_pages() then returns 1
even though nothing was reclaimed. It returns before the retry that
would reclaim memory.low-protected cgroups, so when most memory is
protected, the pass that did run finds next to nothing.
- Compaction at pageblock_order fails or is deferred.
- should_reclaim_retry() takes the reported progress as progress for
the order-0 request and resets no_progress_loops. The request
retries.
Order 1-3 requests loop the same way, and should_compact_retry() also
checks their pageblock_order compaction result against the request order.
On a production host (64G, defrag_mode, memory.low covering most of the
workload), 95% of direct reclaim runs were order-9 runs that returned 1
with nothing reclaimed, at up to 60k runs per second. Across ~200M
should_reclaim_retry() calls in a day, no_progress_loops never left 0.
The spinning allocations were SLUB slab refills for inode and dentry
caches. The time spent registers as memory pressure, and pressure-based
OOM killing takes down both workloads and system services.
For promoted requests:
- Reclaim progress does not reset no_progress_loops, as for costly
orders.
- should_compact_retry() checks the compaction result at the promoted
order and does not retry COMPACT_SKIPPED, since the request can fall
back. The compaction priority floor and the COMPACT_SUCCESS retry
limit stay those of the request order, so a non-costly request still
gets its COMPACT_PRIO_SYNC_FULL pass before it falls back.
When a promoted request takes the fallback, reset the retry counters, so
that the fallback attempt gets a full retry budget before the OOM killer
is considered. A request whose reclaim was not promoted spent its budget
on the request itself and keeps its counters, as Usama Arif pointed out.
__alloc_pages_slowpath() computes the promoted order once per iteration
and keeps it in alloc_context for direct reclaim, direct compaction and
the two retry helpers, so no callee has to recompute it. Harry Yoo asked
for the promoted order to be explicit rather than derived in each callee,
and Vlastimil Babka suggested alloc_context over new function parameters.
In a VM reproducer (32G, defrag_mode, inode churn under memory.low):
before after
should_reclaim_retry() calls 63M 293k
peak memory pressure (PSI some avg10) 99% 12%
File creation runs 5.7x faster.
[kas@kernel.org: s/order_nofrag_promoted/order_defrag_promoted/, per hannes]
Link: https://lore.kernel.org/atDL-nSFP2KVU7cI@thinkstation
Link: https://lore.kernel.org/as5q14OsbK2CmOrh@thinkstation
Fixes: 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm in defrag_mode")
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Assisted-by: LLM
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: <stable@vger.kernel.org>
|
|
hugetlb_sysfs_init() guards the "hugetlb_sysfs_initialized = true"
assignment with an #ifdef CONFIG_NUMA block, while the following
hugetlb_register_all_nodes() call is left unguarded and relies on the
empty !CONFIG_NUMA stub.
Move the assignment into the NUMA version of hugetlb_register_all_nodes(),
before it registers the online nodes, so the #ifdef in
hugetlb_sysfs_init() can go away. Keep the !CONFIG_NUMA stub, so the call
site is unconditional in both configurations.
No functional change; this is an #ifdef cleanup.
Link: https://lore.kernel.org/20261009142010.552614-1-lienze@kylinos.cn
Signed-off-by: Enze Li <lienze@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Muchun Song <muchun.song@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Enze Li <enze.li@gmx.com>
|
|
__discard_anon_folio_pmd_locked() clears a lazyfree THP's PMD before it
determines whether the folio can be discarded, and restores it if the
folio is dirty or has extra references. It is done under only the PMD
lock and the anon_vma read lock inside try_to_unmap_one() and no mmap lock
is held. The temporarily none PMD causes UAF.
A parallel munmap() or mremap() is supposed to serialize with it on the
PMD lock, but zap_pmd_range() (called by munmap()) and move_page_tables()
(called by mremap()) do not take the lock when they see the temporarily
none PMD and skip the PMD. There is no more page table walk after the
none PMD is restored:
- munmap(): free_pgtables() waits in unlink_anon_vmas() for the anon_vma
lock, then free_pmd_range() drops the restored PMD as bad without
removing the rmap, so the folio keeps a stale mapcount.
- whole-VMA MREMAP_DONTUNMAP: dontunmap_complete() unlinks the source VMA
from its anon_vma while the restored PMD still maps the folio.
- plain mremap(): the THP is not moved but unmapped, so the THP's data is
lost.
In the first two cases, the folio stays mapped, but no VMA that maps it is
linked to its anon_vma. When the anon_vma is freed, it is still reachable
when one rmap walks the folio. Racing lazyfree reclaim against
MREMAP_DONTUNMAP as an unprivileged user reproduces a KASAN use-after-free
in folio_lock_anon_vma_read() as reported in the Closes link.
One solution is to use an invalidated PMD instead of a cleared one until
the discard is committed, but it needs arch fixes first.
1. GUP-fast still follows an invalidated PMD on riscv, LoongArch and s390.
2. sparc64 miscounts thp_pte_count when an invalidated PMD is set again or
cleared.
To keep the fix simple for backport, stop discarding lazyfree THPs at PMD
level instead. try_to_unmap_one() splits the PMD and discards the folio at
PTE level.
unmap_huge_pmd_locked() and __discard_anon_folio_pmd_locked() are left in
place, unused, to minimize this fix for backport and the invalidated PMD
approach is planned as a follow-up series to use them again.
Link: https://lore.kernel.org/20261012-lazyfree-stable-fix-v2-1-6387d6e30063@nvidia.com
Fixes: 735ecdfaf4e8 ("mm/vmscan: avoid split lazyfree THP during shrink_folio_list()")
Signed-off-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Kyle Zeng <kylebot@openai.com>
Closes: https://lore.kernel.org/r/20261009165214.40212-2-kylebot@openai.com
Reviewed-by: Kyle Zeng <kylebot@openai.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Tested-by: JongHyuk Kim(MalHyuk) <malhyuk97@gmail.com>
Link: https://lore.kernel.org/CADQRuqvXQ=YE=7T3oR66vTxE33pi=ygEeZKPTOU=cALZL0Figw@mail.gmail.com
Assisted-by: LLM
Cc: Rik van Riel <riel@surriel.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: <stable@vger.kernel.org>
|
