| Age | Commit message (Collapse) | Author |
|---|
|
A compound page of order N must start at a PFN aligned to 2^N pages.
Therefore, a range used to initialize compound pages must start on a
compound-page boundary and contain an integral number of compound pages.
Two recent Device DAX bugs were caused by ranges that violated this
invariant, but neither was detected by memmap_init_zone_device().
Warn if the adjusted zone-device range violates this invariant so future
violations are reported when the compound pages are initialized.
Link: https://lore.kernel.org/20261016131334.3419393-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/r/20261016053944.2579541-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/r/20261016054914.2601510-1-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Assisted-by: LLM
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: Alison Schofield <alison.schofield@intel.com>
Cc: Joao Martins <joao.m.martins@oracle.com>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
|
|
devm_request_resource() calls request_resource_conflict(), which takes
resource_lock, tries to insert the new resource and returns a pointer to
the conflicting resource after dropping the lock. The error message is
then formatted from that pointer: conflict->name and the %pR argument
read the name, start, end, flags and desc of a resource that nothing
pins any more. The owner of the conflicting resource may call
release_resource() and free it the moment the lock is dropped.
The conflicting resource normally belongs to another device. With two
devices whose drivers claim overlapping regions (two PCI host bridges
whose "ranges" windows overlap, two instances of a controller on the
same aperture, or a stale claim an administrator is replacing), the
following interleaving reads freed memory:
CPU 0: driver B probe CPU 1: driver A unbind
devm_request_resource(devB, root, resB)
request_resource_conflict()
write_lock(&resource_lock)
__request_resource() -> resA
write_unlock(&resource_lock)
devres_release_all(devA)
devm_resource_release()
release_resource(resA)
put_device() -> kfree(resA)
dev_err(..., resA->name, resA) use after free
Both halves are reachable from userspace on any kernel: a write to the
"unbind" file of the driver that holds the region and a write to the
"bind" file of the driver that wants it. One in-tree path is
drivers/pci/bus.c devm_request_pci_bus_resources(), called from host
bridge probe for every bridge window; the windows are the embedded
struct resource of kzalloc'ed resource_entry objects that
pci_free_resource_list() frees when the other bridge is released.
In the common case the "resource collision" message is garbled; with
the slab object poisoned (CONFIG_SLUB_DEBUG) or reused, formatting
conflict->name oopses, and KASAN reports a slab-use-after-free read in
devm_request_resource(). All of this happens in a probe that is
already failing with -EBUSY. The window is a few hundred instructions
wide and exists only when two devices actually collide, so the race
has not been observed outside of analysis.
Do the same here: take resource_lock, insert with __request_resource()
and print the diagnostic while the lock still keeps the conflicting
resource in the tree, then drop the lock. The return values and the
devres handling are unchanged.
The change was built for x86_64 and RISC-V; the resource KUnit tests
pass on both.
Found by formal verification of kernel/resource.c against the kernel
binary.
Link: https://lore.kernel.org/20261016140043.790048-1-nickolai@csail.mit.edu
Fixes: 8d38821cbcf5 ("resources: Add device-managed request/release_resource()")
Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Assisted-by: LLM
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Cc: Thierry Reding <treding@nvidia.com>
Cc: <stable@vger.kernel.org>
|
|
A process that extends a file by 4 GiB or more past a dirty folio, on a
file system whose block size is smaller than the page size (ext4 with 1k
blocks), reads back the bytes it wrote through a shared mapping beyond the
old end of file instead of zeroes, and a later write into that block
carries them to the disk. With the example of commit 52aecaee1c26 ("mm:
zero range of eof folio exposed by inode size extension") and the truncate
taken 4 GiB further (pwrite 2k at 0, mmap the first page and write 'X'
over [2k, 4k), truncate(2) to 4 GiB + 2k), the read of [2k, 4k) returns
the 2048 'X' bytes, and a pwrite of 16 bytes at 3k followed by fsync() and
a remount leaves 1008 'X' bytes on the disk in [3k+16, 4k). No privilege
is needed, and every size extension of 4 GiB or more reaches it:
truncate(2)/ftruncate(2) through truncate_setsize(), writes beyond the end
of file through generic_write_end(), ext4, iomap, btrfs, exfat, ntfs, afs
and orangefs. Affected since v6.13.
pagecache_isize_extended() zeroes the part of the dirty folio straddling
the old end of file that the size extension exposes, from the old size to
the new size or the end of the folio, whichever comes first:
end = min_t(unsigned int, to - folio_pos(folio), folio_size(folio));
min_t() converts both operands to unsigned int before comparing, so the
loff_t difference to - folio_pos(folio) is reduced modulo 2^32 first.
When the file grows by 4 GiB or more past the folio, end is that residue
instead of folio_size(folio): below offset nothing is zeroed, above it
only [offset, end) is. The bytes beyond the old end of file stay in the
dirty folio and, now inside i_size, are file data.
Compare in loff_t; the result is at most folio_size(folio) and fits end.
Found while formally verifying pagecache_isize_extended() against the
kernel binary.
Link: https://lore.kernel.org/20261016222721.791785-1-nickolai@csail.mit.edu
Fixes: 52aecaee1c26 ("mm: zero range of eof folio exposed by inode size extension")
Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>
Assisted-by: LLM
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Brian Foster <bfoster@redhat.com>
Cc: Theodore Ts'o <tytso@mit.edu>
Cc: <stable@vger.kernel.org>
|
|
With debug_guardpage_minorder set, expand() turns the unused halves of a
split into guard pages, which page_del_and_expand() leaves out of the free
counts. Freeing the allocated half credits only that half, and
__free_one_page() then merges the guard buddy with clear_page_guard(),
which no longer touches the counters.
The merged block goes on a free list with the guard pages never counted,
so NR_FREE_PAGES falls behind the free lists by every guard that merges.
Before commit e0932b6c1f94 ("mm: page_alloc: consolidate free page
accounting"), __set_page_guard() and __clear_page_guard() adjusted the
counts themselves; that commit dropped both adjustments but kept the
subtraction implicit in expand().
Count the guard pages under the merged block's migratetype when the buddy
is cleared. account_freepages() skips an isolated block as it does for
every other free page, and moving the block off the isolated list counts
it then.
Booting a 16GB VM with debug_pagealloc=on debug_guardpage_minorder=1,
check the difference in free pages reported between /proc/vmstat and
/proc/buddyinfo.
At three points (idle, after a read and file-creation load, after a second
load), check the difference between nr_free_pages and the buddyinfo
numbers in the normal zone, as a number of pages:
idle after load after second load
unpatched -130 79205 99121
patched -16 45 0
The small remaining difference seems to be due to the numbers not being
read at exactly the same time, and is also seen without guard pages.
Link: https://lore.kernel.org/20261016122304.5b1eba8e@fangorn
Fixes: e0932b6c1f94 ("mm: page_alloc: consolidate free page accounting")
Signed-off-by: Rik van Riel <riel@surriel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Dev Jain <dev.jain@arm.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Assisted-by: LLM
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Kairui Song <kasong@tencent.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Frank van der Linden <fvdl@google.com>
Cc: <stable@vger.kernel.org>
|
|
For vmemmap-optimized sections, tail struct pages may be backed by shared
vmemmap pages. Those shared pages must carry the same page zone ID as the
struct pages initialized for the section.
Warn in __init_single_page() if the shared tail page has a different
page_zone_id(), which would indicate inconsistent initialization.
[muchun.song@linux.dev: document ordering of shared and retained struct page initialization]
Link: https://lore.kernel.org/20261010115523.4005071-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20261010075755.83339-7-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
vmemmap_populate_address() no longer has any callers that need the
returned PTE. Its only remaining user, vmemmap_populate_range(), only
checks whether population succeeded.
Open-code vmemmap_populate_address() directly in
vmemmap_populate_basepages(), remove the now-redundant range helper, and
return -ENOMEM directly on failure.
Link: https://lore.kernel.org/20261010075755.83339-6-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
vmemmap_pte_populate() no longer uses ptpfn as an input. Drop the
argument to simplify the code.
Link: https://lore.kernel.org/20261010075755.83339-5-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
The common vmemmap path selects the shared page for optimized mappings
itself, so Device DAX no longer needs vmemmap_populate_compound_pages() to
find a shared tail page and pass its backing PFN through the generic
population helpers.
Remove the Device DAX-specific population path and let section memmap
population always use vmemmap_populate(). The powerpc retains an
architecture-specific compound-page implementation, so select it directly
from radix__vmemmap_populate() for optimizable sections.
Link: https://lore.kernel.org/20261010075755.83339-4-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
The common vmemmap population path cannot yet handle optimized Device DAX
mappings on its own. It uses pfn_to_zone() to find the shared tail page,
but Device DAX populates its vmemmap at runtime before the ZONE_DEVICE
span is initialized.
Teach the common path to use device_zone() for runtime optimized vmemmap
population while retaining pfn_to_zone() for early boot. This allows the
same path to support both early boot mappings and Device DAX.
The backing PFN supplied by the Device DAX-specific population path is no
longer used, allowing the redundant lookup and population code to be
removed later.
Link: https://lore.kernel.org/20261010075755.83339-3-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
Patch series "mm: Unify device DAX and HugeTLB vmemmap population paths",
v4.
This series is split out from the earlier, larger series "mm: Generalize
HVO for HugeTLB and device DAX" [2]. While the parent series generalizes
vmemmap optimization across HugeTLB and device DAX, this subset addresses
a single, self-contained step: unifying their vmemmap population paths.
After the preceding Device DAX conversion, both HugeTLB and Device DAX
describe optimized vmemmap mappings through memory-section metadata and
use per-zone shared tail vmemmap pages. The generic code, however, still
carries a Device DAX-specific population flag and compound-page population
path, along with arguments and helpers needed only by that path.
This series first removes VMEMMAP_POPULATE_DAX and moves selection and
reference handling for the shared tail page into the common vmemmap
population path. It then removes the generic Device DAX-specific
compound-page population path and routes section vmemmap population
through vmemmap_populate(). The powerpc radix path continues to use its
architecture-specific compound-page population implementation for
optimizable sections.
The remaining patches remove the unused ptpfn argument, open-code
vmemmap_populate_address() now that no caller needs its returned PTE, and
add a warning for inconsistent zone initialization of shared tail vmemmap
pages.
This is the fourth smaller step toward the broader HVO generalization.
After this series, HugeTLB and Device DAX use the same population model
instead of parallel generic paths, while powerpc keeps its
architecture-specific implementation.
This patch (of 6):
VMEMMAP_POPULATE_DAX currently distinguishes DAX vmemmap population in two
places: it keeps allocations on the normal path and takes a reference when
a backing page is supplied for reuse.
After Device DAX switched to the common per-zone shared tail page, both
conditions can be determined locally. DAX supplies ptpfn for every shared
tail mapping and requests an allocation only for compound head mappings,
whose PFNs are not optimizable. Therefore, vmemmap_optimizable_pfn()
alone selects the correct allocation path.
When ptpfn is supplied, the caller is reusing an existing backing page.
Once the slab allocator is available, take a reference for each reused
mapping to balance the release performed by vmemmap_free(). Although the
buddy allocator is available before slab, no vmemmap population occurs in
that interval. Earlier mappings are backed by memblock/reserved memory
and do not need page reference accounting.
Remove VMEMMAP_POPULATE_DAX and the flags argument from the vmemmap
population helpers.
Link: https://lore.kernel.org/20261010075755.83339-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20261010075755.83339-2-songmuchun@bytedance.com
Link: https://lore.kernel.org/20260930140627.57431-1-songmuchun@bytedance.com/ [1]
Link: https://lore.kernel.org/20260513130542.35604-1-songmuchun@bytedance.com/ [2]
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
swap_cache_get_folio() reads the folio from the swap table and takes a
reference with folio_try_get(), without any lock. If the folio is removed
from the swap cache and freed in between, and its memory gets reused,
folio_try_get() still succeeds and we return a page that has nothing to do
with the swap entry:
CPU 0 CPU 1
do_swap_page()
swap_cache_get_folio()
swp_tb = swap_table_get()
folio = swp_tb_to_folio(swp_tb)
madvise(MADV_DONTNEED)
zap_nonpresent_ptes()
swap_put_entries_direct()
swap_put_entries_cluster()
__try_to_reclaim_swap()
swap_cache_del_folio()
folio_put() <- freed
page reused, e.g. as an anon
folio of another task
folio_try_get(folio) <- succeeds
return folio
swap_update_readahead() <- may clear PG_readahead on it
folio_lock_or_retry() <- locks it
folio_matches_swap_entry() fails, unlock and put
filemap_get_entry() does the same lockless lookup for the page cache, and
after folio_try_get() it rechecks the xarray, because the folio may have
been freed and reused in between. swap_cache_get_folio() lost that
recheck when the swap cache moved to the swap table.
So CPU 0 writes to a folio it doesn't own: it may clear PG_readahead and
takes the folio lock of someone else's page, before
folio_matches_swap_entry() finds out the folio is not ours.
When the race occurs, a swap fault can acquire a reference to a
physical page that has been freed and reused for an unrelated purpose,
potentially by another process. The fault handler may then clear that
page's PG_readahead/PG_reclaim flag, wait on or acquire its folio lock,
and potentially attempt to remove it from an unrelated swap cache.
These operations can interfere with unrelated workloads and may cause
unexpected stalls and impaired reclaim. The subsequent swap-entry
validation normally prevents the wrong folio from being mapped into the
faulting process.
It's documented that caller always double checks the folio so this part
is harmless from the aspect of data corruption or crash.
Recheck the slot after folio_try_get() and retry if it no longer holds
this folio.
[akpm@linux-foundation.org: add LLM-generated description of userspace impact]
Link: https://lore.kernel.org/20261015060528.302634-1-jiayuan.chen@linux.dev
Fixes: 8578e0c00dcf ("mm, swap: use the swap table for the swap cache and switch API")
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Kairui Song <kasong@tencent.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Youngjun Park <youngjun.park@lge.com>
Cc: <stable@vger.kernel.org>
|
|
min_free_kbytes is recalculated on memory hotplug and on THP sysfs writes.
The new value comes from calculate_min_free_kbytes() or, while THP is
enabled, from set_recommended_min_free_kbytes(). A value written to the
vm.min_free_kbytes sysctl is replaced whenever the new value is higher, so
a lower value does not stick. This matters most when THP is enabled on
configurations with large pageblocks, where the new value can be excessive
[1].
For example, with default arm64 pageblock sizes, the THP recommendation
can reach 5.5 GiB per eligible zone with 64 KiB pages vs 22 MiB with 4 KiB
pages. This keeps a significant amount of memory free, reducing what
applications can use and triggering reclaim much earlier. Users can lower
vm.min_free_kbytes to avoid this, but the lower value only sticks until
the next recalculation.
Stop recalculating min_free_kbytes once a value has been written to the
sysctl. Systems where the sysctl is never written keep the current
behavior.
Link: https://lore.kernel.org/20261014140859.1006967-1-noren@nvidia.com
Link: https://lore.kernel.org/r/20260716173504.760369-1-noren@nvidia.com/ [1]
Signed-off-by: Nimrod Oren <noren@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Link: https://lore.kernel.org/r/d202dcc3-d7d7-4e98-a9a5-3137cfbfd041@kernel.org/
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Shuah Khan <skhan@linuxfoundation.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: David Laight <david.laight.linux@gmail.com>
|
|
So far the assumption has been that mm->mm_lock_seq overflow is benign,
resulting only in readers spuriously falling back to the mmap lock, however
it turns out that this was an unsafe assumption.
If a VMA can be spuriously considered write locked that means read lock
exclusion does not take place, and thus a VMA can end up write-locked with
concurrent readers.
Since overflow only occurs every 2^31 write lock/unlock cycles, resolve the
issue by resetting every VMA's vma->vm_lock_seq each time such an overflow
occurs.
This is done in vma_end_write_all(), which is called when the mmap write
lock is released, i.e. in mmap_write_unlock() and mmap_write_downgrade().
Care must be taken, as resetting the sequence count at overflow means
changing the seqcount of VMAs which are already attached and observable.
The mm's seqcount is guaranteed to be odd when a write lock is held, and
thus any write locked VMA will have an odd seqcount assigned.
Therefore resetting to an even value prevents spurious write locks, since
these can only occur with the mmap write lock held, at which point the mmap
seqcount will be odd.
However, vma_start_read() also checks vma->vm_lock_seq == mm->mm_lock_seq,
bailing if so (indicating a write lock).
This would cause any even reset value chosen to prevent VMA read locks from
being taken - avoid this by also checking whether the VMA seqcount is odd
before bailing on the read lock.
So far, vma->vm_lock_seq has defaulted to UINT_MAX, but this has been
unnecessary since commit e5e7fb278e59 ("mm: convert mm_lock_seq to a proper
seqcount"), since this is assigned when the VMA is still detached, and
cannot be observed.
Therefore perform both reset and initialisation of vma->vm_lock_seq by
simply setting vma->vm_lock_seq to zero.
The reset, performed in __mm_reset_vma_seqcounts(), only needs to
WRITE_ONCE() the value.
However the read of vma->vm_lock_seq in vma_start_read() needs to have
acquire semantics to pair with the smp_wmb() in do_raw_write_seqcount_end()
so a reader reading a reset value also observes VMA modifications made
under the write lock.
Since all VMA write locks were just released and the mmap write lock is
still held, there are no concurrent VMA writers.
Concurrent VMA readers are unaffected. New VMA readers observe either zero,
never a write lock, or the prior seqcount, behaving as before the reset.
Since this only occurs every 2^31 write lock/unlock cycles, the time taken
to update all VMAs is considered to be acceptable.
Also update comments, documentation and VMA userland tests to reflect the
change.
User facing effects are - dangerous races that could cause memory
corruption or NULL pointer derefences (especially on page table traversal)
are avoided.
Link: https://lore.kernel.org/20261016-fix-vma-lock-overflow-v2-1-9eb778b79b58@kernel.org
Fixes: 5e31275cc997 ("mm: add per-VMA lock and helper functions to control it")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Daehyeon Ko <4ncienth@gmail.com>
Closes: https://lore.kernel.org/all/20261015143735.1630957-1-4ncienth@gmail.com/
Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Tested-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: <stable@vger.kernel.org> # 7.3.x
|
|
MEMORY MANAGEMENT - MISC is a list of misc mm sources and also acts as
catch-all for the mm Documentation subdirectories. Documentation for
DAMON is well maintained within its own subsystem and also actively
updated, so there's no need to Cc all MISC maintainers for them as well.
Thus add exclusion entries for them.
Link: https://lore.kernel.org/20261015-maint-mm-core-x-doc-damon-v1-1-226decb5cd0d@kernel.org
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: SJ Park <sj@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Shuah Khan <skhan@linuxfoundation.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
|
|
When the cgroup has no memory pressure at all, writing to
/sys/devices/system/nodeX/reclaim triggers proactive reclaim on
NUMA node, causing increase in the writer cgroup's memory PSI.
Due to this reclaim is performed in the context of the write(),
accounted as memory pressure on the writer, like
commit e22c6ed90aa9 ("mm: memcontrol: don't count limit-setting reclaim
as memory pressure"). This is unexpected, the similar scenarios (e.g.
senpai) will continue to occur, only this time it is for the
/sys/devices/system/nodeX/reclaim knob.
The Documentation/ABI/stable/sysfs-devices-node documentation also
notes that "This interface is equivalent to the memcg variant."
This patch unifies the semantics of the memcg and node interfaces:
per-node proactive reclaim is no longer counted as memory pressure,
and the per-node proactive reclaim interface no longer produces
phantom pressure.
I ran demo[1] that performs per-node proactive reclaim 10000 times
in qemu, writer cgroup memory.pressure as follows:
without patch:
some avg10=31.53 avg60=13.42 avg300=3.31 total=10602985
full avg10=31.53 avg60=13.42 avg300=3.31 total=10602985
with patch:
some avg10=9.59 avg60=3.41 avg300=0.81 total=2686221
full avg10=9.59 avg60=3.41 avg300=0.81 total=2686221
Link: https://lore.kernel.org/20261012022917.1102467-1-vernon2gm@gmail.com
Link: https://github.com/vernon2gh/app_and_module/tree/main/reclaim_node_psi [1]
Fixes: b980077899ea ("mm: introduce per-node proactive reclaim interface")
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Acked-by: Davidlohr Bueso <dave@stgolabs.net>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: <stable@vger.kernel.org>
|
|
Commit a08c7193e4f1 ("mm/filemap: remove hugetlb special casing in
filemap.c") changed mapping->nrpages to count base pages for hugetlb
folios. However, shm_add_rss_swap() still multiplies it by
pages_per_huge_page(), overcounting RSS in both SHM_INFO and
/proc/sysvipc/shm.
Use mapping->nrpages for RSS accounting across all backing types, and
account swap only for non-hugetlb shmem. Use READ_ONCE() for both
counters instead of info->lock, which does not protect nrpages and cannot
provide a consistent snapshot.
Link: https://lore.kernel.org/20261014082218.2236088-1-tongweilin@linux.alibaba.com
Fixes: a08c7193e4f1 ("mm/filemap: remove hugetlb special casing in filemap.c")
Signed-off-by: Weilin Tong <tongweilin@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: David Hildenbrand <david@kernel.org>
Cc: Sidhartha Kumar <sidhartha.kumar@oracle.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Dan Carpenter <error27@gmail.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: <stable@vger.kernel.org>
|
|
MGLRU evict_folios() isolates folios from the LRU without updating the
NR_ISOLATED_ANON/FILE counters, unlike the legacy shrink_inactive_list()
path. This causes compaction's too_many_isolated() check to under-count
isolated pages when MGLRU reclaim is active.
Add NR_ISOLATED counter updates in evict_folios(): increment after
isolate_folios() and decrement after all retry passes complete, using the
existing nr_isolated which holds the original isolated count.
Link: https://lore.kernel.org/141901355a7d9e4bbca5c4840ae32ea5c65fe3b6.1791968227.git.zhuhui@kylinos.cn
Link: https://lore.kernel.org/cover.1791968227.git.zhuhui@kylinos.cn
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
|
|
As-is, madvise_cold_or_pageout_pte_range() discards the hardware young bit
by invalidating the PMD, doing pmd_mkold() and subsequently writing to the
pmd again. However, it does so in a racy manner, looking at a
possibly-stale snapshot. It's entirely possible that a write elsewhere
set the hardware D bit, which this logic now loses.
Fix it by using pmdp_test_and_clear_young(). This is an atomic
modification of the PMD entry, and thus safe against any possible racing
writes. As a (nice) side-effect, this elides one non-required TLB flush
in pmdp_invalidate().
This bug can interact with the swapping system in complicated ways,
leading to data loss. For instance, if a folio is marked lazyfree, but
was later read from, and then MADV_COLD'd while racing a write to that
same THP, dropping the dirty bit may lead to data loss as
reclaim/MADV_PAGEOUT drops the "clean" anon folio:
T0 T1
================================================================
MADV_FREE(thp) |
folio marked lazyfree |
read(thp) |
PMD marked young by hardware |
| MADV_COLD(thp)
write(thp) | if (pmd_young(pmd)) {
PMD marked dirty by hardware | set(pmdp, pmd_mkold(pmd));
MADV_PAGEOUT/reclaim |
dirty bit may or may not be |
set |
reclaim/pageout may drop |
the folio |
Link: https://lore.kernel.org/20261013191719.95502-1-pfalcato@suse.de
Fixes: 9c276cc65a58 ("mm: introduce MADV_COLD")
Signed-off-by: Pedro Falcato <pfalcato@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Samarth Narang <samarth.colleges@gmail.com>
Closes: https://lore.kernel.org/linux-mm/20261011132227.413217-1-samarth.colleges@gmail.com/
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: <stable@vger.kernel.org>
|
|
When memory allocation profiling is enabled, the pages backing percpu
memory chunks are attributed to func:pcpu_alloc_pages. However, the
accounting of individual percpu memory allocations via alloc_percpu()
and friends record only the size of a single object rather than its
full size across all CPUs.
As a result, it is often not straightforward to identify which kernel code
is actually responsible for percpu memory, because the size is
under-accounted by a factor of num_possible_cpus().
Account full object size with pcpu_obj_full_size(size) instead, which
multiplies the object size by num_possible_cpus() and adds the objcg
pointer overhead. This is consistent with what memcg already does for
charging percpu allocations.
Verified with 64-CPU VM by spawning 400 processes in a memory cgroup and
measuring mm_init()'s rss_stat counters.
Delta of kernel/fork.c:1142 func:mm_init in /proc/allocinfo:
bytes calls per allocation
unpatched 6320 395 16
patched 418176 396 1056
'percpu' field in memory.stat: 422400 (1056 bytes per process)
Memory allocation profiling now accounts the same amount of memory as
what memcg charges.
Link: https://lore.kernel.org/20261013-fix-alloc-percpu-accounting-v1-1-03e788a1e41f@kernel.org
Fixes: 60fa4a9e2323 ("mm: percpu: add codetag reference into pcpuobj_ext")
Signed-off-by: Harry Yoo (Meta) <harry@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Assisted-by: LLM # Verification
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Hao Ge <hao.ge@linux.dev>
Cc: <stable@vger.kernel.org>
|
|
To move one page out of a large folio, move_pages_ptes() first has to
split the folio. __folio_split() refuses if the folio has any reference
beyond its page table mappings and the caller's own, and it reports the
refusal as -EAGAIN. move_pages() treats returned -EAGAIN as a transient
race and keeps retrying the same address, holding the VMA read locks the
whole time. This is incorrect since in some cases userspace can hold such
a reference indefinitely, such as by registering the page as an io_uring
buffer.
An unprivileged user can trigger this with three threads: first holds such
a reference on a page of a THP, second calls UFFDIO_MOVE on another page
of the same THP and spins in the kernel forever holding the VMA read
locks, and third calls fork() or mprotect(), which takes mmap_lock for
writing and then waits forever for those read locks. From then on every
mmap_lock user of that mm hangs, including other processes inspecting it
through /proc/<pid>/, until the process is SIGKILLed. The hung task
detector reports the blocked tasks (third one), and with
kernel.hung_task_panic=1 the machine panics:
INFO: task wedge:95 blocked for more than 122 seconds.
task:wedge state:D stack:12528 pid:95 tgid:95 ppid:94
Call Trace:
__schedule+0x596/0x1110
schedule+0x36/0xf0
__vma_start_exclude_readers+0x100/0x170
__vma_start_write+0x51/0xd0
__split_vma+0x208/0x3d0
vma_modify+0x6a8/0x840
vma_modify_flags+0xce/0x110
mprotect_fixup+0xbe/0x480
do_mprotect_pkey+0x2d9/0x580
__x64_sys_mprotect+0x1a/0x30
Fix by returning -EBUSY instead of -EAGAIN in move_pages_ptes(), the
caller of split_folio(), when the split fails. Thus, move_pages() does
not retry -EBUSY and the ioctl drops its locks and returns to userspace
instead of spinning.
Link: https://lore.kernel.org/20261012201514.97015-1-cenzhang@linux.microsoft.com
Fixes: adef440691ba ("userfaultfd: UFFDIO_MOVE uABI")
Signed-off-by: Cen Zhang (Microsoft) <cenzhang@linux.microsoft.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: <AutonomousCodeSecurity@microsoft.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Assisted-by: LLM
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Peter Xu <peterx@redhat.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: <tgopinath@linux.microsoft.com>
Cc: <stable@vger.kernel.org>
|
|
mlock_fixup() adjusts mm->locked_vm, an unsigned long, by the number of
pages whose VM_LOCKED state it changes, but computes that number in an
int:
int nr_pages;
...
nr_pages = (end - start) >> PAGE_SHIFT;
...
mm->locked_vm += nr_pages;
When a single VMA range of 2^31 pages (8 TiB with 4 KiB pages) or more
is locked or unlocked, the count is truncated to 32 bits and then
sign-extended, so locked_vm is changed by the wrong amount, while
munmap(), mremap() and the other paths that maintain locked_vm use the
full unsigned long page count.
Such a range is reachable on 64-bit architectures with a 47-bit or
larger user address space (x86_64, arm64 with 48-bit VA, riscv64 Sv48)
by a task with CAP_IPC_LOCK or a raised RLIMIT_MEMLOCK, which do_mlock()
checks against the full untruncated length. It costs nothing with
MAP_NORESERVE and MLOCK_ONFAULT/MCL_ONFAULT, since then no page is
populated. On x86_64, after
p = mmap(NULL, 8UL << 40, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
mlock2(p, 8UL << 40, MLOCK_ONFAULT);
VmLck in /proc/self/status reads 18446744065119617024 kB instead of
8589934592 kB; munmap() of the region then subtracts the true size and
leaves VmLck at 18446744056529682432 kB instead of 0. With 16 TiB,
locked_vm is not changed at all, VmLck stays 0 kB, and after the task
drops CAP_IPC_LOCK or lowers RLIMIT_MEMLOCK, mlock(), mmap(MAP_LOCKED)
and the other mlock_future_ok() and account_locked_vm() users enforce
RLIMIT_MEMLOCK against that count, so they let the task lock memory
beyond its limit (or, in the 8 TiB case, refuse every further lock).
The error persists for the life of the mm; unlocking the range in pieces
skews it further. The wrong count is per-mm accounting only: it does
not change which pages are locked.
The int has been there since before git history (the variable was named
"pages" until it was renamed in 2.6.28); every other updater of
locked_vm uses an unsigned long or long count. Make nr_pages a long,
which holds the page count of any user range and its negation, so the
existing sign handling is unchanged.
Found by formal verification of mm/mlock.c against the kernel binary.
Link: https://lore.kernel.org/20261014131358.780240-1-nickolai@csail.mit.edu
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Assisted-by: LLM
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: <stable@vger.kernel.org>
|
|
The mm Makefile includes local_config.mk, which is generated by a rule
that announces itself with "CHK [config] local_config.mk" on standard
output, which make -s does not suppress. When make has to regenerate the
file while it reads the Makefile for emit_tests, for example on a fresh
tree or when the selftests are installed with make -B, the message
becomes part of the emit_tests output that the selftests install target
appends to kselftest-list.txt, so the runlist gains a line that is not a
test.
Print the message on standard error. With V=1 the message is empty and
only the redirection of an empty command remains.
Link: https://lore.kernel.org/20261014200137.1364121-1-sashal@kernel.org
Fixes: 04cf82a741e0 ("selftests/mm: suppress compiler error in liburing check")
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Li Wang <li.wang@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
|
|
remap_region() checks that the destination region is free before mapping
it, but not the preamble region placed right before it, so mapping the
preamble can intermittently fail with EEXIST:
# Failed to map dest preamble region: File exists
not ok 15 5MB mremap - Source 1MB-aligned, Dest 1MB-aligned with 40MB Preamble
This happens when the kernel's heap ASLR (arch_randomize_brk()) lands the
process heap inside that unchecked range. Confirmed by correlating
brk(NULL) at startup with pass/fail across 60 runs: every failure had brk
land inside that range.
Fix it by including the preamble in the free-region check.
Link: https://lore.kernel.org/20261014-selftests-mm-mremap_preamble-v1-1-2e97f501cb56@marliere.net
Signed-off-by: Ricardo B. Marlière (SUSE) <ricardo@marliere.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Sarthak Sharma <sarthak.sharma@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Shuah Khan <shuah@kernel.org>
|
|
A collapse is two jobs. One reads a PTE table under mmap_lock and decides
whether the range is worth collapsing. The other allocates, isolates,
copies and flushes, and wants the lock given up first.
collapse_single_pmd() did both, so the boundary between them was somewhere
in the middle of a function.
Give each half its own function:
- collapse_scan_pmd() scans one table. It picks the side:
collapse_scan_anon(), the anonymous scan that used to carry the
collapse_scan_pmd() name, or collapse_scan_file().
- collapse_pmd() does the collapse the scan asked for, and is handed
what the scan returned. SCAN_SUCCEED means there is something to
collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page cache already
holds the PMD folio and only the PTE table is left to retract.
Anything else is why there is nothing to do. It calls
collapse_anon(), formerly mthp_collapse(), or collapse_file().
For now collapse_single_pmd() stays, as the two of them with the
mmap_lock drop in between, so its callers see what they saw before.
collapse_control_init() sets a control up before its first scan.
What the scan found and the collapse needs travels in collapse_control.
For an anonymous table that is the orders and the referenced and
swapped-out counts. collapse_anon() and collapse_huge_page() now read
them from there instead of taking them as arguments.
For a file it is the file itself and the offset in it. A file collapse
works on the page cache and never sees a VMA, so the scan takes the
reference while it still has one and the collapse gives it back.
The file scan moves under mmap_lock with the anonymous one, where before
the lock was given up first. The lock is now held over the page cache
walk, an RCU walk over one table's worth of slots with no PTL, and taken
fewer times. collapse_scan_mm_slot() ends its walk whenever the lock was
dropped, so a refused file table used to cost khugepaged an unlock, a
trip back through khugepaged_do_scan(), a relock and a VMA lookup. Now
only a table that goes on to be collapsed does.
Link: https://lore.kernel.org/20261014171145.3292736-10-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
collapse_scan_file() reads the page cache to decide whether a table is
worth collapsing and, when it is, calls collapse_file() itself. The
caller cannot get between the decision and the collapse.
Move the collapse_file() call up into collapse_single_pmd(), so the scan
stops at the decision.
Two things change with it. The writeback retry re-runs collapse_file()
alone instead of rescanning first; collapse_file() repeats the scan's
checks under the page cache lock anyway. And mm_khugepaged_scan_file
fires before the collapse, so for an accepted table its status reads
SCAN_SUCCEED; what the collapse made of the table is for
mm_khugepaged_collapse_file to report.
Preparation for splitting a collapse into a scan under mmap_lock and a
run without it. The file scan has to stop where the anonymous one will.
Link: https://lore.kernel.org/20261014171145.3292736-9-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
collapse_control_init_scan() resets what one scan accumulates: the node
load, the allocation nodemask and the eligible-PTE bitmap. It runs before
every PTE table a scan is given, not once per control, so its name points
at the wrong thing.
Call it collapse_scan_reset().
Preparation for giving a control a real init, run once for a whole series
of scans. Two names a word apart would then stand for two different jobs.
No functional change.
Link: https://lore.kernel.org/20261014171145.3292736-8-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
collapse_possible() only forwards to collapse_possible_orders() and turns
its mask into a bool. Its three callers can test the mask themselves.
No functional change.
Link: https://lore.kernel.org/20261014171145.3292736-7-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
Tests scattered through the collapse path decide what a collapse is
allowed to do by asking whether khugepaged started it. Between them they
settle:
- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, and
whether a sub-PMD window is held to a stricter rule than a PMD;
- whether a range has to look used, and whether a MADV_FREE'd page is
left alone;
- whether the PMD is mapped as part of the request, and whether dirty
pages are worth writing back and retrying.
None of those is a fact about khugepaged. Each is something the caller
decided before asking, and the collapse code should not have to look up
who called to find out.
Add struct collapse_policy for the caller to fill: khugepaged from its
own settings, MADV_COLLAPSE from the fact that a user asked explicitly.
Every test becomes a read of a field, and cc->is_khugepaged goes, having
no reader left.
The PTE limits come as two sets, one for a PMD-sized window and one for
anything smaller, so that the helpers pick a set for the order and read
it. khugepaged takes no swapped-out or shared PTE into a sub-PMD window,
and empty PTEs only when the knob says all or nothing; the warning for a
knob value in between moves to where khugepaged fills its policy. The
fields only one side reads say which: anon_ or file_. David Hildenbrand
asked for both.
khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per
call. That is the one change in behaviour. The max_ptes_* limits and the
defrag setting behind the allocation mask are sampled once per pass rather
than on every table. A table scanned early in a pass and one scanned late
are then treated alike.
collapse_file() also drops a NULL check on the collapse_control. It has
one call site, reached only from collapse_single_pmd(), which dereferences
cc unconditionally, so the check was already dead.
Link: https://lore.kernel.org/20261014171145.3292736-6-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
khugepaged.c holds both the users of collapse and the machinery that
performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's
entry point and the collapse itself all sit in one file and reach into
each other freely. Nothing marks where a user ends and the engine
begins.
Start drawing that line.
Add mm/collapse.h for what the two sides have to agree on:
- enum scan_result - what the engine hands back;
- struct collapse_control - the state a request carries.
And two constants move with them:
- KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT;
- KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER.
Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix.
No functional change.
Link: https://lore.kernel.org/20261014171145.3292736-5-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
The name says less than the bit means. A set bit means not only that the
PTE is present, but also that it passed the other checks: uffd,
lazyfree, anonymity, sharing. The PTE can be considered a collapse
source.
mthp_collapse() then reads the bitmap starting at the PMD order, so the
bitmap is not specific to mTHP either.
Name it for what a set bit means, and update the comments that named it.
No functional change.
Link: https://lore.kernel.org/20261014171145.3292736-4-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and
tests cc->is_khugepaged to know whether it should: the counter belongs to
the daemon, and MADV_COLLAPSE must not touch it.
The daemon sees every result of every collapse it asks for, so it can keep
its own counter without the shared path testing who called.
Link: https://lore.kernel.org/20261014171145.3292736-3-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
Patch series "mm/collapse: separate a collapse from its callers", v5.
There is no line between the collapse engine and the callers that ask for
a collapse. khugepaged.c holds both, and they reach into each other.
- Sixteen tests through the collapse path read cc->is_khugepaged to work
out what they are allowed to do, when every one of those decisions was
made by the caller before it asked.
- collapse_single_pmd() does both halves of a collapse behind one call.
Where it gives up mmap_lock between them is buried in the anonymous
scan on one side and ahead of the file scan on the other.
Start drawing the line. The caller states what it allows in a policy,
and the collapse code reads the policy instead of asking who called. The
scan of a PTE table and the collapse that follows get a function each,
with the mmap_lock drop between them in one place.
This patch (of 9):
madvise_collapse() holds an mmgrab() reference across its work. It is
redundant. Every caller already holds mm_users:
- madvise(2) works on current->mm, which lives as long as the task is in
the syscall;
- process_madvise(2) reaches a remote mm through mm_access(), which takes
an mm_users reference and holds it until the syscall returns;
- io_uring passes current->mm;
- DAMON takes one with get_task_mm() and drops it after the call.
Drop the mmgrab()/mmdrop() pair.
Link: https://lore.kernel.org/20261014171145.3292736-1-kas@kernel.org
Link: https://lore.kernel.org/20261014171145.3292736-2-kas@kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Assisted-by: LLM
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
|
|
find_vma_and_prepare_anon() can end up allocating an anon_vma in VMAs that
are not CoW mappings, which is wrong.
If we have a MAP_SHARED R/O file mapped, we lack VM_SHARED (but do have
VM_MAYSHARE) and lack VM_MAYWRITE.
While uffd later rejects such mappings (uffd registration requires
VM_MAYWRITE, see comment in validate_dst_vma) find_vma_and_prepare_anon()
still wrongly allocates an anon vma before it validates the vma.
This in turn later confuses copy_page_range()->vma_needs_copy() such
that we end up calling copy_hugetlb_page_range() with ranges that
actually do not contain *any* per-pagetable information that would
require copying, which is the basis for PMD-table sharing in the first
place.
copy_hugetlb_page_range() should not have to worry about shared
PMD tables at all: it doesn't make any sense to copy page tables that are
shared, as they can be fully recreated later. Ending up in the function
with shareable page tables is the real issue.
If a destination PMD table is shared in copy_hugetlb_page_range(), it
might get concurrently unshared as we are not holding relevant
locks, resulting in us using a page table that is no longer mapped and
possibly freed after calling:
dst_pte = huge_pte_alloc(dst, dst_vma, addr, sz);
To fix it, do not allocate an anon_vma in non-CoW mappings and add a
safety guard to copy_hugetlb_page_range(), making sure that we never
end up messing with shared PMD tables in there.
Before commit 867a43a34ff8a ("userfaultfd: use per-vma locks in userfaultfd
operations") we would always validate the VMA before allocating an
anon_vma, and consequently never wrongly allocate an anon_vma.
As uffd_lock_vma() is part of the same lookup machinery and
also contains a misleading VM_SHARED check, change that one as well
directly for consistency.
A follow-up cleanup will stop taking the hugetlb_vma_lock_read(src_vma)
lock in copy_hugetlb_page_range() entirely. Further, the usage of
VM_SHARED in userfaultfd code should be cleaned up later to focus on
CoW vs. non-CoW mappings.
Link: https://lore.kernel.org/20261014-copy_hugetlb_page_range-v1-1-5b68e03fe7fe@kernel.org
Fixes: 867a43a34ff8 ("userfaultfd: use per-vma locks in userfaultfd operations")
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Kyle Zeng <kylebot@openai.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Kyle Zeng <kylebot@openai.com>
Tested-by: Kyle Zeng <kylebot@openai.com>
Acked-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Tested-by: Lance Yang <lance.yang@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Lokesh Gidra <lokeshgidra@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: <stable@vger.kernel.org>
|
|
Replace the hardcoded 32 with sizeof(buf) at the remaining four call
sites, for consistency with the rest of the file.
Link: https://lore.kernel.org/20261014031147.444733-1-lienze@kylinos.cn
Signed-off-by: Enze Li <lienze@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: David Hildenbrand <david@kernel.org>
|
|
Since commit 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm
in defrag_mode"), direct reclaim and compaction for non-movable requests
under defrag_mode run at pageblock_order, to produce the whole blocks that
ALLOC_NOFRAGMENT needs. The retry decisions that follow still use the
request order. An order-0 request can therefore retry indefinitely
without ever reaching the ALLOC_NOFRAGMENT fallback:
- Reclaim at pageblock_order gives up after one pass as soon as a zone
looks compaction_ready(), and do_try_to_free_pages() then returns 1
even though nothing was reclaimed. It returns before the retry that
would reclaim memory.low-protected cgroups, so when most memory is
protected, the pass that did run finds next to nothing.
- Compaction at pageblock_order fails or is deferred.
- should_reclaim_retry() takes the reported progress as progress for
the order-0 request and resets no_progress_loops. The request
retries.
Order 1-3 requests loop the same way, and should_compact_retry() also
checks their pageblock_order compaction result against the request order.
On a production host (64G, defrag_mode, memory.low covering most of the
workload), 95% of direct reclaim runs were order-9 runs that returned 1
with nothing reclaimed, at up to 60k runs per second. Across ~200M
should_reclaim_retry() calls in a day, no_progress_loops never left 0.
The spinning allocations were SLUB slab refills for inode and dentry
caches. The time spent registers as memory pressure, and pressure-based
OOM killing takes down both workloads and system services.
For promoted requests:
- Reclaim progress does not reset no_progress_loops, as for costly
orders.
- should_compact_retry() checks the compaction result at the promoted
order and does not retry COMPACT_SKIPPED, since the request can fall
back. The compaction priority floor and the COMPACT_SUCCESS retry
limit stay those of the request order, so a non-costly request still
gets its COMPACT_PRIO_SYNC_FULL pass before it falls back.
When a promoted request takes the fallback, reset the retry counters, so
that the fallback attempt gets a full retry budget before the OOM killer
is considered. A request whose reclaim was not promoted spent its budget
on the request itself and keeps its counters, as Usama Arif pointed out.
__alloc_pages_slowpath() computes the promoted order once per iteration
and keeps it in alloc_context for direct reclaim, direct compaction and
the two retry helpers, so no callee has to recompute it. Harry Yoo asked
for the promoted order to be explicit rather than derived in each callee,
and Vlastimil Babka suggested alloc_context over new function parameters.
In a VM reproducer (32G, defrag_mode, inode churn under memory.low):
before after
should_reclaim_retry() calls 63M 293k
peak memory pressure (PSI some avg10) 99% 12%
File creation runs 5.7x faster.
[kas@kernel.org: s/order_nofrag_promoted/order_defrag_promoted/, per hannes]
Link: https://lore.kernel.org/atDL-nSFP2KVU7cI@thinkstation
Link: https://lore.kernel.org/as5q14OsbK2CmOrh@thinkstation
Fixes: 7e8756d7ad22 ("mm: page_alloc: fix non-movable reclaim storm in defrag_mode")
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Assisted-by: LLM
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: <stable@vger.kernel.org>
|
|
uffd_move_pmd_test()/uffd_move_test_common() splits the test area into
chunks of chunk_size and moves them one by one. On arm64 with 64K base
pages the chunk_size (read_pmd_pagesize()) is 512MB, which is larger than
the whole 32MB (UFFD_TEST_MEM_SIZE) test area:
step_count = nr_pages / (chunk_size / page_size) == 0
Because chunk_size > page_size, the test then aligns both areas up to
512MB, which points far outside of the 32MB mappings, and decrements
step_count, which underflows to ULONG_MAX. The verification loop then
runs away and the first dereference of area_src faults with a SIGSEGV.
Skip the test when a single chunk cannot fit into the test area.
Link: https://lore.kernel.org/20261011163527.95988-1-zenghui.yu@linux.dev
Fixes: a2bf6a9ca805 ("selftests/mm: add UFFDIO_MOVE ioctl test")
Signed-off-by: Zenghui Yu (Huawei) <zenghui.yu@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Tested-by: Sarthak Sharma <sarthak.sharma@arm.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Peter Xu <peterx@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
hugetlb_sysfs_init() guards the "hugetlb_sysfs_initialized = true"
assignment with an #ifdef CONFIG_NUMA block, while the following
hugetlb_register_all_nodes() call is left unguarded and relies on the
empty !CONFIG_NUMA stub.
Move the assignment into the NUMA version of hugetlb_register_all_nodes(),
before it registers the online nodes, so the #ifdef in
hugetlb_sysfs_init() can go away. Keep the !CONFIG_NUMA stub, so the call
site is unconditional in both configurations.
No functional change; this is an #ifdef cleanup.
Link: https://lore.kernel.org/20261009142010.552614-1-lienze@kylinos.cn
Signed-off-by: Enze Li <lienze@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Muchun Song <muchun.song@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Enze Li <enze.li@gmx.com>
|
|
MREMAP_DONTUNMAP keeps the source VMA in place, but clears its mlock flags
while setting them on the destination VMA. The "mm/mremap: fix two issues
with MREMAP_DONTUNMAP" series fixed two cases where this leaked
mm->locked_vm:
- an unfaulted mlock-on-fault VMA moved behind itself self-merges, so the
single resulting VMA loses the flags without the accounting being
dropped;
- a partial mremap() moves only part of the range, leaving the pages which
are not moved accounted as locked in a VMA whose flags were cleared.
Add a test which mlock2()s a source mapping, moves all or part of it with
MREMAP_DONTUNMAP, unmaps everything and checks that VmLck is back to its
value from before, and run it for the self-merge case and for a partial
move with and without MLOCK_ONFAULT.
Verified on x86_64: the three cases fail on mm-unstable with the two fixes
reverted and pass with them applied.
Link: https://lore.kernel.org/20261006064120.6796-1-azpijr@gmail.com
Signed-off-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
|
__discard_anon_folio_pmd_locked() clears a lazyfree THP's PMD before it
determines whether the folio can be discarded, and restores it if the
folio is dirty or has extra references. It is done under only the PMD
lock and the anon_vma read lock inside try_to_unmap_one() and no mmap lock
is held. The temporarily none PMD causes UAF.
A parallel munmap() or mremap() is supposed to serialize with it on the
PMD lock, but zap_pmd_range() (called by munmap()) and move_page_tables()
(called by mremap()) do not take the lock when they see the temporarily
none PMD and skip the PMD. There is no more page table walk after the
none PMD is restored:
- munmap(): free_pgtables() waits in unlink_anon_vmas() for the anon_vma
lock, then free_pmd_range() drops the restored PMD as bad without
removing the rmap, so the folio keeps a stale mapcount.
- whole-VMA MREMAP_DONTUNMAP: dontunmap_complete() unlinks the source VMA
from its anon_vma while the restored PMD still maps the folio.
- plain mremap(): the THP is not moved but unmapped, so the THP's data is
lost.
In the first two cases, the folio stays mapped, but no VMA that maps it is
linked to its anon_vma. When the anon_vma is freed, it is still reachable
when one rmap walks the folio. Racing lazyfree reclaim against
MREMAP_DONTUNMAP as an unprivileged user reproduces a KASAN use-after-free
in folio_lock_anon_vma_read() as reported in the Closes link.
One solution is to use an invalidated PMD instead of a cleared one until
the discard is committed, but it needs arch fixes first.
1. GUP-fast still follows an invalidated PMD on riscv, LoongArch and s390.
2. sparc64 miscounts thp_pte_count when an invalidated PMD is set again or
cleared.
To keep the fix simple for backport, stop discarding lazyfree THPs at PMD
level instead. try_to_unmap_one() splits the PMD and discards the folio at
PTE level.
unmap_huge_pmd_locked() and __discard_anon_folio_pmd_locked() are left in
place, unused, to minimize this fix for backport and the invalidated PMD
approach is planned as a follow-up series to use them again.
Link: https://lore.kernel.org/20261012-lazyfree-stable-fix-v2-1-6387d6e30063@nvidia.com
Fixes: 735ecdfaf4e8 ("mm/vmscan: avoid split lazyfree THP during shrink_folio_list()")
Signed-off-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Kyle Zeng <kylebot@openai.com>
Closes: https://lore.kernel.org/r/20261009165214.40212-2-kylebot@openai.com
Reviewed-by: Kyle Zeng <kylebot@openai.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Tested-by: JongHyuk Kim(MalHyuk) <malhyuk97@gmail.com>
Link: https://lore.kernel.org/CADQRuqvXQ=YE=7T3oR66vTxE33pi=ygEeZKPTOU=cALZL0Figw@mail.gmail.com
Assisted-by: LLM
Cc: Rik van Riel <riel@surriel.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: <stable@vger.kernel.org>
|
|
madvise_cold_or_pageout_pte_range()
During stress testing with stress-ng on a 256-CPU system, a softlockup was
observed when madvise(MADV_COLD) was called on a 31GB memory region backed
by transparent hugepages. The region contains 15935 THP PMD entries, and
the softlockup was triggered after 14485 of them had been processed.
The THP branch of madvise_cold_or_pageout_pte_range() has no scheduling
point. walk_pmd_range() invokes it once per 2MB PMD entry with no
cond_resched() anywhere in the loop. Each iteration calls
pmdp_invalidate() which issues a synchronous TLB shootdown (dsb ish)
waiting for all CPUs to acknowledge. This operation can be time-consuming
under heavy system load, causing the accumulated time across all PMD
entries to exceed the softlockup threshold (20s) and panic when
softlockup_panic=1. On a CONFIG_PREEMPT_NONE=y kernel, there is no other
opportunity to reschedule during this entire traversal.
The regular PTE path already has an unconditional cond_resched() at the
function tail (introduced by commit 9c276cc65a58 ("mm: introduce
MADV_COLD") and commit 1a4e58cce84e ("mm: introduce MADV_PAGEOUT") when
MADV_COLD and MADV_PAGEOUT were first introduced) and a conditional one
inside the loop (commit b2f557a21bc8 ("mm/madvise: add cond_resched() in
madvise_cold_or_pageout_pte_range()")). The THP branch never received
equivalent treatment.
Fix by adding cond_resched() at the huge_unlock exit path after
spin_unlock(ptl) and reclaim_pages(), which is the single exit point for
all goto paths in the THP branch.
Crash log:
watchdog: BUG: soft lockup - CPU#63 stuck for 23s! [stress-ng-vm:1665914]
...
Call trace:
pmdp_invalidate+0x48/0x68
madvise_cold_or_pageout_pte_range+0x84c/0xb00
walk_pmd_range.isra.0+0x1ac/0x3b0
walk_pud_range+0x120/0x190
walk_pgd_range+0x170/0x1b8
__walk_page_range+0x184/0x198
walk_page_range_vma+0x94/0x128
madvise_cold+0xd8/0x198
madvise_vma_behavior+0x234/0x638
madvise_walk_vmas.constprop.0+0xdc/0x210
madvise_do_behavior+0xa8/0x328
do_madvise+0xd8/0x140
__arm64_sys_madvise+0x2c/0xe0
Link: https://lore.kernel.org/20261010024042.4070048-1-albin_yang@163.com
Signed-off-by: Wei Yang <albinwyang@tencent.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: <stable@vger.kernel.org>
|
|
Pretty much every pr_* message in this file starts with "HugeTLB: ", typed
out by hand, and one of them even drifted to lowercase ("hugetlb:").
hugetlb_vmemmap.c already handles this with pr_fmt, so let's do the same
here instead of repeating the prefix over and over.
The output stays the same for almost everything; the only tweaks are that
the lowercase message now matches the others, and the ratelimited "killed
due to inadequate hugepage pool" warning picks up the prefix like
everything else.
Link: https://lore.kernel.org/20261010092333.137126-1-lienze@kylinos.cn
Signed-off-by: Enze Li <lienze@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: David Hildenbrand <david@kernel.org>
Cc: Enze Li <enze.li@gmx.com>
|
|
snapshot_set_swap_area() unpins the previously selected swap device and
pins the new one in two separate swap_lock critical sections. In the gap
between them, swapoff() observes SWP_HIBERNATION cleared, bypasses the
guard, and tears down the device, reopening the race the SWP_HIBERNATION
pin was meant to close. The window is reachable on any
SNAPSHOT_SET_SWAP_AREA call after the snapshot device is opened for
hibernation, and on any retry after the resume path's first selection.
Add repin_hibernation_swap_type(), which looks up the new device, clears
the old SWP_HIBERNATION flag and sets the new one under a single swap_lock
acquisition. The same-device case is short- circuited so userspace can
re-select the same swap area without tripping WARN_ON_ONCE and -EBUSY.
Switch snapshot_set_swap_area() to the new helper.
A failed lookup now preserves the previous pin instead of dropping it, so
a bad SNAPSHOT_SET_SWAP_AREA leaves the prior selection intact. The open
and release paths keep using pin_hibernation_swap_type() and
unpin_hibernation_swap_type().
The race was identified during AI-assisted review of the SWP_HIBERNATION
pinning series.
[akpm@linux-foundation.org: coding-style cleanups]
Link: https://lore.kernel.org/20260430195651.287659-1-devnexen@gmail.com
Fixes: 8e6e0d845823 ("mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device")
Signed-off-by: David Carlier <devnexen@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Assisted-by: Codex (gpt-5-codex)
Cc: Baoquan He <bhe@redhat.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Len Brown <lenb@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: DaeMyung Kang <charsyam@gmail.com>
Cc: Youngjun Park <youngjun.park@lge.com>
|
|
If hugetlb_add_to_page_cache() in memfd_alloc_folio() fails with -EEXIST,
a concurrent fault has already instantiated the folio in the page cache,
and the reservation now belongs to that folio. Calling
hugetlb_unreserve_pages() in that case incorrectly removes the region
backing the cached folio and releases one reservation more than it should,
leaving that folio in the page cache with no region recording it.
That leaves resv_huge_pages one page short until that folio is removed
from the page cache, so available_huge_pages() reports a page that is not
actually free and the pool can grant one reservation more than it can
back. Applications using hugetlb memfds can then fail to allocate or
fault in a page they already reserved. They see -ENOMEM or -ENOSPC from
the allocation or fault path even though the reservation and
HugePages_Free still look healthy.
So hold the hugetlb fault mutex from hugetlb_reserve_pages() until the
error-path unreserve completes to make the reserve, allocate and
instantiate steps atomic against concurrent faults. With the mutex held
from the start, a concurrent fault can no longer consume the reservation
between reserve and allocate/instantiate. If a fault completed before the
mutex was taken, it has already added the region for that index, so
hugetlb_reserve_pages() returns 0 and the error path leaves the region in
place.
Link: https://lore.kernel.org/20260927094757.31665-1-hongfu.li@linux.dev
Link: https://lore.kernel.org/20260903030134.7407-1-hongfu.li@linux.dev
Fixes: 717cf9357325 ("mm/memfd: reserve hugetlb folios before allocation")
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Vivek Kasireddy <vivek.kasireddy@intel.com>
Cc: <stable@vger.kernel.org>
|
|
Commit b3723b596b54 ("mm/damon/core: fix unconditionally skip last
region") fixed a minor bug.
Add a KUnit test case to cover this behavior and prevent regressions in
the future.
Link: https://lore.kernel.org/20261006065209.41507-4-sj@kernel.org
Signed-off-by: Liew Rui Yan <aethernet65535@gmail.com>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: David Gow <davidgow@davidgow.net>
|
|
Hugetlb folio acquisition is open-coded in three places:
- damon_hugetlb_mkold()
- damon_young_hugetlb_entry()
- damon_va_probe_hugetlb_entry()
DAMON already has damon_get_monitor_folio(), which paddr uses to get
monitored LRU and hugetlb folios and to filter them through
damon_folio_acceptable(). These paths bypass that check.
Use that helper in these paths too, so that DAMON acquires hugetlb folios
and applies its acceptance policy in one place.
No functional change intended. All callers reach these paths with a
present hugetlb PTE under the page table lock, so the folio is always a
mapped hugetlb folio and the helper always returns it.
Link: https://lore.kernel.org/20261006065209.41507-3-sj@kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
|
|
Patch series "mm/damon: fix a minor bug, clean code, and add a test".
Patch 1 from Jaeyeon Lee fixes a minor overflow issue in DAMOS. Patch 2
from Gregory Price (Meta) cleans up the code for handling of hugetlb
pages. Patch 3 from Liew Rui Yan adds a kunit test for a recently fixed
bug.
This patch (of 3):
damon_hot_score() calculates the age of a region by multiplying r->age and
the aggregation interval. On 32-bit architectures, unsigned long is 32
bits, so the multiplication overflows once the age exceeds about 4295
seconds. Then old regions are scored as if they were young.
Fix it by doing the calculation in 64-bit arithmetic using div_u64().
Also make age_in_sec u64, so the result is not truncated back to 32 bits.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20261006065209.41507-1-sj@kernel.org
Link: https://lore.kernel.org/20261006065209.41507-2-sj@kernel.org
Link: https://lore.kernel.org/20260623011652.1354-1-sj@kernel.org [1]
Fixes: 198f0f4c58b9 ("mm/damon/vaddr,paddr: support pageout prioritization")
Signed-off-by: Jaeyeon Lee <jaeyeon.lee.dev@gmail.com>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Assisted-by: LLM
|
|
Introduce a kselftest utility to validate memory mapping capabilities
under nommu kernels, aligned with
Documentation/admin-guide/mm/nommu-mmap.rst.
The test implements basic checks into a generic architecture-agnostic
test matrix applicable across nommu targets. It evaluates:
1. MAP_FIXED allocation rejections.
2. Standard MAP_PRIVATE and MAP_ANONYMOUS allocation resilience.
3. Regular file mappings via standard filesystem storage.
4. Block device subsystem mappings (gracefully skipping if node is
missing).
5. Shared vs Private backing discrepancies under nommu conditions.
6. mremap limits, ensuring non-expandable restrictions behave properly.
Link: https://lore.kernel.org/20261006085642.3852152-3-thehajime@gmail.com
Signed-off-by: Hajime Tazaki <thehajime@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260825015945.141739-1-thehajime@gmail.com
Assisted-by: Gemini:Pro [AI_Reviewer] [Sashiko_Linter]
Assisted-by: cubic.dev:unspecified
Cc: David Hildenbrand <david@kernel.org>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Ricardo B . Marlière <rbm@suse.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Hangbin Liu <liuhangbin@gmail.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kees Cook <kees@kernel.org>
Cc: Mark Brown <broonie@kernel.org>
Cc: Will Drewry <wad@chromium.org>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Daniel Palmer <daniel@thingy.jp>
|
|
Patch series "support kselftest on nommu platform", v5.
This patchset adds an ability to execute kselftest on nommu platforms.
Currently there are several issues if we wish to run kselftests on nommu
targets:
- it cannot compile/build test binaries because the current files mainly
assume to build with glibc,
- some of the tests are not able to run on nommu targets as there are no
fork(2) syscall.
The first issue can be avoided if we can build static PIE binaries (if
targets support it), but in our case (build on ubuntu/glibc and run on
alpine/musl-libc), it fails to invoke due to lack of the GNU ifunc
mechanism. Thus, we need to cross-compile with musl toolchain, which
needs to be solved the first issue.
The second issue is the lack of fork(2) syscall on those platforms.
Especially the test harness helper (kselftest_harness.h) uses the
syscall, which cannot be simply with vfork(2). `timeout` command used
in `runner.sh` never works for nommu platform as it uses fork(2).
nommu component in the mm subsystem has several known issues and having
test cases should help this situation, thus this patchset is very first
step toward enriching test environment which has not been well tested
for a while. The test cases is implemented based on the document
(Documentation/admin-guide/mm/nommu-mmap.rst).
Initial shape for this patches is implemented as an opt-in TARGET, not
built/invoked automatically if users don't specify this nommu target.
This patch (of 2):
Architectures lacking MMU don't support fork(2) syscall and only vfork(2)
is available with limitations. Thus, we cannot run kselftest on nommu
architecture as is.
This commit addresses this issue with the following changes:
- Detect NOMMU at runtime from the MmapCopy field in /proc/meminfo
and avoid timeout, which requires fork(2).
- Document how to build and run the NOMMU tests, including the procfs
prerequisite for runtime detection.
Build and run the tests with:
$ make ARCH=um O=build kselftest-all TARGETS=nommu
$ make ARCH=um O=build kselftest-install TARGETS=nommu
$ ./build/kselftest/kselftest_install/run_kselftest.sh -p -c nommu
Link: https://lore.kernel.org/20261006085642.3852152-1-thehajime@gmail.com
Link: https://lore.kernel.org/20261006085642.3852152-2-thehajime@gmail.com
Signed-off-by: Hajime Tazaki <thehajime@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Closes: https://sashiko.dev/#/patchset/20260813063401.1786548-1-thehajime%40gmail.com
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kees Cook <kees@kernel.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Will Drewry <wad@chromium.org>
Cc: Mark Brown <broonie@kernel.org>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Hangbin Liu <liuhangbin@gmail.com>
Cc: "Ricardo B. Marliere" <rbm@suse.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Daniel Palmer <daniel@thingy.jp>
|
|
On systems/VMs with a large number of CPUs (e.g. 4096), hugetlb_init()
computes num_fault_mutexes = roundup_pow_of_two(8 * num_possible_cpus()) =
32768. The subsequent kmalloc for the mutex table can exceed
MAX_PAGE_ORDER when struct mutex is enlarged by debug options like
CONFIG_DEBUG_MUTEXES and CONFIG_DEBUG_LOCK_ALLOC, resulting in:
WARNING: at __alloc_frozen_pages_noprof (order > MAX_PAGE_ORDER)
kernel BUG at mm/hugetlb.c (BUG_ON(!hugetlb_fault_mutex_table))
Kernel panic - not syncing: Fatal exception
Switch to kvmalloc_objs() so the allocation falls back to vmalloc when the
contiguous physical allocation is too large. The table is only accessed
by index, so virtual contiguity is sufficient.
Link: https://lore.kernel.org/20261006160128.31447-1-dheerajkumar.srivastava@amd.com
Fixes: 8382d914ebf7 ("mm, hugetlb: improve page-fault scalability")
Signed-off-by: Dheeraj Kumar Srivastava <dheerajkumar.srivastava@amd.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: David Hildenbrand <david@kernel.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Vasant Hegde <vasant.hegde@amd.com>
Cc: Bharata B Rao <bharata@amd.com>
|
|
Every test is expected to reserve the HugeTLB pages it needs.
hugetlb-read-hwpoison was not converted, so on a system without
pre-reserved huge pages the MAP_POPULATE mmap() fails with ENOMEM and
every test case is skipped.
Reserve the pages with hugetlb_setup_default(). Each chunk size runs two
HWPOISON tests and a poisoned huge page can't be reused, so reserve two
pages per chunk size. The pool size is restored on exit. The poisoned
pages remain allocated until reboot.
With the patch, all 12 cases pass on a system with no huge pages
reserved (tested on Fedora, x86_64, CONFIG_MEMORY_FAILURE=y).
Link: https://lore.kernel.org/20261004205458.119608-1-jaeyeon.lee.dev@gmail.com
Signed-off-by: Jaeyeon Lee <jaeyeon.lee.dev@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Tested-by: Sarthak Sharma <sarthak.sharma@arm.com>
Reviewed-by: Sarthak Sharma <sarthak.sharma@arm.com>
Assisted-by: LLM
Cc: David Hildenbrand <david@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
|
