| Age | Commit message (Collapse) | Author |
|---|
|
Booting with a boot-time function tracer and a filter, for example
ftrace=function ftrace_filter=pud_free_pmd_page
panics on 7.3-rc4 as soon as the tracer starts:
[ 23.531178] Starting tracer 'function'
[ 23.675800] Oops: general protection fault, probably for non-canonical address 0xdffffc0000000038: 0000 [#1] SMP KASAN NOPTI
[ 23.819917] KASAN: null-ptr-deref in range [0x00000000000001c0-0x00000000000001c7]
[ 23.964025] CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 7.3.0-rc4-fe2ec83746e5-with-fixes-v2+ #195 PREEMPT(undef)
[ 24.252248] RIP: 0010:__queue_work+0xab/0xf00
[ 25.981629] Call Trace:
[ 26.125727] <TASK>
[ 26.413912] ? pagetable_free_kernel+0x20/0x120
[ 26.990283] queue_work_on+0x97/0xf0
[ 27.134382] __cpa_collapse_large_pages+0x501/0x6f0
[ 27.566662] cpa_flush+0x394/0x620
[ 27.998953] change_page_attr_set_clr+0x321/0x4a0
[ 29.151729] set_memory_rox+0xa2/0xf0
[ 29.584018] create_trampoline+0x431/0x6f0
...
[ 44.343347] Kernel panic - not syncing: Attempted to kill the idle task!
The boot-time tracer is started from early_trace_init(), which runs
before workqueue_init_early(). Making its trampoline read-only splits a
large page, and CPA collapses it again right away. The split table has
been a kernel page table since commit 9e4a3ec3411b
("x86/mm/pat: Allocate split page tables as kernel page tables"), so the
collapse frees it through pagetable_free_kernel(), which queues work on
system_percpu_wq - still NULL at that point. That commit is correct in
itself; it only lets CPA reach pagetable_free_kernel() before the
workqueue that function relies on exists.
Keep putting the table on the list, but don't schedule the work while
the system is still booting. A core_initcall schedules it once to free
whatever was queued by then.
Link: https://lore.kernel.org/20260925050647.86913-1-mikhail.v.gavrilov@gmail.com
Link: https://lore.kernel.org/20260924064321.23787-1-mikhail.v.gavrilov@gmail.com
Fixes: 9e4a3ec3411b ("x86/mm/pat: Allocate split page tables as kernel page tables")
Signed-off-by: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Suggested-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Tested-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Vishal Moola (Oracle) <vishal.moola@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Lu Baolu <baolu.lu@linux.intel.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: <stable@vger.kernel.org>
|
|
The MREMAP_DONTUNMAP feature is highly unusual in that it permits mremap()
operations that keep the original VMA in place.
Historically this has led to a lot of bugs where non-obvious interactions
occur between existing mremap() operations and the original VMA.
Fix another of these - partial copies.
The long-standing mremap() partial VMA logic has the baked-in assumption
that the originating VMA is unmapped and thus moved.
However MREMAP_DONTUNMAP defeats this by performing a partial copy
instead since it keeps the source VMA around.
An mremap(..., MREMAP_DONTUNMAP) operation disallows resizing of the VMA,
but the operation can be performed partially:
|-----------------|
| |
| v
<------> <------>
.new_sz. new_sz
|--.------.--| |------|
| .source. | | dest |
|--.------.--| |------|
<------------>
old_sz
The page tables in the specified range are moved, but the original VMA is
kept intact.
This interacts poorly with mlock()'d VMAs, as the VMA_LOCKED_BIT flag is
cleared for the entire source VMA and set for the entire destination VMA.
This results in an mm->locked_vm leak as the change is therefore not
accounted correctly.
The clear solution here is to make the portion of the source VMA which is
mremap()'d distinct from the rest of it, a.k.a. split it.
Therefore resolve this issue by splitting it ahead of the rest of the
mremap() operation if the VMA is mlock()'d.
This is valid, as the source VMA will lose its VMA_LOCKED_BIT flag, so if a
partial remap it will become distinct from the rest of the VMA.
In order to make this change re-expose split_vma() in vma.h for
CONFIG_MMU (nommu doesn't compile mremap.c and uses a static helper
instead).
Finally, update the sys_map_count check to account for this case.
Note that the early check does not use needs_pre_split() - this is because
the VMA has not been looked up by this point, so be conservative and assume
that the VMA is mlock()'d in this case for the purposes of the
sys_map_count check.
Link: https://lore.kernel.org/20260930-fix-dontunmap-partial-self-merge-v2-2-f388985a0f0a@kernel.org
Fixes: e346b3813067 ("mm/mremap: add MREMAP_DONTUNMAP to mremap()")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Tested-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Brian Geffon <bgeffon@google.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: <stable@vger.kernel.org>
|
|
Patch series "mm/mremap: fix two issues with MREMAP_DONTUNMAP", v2.
The MREMAP_DONTUNMAP feature is highly unusual in that it permits mremap()
operations that keep the original VMA in place.
Historically this has led to a lot of bugs where non-obvious interactions
occur between existing mremap() operations and the original VMA.
Commit 397432cab17b ("mm/mremap: account mm->locked_vm correctly for
MREMAP_DONTUNMAP") fixed an accidentally introduced bug around
mm->locked_vm accounting, but this wasn't the only issue.
And thus history repeats itself, as it turns out that mm->locked_vm
accounting is broken by MREMAP_DONTUNMAP yet again by two further cases,
and has been broken ever since the feature was introduced.
Both relate to the fact that VMA_LOCKED_BIT is cleared on the source VMA
(it has to be as all page tables are moved):
1. If an unfaulted VMA_LOCKONFAULT_BIT anonymous VMA self-merges it
clears the VMA_LOCKED_BIT flag and permanently leaks mm->locked_vm
pages.
2. If a partial mremap() is performed on a locked VMA there is a leak equal
to the number of pages not copied.
(Both for MREMAP_DONTUNMAP operations only)
Both issues can be fixed by treating the source range as distinct from the
destination range, which is the definition of what MREMAP_DONTUNMAP does
so is appropriate.
In case 1, simply disallow the self-merge, keeping adjacent source and
destination VMAs distinct.
In case 2, split the source range ahead of time if the VMA is mlock()'d,
so accounting is always correct.
Both changes were tested locally and confirmed to fix the issues.
For the purposes of a backport, the fixes are kept distinct, a follow-up
series can add self-tests.
This patch (of 2):
The MREMAP_DONTUNMAP feature is highly unusual in that it permits mremap()
operations that keep the original VMA in place.
Historically this has led to a lot of bugs where non-obvious interactions
occur between existing mremap() operations and the original VMA.
Fix another of these - self-merge.
Self-merge occurs when a VMA is moved in front of or behind itself and the
attributes of the VMA permit such a merge.
Practically this can only happen for unfaulted anonymous VMAs due to the
page offset equality requirement for merge:
|------------|
| |
| v
|...........||-----------||...........|
| || unfaulted || |
|...........||-----------||...........|
^ |
| |
|------------|
This becomes problematic if the VMA is configured by the user to
mlock-on-fault, i.e. the VMA_LOCKED_BIT, VMA_LOCKONFAULT_BIT VMA flags are
set.
MREMAP_DONTUNMAP clears mlock flags for the source VMA and maintains them
for the destination VMA.
Self-merge makes this impossible (there is only one VMA) and incorrectly
clears the destination VMA's mlock flags.
This causes a leak in mm->locked_vm as clearing this flag does not
decrement the counter and the VMA no longer has VMA_LOCKED_BIT set so it
is not decremented on unmap.
Resolve this by simply disallowing a self-merge in this case - the source
and destination VMAs are kept distinct and then are able to have distinct
mlock() flags.
Update dontunmap_complete() to make the now-redundant self-merge check a
VM_WARN_ON_ONCE() instead to guard against future regressions.
Also update the VMA userland tests to reflect the change.
Link: https://lore.kernel.org/20260930-fix-dontunmap-partial-self-merge-v2-0-f388985a0f0a@kernel.org
Link: https://lore.kernel.org/20260930-fix-dontunmap-partial-self-merge-v2-1-f388985a0f0a@kernel.org
Fixes: e346b3813067 ("mm/mremap: add MREMAP_DONTUNMAP to mremap()")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
Tested-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Brian Geffon <bgeffon@google.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: <stable@vger.kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/mm/slab
Pull slab fixes from Vlastimil Babka:
- Stable fix for a potential deadlock in kfree_rcu() when called
from set_cpus_allowed_force() (Harry Yoo)
- MAINTAINERS update of the slab.git URL
* tag 'slab-for-7.3-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/mm/slab:
mm/slab: do not wake up kswapd in __kfree_rcu_sheaf()
MAINTAINERS: update slab.git URL
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull MM fixes from Andrew Morton:
- Fix module loading incorrectly returning success after alloc_tag
codetag setup failed
- Restore MADV_COLLAPSE semantics for shmem so forced collapse ignores
the shmem THP/mTHP sysfs settings
- Make alloc_tag UAPI structure padding explicit
- Fix DAMON schemes unexpectedly stopping after quotas are disabled
through the online parameter update interface
- Fix a false SW_TAGS KASAN invalid-access report when freeing vmapped
task stacks
- Split the MEMORY MANAGEMENT - MEMORY POLICY AND MIGRATION MAINTAINERS
entry into separate MIGRATION and NUMA PLACEMENT entries
- Move memory tiering maintenance under NUMA PLACEMENT
- Add Gregory Price as a NUMA PLACEMENT co-maintainer
- Add Heming Zhao as an ocfs2 reviewer
- Fix mmap_prepare() state being copied onto a merged VMA rather than
only onto a newly allocated VMA
* tag 'mm-hotfixes-stable-2026-09-27-19-12' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm:
mm/vma: predicate setting mmap_prepare VMA fields on new vma alloc
MAINTAINERS: add Heming Zhao as ocfs2 reviewer
MAINTAINERS: make Gregory a co-maintainer of MEMORY MANAGEMENT - NUMA PLACEMENT
MAINTAINERS: move memory tiering under MEMORY MANAGEMENT - NUMA PLACEMENT
MAINTAINERS: split up MEMORY MANAGEMENT - MEMORY POLICY AND MIGRATION
kasan: unpoison task stack below watermark only in generic mode
mm/damon/core: don't skip damos_adjust_quota() while esz is not zero
alloc_tag: avoid implicit padding in uapi
mm: shmem: ignore sysfs configs for shmem forced collapse
module: fix lost error code from codetag_load_module()
|
|
It only makes sense to manipulate VMA fields if a new VMA was allocated,
rather than merged.
VMA merging does not compare vm_ops or vm_private_data, so a merged VMA
keeps its own, which is also what the legacy f_op->mmap path does since it
never touches an existing VMA.
Currently, these fields will get overwritten by whatever state is
established in the mmap_prepare hook, and if the VMA was merged,
vm_ops->mapped will not have been called, so this could destructively
clear existing state without replacing it with anything valid.
There is an implicit requirement that vm_private_data and vm_ops are
fungible across VMAs which means that losing the 'new' state is fine.
However in this case the 'old' state is being overwritten by potentially
invalid 'new' state, so this must be rectified.
Additionally constify have_mmap_prepare while here.
All existing in-tree users either derive state from the file or are
unmergeable due to VMA flags, so this has no direct impact.
Link: https://lore.kernel.org/20260923-fix-mmap-prepare-overwrite-v1-1-3b3f1bfcdf5e@kernel.org
Fixes: c84bf6dd2b83 ("mm: introduce new .mmap_prepare() file callback")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Acked-by: Zi Yan <ziy@nvidia.com>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: <stable@vger.kernel.org>
|
|
CPU resume and BPF exception handling can discard stack frames without
running their compiler-generated epilogues. Generic KASAN needs
kasan_unpoison_task_stack_below() to clear the redzones left behind by
those frames before the stack is reused.
CONFIG_KASAN_STACK also enables this helper for SW_TAGS. The helper
derives the stack base from an untagged stack pointer, so kasan_unpoison()
writes KASAN_TAG_KERNEL (0xff) into the shadow for [base, watermark). For
a vmapped task stack, vm_area->addr still carries the original random
allocation tag. This creates a tag mismatch at the stack base even if
that memory has never held an instrumented stack object.
When the task exits and its stack is not cached, thread_stack_free_rcu()
passes vm_area->addr to vfree(). In RCU callback context this reaches
vfree_atomic(), whose llist_add() writes to the allocation base through
the tagged pointer and triggers a false KASAN invalid-access report:
BUG: KASAN: invalid-access in vfree_atomic+0x90/0x150
Write of size 8 at addr c2ffffc0a8f70000 by task rcuop/7/75
Pointer tag: [c2], memory tag: [ff]
Call trace:
__hwasan_store8_noabort+0xe8/0xf8
vfree_atomic+0x90/0x150
vfree+0x220/0x298
thread_stack_free_rcu+0x3c/0x4c
rcu_do_batch+0x308/0xaf0
rcu_nocb_cb_kthread+0x33c/0x708
The deferred-free path started using the tagged vm_area->addr in commit
449e0b4ed5a1 ("fork: clean-up naming of vm_stack/vm_struct variables in
vmap stacks code"). Previously it freed the untagged pointer derived from
tsk->stack, so the write was never tag-checked.
SW_TAGS does not need the blanket shadow reset to make subsequent stack
accesses valid: untagged kernel pointers have the match-all 0xff tag, and
the compiler initializes the shadow tags of instrumented stack objects
before use.
Return early unless CONFIG_KASAN_GENERIC is enabled. Keep the mode check
in the common helper so it covers both resume and bpf_throw().
Link: https://lore.kernel.org/20260916175113.1327454-1-ryabinin.a.a@gmail.com
Fixes: 449e0b4ed5a1 ("fork: clean-up naming of vm_stack/vm_struct variables in vmap stacks code")
Signed-off-by: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Shaobo Huang <huangshaobo3@xiaomi.com>
Closes: https://lore.kernel.org/all/20260806123020.90869-1-huangshaobo3@xiaomi.com/
Assisted-by: LLM
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: <stable@vger.kernel.org>
|
|
DAMOS could unexpectedly stop working when a user disables quota using the
online parameters commit feature. Fix it by correcting a wrong quota
unset check in damos_adjust_quota().
DAMON users could disable all quotas by unsetting time and size quotas,
and removing all quota goals. The intention of disabling quotas would be
making DAMOS run at full speed. When such quota disabled setup is
detected, damos_adjust_quota() skips all its work. The skipped works
include effective size quota (damos_quota->esz) updates and charged quota
amount (damos_quota->charged_sz) resets. The intention is to avoid doing
unnecessary work when quotas are disabled.
However, users could do the setup while effective size quota is non-zero,
by doing the disabling with the online DAMON parameters commit feature.
In this case, because the effective size quota exists, DAMOS will keep
working with the quota until it is fully charged. After the effective
quota is fully charged, the charged quota amount (damos_quota->charged_sz)
cannot be reset because damos_adjust_quota() skips it. Then, DAMOS stops
working until the quota is newly set or DAMON is entirely restarted.
The problem happens because damos_adjust_quota() assumes the user setup
for disabling quota immediately disabled it. In reality, the quota is
still working until the effective size quota is also updated to zero.
Other logic for catching that uses damos_quota_is_set(), which understands
the fact and therefore checks the effective size quota in addition to the
user setup. Fix the issue by using damos_quota_is_set() in
damos_adjust_quota() to determine if its works should be skipped.
The user impact is a non-deterministic and unexpected DAMOS stop behavior.
That is, users would disable quotas using the online parameters commit
feature, expecting DAMOS will run at full speed. However, depending on
the timing, the setup can be updated while the effective size quota is
non-zero. Due to the above mentioned internal mechanism, DAMOS stops
working instead of running at full speed. It is unexpected behavior. It
is also non-deterministic because sometimes the setup is done when the
effective size quota is zero, depending on the timing. It doesn't cause
critical issues like crashes or leaks. Users can simply set a reasonable
quota again, or restart DAMON. But definitely it is an unexpected and
non-deterministic behavior that makes it difficult to reliably use. Also
investigating the root cause of the behavior would be quite difficult.
Link: https://lore.kernel.org/20260916135020.86483-1-sj@kernel.org
Fixes: da87878010e5 ("mm/damon/sysfs: support online inputs update")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Kunwu Chan <kunwu.chan@gmail.com>
Cc: <stable@vger.kernel.org> # 5.19.x
|
|
According to Documentation/mm/transhuge.rst, MADV_COLLAPSE is expected to
ignore any THP or mTHP interface settings.
However, after commit 26c7d8413aaf ("mm: thp: support "THPeligible"
semantics for mTHP with anonymous shmem"), performing MADV_COLLAPSE on
shmem will depend on /sys/.../hugepages-2048kB/shmem_enabled being set to
"inherit" (although that is the default), which can cause MADV_COLLAPSE to
fail unexpectedly.
This could cause a userspace-visible performance regression.
Fix this by returning the result of shmem_huge_global_enabled() directly
when MADV_COLLAPSE is requested, allowing PMD-order collapse while
ignoring shmem THP/mTHP settings.
Link: https://lore.kernel.org/063f655b4d6c4234f3aa27ed6ecab10283ecb880.1789351825.git.baolin.wang@linux.alibaba.com
Fixes: 26c7d8413aaf ("mm: thp: support "THPeligible" semantics for mTHP with anonymous shmem")
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Closes: https://lore.kernel.org/all/20260908125105.1510704-7-kirill@shutemov.name/
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Barry Song <baohua@kernel.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: <stable@vger.kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/mm/memblock
Pull memblock fix from Mike Rapoport:
"The rework of HugeTLB bootmem allocation forgot to add the name for a
new RSV_HUGETLB flag for proper display in debugfs.
Add the flag name now"
* tag 'fixes-2026-09-24' of git://git.kernel.org/pub/scm/linux/kernel/git/mm/memblock:
mm: memblock: add missing HugeTLB flag name
|
|
Since kfree_rcu() can be called under pi_lock (a raw spinlock in the
scheduler), kfree_rcu() itself should never allocate memory with
__GFP_KSWAPD_RECLAIM as waking up kswapd ends up acquiring pi_lock,
which leads to a deadlock.
Reproducing the issue even intentionally was not straightforward.
The set_cpus_allowed_force() path that is called under pi_lock is
exercised very rarely, and competing tasks that allocate memory will
most likely wake kswapd up.
Therefore the existence of the deadlock was verified with a modified
kernel that has a lockdep map for waking up kswapd:
======================================================
WARNING: possible circular locking dependency detected
7.2.0-rc1-slab-for-next+ #10 Not tainted
------------------------------------------------------
git/6660 is trying to acquire lock:
ffff8e524533c550 (&p->pi_lock){-.-.}-{2:2}, at: _raw_spin_lock_irqsave+0x12/0x20
but task is already holding lock:
ffff8e57bffff1a0 (&pgdat->kswapd_wait){....}-{3:3}, at: _raw_spin_lock_irqsave+0x12/0x20
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (&pgdat->kswapd_wait){....}-{3:3}:
__lock_acquire+0x5a4/0xc50
lock_acquire.part.0+0xb7/0x240
lock_acquire+0x70/0x170
__raw_spin_lock_irqsave+0x44/0x80
_raw_spin_lock_irqsave+0x12/0x20
__wake_up_common_lock+0x31/0xa0
__wake_up+0x20/0x40
kswapd_wakeup_wake+0x7e/0x110
wakeup_kswapd+0x295/0x320
wake_all_kswapds+0xac/0x1a0
__alloc_pages_slowpath.constprop.0+0x282/0xf80
__alloc_frozen_pages_noprof+0x32a/0x360
alloc_slab_page+0x2e/0x160
allocate_slab+0x82/0x420
new_slab+0x52/0xb0
refill_objects+0x13d/0x190
refill_sheaf+0x5d/0xd0
__pcs_replace_empty_main+0x230/0xb10
[...]
-> #1 (kswapd_wakeup){-.-.}-{0:0}:
__lock_acquire+0x5a4/0xc50
lock_sync.part.0+0x75/0x100
lock_sync+0x36/0x70
might_wakeup_kswapd+0x61/0xa0
__kfree_rcu_sheaf+0x33/0xd90
kvfree_call_rcu+0x1d4/0x3b0
set_cpus_allowed_force+0x163/0x220
cpuset_cpus_allowed_fallback+0x18b/0x240
select_fallback_rq+0x1e6/0x250
[...]
-> #0 (&p->pi_lock){-.-.}-{2:2}:
check_prev_add+0xe6/0xe00
validate_chain+0x51e/0x6e0
__lock_acquire+0x5a4/0xc50
lock_acquire.part.0+0xb7/0x240
lock_acquire+0x70/0x170
__raw_spin_lock_irqsave+0x44/0x80
_raw_spin_lock_irqsave+0x12/0x20
try_to_wake_up+0x77/0xa90
default_wake_function+0x27/0x60
autoremove_wake_function+0x23/0xb0
__wake_up_common+0xb8/0x170
__wake_up_common_lock+0x51/0xa0
__wake_up+0x20/0x40
kswapd_wakeup_wake+0x7e/0x110
wakeup_kswapd+0x295/0x320
wake_all_kswapds+0xac/0x1a0
__alloc_pages_slowpath.constprop.0+0x282/0xf80
__alloc_frozen_pages_noprof+0x32a/0x360
alloc_slab_page+0x2e/0x160
allocate_slab+0x82/0x420
new_slab+0x52/0xb0
refill_objects+0x13d/0x190
refill_sheaf+0x5d/0xd0
__pcs_replace_empty_main+0x230/0xb10
[...]
other info that might help us debug this:
Chain exists of:
&p->pi_lock --> kswapd_wakeup --> &pgdat->kswapd_wait
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&pgdat->kswapd_wait);
lock(kswapd_wakeup);
lock(&pgdat->kswapd_wait);
lock(&p->pi_lock);
*** DEADLOCK ***
3 locks held by git/6660:
#0: ffff8e519f0e9e20 (&type->i_mutex_dir_key#6){++++}-{4:4}, at: lookup_slow+0x2d/0x60
#1: ffffffff8a31c460 (kswapd_wakeup){-.-.}-{0:0}, at: kswapd_wakeup_wake+0x4d/0x110
#2: ffff8e57bffff1a0 (&pgdat->kswapd_wait){....}-{3:3}, at: _raw_spin_lock_irqsave+0x12/0x20
[...]
Fix this by always avoiding waking up kswapd in __kfree_rcu_sheaf().
Note that there are two paths that might wake up kswapd:
1) __kfree_rcu_sheaf()
// __GFP_KSWAPD_RECLAIM might wake up kswapd
-> alloc_empty_sheaf(GFP_NOWAIT)
2) __kfree_rcu_sheaf()
// Let's say __kfree_rcu_sheaf() doesn't pass GFP_NOWAIT
-> alloc_empty_sheaf(__GFP_NOWARN)
-> kmalloc_flags()
-> slab_alloc_node()
-> alloc_from_pcs()
-> __pcs_replace_empty_main()
// Free a sheaf in an allocation path when the sheaf becomes empty
// and refilling the sheaf fails
-> free_empty_sheaf()
-> slab_free()
-> free_to_pcs()
-> __pcs_replace_full_main()
// However free path always assumes it's safe to wake up kswapd
-> alloc_empty_sheaf(GFP_NOWAIT)
Drop __GFP_KSWAPD_RECLAIM in both cases. Note that the kfree_rcu() is
not the only user of free_to_pcs() path, but it should be fixed as it
can be invoked under pi_lock.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260831-b4-kfree_rcu_hotfix-v1-1-4f0fb882638b%40kernel.org
Fixes: ec66e0d59952 ("slab: add sheaf support for batching kfree_rcu() operations")
Link: https://lore.kernel.org/linux-mm/20260831143500.x-saxdAs@linutronix.de
Assisted-by: LLM # dicovery and verification
Signed-off-by: Harry Yoo (Meta) <harry@kernel.org>
Link: https://patch.msgid.link/20260922-kfree-rcu-dont-wakeup-kswapd-v1-1-42d7e2636f9e@kernel.org
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull MM fixes from Andrew Morton:
"14 hotfixes. 10 are cc:stable. 11 are for MM.
Five DAMON fixes: one fixes an arm64 contpte bug where DAMON can write
past the end of a page-table page, resulting in memory corruption and
possible crashes.
Two hugetlb fixes: one fixes an mremap() address calculation bug which
can panic x86-64.
There's also a missing anon_vma publication barrier which can result
in hung tasks, and a writeback fix to keep long cgroup writeback
drains from delaying Tasks-RCU grace periods.
The remainder are smaller fixes and maintenance changes"
* tag 'mm-hotfixes-stable-2026-09-21-17-08' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm:
MAINTAINERS: update Xu Xin's email
writeback: report a Tasks-RCU quiescent state per cgwb drain pass
mm/damon/core: reset invalid quota->charge_target_from
MAINTAINERS: add Baoquan and Baolin as MGLRU reviewers
mm/rmap: fix missing barrier between anon_vma init and vma->anon_vma publish
mm/hugetlb: preserve mremap address delta when skipping page tables
mm/damon/core: fix unconditionally skip last region
mm/damon/vaddr: avoid hw-driven pte updates during damon_hugetlb_mkold()
mm/damon/core: allow esz to be set to zero
mm/damon/ops-common: use a page-aligned address in damon_ptep_mkold()
ocfs2: make ocfs2_calc_xattr_init() return void
mailmap: update Haowen Bai's email address
selftests/cgroup: account for zswap shrinker writeback
mm/hugetlb: do not dissolve gigantic pages without runtime support
|
|
cleanup_offline_cgwbs_workfn() drains a dying cgwb by calling
cleanup_offline_cgwb() until it returns false, with a cond_resched()
between passes. On a CONFIG_PREEMPTION kernel that cond_resched() does
nothing: _cond_resched() is a plain "return 0", and under PREEMPT_DYNAMIC
the full and lazy modes disable it. Since commit 7dadeaa6e851 ("sched:
Further restrict the preemption modes") those are the only two models on
the architectures with PREEMPT_LAZY support, arm64 and x86 among them, so
the drain loop never reports a Tasks-RCU quiescent state.
A worker draining a cgwb with millions of attached inodes runs for
minutes. On a 6.18 arm64 host in lazy mode the cgwb worker drained one
dying cgroup's writeback domain for over 11 minutes. A BPF program unlink
(bpf_trampoline_unlink_prog -> bpf_trampoline_update ->
unregister_ftrace_direct -> ftrace_shutdown -> synchronize_rcu_tasks())
waited on that grace period while holding the trampoline mutex, 42 tasks
queued behind it in D state, and the hung task detector fired at 614 s and
panicked the host. Any BPF or ftrace detach during a long drain inherits
the drain's length.
Fix this by calling cond_resched_tasks_rcu_qs() so we do not stall out
anybody who calls sycnrhonize_rcu_tasks(). We put this in a do { } while
loop because if we have many small cgroups cleanup_offline_cgwb() will
return false and we will never call cond_resched_tasks_rcu_qs(), creating
the same problem.
Link: https://lore.kernel.org/20260909-cgwb-tasks-rcu-qs-v1-1-967a7754771f@toxicpanda.com
Fixes: c22d70a162d3 ("writeback, cgroup: release dying cgwbs by switching attached inodes")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Link: https://lore.kernel.org/bpf/9d444098-7c03-4163-af12-bd0a79a51443@paulmck-laptop/
Assisted-by: LLM
Acked-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Roman Gushchin <roman.gushchin@linux.dev>
Reviewed-by: Jan Kara <jack@suse.cz>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: "Paul E . McKenney" <paulmck@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: <stable@vger.kernel.org>
|
|
DAMOS can suddenly stop working if a target process that the quota is just
fully charged on is terminated. Fix by catching and processing the corner
case.
When DAMOS quota is fully charged, the target and the region to continue
applying the action in the next round is saved in
damos_quota->charge_{target,addr}_from. In the next round, DAMOS iterates
targets and regions from the beginning. It skips applying the action to
the regions until it visits and skips the saved target/region.
Virtual address space targets become invalid if the process is terminated.
Trying to apply the scheme to invalid target is just a waste of time.
Hence commit 6e4930e33329 ("mm/damon/core: fix wasteful CPU calls by
skipping non-existent targets") made the logic to skip invalid targets.
However, it does skip before the charged target/region skipping/updating.
Let's suppose the user runs DAMOS for multiple virtual address spaces with
a quota. The quota exceeded in the middle of a virtual address space.
And the process of the address space is terminated. Then the
charge_target_from points to the invalid target. The pointer update logic
is skipped for the invalid target, so the charge_target_from is never
updated. DAMOS action to every target/region is skipped. From the user's
perspective, it would look like suddenly DAMOS has stopped working.
No critical leak or crash can happen. The user could reinstall the
scheme. But this makes use of DAMOS under certain setups quite
unreliable.
When the invalid target is found, further check the corner case and reset
the pointer.
This issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260910142846.172957-1-sj@kernel.org
Link: https://lore.kernel.org/20260830064708.40CA61F000E9@smtp.kernel.org [1]
Fixes: 6e4930e33329 ("mm/damon/core: fix wasteful CPU calls by skipping non-existent targets")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Enze Li <lienze@kylinos.cn>
Cc: <stable@vger.kernel.org> # 7.0.x
|
|
On arm64 server, we find that a task trying to grab the anon_vma lock
triggers hungtask.
INFO: task main:2354726 blocked for more than 120 seconds.
Tainted: G E 5.10.0-0021.aarch64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:main state:D stack: 0 pid:2354726 ppid:2350673 flags:0x00000a01
Call trace:
__switch_to+0x7c/0xbc
__schedule+0x3b4/0x8a0
schedule+0x50/0xe0
rwsem_down_write_slowpath+0x3cc/0x6cc
down_write+0x60/0x260
__anon_vma_prepare+0x6c/0x210
do_anonymous_page+0x258/0x660
handle_pte_fault+0x188/0x214
__handle_mm_fault+0x1b0/0x380
handle_mm_fault+0xf4/0x284
do_page_fault+0x19c/0x494
do_translation_fault+0xcc/0xf8
do_mem_abort+0x48/0xac
el0_da+0x44/0x80
el0_sync_handler+0x88/0xb4
el0_sync+0x160/0x180
After analyzing the vmcore, we found the anon_vma->root->rwsem.count is
-1. There is another anon_vma whose anon_vma->root->rwsem.count is 1, the
anon_vma->root->rwsem.owner shows the lock is held, but the stack of the
task shows the task doesn't hold the anon_vma lock.
After adding more debugging info, we found __anon_vma_prepare() reuses
anon_vma and triggers the UAF of anon_vma->root due to missing memory
barrier, leading to locking and unlocking two different anon_vma->root,
thus leading to an anon_vma will never be unlocked, and another anon_vma
couldn't be locked anymore.
This race requires two adjacent VMAs that are not merged but are
anon_vma-compatible (e.g., they differ in VMA_ACCESS_FLAGS that can be
changed by mprotect()). Two threads fault on each VMA concurrently, both
calling __anon_vma_prepare() with only mmap_lock held for reading.
THREAD A THREAD B
__anon_vma_prepare __anon_vma_prepare
find_mergeable_anon_vma() -> NULL
anon_vma = anon_vma_alloc();
anon_vma->root = anon_vma;
// the two stores may be reordered
vma->anon_vma = anon_vma;
// finds A's anon_vma
anon_vma = find_mergeable_anon_vma(vma);
anon_vma_lock_write(anon_vma);
// may still see the old root
down_write(&anon_vma->root->rwsem);
anon_vma_unlock_write(anon_vma);
// see the new root, never unlock old
up_write(&anon_vma->root->rwsem);
thread A triggers page fault and calls __anon_vma_prepare() to prepare
anon_vma for the faulting vma. __anon_vma_prepare() allocates and
initializes a new anon_vma, and then publishes it to the vma with a plain
store. anon_vma_prepare() only requires the mmap_lock to be held for
reading, so two threads can fault on adjacent VMAs at the same time.
While thread A publishes a new anon_vma, thread B could find the anon_vma
via find_mergeable_anon_vma() and then locks anon_vma->root->rwsem.
The store to anon_vma->root in anon_vma_alloc() and the store to
vma->anon_vma can be reordered. The anon_vma_lock_write() and spin_lock()
only provide acquire semantics, which do not prevent prior stores from
being reordered after them. The release semantics of the corresponding
spin_unlock() and anon_vma_unlock_write() come too late, the store to
vma->anon_vma is already published before they take effect. As a result,
thread B can observe the following order:
vma->anon_vma = anon_vma;
anon_vma->root = anon_vma;
The anon_vma slab is SLAB_TYPESAFE_BY_RCU, so a newly allocated anon_vma
may reuse memory from a previously freed one. The constructor
(anon_vma_ctor) does not reset anon_vma->root, and __put_anon_vma()
doesn't clear it either, so the old root value persists until
anon_vma_alloc() overwrites it. If that store isn't visible, thread B
reads a root that points to the old anon_vma and locks it.
As a result, thread B can call anon_vma_lock_write() with the old root,
and call anon_vma_unlock_write() with the new root, leading to an anon_vma
will never be unlocked, and another anon_vma couldn't be locked anymore
(its count is dropped from 0 to -1 due to wrong unlock).
To fix it, change the plain store `vma->anon_vma = anon_vma` to store
release, so that the fields of anon_vma are visible before anon_vma is
published to vma->anon_vma.
At read side, the load of anon_vma and anon_vma->root have address
dependency. According to Documentation/memory-barriers.txt and some
investigations, only Alpha needs address-dependency barriers and it has
been handled by READ_ONCE() in reusable_anon_vma().
We reproduced this issue in v5.10 with KSM enabled. The kernel doesn't
merge commit cf7e7a3503df ("mm: prevent KSM from breaking VMA merging for
new VMAs"), so there are many adjacent VMAs that aren't merged but are
compatible for anon_vma.
Without this fix, our production environment could reproduce this issue
about 2-5 times each month. After adding a smp_mb() before
anon_vma_lock_write(anon_vma) in __anon_vma_prepare(), which is different
to this patch, this issue hasn't been reproduced for one month.
Link: https://lore.kernel.org/20260908122924.554373-1-tujinjiang@huawei.com
Fixes: 5c341ee1dfc8 ("mm: track the root (oldest) anon_vma")
Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Hiroyouki Kamezawa <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Jann Horn <jannh@google.com>
Cc: Jinjiang Tu <tujinjiang@huawei.com>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Larry Woodman <lwoodman@redhat.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nanyong Sun <sunnanyong@huawei.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: <stable@vger.kernel.org>
|
|
move_hugetlb_page_tables() optimizes mremap() by advancing to the last
entry in the page table when the source page table does not exist, either
initially or after unsharing a PMD table. The common loop increment then
steps to the first entry in the next page table.
However, the code advances both the source and destination addresses to
the last entries in their respective page tables, which is wrong. The
destination address must be advanced only by the same amount as the source
address.
If the source and destination offsets within their page tables differ, the
destination address can be advanced too far, causing follow-up issues.
Fix this by advancing the destination address by the source advance
distance.
With a reproducer, we were able to trigger a kernel panic on x86-64. With
this fix in place, we can no longer reproduce the issue.
Link: https://lore.kernel.org/20260914132352.472-1-jaewook376@gmail.com
Fixes: e95a9851787b ("hugetlb: skip to end of PT page mapping when pte not present")
Fixes: 4ddb4d91b82f ("hugetlb: do not update address in huge_pmd_unshare")
Signed-off-by: Jaewook You <jaewook376@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Johan Hovold <johan@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: <stable@vger.kernel.org>
Assisted-by: LLM
|
|
Once quota set, the charge_{target,addr}_from unconditionally skips and
resets at the last region of the tracked target, so the last region can be
skipped even when it has not been processed.
Example:
1. Target has 2 regions: R1 (0-100 bytes) and R2 (100-200 bytes).
2. Quota is configured to process only 100 bytes per window.
3. Window 1: Processes R1 (0-100). Quota is full. charge_{target,
addr}_from is saved at (Target, 100).
4. Window 2: The loop reaches R2. Because R2 is
damon_last_region(t), the old code unconditionally returns true,
skipping R2 entirely and resetting the charge_{target,addr}_from.
Result: R2 is permanently skipped even though it has never been
processed.
However, it is important to note that this is a very minor issue. This is
because it is triggered only when the previous window saved/kept
charge_{target,addr}_from, and in the next window, all regions except the
last region were skipped by damos_skip_charged_region().
Fix this by only resetting the charge_{target,addr}_from when last region
is reached, only skipping when it is applied or cannot split.
Link: https://lore.kernel.org/20260908134739.96919-1-sj@kernel.org
Fixes: 50585192bc2e ("mm/damon/schemes: skip already charged targets and regions")
Signed-off-by: Liew Rui Yan <aethernet65535@gmail.com>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: <stable@vger.kernel.org> # v5.16.x
|
|
damon_hugetlb_mkold() reads the page table entry into a local variable,
unsets the accessed bit in the variable, and updates the page table entry
with the updated variable value. If hardware updates the same page table
entry in parallel, the hw updates could be lost. For example,
hardware-updated dirty bits might be lost.
Avoid the parallel updates by clearing the page table entry when reading
it together, using huge_ptep_get_and_clear(). If a parallel write to the
memory is made after the clearing, the hw will see the page table entry is
cleared, trigger page fault and wait until it is handled. The page fault
handling will wait for damon_hugetlb_mkold() due to the page table lock.
Because hugetlbfs is an in-memory file system and hugetlb pages cannot be
reclaimed, no critical issue is expected to my best knowledge. But
definitely this is a nasty bug that should be fixed sooner rather than
later.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260907170358.100168-1-sj@kernel.org
Link: https://lore.kernel.org/20260830160545.98969-1-sj@kernel.org [1]
Fixes: 49f4203aae06 ("mm/damon: add access checking for hugetlb pages")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: <stable@vger.kernel.org> # 5.17.x
|
|
When the temporal quota goal tuner determines that the goal has been
achieved (score >= 10000), it sets esz_bp to zero so that the esz becomes
zero. However, damos_set_effective_quota() clamps the esz to
min_region_sz when quota->ms is set.
This is a minor issue, the main problem is that it doesn't match the
description in the documentation, which state that if the goal has already
been [over-]achieved, the quota will be set to zero.
Fix this by set quota (esz) as minimum as possible.
Link: https://lore.kernel.org/20260908135413.97570-1-sj@kernel.org
Fixes: 8bbde987c2b8 ("mm/damon/core: disallow time-quota setting zero esz")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Liew Rui Yan <aethernet65535@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: <stable@vger.kernel.org> # v7.1.x
|
|
__damon_va_prepare_access_check() picks a random byte address within the
region and stores it in r->sampling_addr. damon_va_mkold() passes it into
a page table walk, which hands it to damon_ptep_mkold() as the address of
the page to sample:
damon_va_mkold(mm, r->sampling_addr)
damon_va_walk_page_range(mm, addr, addr + 1)
damon_mkold_pmd_entry()
damon_ptep_mkold(pte, vma, addr)
ptep_test_and_clear_young(vma, addr, pte)
mmu_notifier_clear_young(mm, addr, addr + PAGE_SIZE)
For arm64, before commit 6f0e1142173a ("arm64: mm: support batch clearing
of the young flag for large folios"), the contpte helper walked exactly
CONT_PTES entries from the aligned-down page table pointer and used @addr
only to pass down to each entry, so an unaligned value was harmless:
ptep = contpte_align_down(ptep);
addr = ALIGN_DOWN(addr, CONT_PTE_SIZE);
for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE)
Now the range to walk is derived from @addr instead: end = addr + nr *
PAGE_SIZE, rounded up to CONT_PTE_SIZE. For a sample in the last page of
a contpte block, the sub-page offset puts end just past the block
boundary, so the round-up lands a whole block further and the walk clears
PTE_AF in CONT_PTES entries beyond the sampled block. For the last block
in a page table page, those entries are past the end of that page, so the
walk writes into the page that follows.
Triggered by the full 7.1/7.2 kernel selftest suite on arm64 (EC2
c/m6g.4xlarge). The kernel sometimes crashes at or shortly after the
DAMON test.
What the overrun does depends on the page that happens to follow the page
table, so there is no single signature. If that page is read-only, the
write faults in the sampling path itself:
Unable to handle kernel write to read-only memory at virtual address ffff0003c5d2d000
FSC = 0x0f: level 3 permission fault
CM = 0, WnR = 1, TnD = 0, TagAccess = 0
CPU: 10 UID: 0 PID: 3487 Comm: kdamond.2
pc : contpte_test_and_clear_young_ptes+0x70/0xc0
lr : damon_ptep_mkold+0x1e8/0x1f8
Call trace:
contpte_test_and_clear_young_ptes+0x70/0xc0 (P)
damon_mkold_pmd_entry+0x150/0x170
walk_pmd_range+0x110/0x2b0
walk_pud_range+0x10c/0x208
walk_pgd_range+0x134/0x258
__walk_page_range+0x98/0x1b0
walk_page_range_vma_unsafe+0x90/0x148
walk_page_range_vma+0x28/0x40
damon_va_walk_page_range+0x114/0x2b8
damon_va_prepare_access_checks+0xec/0x1a8
kdamond_fn+0x534/0x770
kthread+0x128/0x138
ret_from_fork+0x10/0x20
Otherwise the page is writable, the PTE_AF clearing succeeds silently and
the damage only surfaces later, in whatever happened to own the page, so
the backtrace is unrelated to DAMON and differs between runs.
Pass a page-aligned address to the ptep_test_and_clear_young() call in
damon_ptep_mkold(), which is the only place DAMON can reach
contpte_test_and_clear_young_ptes() from. Nothing else sees the aligned
address, and r->sampling_addr itself is left as is, so the sampling and
region bookkeeping semantics are unchanged.
Link: https://lore.kernel.org/20260904002829.116381-1-sj@kernel.org
Fixes: 6f0e1142173a ("arm64: mm: support batch clearing of the young flag for large folios")
Signed-off-by: Nathan Gao <zcgao@amazon.com>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: <stable@vger.kernel.org>
|
|
dissolve_free_hugetlb_folio() doesn't check
hstate_is_gigantic_no_runtime(h) though remove_hugetlb_folio()/
update_and_free_hugetlb_folio() silently bail for such folios, so it frees
a still-listed folio and, on vmemmap restore failure, the
add_hugetlb_folio() rollback corrupts the free list.
Link: https://lore.kernel.org/20260823044118.1097121-2-xialonglong2025@163.com
Fixes: 6eb4e88a6d27 ("hugetlb: create remove_hugetlb_page() to separate functionality")
Signed-off-by: Longlong Xia <xialonglong@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Assisted-by: Codex:gpt-5.6-sol
Acked-by: Muchun Song <muchun.song@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: <stable@vger.kernel.org>
|
|
Commit 7d163a75f821 ("memblock: make HugeTLB bootmem allocation work
with KHO") added MEMBLOCK_RSRV_HUGETLB but did not add the corresponding
entry to flagname[]. As a result, memblock debugfs cannot report the
flag by name.
Add the missing RSV_HUGETLB entry.
Fixes: 7d163a75f821 ("memblock: make HugeTLB bootmem allocation work with KHO")
Signed-off-by: Meijing Zhao <zhaomeijing@lixiang.com>
Signed-off-by: Wandun Chen <chenwandun@lixiang.com>
Link: https://lore.kernel.org/lkml/20260821020910.3428585-2-zhaomeijing100@gmail.com/
Link: https://patch.msgid.link/20260915124641.2498280-1-chenwandun1@gmail.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/mm/memblock
Pull memblock fixes from Mike Rapoport:
"Fix for regions display in debugfs and MAINTAINERS update:
- Make sure that multiple flags on a memblock region are all
displayed in debugfs
- Update memblock tree tags in MAINTAINERS"
* tag 'fixes-2026-09-14' of git://git.kernel.org/pub/scm/linux/kernel/git/mm/memblock:
MAINTAINERS: update memblock tree URLs
mm: memblock: show all region flags in debugfs
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull misc fixes from Andrew Morton:
"14 hotfixes. 10 are cc:stable. 11 are for MM.
All are singletons - please see the changelogs for details"
* tag 'mm-hotfixes-stable-2026-09-13-21-50' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm:
mm/folio: EXPORT_SYMBOL_FOR_KVM(lru_cache_drain_for_folio)
mm/shrinker: fix bogus set_shrinker_bit() with cgroup.memory=nokmem
mm/vma: correctly unaccount on mmap_prepare() failure
mm/mlock: use the IRQ-safe accessor for NR_MLOCK in __munlock_folio()
remove old lib/alloc_tag.c
fs/dax: check zero or empty entry before converting xarray entry
fs: fix missed removal of super_fs_objects_eligible()
mm: filemap: retain mapped dropbehind folios
mailmap: update entry for Christopher Obbard
memcg: avoid charging the root memcg from obj_cgroup_charge_pages()
mm, swap: fix SWAP_USAGE_OFFLIST_BIT collision with real usage count
mailmap: map Coiby Xu's address
mm/mremap: account mm->locked_vm correctly for MREMAP_DONTUNMAP
mm/huge_memory: bypass THP tuneables for huge pfnmap mappings
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/mm/slab
Pull slab fixes from Vlastimil Babka:
- Stable fix for an ABA issue causing slab list corruption introduced
in 7.2 (Harry Yoo, with big thanks to Hyunwoo Kim for the thorough
report and initial version of the fix)
- Fix for 7.3 regression of kvfree_rcu() on PREEMPT_RT which can cause
a deadlock from the set_cpus_allowed_force() caller (Vlastimil Babka)
* tag 'slab-for-7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/mm/slab:
mm/slab: take n->list_lock in __slab_try_return_freelist() to avoid race
mm/slab: disallow kfree_rcu_sheaf() on PREEMPT_RT again
|
|
Commit 493f349e38d0 ("memblock: Add flags and nid info in memblock
debugfs") made memblock_debug_show() stop after finding the first set
flag. A memblock region can carry multiple flags, so the remaining flags
are hidden from debugfs.
Walk all bits in the region flags and print every set flag separated by
"|". Keep walking beyond flagname[] so that a set flag without a known
name is reported as UNKNOWN rather than silently ignored.
Fixes: 493f349e38d0 ("memblock: Add flags and nid info in memblock debugfs")
Signed-off-by: Meijing Zhao <zhaomeijing@lixiang.com>
Link: https://patch.msgid.link/20260902075944.3742866-1-zhaomeijing100@gmail.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
To simplify independent development in the KVM and MM subsystems, now
export to KVM the lru_cache_drain_for_folio() which MM added in 7.3-rc1.
Link: https://lore.kernel.org/lkml/bd6c9c74-e374-a9d3-ba1f-8b6f430894fc@google.com/T/#u
Link: https://lore.kernel.org/02876cea-5727-2ca4-bead-73659ea6fec4@google.com
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Hugh Dickins <hughd@google.com>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Suggested-by: David Hildenbrand <david@kernel.org>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Sean Christopherson <seanjc@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
With cgroup.memory=nokmem, shrinker_memcg_alloc() bails out early and
never allocates an id, so shrinker->id keeps the 0 it got from the
kzalloc() in shrinker_alloc(). __list_lru_init() then copies that 0 into
lru->shrinker_id, where it looks like a valid bit index.
Nothing calls expand_shrinker_info() on nokmem either, so shrinker_nr_max
stays 0 and every memcg ends up with an empty map (map_nr_max == 0).
deferred_split_folio() hands a real memcg to __list_lru_add() regardless
of whether the lru is memcg aware, so the first THP queued in a cgroup
does set_shrinker_bit(memcg, nid, 0) and trips the bounds check:
WARNING: mm/shrinker.c:212 at set_shrinker_bit+0x7d/0x90, CPU#126
Call Trace:
<TASK>
deferred_split_folio+0x18c/0x220
map_anon_folio_pmd_nopf+0xdd/0x130
map_anon_folio_pmd_pf+0x14/0xb0
do_huge_pmd_anonymous_page+0x1a1/0x620
__handle_mm_fault+0xea9/0x10d0
handle_mm_fault+0xe5/0x320
do_user_addr_fault+0x1cc/0x870
exc_page_fault+0x81/0x1b0
asm_exc_page_fault+0x27/0x30
</TASK>
Harmless, the WARN_ON_ONCE() is what keeps the out of bounds unit[] read
from happening, but the id should not look valid in the first place.
Clear it before returning.
Two other spots could paper over this: drop the id in __list_lru_init()
when nokmem turns memcg_aware off, or make deferred_split_folio() pass
NULL like list_lru_add_obj() does. Both leave shrinker->id lying around
for the next caller, so fix it where the id is handed out.
Link: https://lore.kernel.org/20260902073800.305481-1-jiayuan.chen@linux.dev
Fixes: fafaeceb89a5 ("mm: switch deferred split shrinker to list_lru")
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
__mmap_setup() accounts memory for relevant mappings via:
security_vm_enough_memory_mm()
-> __vm_enough_memory()
-> vm_acct_memory()
If __mmap_setup() fails, this indicates that this accounting did not take
place, and thus it's appropriate for __mmap_region() to jump to
abort_munmap.
However if call_mmap_prepare() fails, it also jumps there and any accounted
memory is not correctly unaccounted.
Fix this by handling each error separately.
Link: https://lore.kernel.org/20260902-fix-unaccount-mmap_prepare-v1-1-ea070189fdfb@kernel.org
Fixes: c84bf6dd2b83 ("mm: introduce new .mmap_prepare() file callback")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
NR_MLOCK is updated from interrupt context. __free_pages_prepare() clears
a stray PG_mlocked and adjusts NR_MLOCK, and a folio can reach it with the
flag still set from a bio completion handler:
__free_pages_ok+0x6af/0x7a0
<IRQ>
__bio_release_pages+0xde/0x260
__iomap_dio_bio_end_io+0x16e/0x1a0
blk_update_request+0x14b/0x3d0
blk_mq_end_request+0x18/0x30
blk_done_softirq+0x49/0x60
The folio gets there like this. A MAP_SHARED file mapping is mlocked, so
its page cache folios carry PG_mlocked, and an O_DIRECT write sourced from
that mapping GUP-pins those same folios. munlock() then runs
mlock_vma_pages_range(), which clears VM_LOCKED before walking the page
tables to munlock each folio. A concurrent hole punch reaches the folio
through the rmap (i_mmap_rwsem, not mmap_lock) and can land inside that
window: __folio_remove_rmap() -> munlock_vma_folio() sees VM_LOCKED
already clear, so it neither queues the folio on the mlock batch nor takes
a reference, and the pte it clears makes the pending mlock_pte_range()
walk skip the folio at its !pte_present() check. filemap_remove_folio()
then drops the page cache reference, leaving the bio's pin as the last
one, released from the completion handler above.
So __zone_stat_mod_folio() here needs interrupts disabled, not merely
preemption, and __munlock_folio() has a path where they are not: when the
folio has already been taken off the LRU by somebody else the function
jumps straight to the counter update without taking the lruvec lock. The
read-modify-write of the per-CPU NR_MLOCK diff can then be interrupted by
the softirq above, and one of the two decrements is lost, leaving Mlocked
in /proc/meminfo permanently overstated.
Use zone_stat_mod_folio(). mod_zone_state()'s this_cpu_try_cmpxchg() is
atomic against a same-CPU interrupt and retries, and on the path where the
lruvec lock is held its cost is negligible next to the lock itself.
The UNEVICTABLE_PG* events are deliberately left on the __ accessors: they
occupy different vm_event_states slots from the UNEVICTABLE_PGCLEARED that
__free_pages_prepare() bumps, and nothing updates those two from interrupt
context.
Link: https://lore.kernel.org/20260901180109.3797944-1-shakeel.butt@linux.dev
Fixes: 2fbb0c10d1e8 ("mm/munlock: mlock_page() munlock_page() batch by pagevec")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Reported-by: syzbot+cd2073ee6d958a8d0fcd@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/linux-mm/6a931c5a.08e933ee.dbf97.0093.GAE@google.com/
Acked-by: Hugh Dickins <hughd@google.com>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Fault-around can map ready dropbehind folios without going through the
normal page-cache lookup that clears dropbehind. A mapping represents a
competing cached user, so retain the folio instead of forcibly unmapping
it when writeback completes.
For a mapped folio, folio_unmap_invalidate() can call
unmap_mapping_folio(), which takes i_mmap_rwsem and may sleep. Retaining
mapped folios avoids this path when folio_end_dropbehind() runs in
non-preemptible task context.
Tal was able to trigger a sleeping-in-atomic warning due to this [1].
Unmapped dropbehind folios continue through the existing invalidation path.
Link: https://lore.kernel.org/4aba05e1a2c3b61cb337d373eb9b7a8db4ddd822.1788024049.git.qiwenjie@xiaomi.com
Link: https://lore.kernel.org/076bb01b-6fcf-4691-be8c-0e8507c9fe64@columbia.edu [1]
Fixes: fb7d3bc41493 ("mm/filemap: drop streaming/uncached pages when writeback completes")
Signed-off-by: Wenjie Qi <qiwenjie@xiaomi.com>
Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Tal Zussman <tz2294@columbia.edu>
Tested-by: Tal Zussman <tz2294@columbia.edu>
Cc: Barry Song <baohua@kernel.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Trond Myklebust <trond.myklebust@hammerspace.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
obj_cgroup_charge_pages() resolves the objcg to its memcg and calls
try_charge_memcg(), which does not short circuit the root memcg. That
memcg can be the root memcg: obj_cgroup_is_root() reflects the memcg the
objcg was created for and is never updated, while memcg_reparent_objcgs()
does redirect objcg->memcg to the parent on rmdir. An objcg of a dying
child of root therefore passes every obj_cgroup_is_root() filter but
resolves to the root memcg.
Folios keep the objcg they were charged with, so this is easy to reach
through zswap: allocate anon memory in a cgroup, move the task out, remove
the cgroup, then write to the root cgroup's memory.reclaim. The reclaimed
folios are charged through the reparented objcg and end up in
refill_stock() with the root memcg:
WARNING: mm/memcontrol.c:2198 at refill_stock+0x644/0x940
refill_stock+0x644/0x940
try_charge_memcg+0x12d6/0x1570
__obj_cgroup_charge+0x35/0xf0
obj_cgroup_charge+0x1de/0x210
obj_cgroup_charge_zswap+0x83/0x270
zswap_store+0x1620/0x2000
swap_writeout+0x94c/0x14c0
shrink_folio_list+0x3388/0x52b0
[...]
try_to_free_mem_cgroup_pages+0x30d/0x830
user_proactive_reclaim+0x504/0x840
memory_reclaim+0x1f/0x30
Beyond the warning, the charge is asymmetric: obj_cgroup_uncharge_pages()
skips refill_stock() for the root memcg, so the root's page counter grows
and is never uncharged. It is not user visible, since memory.current is
not exposed on the root, but it is a leak.
Use try_charge(), which returns early for the root memcg, restoring the
symmetry with obj_cgroup_uncharge_pages().
The above sequence was scripted into a standalone reproducer (zswap on,
swap on a virtio disk, 512MB of anon memory faulted in inside a child of
the root cgroup, the task then migrated to the root cgroup, the child
removed, followed by "echo 600M swappiness=max > memory.reclaim" on the
root) and run in a CONFIG_DEBUG_VM=y VM. It reproduces the splat on the
first zswap store of a reparented folio, with the same call chain as the
report. With this patch applied the splat is gone while the zswap store
count over the run is unchanged, so the same path is still exercised.
cgroup selftests test_zswap, test_kmem and test_memcontrol show no new
failures.
Link: https://lore.kernel.org/20260829023251.474083-1-shakeel.butt@linux.dev
Fixes: 20d6c1725228 ("memcg: avoid refill_stock for root memcg")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Reported-by: Farhad Alemi <farhad.alemi@berkeley.edu>
Closes: https://lore.kernel.org/all/CA+0ovCgWzUMK+nNbbtH7eV65Ca=fDN4Ozu7iASgryjvv8Tk8zQ@mail.gmail.com/
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
SWAP_USAGE_OFFLIST_BIT is embedded in the si->inuse_pages usage counter,
and is meant to sit above any value that counter can reach. However, it
is defined from BITS_PER_TYPE(atomic_t), so it is bit 30. On a system
with 4 KiB pages the flag collides with the usage count once that count
reaches 4 TiB.
swap_usage_in_pages() masks bit 30 out, so whenever the real count has
that bit set, every caller of it reads 4 TiB low:
* /proc/swaps understates Used by 4 TiB.
* A raw count of exactly 2^30 masks to zero, so try_to_unuse() takes its
"if (!swap_usage_in_pages(si)) goto success;" early exit and swapoff
tears the device down while pages are still swapped out. Nothing in
the rest of swapoff aborts the teardown, so those pages are lost.
Independently of swapoff, the collision also corrupts the counter and the
plist. On a device in normal use, a free that leaves bit 30 set in the
count makes swap_usage_sub() see the flag where there is only count, and
call add_to_avail_list(). It clears the bit with
fetch_and(~SWAP_USAGE_OFFLIST_BIT), leaving the stored count 4 TiB below
the real one, and calls plist_add() on a device that is already listed,
tripping the WARN_ON(!plist_node_empty(node)) in plist_add() and linking
the node a second time.
Change the definition of SWAP_USAGE_OFFLIST_BIT to be based on
atomic_long_t instead. Note that the usage counter field itself is of
this same type, so it is still a valid bit.
Link: https://lore.kernel.org/20260828191433.3304458-1-nphamcs@gmail.com
Fixes: b228386cf237 ("mm, swap: clean up plist removal and adding")
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260825153238.2695446-1-nphamcs%40gmail.com
Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Kairui Song <kasong@tencent.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Gregory Price <gourry@gourry.net>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Youngjun Park <youngjun.park@lge.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
When a VMA is mremap()'d with MREMAP_DONTUNMAP set, that results in the
VMA being copied, but the source VMA not being unmapped.
If the VMA is mlock()'d this is a legal operation, though the source VMA
has its VMA_LOCKED_BIT cleared.
However this is done in dontunmap_complete(), after mm->locked_vm was
incremented via vrm_stat_account(), resulting in double-counting.
Worse, this is not even corrected when source VMA is unmapped, due to the
VMA_LOCKED_BIT flag having been cleared.
This all works fine in the usual mremap() case (without MREMAP_DONTUNMAP),
as the source VMA is unmapped with VMA_LOCKED_BIT intact, at which time
mm->locked_vm is decremented accordingly.
Resolve the issue by invoking vrm_stat_account() only after
dontunmap_complete() has run.
Note that MREMAP_DONTUNMAP requires old_len == new_len, so no need to
account for a delta in size in this case.
The bug was introduced by commit b714ccb02a76 ("mm/mremap: complete
refactor of move_vma()") which incorrectly reordered the accounting and
the clearing of the VMA_LOCKED_BIT flag.
Link: https://lore.kernel.org/20260828-mremap-fix-locked-vm-v1-1-c80be7505d1e@kernel.org
Fixes: b714ccb02a76 ("mm/mremap: complete refactor of move_vma()")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reported-by: sashiko-bot <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260825-fix-mremap-dontunmap-pgoff-v1-1-39a40b2c98b3@kernel.org
Reported-by: Kunwu Chan <kunwu.chan@gmail.com>
Closes: https://lore.kernel.org/all/20260828094823.594279-1-kunwu.chan@linux.dev/
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Tested-by: Kunwu Chan <kunwu.chan@gmail.com>
Reviewed-by: Kunwu Chan <kunwu.chan@gmail.com>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The sysfs THP tuneables at /sys/kernel/mm/transparent_huge_pages/ rather
confusingly only control the behaviour of THP in some instances.
They are not applicable to MADV_COLLAPSE operations, nor to DAX mappings.
Long-term, THP is predicated upon compaction being able to obtain large
folios to populate THP ranges.
However, vm_normal_folio() returns NULL for PFN map mappings, thus their
reference count is maintained by the driver, not core mm.
As a consequence, the folios are not subject to reclaim nor compaction, so
are not truly part of the THP mechanism at all.
However, since commit 5dd40721f147 ("mm: allow THP orders for PFNMAPs")
introduced the ability to establish huge PFN maps, they have been subject
to THP tuneables.
This is incorrect - if a huge PFN map is available (defined by
vma->vm_ops->huge_fault being non-NULL for a VMA_PFNMAP_BIT VMA), then it
should be mapped huge upon fault-in.
Correct this by explicitly checking for this while ensuring that smaps
continues to accurately report THPeligible statistics.
While here, abstract the entire file-backed THP check in
vma_can_map_huge_file(), with sensible separation of logic into helper
functions.
Note that drm_gem_shmem_mmap() and panthor_gem_mmap() establish huge PFN
maps of shmem folios, however they are marked unevictable in
drm_gem_get_pages(), and in any case would fail the reference check in
__remove_mapping() even if they weren't.
Failing to map huge PFN maps has resulted in significant real-world
performance degradation, see links for details.
[ziy@nvidia.com: rename some functions]
Link: https://lore.kernel.org/DL1HIHWYJ7TB.1CY76SJS0V03L@nvidia.com
Link: https://lore.kernel.org/20260827-hugepfn-allowable-orders-v1-1-94819c8807c8@kernel.org
Fixes: 5dd40721f147 ("mm: allow THP orders for PFNMAPs")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Zi Yan <ziy@nvidia.com>
Reported-by: Cedric Le Goater <clg@redhat.com>
Closes: https://lore.kernel.org/linux-mm/20260805055544.1568534-1-clg@redhat.com/
Reported-by: Saravanan D <saravanand@crusoe.ai>
Closes: https://lore.kernel.org/linux-mm/20260821070520.25759-1-saravanand@crusoe.ai/
Reviewed-by: Zi Yan <ziy@nvidia.com>
Tested-by: Saravanan D <saravanand@crusoe.ai>
Tested-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: SJ Park <sj@kernel.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Peter Xu <peterx@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
|
|
Commit ba7425312607 ("mm, slab: add an optimistic
__slab_try_return_freelist()") incorrectly assumed that nobody has freed
an object to the slab as long as slab->freelist is NULL and cmpxchg
succeeds.
However, as reported by Hyunwoo Kim [1], other CPUs might have freed
an object to the slab, insert the slab to the partial list, then
allocated an object from the slab, and be in the middle of removing
the slab from the list under n->list_lock.
Since __refill_objects_node() puts the slab back on pc.slabs
outside n->list_lock, it might insert the slab into that list while
the slab is concurrently being removed from n->partial.
This led to a list corruption [1]:
list_add corruption. next->prev should be prev
(ffff888100000248), but was dead000000000122.
(next=ffffea000416e410).
kernel BUG at lib/list_debug.c:29!
Oops: invalid opcode: 0000 [#1] SMP NOPTI
CPU: 1 UID: 65534 PID: 144 Comm: poc Not tainted
7.2.0-16172-gcf72cbb39da8-dirty #1 PREEMPT(lazy)
RIP: 0010:__list_add_valid_or_report+0x80/0xd0
...
Call Trace:
alloc_from_new_slab+0x183/0x300
___slab_alloc+0x31c/0x890
__kmalloc_noprof+0x3d4/0x800
lsm_blob_alloc+0x2d/0x50
security_msg_msg_alloc+0x26/0x90
load_msg+0x1aa/0x210
do_msgsnd+0x91/0x800
do_syscall_64+0x109/0x5d0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
...
Kernel panic - not syncing: Fatal exception
This is a classic ABA problem where cmpxchg succeeds but the state has
changed since __refill_objects_node() took the freelist from the slab.
As Vlastimil Babka mentioned [2], it should be rare to return more than
one slab (due to the racy read of slab->counters in
get_partial_node_bulk()). Therefore, instead of introducing additional
complexity, acquire and release n->list_lock twice in the worst case.
Return the slab directly to the partial list and hold n->list_lock
across the cmpxchg and add_partial(). This is similar to the initial
version of commit ba7425312607 [3]. This is enough to avoid the race as
the list manipulation is serialized by n->list_lock. While at it,
bring back unlikely() hint now that the condition is unlikely.
Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Closes: https://lore.kernel.org/linux-mm/apPa-cGLcyt90l-E@v4bel [1]
Link: https://lore.kernel.org/linux-mm/ae25c193-b95f-40c1-83b6-1c2546467e41@kernel.org [2]
Link: https://lore.kernel.org/all/20260421-b4-refill-optimistic-return-v1-1-24f0bfc1acff@kernel.org [3]
Fixes: ba7425312607 ("mm, slab: add an optimistic __slab_try_return_freelist()")
Cc: stable@vger.kernel.org
Signed-off-by: Harry Yoo (Meta) <harry@kernel.org>
Link: https://patch.msgid.link/20260903-slab-fix-aba-v3-1-b44cb6badd54@kernel.org
Reviewed-by: Hao Li <hao.li@linux.dev>
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
|
|
This partially reverts commit 2a8bb29ec9b2 ("mm/slab: allow
kfree_rcu_sheaf() on PREEMPT_RT"). It was based on the assumption that
local_trylock() is safe on PREEMPT_RT from any context.
However kvfree_rcu() is also called by set_cpus_allowed_force() with
task_struct::pi_lock acquired and there it's not safe, as syzbot
has reported.
For the immediate fix, skip kfree_rcu_sheaf() on PREEMPT_RT again from
kvfree_call_rcu(). In theory, kfree_rcu_nolock() would have the same
problem when called from under pi_lock on PREEMPT_RT but that can
be addressed if such a caller is proposed.
Add an explanation comment, courtesy of Sebastian.
Reported-by: syzbot+acf142088e0182172e58@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=acf142088e0182172e58
Reported-by: ThangNN99 <ngocthang2710.1999@gmail.com>
Fixes: 2a8bb29ec9b2 ("mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT")
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260831-b4-kfree_rcu_hotfix-v1-1-4f0fb882638b@kernel.org
Reviewed-by: Harry Yoo (Meta) <harry@kernel.org>
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
|
|
secretmem accounts folios by treating memory as if it were mlock()'d and
thus limited by the RLIMIT_MEMLOCK limit.
However the folios are unevictable and remain so until the inode is
evicted, eliminating usual mlock() semantics - mapping folios then
unmapping them does not clear their unevictable state, since it depends on
AS_UNEVICTABLE, not PG_mlocked.
A user can therefore easily work around the RLIMIT_MEMLOCK limit - simply
map then unmap and VmLck no longer counts the secretmem range. Worse,
folios are not accounted in the process's RSS, meaning the OOM killer
won't know to kill the process.
Repeatedly mapping/unmapping (or forking) can then result in the
consumption of all available system memory with unevictable folios and
cause system instability.
A secretmem fd can be passed between processes and over fork so a
per-process limit simply does not make sense, so follow the precedent set
by io_uring, perf, skbuff, iommufd and xdp by tracking the number of
locked pages in user_struct->locked_vm.
Since the scope tracked is actually inode lifetime, the RLIMIT_MEMLOCK
applies per-user not per-process, so it doesn't make sense to bypass for
users with CAP_IPC_LOCK, therefore remove this bypass.
There is simply no reason to carry on marking the mapping as mlock()'d
since it's misleading and the lifecycle is now correctly handled, so
remove this too.
Note that secretmem does not support any form of truncation (including
hole punching) and the folios are unreclaimable, so the folios need only
be accounted on fault and unaccounted on inode destruction.
__secretmem_account_pages() is more or less a duplicate of the code that
io_uring etc. use, but since this is a bug fix that needs backporting,
defer any de-duplication efforts to a follow-up.
test_mlock_limit() asserts mlock_future_ok() on mmap(), however this has
been removed, so remove the test altogether for the fix. A new test will
be sent separately for upstream.
Link: https://lore.kernel.org/20260826-secretmem-accounting-v3-1-94cb04399510@kernel.org
Fixes: 1507f51255c9 ("mm: introduce memfd_secret system call to create "secret" memory areas")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reported-by: Daehyeon Ko <4ncienth@gmail.com>
Closes: https://lore.kernel.org/linux-mm/20260813225328.2010303-1-4ncienth@gmail.com/
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Daehyeon Ko <4ncienth@gmail.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: David S. Miller <davem@davemloft.net>
Cc: Hagen Paul Pfeifer <hagen@jauu.net>
Cc: Jakub Kacinski <kuba@kernel.org>
Cc: James Bottomley <james.bottomley@HansenPartnership.com>
Cc: Jesper Dangaard Brouer <hawk@kernel.org>
Cc: John Fastabend <john.fastabend@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Stanislav Fomichev <sdf@fomichev.me>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Uniquely an mremap() invocation using the MREMAP_DONTUNMAP flag can reset
a faulted VMA into an unfaulted one.
It does so after the page tables have been moved to the copied VMA with
MREMAP_DONTUNMAP leaving the old VMA in place which is naturally unfaulted
as the page tables it had are no longer present.
However, in doing so, it violates the invariant that the anonymous page
offset of an unfaulted VMA is vma->vm_start >> PAGE_SHIFT.
This is because a VMA may have been faulted in, mremap()'d (causing a
delta between its page offset and vma->vm_start >> PAGE_SHIFT), and then
mremap()'d again with MREMAP_DONTUNMAP resulting in the unfaulting.
This condition is a violation of a fundamental assumption in mm, but now
also triggers an assert in assert_sane_pgoff() which explicitly checks for
this condition.
Correct it by resetting the VMA's page offset at the point of completing
the MREMAP_DONTUNMAP operation.
Link: https://lore.kernel.org/20260825-fix-mremap-dontunmap-pgoff-v1-1-39a40b2c98b3@kernel.org
Fixes: 1583aa278f5f ("mm: mremap: unlink anon_vmas when mremap with MREMAP_DONTUNMAP success")
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reported-by: syzbot+f12658786a4153df5113@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/6a87853b.ae6ddae5.3da009.0023.GAE@google.com/
Tested-by: syzbot+f12658786a4153df5113@syzkaller.appspotmail.com
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Kunwu Chan <kunwu.chan@gmail.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Li Xinhai <lixinhai.lxh@gmail.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Commit ba23f58de896 ("mm/migrate: don't call
folio_putback_active_hugetlb() on dst hugetlb folio") moved setting of the
migratable flag and active-list placement from
folio_putback_active_hugetlb(dst) into move_hugetlb_state(), so that the
freshly allocated destination folio is handled where allocation is known
to have succeeded.
Unfortunately, the new code was appended after the existing
temporary-folio block in move_hugetlb_state(), which contains an early
return added earlier by commit 5af1ab1d24e08 ("mm/hugetlb: optimize the
surplus state transfer code in move_hugetlb_state()"):
if (folio_test_hugetlb_temporary(new_folio)) {
...
if (new_nid == old_nid)
return; <-- skips the new code
...
}
/* added by ba23f58 */
folio_set_hugetlb_migratable(new_folio);
list_move_tail(&new_folio->lru, ...&h->hugepage_activelist);
When the destination folio is temporary (i.e. the hugetlb pool was
exhausted and the migration callback fell back to
alloc_migrate_hugetlb_folio()) and the migration does not cross a node --
the common case, and always true on a single-NUMA system --
move_hugetlb_state() returns before setting the migratable flag or adding
the new folio to the active list. The destination folio is then installed
in the page table but cannot be isolated afterwards, since
folio_isolate_hugetlb() rejects folios without the migratable flag; a
subsequent soft-offline, hard-offline or memory-hotplug offline of that
folio fails with -EBUSY.
This was reproduced on a single-NUMA arm64 VM: a second MADV_SOFT_OFFLINE
on an already-migrated hugetlb page returned EBUSY and logged "hugepage
isolation failed".
Keep the surplus adjustment, which is the only part that depends on the
node crossing, guarded by `if (new_nid != old_nid)', while making the
migratable flag and active-list placement unconditional. This preserves
the cleanup intent of ba23f58 and closes the early-return hole.
Link: https://lore.kernel.org/20260707110254.3147686-1-mawupeng1@huawei.com
Fixes: ba23f58de896 ("mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio")
Signed-off-by: Wupeng Ma <mawupeng1@huawei.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
syzbot reported a sleeping function called from invalid context splat in
bucket_table_alloc().
When rhashtable_insert_slow() rehashes the table under rcu_read_lock(), it
calls bucket_table_alloc(..., GFP_ATOMIC | __GFP_NOWARN). If the bucket
table allocation uses vmalloc, __vmalloc_node_range_noprof() invokes
vm_area_alloc_pages() -> alloc_pages_bulk_mempolicy_noprof() with the
passed GFP_ATOMIC flags.
If the current task has an MPOL_WEIGHTED_INTERLEAVE mempolicy,
alloc_pages_bulk_weighted_interleave() is called and currently hardcodes
GFP_KERNEL when allocating the temporary weights array, triggering a
might_alloc() splat in atomic/RCU contexts.
Pass the gfp flags (masked with GFP_RECLAIM_MASK to strip page-allocator
zone modifiers like __GFP_HIGHMEM) received by
alloc_pages_bulk_weighted_interleave() to kmalloc() instead of hardcoding
GFP_KERNEL. Since the weights buffer is immediately initialized in full,
kmalloc() is sufficient.
Link: https://lore.kernel.org/20260821170407.3721004-1-edumazet@google.com
Fixes: fa3bea4e1f82 ("mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: syzbot+0dbf6d295b3350944f0b@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/lkml/6a88837e.ae6ddae5.3da009.0040.GAE@google.com/T/#u
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Gregory Price (Meta) <gourry@gourry.net>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
zap_huge_pmd_folio() propagates the pmd young bit to the folio for the
file case, but not the dirty bit. The pte path does propagate it, in
zap_present_folio_ptes() and so does the pmd split path, in
__split_huge_pmd_locked().
For most file mappings the omission is harmless, because writing to a
shared file mapping goes through page_mkwrite(), which dirties the folio.
tmpfs is different: it has no page_mkwrite(), and vma_wants_writenotify()
is false for it, so a *read* fault on a MAP_SHARED tmpfs mapping installs
a writable pmd via do_read_fault(). do_read_fault() does not call
fault_dirty_shared_page(), so subsequent stores through that mapping set
only the hardware dirty bit in the pmd and never call folio_mark_dirty().
A shmem folio allocated by a fault is marked uptodate but not dirty (see
the clear: block in shmem_get_folio_gfp()), so PG_dirty is never set at
all.
Unmapping such a folio - munmap(), or exit_mmap() when the process dies -
then loses the only record that it was written, because zap_huge_pmd()
drops the pmd without transferring the dirty bit. Reclaim afterwards sees
a clean shmem folio: the whole swap-out block in shrink_folio_list() is
inside "if (folio_test_dirty(folio))", so pageout() is skipped and the
folio falls into __remove_mapping(). There, folio_is_file_lru() is false
for a swapbacked folio, so no shadow entry is created and
__filemap_remove_folio(folio, NULL) simply empties the i_pages slot. The
data is freed without ever being written to swap, and the next fault on
that index returns a freshly zeroed folio.
This is silent data loss for any process that keeps state in a MAP_SHARED
tmpfs segment across an unmap - for example a cache handed from one
process generation to the next through /dev/shm. It requires the folio to
be PMD-mapped, so it only shows up once shmem THP is enabled (which is
what we did in Meta fleet and started noticing crashes); with THP off the
pte path transfers the dirty bit correctly. It also only becomes visible
when swap is enabled, because with no swap device shmem folios (which are
on the anon LRU) are not scanned by reclaim at all, so the clean folio is
never dropped.
Reproduced on x86_64 with a tmpfs mounted huge=within_size: read-fault a
2MB-backed region, write a known pattern through the resulting mapping,
munmap, force reclaim of the cgroup, then re-map and read back. Without
this patch the region reads back as zeros and vmstat shows zswpout 0 - the
data was discarded rather than swapped. With this patch the region reads
back correctly and the pages are swapped out as expected. With
huge=never, or when the first touch is a write, the test passes either
way.
Link: https://lore.kernel.org/20260819101222.3732660-1-usama.arif@linux.dev
Fixes: b5072380eb61 ("thp: support file pages in zap_huge_pmd()")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Kiryl Shutsemau <kas@kernel.org>
Acked-by: Hugh Dickins <hughd@google.com>
Tested-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
During move_pages() operation, when move_pages_ptes() returns EAGAIN, the
error code is not cleared even after we processed it. This leads to a
successful retry but then the same pages are retried again due to the
stale error code. This time move fails because pages are already moved,
loop is terminated and move_pages() reports a failure. Clear the error
code once we processes EAGAIN.
Link: https://lore.kernel.org/e1e0b5f8-c3c6-0537-670b-4397f822f980@gmail.com
Fixes: 50944692052b ("userfaultfd: opportunistic TLB-flush batching for present pages in MOVE")
Assisted-by: ChatGPT:GPT-5.6-Luna
Signed-off-by: Bryan Lim <foxieflakey@gmail.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Peter Xu <peterx@redhat.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Lance reported that when nothing else causes the mm to be considered for
khugepaged collapse, an MADV_HUGEPAGE-advised tmpfs VMA alone does not
trigger scanning.
After commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled()
into shmem_allowable_huge_orders()"), the shmem/tmpfs allowable order
check reads vma->flags directly. However, when MADV_HUGEPAGE is handled,
khugepaged_enter_vma() is called before the VMA's flags have been updated,
so the check uses stale flags and incorrectly rejects the VMA for
collapse. As a result, khugepaged does not collapse the tmpfs file into
PMD order in time.
Fix this by calling khugepaged_enter_vma() with the new VMA flags in
madvise_update_vma(). Meanwhile we can remove the khugepaged_enter_vma()
in hugepage_madvise().
Link: https://lore.kernel.org/7d5b5eb27be798f89d563b06254c947ff53db0b2.1787020910.git.baolin.wang@linux.alibaba.com
Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reported-by: Lance Yang <lance.yang@linux.dev>
Closes: https://lore.kernel.org/all/20260815181632.21453-1-lance.yang@linux.dev/
Suggested-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
dissolve_free_hugetlb_folio() can remove a free folio as surplus when its
node has surplus pages. In that case remove_hugetlb_folio() decrements
both nr_huge_pages and surplus_huge_pages, leaving the persistent pool
size unchanged.
Updating max_huge_pages as if a persistent folio had been removed can
therefore corrupt the persistent pool target and underflow it when
max_huge_pages is zero. Keep max_huge_pages unchanged for surplus folios,
including the vmemmap restoration rollback path.
Link: https://lore.kernel.org/20260814083027.1419487-1-xialonglong2025@163.com
Fixes: cb402bbdabca ("mm/hugetlb: fix surplus pages in dissolve_free_huge_page()")
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Longlong Xia <xialonglong@kylinos.cn>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jinjiang Tu <tujinjiang@huawei.com>
Cc: Longlong Xia <xialonglong@kylinos.cn>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
migrate_device_range() and migrate_device_pfns() clear the entries
following a compound folio so that the PFN arrays retain their
page-granular representation.
If a compound folio extends beyond the end of the caller-provided range,
the loops clear all following folio entries without limiting them to the
number of slots remaining in the npages-sized array, causing an
out-of-bounds write.
Do not proceed with a compound folio if its page-granular representation
does not fit entirely in the remaining PFN array. If this happens, drop
any reference and lock acquired for the folio, clear the remaining
entries, and stop collecting.
Observed with a KASAN x86 QEMU kernel using the HMM migrate_anon_huge_zero
selftest. Closing /dev/hmm_dmirror0 after migrating an anonymous huge
page to device memory exercises:
dmirror_fops_release()
-> dmirror_device_evict_chunk()
-> migrate_device_range()
Link: https://lore.kernel.org/20260817120758.669807-3-sh_def@163.com
Fixes: a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
Signed-off-by: Hui Su <sh_def@163.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
hugetlb_cgroup_css_alloc() rounds the counter limit down to a multiple of
the huge page size and then applies it inside an assertion:
VM_BUG_ON(page_counter_set_max(fault, limit));
VM_BUG_ON(page_counter_set_max(rsvd, limit));
With CONFIG_DEBUG_VM=n, VM_BUG_ON(cond) is BUILD_BUG_ON_INVALID(cond),
i.e. ((void)(sizeof((__force long)(cond)))), whose operand is never
evaluated. page_counter_set_max() is not a predicate - it performs
xchg(&counter->max, nr_pages) - so on every non-debug kernel the limit is
never applied and the counters keep page_counter_init()'s
PAGE_COUNTER_MAX.
That is user-visible, because hugetlb_cgroup_read_u64_max() recomputes the
same rounded value and uses equality as its "unlimited" sentinel.
PAGE_COUNTER_MAX is LONG_MAX / PAGE_SIZE = 2251799813685247, which is odd,
so round_down() really does change it and the two sides disagree. With
CONFIG_DEBUG_VM=n:
$ cat /sys/fs/cgroup/t/hugetlb.2MB.max
9223372036854771712
and with this patch:
$ cat /sys/fs/cgroup/t/hugetlb.2MB.max
max
A debug option should not change cgroup output.
Call the function, then assert the result, as v6.12 did. Use
VM_WARN_ON_ONCE() rather than restoring VM_BUG_ON(): the two are identical
under CONFIG_DEBUG_VM=n, and checkpatch asks that new code not use BUG()
variants.
Link: https://lore.kernel.org/20260817103433.191266-1-njilav@gmail.com
Fixes: 0e2759afcaf9 ("page_counter: track failcnt only for legacy cgroups")
Signed-off-by: Narek Jilavyan <njilav@gmail.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The v1 soft limit has been deprecated since v6.12 and nobody has reported
depending on it. Start the removal by decoupling the interface from the
implementation: keep memory.soft_limit_in_bytes, but ignore writes to it
and always report the maximum value on read similar to what
memory.kmem.limit_in_bytes already does.
Writes are still parsed, so malformed input keeps returning -EINVAL. The
knob now also behaves the same everywhere: it used to return -EOPNOTSUPP
on PREEMPT_RT, where soft limit reclaim has always been disabled.
This also fixes the syzbot report linked below. Soft limit reclaim is the
only caller that runs shrink_lruvec() from kswapd against a specific
memcg, so it is the only way to reach lru_gen_shrink_lruvec() and in turn
set_mm_walk(), which warns when called from kswapd.
Link: https://lore.kernel.org/20260811203203.3456029-2-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Reported-by: syzbot+12ee2725d5fde63a9c96@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/6a7a6929.b50370da.49fe0.005e.GAE@google.com/
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
alloc_buddy_hugetlb_folio_with_mpol() can pass a NULL nodemask to
alloc_fresh_hugetlb_folio() as a fallback to allocate from all nodes. If
order is gigantic, alloc_fresh_hugetlb_folio() propagates the NULL
nodemask down to hugetlb_cma_alloc_frozen_folio() via
alloc_gigantic_frozen_folio().
Additionally, hugetlb_cma_alloc_frozen_folio() previously attempted
allocation on hugetlb_cma[nid] without verifying if nid is included in the
caller's nodemask. Adding a node_isset(nid, *nodemask) check ensures the
initial preferred node allocation honors the memory policy / nodemask.
However, hugetlb_cma_alloc_frozen_folio() dereferences the nodemask in
node_isset(nid, *nodemask) and for_each_node_mask(node, *nodemask),
leading to a null pointer dereference kernel panic when nodemask is NULL.
Fix this by checking if nodemask is NULL in
hugetlb_cma_alloc_frozen_folio() and defaulting it to
cpuset_current_mems_allowed. Enclose the allocation attempts within the
cpuset seqcount retry loop so that if the cpuset changes concurrently
during allocation, the attempts are retried using the updated nodemask.
This ensures that the initial node check and fallback loop safely honor
the task's cpuset without violating cpuset constraints or causing NULL
pointer dereferences or unexpected allocation failures.
From a userspace perspective, this bug allows an unprivileged user to
crash the kernel (trigger a panic) by requesting a gigantic hugepage
allocation with MPOL_PREFERRED_MANY on a system where CMA is only
configured on a subset of NUMA nodes.
This can be reproduced by booting a VM with two NUMA nodes, restricting
CMA to Node 1 (e.g., hugetlb_cma=1:1G default_hugepagesz=1G hugepagesz=1G
hugepages=0), and running a program that allocates a 1GB hugepage area
without reserving, restricts allocation to Node 0 using mbind() with
MPOL_PREFERRED_MANY, and triggers a page fault:
void *ptr = mmap(NULL, 1UL << 30, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB |
MAP_HUGE_1GB | MAP_NORESERVE, -1, 0);
unsigned long nodemask = 1; /* Node 0 */
mbind(ptr, 1UL << 30, MPOL_PREFERRED_MANY, &nodemask,
sizeof(nodemask) * 8, 0);
memset(ptr, 0, 1UL << 30); /* Trigger fault */
This results in a NULL pointer dereference:
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: Oops: 0000 [#1] SMP NOPTI
RIP: 0010:hugetlb_cma_alloc_frozen_folio+0x75/0x120
Call Trace:
<TASK>
only_alloc_fresh_hugetlb_folio.isra.0+0x2c/0x160
alloc_surplus_hugetlb_folio+0x6d/0x100
alloc_hugetlb_folio+0x3c5/0x660
hugetlb_no_page+0x3d9/0x650
Link: https://lore.kernel.org/20260811052909.475635-1-souravpanda@google.com
Fixes: eb02f14c4a2b ("mm/hugetlb: allow overcommitting gigantic hugepages")
Signed-off-by: Sourav Panda <souravpanda@google.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Greg Thelen <gthelen@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Rik van Riel <riel@surriel.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
