| Age | Commit message (Collapse) | Author |
|---|
|
DEFINE_IDA() uses IDA_INIT(), which initializes the embedded XArray lock
with a static spinlock initializer. For an automatic local IDA, lockdep
cannot use the lock address as a persistent class key and reports "INFO:
trying to register non-static key" before disabling itself.
Apply ASSERT_STATIC_STORAGE() to DEFINE_IDA() so that this misuse is
rejected at compile time. For example:
void example(void)
{
DEFINE_IDA(ida);
ida_destroy(&ida);
}
GCC reports:
error: initializer element is not constant
name##_storage_check = &(name)
^
note: in expansion of macro 'ASSERT_STATIC_STORAGE'
ASSERT_STATIC_STORAGE(name)
note: in expansion of macro 'DEFINE_IDA'
DEFINE_IDA(ida);
File-scope definitions and static DEFINE_IDA() within a function remain
valid. Automatic local IDAs must instead be initialized with ida_init().
Direct uses of IDA_INIT() are not covered by this declaration check.
Convert the five automatic local IDAs in the userspace radix-tree tests to
ida_init() so they satisfy the new requirement.
Validated file-scope and static local definitions with a kernel object
build, and confirmed that an automatic local definition fails to compile.
Link: https://lore.kernel.org/20260911221444.1523311-3-ynorov@nvidia.com
Signed-off-by: Yury Norov <ynorov@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Alice Ryhl <aliceryhl@google.com>
Cc: Andrew Ballance <andrewjballance@gmail.com>
Cc: Christopher Li <sparse@chrisli.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
|
|
Patch series "Catch automatic storage in IDA and Maple Tree definitions".
A 0day report [1] from the region allocation benchmark exposed a lockdep
initialization bug: a stack-local Maple Tree used MTREE_INIT(), whose
embedded lock has a static initializer. On the first allocation, lockdep
rejected the lock address as a non-static class key and disabled locking
validation. The IDA benchmark had the same issue, masked because it ran
after Maple Tree had already disabled lockdep.
The fix [2] switches the test to using mt_init_flags() and ida_init().
This series adds a compile-time check to the related DEFINE_IDA() and
DEFINE_MTREE() declaration macros to catch the same class of mistake
earlier.
Patch 1 introduces ASSERT_STATIC_STORAGE(). It declares an unused static
pointer initialized with the object's address, requiring that address to
be a valid static initializer. Patches 2 and 3 apply the helper to IDA
and Maple Tree definitions, respectively.
The helper is mirrored in the tools compiler header. The existing
automatic local IDAs and Maple Trees in the userspace radix-tree tests are
converted to runtime initialization. The interval-tree span test keeps
its existing mt_init_flags() call and uses a plain Maple Tree declaration.
For example, an automatic local definition:
void example(void)
{
DEFINE_IDA(ida);
ida_destroy(&ida);
}
now produces:
error: initializer element is not constant
note: in expansion of macro 'ASSERT_STATIC_STORAGE'
note: in expansion of macro 'DEFINE_IDA'
File-scope definitions and static local definitions remain valid.
Automatic local objects should use ida_init(), mt_init(), or
mt_init_flags().
The check is limited to declaration macros. Direct uses of IDA_INIT(),
MTREE_INIT(), and MTREE_INIT_EXT() remain unchanged. The helper cannot be
inserted directly into those initializer expressions because it expands to
a declaration.
Validated by GCC and Clang checks accepting static storage and rejecting
automatic storage The userspace IDR/IDA and Maple Tree test are passed as
well.
This patch (of 3):
Static lock initializers rely on a persistent object address when lockdep
assigns a lock-class key. Using such an initializer for an automatic
local object can compile successfully but disable lockdep on the first
lock acquisition.
Add ASSERT_STATIC_STORAGE() for declaration macros that require static
storage duration. It declares an unused static pointer initialized with
the object's address. An automatic local object's address is not a valid
static initializer, so the compiler rejects it.
Mirror the helper in tools/include/linux/compiler.h because the userspace
radix-tree tests include the kernel IDA and Maple Tree headers with the
tools compiler definitions.
For example:
void example(void)
{
int object;
ASSERT_STATIC_STORAGE(object);
}
GCC reports:
error: initializer element is not constant
name##_storage_check = &(name)
^
note: in expansion of macro 'ASSERT_STATIC_STORAGE'
ASSERT_STATIC_STORAGE(object);
File-scope objects and static local objects remain valid. The helper
takes an object identifier and must be used as a declaration after that
object has been declared.
Link: https://lore.kernel.org/all/20260911155244.1406122-1-ynorov@nvidia.com/
Link: https://lore.kernel.org/20260911221444.1523311-2-ynorov@nvidia.com
Link: https://download.01.org/0day-ci/archive/20260910/202609101106.771b567e-lkp@intel.com/ [1]
Link: https://lore.kernel.org/all/20260911155244.1406122-1-ynorov@nvidia.com/ [2]
Signed-off-by: Yury Norov <ynorov@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Alice Ryhl <aliceryhl@google.com>
Cc: Andrew Ballance <andrewjballance@gmail.com>
Cc: Christopher Li <sparse@chrisli.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
|
|
damon_probe_hits_wsum() is being called only in aggregation time. In
future, it could also be used by DAMOS. In this case, since DAMOS uses
its own apply_interval, it could be called in sampling time. Then using
the not yet fully aggregated probe_hits could result in suboptimum
outcomes. Extend damon_probe_hits_wsum() to get the weighted sum based on
moving sum to prepare the DAMOS usage.
Link: https://lore.kernel.org/20260911135510.96914-3-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
Patch series "mm/damon: introduce probe_hits_wsum DAMOS core filter", v2.
DAMON can do flexible data attributes monitoring. The classical data
access monitoring can also be done using the attributes monitoring. The
access event is just one of the data attributes that DAMON supports.
Users do monitoring to make some actions based on it. DAMOS is a feature
for automating that. However, DAMOS cannot utilize the data attributes
monitoring results. It is still Data "Access" Monitoring-based Operation
Schemes. It requires users to set the target "access" pattern.
DAMOS core filter is effectively the same as the target access pattern.
It is just a more generalized and flexible way of describing the operation
action target region. Introduce a new DAMOS core filter type,
probe_hits_wsum. It specifies the filter target based on a range of the
probe hits weighted sum. Using this, users can apply DAMOS actions to
regions of specific data attributes pattern.
Note that the classic target access pattern still works. Hence the target
nr_accesses range should still be properly configured. The new filter
would be used in only data attributes-only mode. In the mode, classic
access monitoring is just turned off, and therefore nr_accesses of regions
are always zero. Users could simply set the target nr_accesses range to
include the zero nr_Accesses regions.
Patches Sequence
================
Patch 1 updates the DAMON kernel API for the new filter type. Patch 2
extends damon_probe_hits_wsum() to do the calculation based on moving sum.
Patch 3 implements the filter type in the core layer. Patch 4 refactors
DAMON sysfs interface internal data structure for efficient reuse of data
structure for the probe hits weighted sum range user inputs. Patch 5
updates DAMON sysfs interface to support the new filter type. Patches 6
and 7 update design and usage documents for the new filter type,
respectively.
This patch (of 7):
Update DAMON kernel API to introduce new DAMOS core filter type,
PROBE_HITS_WSUM. It will allow API callers to describe the DAMOS action
target regions based on their probe_hits weighted sum. For describing the
filtering target weighted sum range, add two type-dependent union fields
to damos_filter.
[sj@kernel.org: clarify DAMOS_FILTER_TYPE_PROBE_HITS_WSUM behavior]
Link: https://lore.kernel.org/20260911142522.98013-1-sj@kernel.org
Link: https://lore.kernel.org/20260911135510.96914-1-sj@kernel.org
Link: https://lore.kernel.org/20260911135510.96914-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
The device DAX vmemmap population still reserves one extra tail vmemmap
page after the head page.
Drop that extra reservation and let the shared tail page cover all tail
vmemmap pages after the head page, so DAX follows the same reservation
model as HugeTLB.
This reduces the reserved vmemmap pages for optimized DAX mappings to one
and removes the now-unneeded first-tail population from the generic and
powerpc paths to simplify the code as well.
Link: https://lore.kernel.org/20260930140627.57431-11-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Lance Yang <lance.yang@linux.dev>
|
|
The powerpc radix compound vmemmap population path still finds a reusable
tail page by walking the vmemmap page tables.
Switch it to the common vmemmap_shared_tail_page() helper instead, so it
can use the shared vmemmap page directly to simplify the code.
This removes the powerpc-specific tail-page lookup and its fallback path
and aligns the device DAX vmemmap optimization path with HugeTLB.
[muchun.song@linux.dev: use try_get_page()]
Link: https://lore.kernel.org/20260930152826.76084-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20260930140627.57431-10-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Lance Yang <lance.yang@linux.dev>
|
|
The vmemmap optimization helpers currently live in mm/sparse.h, which is
an internal MM header. That works for MM code, but prevents powerpc from
using the same interfaces without including a private header.
Move the declarations and inline helpers to vmemmap-optimization.h. This
is a preparatory change for powerpc, which has its own vmemmap
optimization implementation and needs to use the common vmemmap
optimization interfaces from architecture code.
Link: https://lore.kernel.org/20260930140627.57431-9-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Lance Yang <lance.yang@linux.dev>
|
|
HugeTLB vmemmap optimization now uses per-zone shared tail vmemmap pages.
Device DAX has not been switched to that mechanism yet.
Switch device DAX to vmemmap_shared_tail_page() as well. This aligns DAX
with HugeTLB by using the common per-zone shared tail vmemmap page.
The optimization is enabled only for DEV-DAX through pgmap->vmemmap_shift,
which supplies the compound page order recorded in section metadata before
vmemmap population. Unlike FS-DAX, DEV-DAX does not modify tail struct
pages, so sharing them is safe.
Since the shared tail page can now back ZONE_DEVICE vmemmap mappings,
initialize its entries with PG_reserved for device zones. Also skip
poisoning vmemmap-optimizable sections while their struct pages may be
shared.
Each PTE mapping the shared device DAX tail page takes a page reference.
A sufficiently large range could therefore cycle the reference count back
to zero if population were allowed to continue after it became
non-positive. Use try_get_page() so further mappings fail at that point.
The section population error path tears down mappings created for the
failed section, while the warning makes this currently impractical limit
visible.
[muchun.song@linux.dev: set PG_reserved before preparing the compound tail]
Link: https://lore.kernel.org/20260930150748.1134516-1-songmuchun@bytedance.com
Link: https://lore.kernel.org/20260930140627.57431-8-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Lance Yang <lance.yang@linux.dev>
|
|
The section-based vmemmap optimization infrastructure is guarded by
CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, but it can also be used by
ZONE_DEVICE users that set dev_pagemap::vmemmap_shift. Introduce
CONFIG_VMEMMAP_OPTIMIZATION as a common config for the shared
infrastructure.
Select the new option from HUGETLB_PAGE_OPTIMIZE_VMEMMAP and from
ZONE_DEVICE when the architecture opts in to DAX vmemmap optimization, and
use it to guard the generic sparse-vmemmap state and helpers.
Link: https://lore.kernel.org/20260930140627.57431-4-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Lance Yang <lance.yang@linux.dev>
|
|
Commit 622026e87c40 ("mm/hugetlb: remove fake head pages") added the
per-zone vmemmap_tails array. Its size depends on MAX_FOLIO_ORDER, which
had been moved to mmzone.h in preparation for the array.
PUD_ORDER is defined by linux/pgtable.h, which cannot be included from
mmzone.h without creating an include cycle. It was therefore open-coded
as PUD_SHIFT - PAGE_SHIFT.
This removed the dependency on PUD_ORDER, but not the underlying
dependency on architecture page-table definitions. PUD_SHIFT is generally
provided by architecture page-table headers, which are not guaranteed to
have been included when mmzone.h is parsed.
The dependency remained hidden because vmemmap_tails was originally
guarded by CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP. Under that condition,
MAX_FOLIO_ORDER resolves to either MAX_PAGE_ORDER or the fixed HugeTLB
limit, rather than the PUD_SHIFT-based definition.
Device DAX, however, does not require CONFIG_HUGETLB_PAGE. When it is
converted to use section-based vmemmap optimization, MAX_FOLIO_ORDER can
resolve to PUD_SHIFT - PAGE_SHIFT while it is being used to size
vmemmap_tails. This would make struct zone depend on architecture
page-table definitions being available when mmzone.h is parsed.
Replace the embedded array with a pointer and allocate it on first use.
This moves the order-count evaluation into sparse-vmemmap.c, after the
architecture page-table definitions are available, and removes the
dependency from mmzone.h.
Removing the compile-time array also removes the original reason for
keeping MAX_FOLIO_ORDER and the vmemmap optimization sizing definitions in
mmzone.h. Follow-up cleanups can place each definition in the header
owned by its respective subsystem.
Link: https://lore.kernel.org/20260930140627.57431-3-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Lance Yang <lance.yang@linux.dev>
|
|
Patch series "mm/damon: introduce pgidle_set probe filter type".
Users can do effective access monitoring with DAMON, using set_pgidle
probe prep operation and pgidle_unset allowing probe filter. In the
setup, high probe_hits means the region was accessed frequently. Zero
probe_hits means the region was not accessed. The filter, however,
matches only the memory that can show the idleness. Regions of zero
probe_hits may contain memory that was just unable to show the idleness.
On virtual address space based monitoring, for example, non-present pages
are reported as cold. It is truly cold, but if the purpose is to make
some operations like reclaiming or compressing the data, this is not
really useful. Introduce a new probe filter type, pgidle_set, that
confirms the page is marked as idle. Using it instead of pgidle_unset,
users can find cold pages that can effectively be processed.
Patch 1 introduces the pgidle_set probe filter type to DAMON kernel API.
Patches 2 and 3 add support for it on physical and virtual address space
operation sets, respectively. Patch 4 adds support for it on the DAMON
sysfs interface. Finally, patch 5 updates the documentation for the new
filter type.
This patch (of 5):
DAMON_FILTER_TYPE_PGIDLE_UNSET matches only memory that was able to
confirm if it is marked as idle. If the memory cannot be marked as idle,
it simply doesn't match. Hence, finding memory that was marked as idle
with only DAMON_FILTER_TYPE_PGIDLE_UNSET is impossible. Introduce a new
filter for the purpose, DAMON_FILTER_TYPE_PGIDLE_SET.
Link: https://lore.kernel.org/20260910142234.171562-1-sj@kernel.org
Link: https://lore.kernel.org/20260910142234.171562-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
For MGLRU, memory.min/low is not honored during global proactive reclaim
(writing to the root memory.reclaim) and global direct reclaim, because
these paths shrink memcgs using stale protection (emin/elow). It can be
reproduced as follows:
# echo 7 > /sys/kernel/mm/lru_gen/enabled
# cd /sys/fs/cgroup
# mkdir -p a/b
# echo 100M > a/memory.min
# echo +memory > a/cgroup.subtree_control
# echo 100M > a/b/memory.min
# echo $$ > a/b/cgroup.procs
# dd if=/dev/zero of=/tmp/testfile bs=1M count=200
# cat a/b/memory.current
222650368
# echo 500M > memory.reclaim
-bash: echo: write error: Resource temporarily unavailable
# cat a/b/memory.current
6070272
memory.min is 100M, yet reclaim drops a/b down to 6M, breaking the
protection. The traditional LRU path is not affected because
shrink_node() calls mem_cgroup_calculate_protection() for each memcg it
visits during a top-down tree walk.
Commit 30d77b7eef01 ("mm/mglru: fix ineffective protection calculation")
moved the protection computation into lru_gen_age_node(), which only runs
for kswapd. Non-kswapd global reclaim reaches shrink_one() through
lru_gen_shrink_node() -> shrink_many() without any protection computation,
so emin/elow are whatever a previous kswapd run left behind - or zero if
kswapd never ran on this node. Relying on a prior kswapd pass is not
correct either: a memcg's emin/elow are derived from its ancestors'
memory.min/low settings and from children_min_usage, both of which change
over time, so emin/elow go stale even after kswapd has run and must be
recomputed at the point of reclaim.
Introduce mem_cgroup_calculate_protection_path() which computes emin/elow
along the root-to-target path only, by iterating through the cgroup
ancestors array top-down. This avoids the full tree traversal that would
be needed with mem_cgroup_calculate_protection(), limiting the cost to
O(depth) per memcg - typically 3-5 levels.
Call it from shrink_one() for the non-kswapd path so that each memcg about
to be shrunk has correct protection values.
Link: https://lore.kernel.org/20260907025445.1836238-3-ridong.chen@linux.dev
Fixes: e4dde56cd208 ("mm: multi-gen LRU: per-node lru_gen_folio lists")
Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Assisted-by: Claude:claude-opus-4-8
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Chris Down <chris@chrisdown.name>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Tejun Heo <tj@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Yu Zhao <yuzhao@google.com>
Cc: <stable@vger.kernel.org>
|
|
When mapping /dev/zero with MAP_PRIVATE, one ends up with strange VMAs
originating from Linux's distant past.
These have vma->vm_file set but NULL vma->vm_ops, meaning they satisfy
vma_is_anonymous() but otherwise resemble a file-backed VMA.
The introduction of anonymous page offsets and their subsequent use as
indexes for MAP_PRIVATE-file-backed mappings mean the rmap does the right
thing with these but we are left with inconsistencies.
The vma_start_pgoff(vma) == vma_start_anon_pgoff(vma) invariant is true for
all other anonymous VMAs, but not these.
These VMAs are also observable as files in /proc/<pid>/[maps, smaps,
map_files] but otherwise behave like anonymous mappings.
Therefore let's make these VMAs actually anonymous at mapping time which
will activate the anonymous code path for mappings.
This means we no longer have to account for this discrepancy anywhere and
no longer have to think about these at all.
This is user-observable, as MAP_PRIVATE-/dev/zero will no longer appear in
procfs as a file-backed mapping, but the impact of this change should be
low as likely nobody is relying upon this.
However in any case, in using MAP_PRIVATE-/dev/zero they are explicitly
asking anonymous memory, so no longer seeing these as file mappings is in
fact correct.
A previous commit gave us file_is_dev_zero() to positively identify these
mappings, so we expressly only do so for these alone.
Update assert_sane_pgoff(), the comment for vma_start_pgoff() and
linear_anon_page_index() to reflect the change.
We make this change in call_mmap_prepare() alone as /dev/zero has been
converted to an mmap_prepare hook and we do not permit nested MAP_PRIVATE
mapping of /dev/zero.
We also remove the now defunct vma_desc_set_anonymous() and eliminate the
temporary bisection hazard fix from the previous commit.
Also update the VMA userland tests to reflect the change.
Finally, update the procfs self tests proc-self-map-files-001 and
proc-self-map-files-002 which both intend to map an arbitrary file
MAP_PRIVATE then assert procfs state, but happen to choose /dev/zero.
Fix them by updating these to /proc/self/exe which is guaranteed to be
present if procfs is mounted.
Link: https://lore.kernel.org/20260926-map-private-dev-zero-v3-4-d4781e84ccfc@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Jan Kara <jack@suse.cz>
|
|
There are no users left of the old API, all have switched to SG-list
object read API. Remove it.
Link: https://lore.kernel.org/20260907105739.1793316-3-senozhatsky@chromium.org
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
|
|
Patch series "mm: zswap: optimize zswap invalidate and store", v3.
This series range-ifies zswap_invalidate() to skip xarray lookups when
zswap is unused, and reuses it in zswap_store() to eliminate redundant
per-slot lookups and open-coded logic.
This patch (of 3):
zswap_invalidate() takes a swp_entry_t only to unpack it right back into
type and offset, and both callers already have those values in hand. Pass
type, offset, and nr_entries directly so swap_range_free() can invalidate
an entire range in one call instead of looping in the caller.
Link: https://lore.kernel.org/20260910123544.818146-1-wangkefeng.wang@huawei.com
Link: https://lore.kernel.org/20260910123544.818146-2-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Kairui Song <kasong@tencent.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Yosry Ahmed <yosryahmed@google.com>
|
|
Patch series "mm/swap: skip empty clusters in the swapoff scan", v4.
Speed up swapoff and reduce scanning stalls on large, mostly empty swap
devices by skipping empty clusters. Reduce swapoff time on a 1 TiB device
from 158 ms to 94 ms after filling 128 GiB, and from 392 ms to 73 ms after
filling 512 GiB; expect little benefit when substantial swap data remains.
find_next_to_unuse() walks a swap device one offset at a time. Slot state
now lives in a per cluster swap table, so patch 2 dismisses an empty
cluster with one counter read instead of SWAPFILE_CLUSTER table reads.
Patch 1 is an unrelated one line comment fix noticed on the way.
A debug test confirmed the skip path runs, and swapoff completed under
load with no DEBUG_VM or lockdep splats.
This patch (of 2):
setup_swap_clusters_info() allocates cluster_info for every swap area, not
only for SSDs.
Link: https://lore.kernel.org/20260909161552.2335971-1-youngjun.park@lge.com
Link: https://lore.kernel.org/20260909161552.2335971-2-youngjun.park@lge.com
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
|
|
Now every architecture has been converted to support
CONFIG_MMU_GATHER_RCU_TABLE_FREE, this configuration option no longer
makes any sense to keep around.
Therefore remove it, and remove all the dead code that existed for
!CONFIG_MMU_GATHER_RCU_TABLE_FREE architectures previously.
Additionally, CONFIG_MMU_GATHER_TABLE_FREE is no longer necessary, as all
architectures instead use CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE when a custom
__tlb_remove_table() is required, so remove this too.
A number of architectures only enabled CONFIG_MMU_GATHER_RCU_TABLE_FREE if
CONFIG_MMU was set, however the mmu_gather logic only actually does
something meaningful if CONFIG_MMU is set (mmu_gather.c is only compiled
in this case, for instance).
As a result, there's no need to gate any of this logic on CONFIG_MMU
explicitly.
CONFIG_PT_RECLAIM however does have a strict dependency on CONFIG_MMU, so
make this dependency explicit.
Additionally, correct comments to remove references to non-RCU page table
gathering and make it clear that this is not 'semi-RCU', nor has it been
since commit 1fb3d8c20bfa ("mm/mmu_gather: replace IPI with
synchronize_rcu() when batch allocation fails").
With this change in place the kernel policy is now that userspace page
tables are freed after an RCU grace period, and thus it is now safe to
unconditionally perform page table walks under RCU, safe in the knowledge
that page tables will not be freed underneath the walker.
This is all that is guaranteed, however, so naturally it is still
incumbent upon page table walkers to ensure that the page table entries
are as expected.
Link: https://lore.kernel.org/20260925-rcu-pagetable-freeing-v5-11-31e91065fea4@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Guo Ren <guoren@kernel.org>
Cc: Brian Cain <bcain@kernel.org>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Dinh Nguyen <dinguyen@kernel.org>
Cc: Simon Schuster <schuster.simon@siemens-energy.com>
Cc: Jonas Bonn <jonas@southpole.se>
Cc: Stefan Kristiansson <stefan.kristiansson@saunalahti.fi>
Cc: Stafford Horne <shorne@gmail.com>
Cc: Rich Felker <dalias@libc.org>
Cc: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Cc: Paul Walmsley <pjw@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Vineet Gupta <vgupta@kernel.org>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Chris Zankel <chris@zankel.net>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Will Deacon <will@kernel.org>
Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: David S. Miller <davem@davemloft.net>
Cc: Andreas Larsson <andreas@gaisler.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Magnus Lindholm <linmag7@gmail.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: WANG Xuerui <kernel@xen0n.name>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Cc: James Bottomley <james.bottomley@hansenpartnership.com>
Cc: Helge Deller <deller@gmx.de>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Richard Weinberger <richard@nod.at>
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com>
Cc: Johannes Berg <johannes@sipsolutions.net>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
|
|
All callers have been converted to folio_swap_entry() and
folio_page_swap_entry(), so remove it.
Link: https://lore.kernel.org/20260908-folio_swap_entry-v2-8-ee6d01dfa5e1@columbia.edu
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Marc Rutland <mark.rutland@arm.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
|
|
Patch series "mm: remove page_swap_entry()", v2.
A folio in the swap cache occupies folio_nr_pages() contiguous swap
entries starting at folio->swap, so a page's swap entry is just
folio->swap plus the page's index in the folio. The swap entry is folio
state, but the only helper for it is page-based: page_swap_entry() takes a
page and recomputes the folio its callers already hold. Several callers
avoid it by open-coding the arithmetic on folio->swap instead.
Add folio_swap_entry() and folio_page_swap_entry(), convert all users, and
remove page_swap_entry(), with a few other cleanups along the way.
This patch (of 8):
A folio in the swap cache occupies folio_nr_pages() contiguous swap
entries starting at folio->swap, so a page's swap entry is just
folio->swap plus the page's index in the folio. page_swap_entry() hides
this behind a compound_head() call, and callers that already have the
folio sometimes open-code the arithmetic instead.
Add folio_swap_entry(), which takes a folio and a page index, and
folio_page_swap_entry() for callers that have the page.
[tz2294@columbia.edu: adjust folio_swap_entry() kerneldoc summary, per David]
Link: https://lore.kernel.org/20260913-folio_swap_entry-doc-fix-1@columbia.edu
Link: https://lore.kernel.org/20260908-folio_swap_entry-v2-0-ee6d01dfa5e1@columbia.edu
Link: https://lore.kernel.org/20260908-folio_swap_entry-v2-1-ee6d01dfa5e1@columbia.edu
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Marc Rutland <mark.rutland@arm.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
|
|
... and use it in hid-core, the only place with a cleanup for a vmalloc()
allocation.
Link: https://lore.kernel.org/20260903-execmem-rox-cache-pmd-v1-v1-4-11beb2a3d249@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Tissoires <bentiss@kernel.org>
Cc: Jiri Kosina <jikos@kernel.org>
Cc: Luis Chamberalin <mcgrof@kernel.org>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
|
|
The ROX cache relies on its chunks being PMD mapped. For a PMD mapped
chunk, set_memory_rox() updates the direct map alias one PMD at a time and
the large mappings there survive.
When execmem refills the cache, it rounds up the requested allocation size
to PMD_SIZE and tries to allocate that with vmalloc(VM_ALLOW_HUGE_VMAP).
If that allocation fails, execmem falls back to vmalloc() of the original
size.
There are two issues with this approach:
* If huge pages are not available, __vmalloc_node_range() silently falls
back to base pages. The area execmem gets is virtually contiguous, but it
is backed by 512 scattered base pages.
Permission updates on such areas split large mappings in the direct map
that contain those base pages, up to 512 PMD splits in the worst case.
* execmem's own fallback adds a small base page mapped area to the cache.
This adds the overhead of cache management to these allocations with no
benefit of reducing fragmentation either in the vmalloc/modules address
space or in the direct map.
Worse, these areas are never freed from the cache, because
execmem_cache_clean() releases only chunks that are a multiple of
PMD_SIZE and aligned to PMD_SIZE, exactly to minimize the number of base
page mappings.
Add VM_REQUIRE_HUGE_VMAP option to vmalloc that fails if the allocation of
huge pages fails or if such an allocation is not possible because huge
page allocations in vmalloc were disabled or the architecture does not
support them.
Use this option when populating the ROX cache. If
vmalloc(VM_REQUIRE_HUGE_VMAP) fails or vmalloc of huge pages is
unavailable, handle the memory allocation outside the ROX cache with plain
vmalloc().
Since the fallback allocation has to return ROX memory, add an
execmem_alloc_rox() helper and use it for both populating the ROX cache
and dealing with a fallback allocation in a ROX execmem_range.
With that, the cache only ever contains PMD aligned chunks sized as a
multiple of PMD_SIZE, and the PMD checks in execmem_cache_clean() become a
VM_WARN_ON_ONCE() to ensure that the PMD mapping invariant does not
change.
Link: https://lore.kernel.org/20260903-execmem-rox-cache-pmd-v1-v1-3-11beb2a3d249@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Assisted-by: copilot:claude-opus-5
Cc: Benjamin Tissoires <bentiss@kernel.org>
Cc: Jiri Kosina <jikos@kernel.org>
Cc: Luis Chamberalin <mcgrof@kernel.org>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
|
|
memcg_vmstats keeps a non-hierarchical copy of every memcg-wide stat item
and event alongside the hierarchical one. The only readers however are
the legacy memory.stat and memory.numa_stat, and reparenting on offline.
All of them live under CONFIG_MEMCG_V1, and their accessors
(memcg_page_state_local() and memcg_events_local()) are already compiled
out with it. This means on a CONFIG_MEMCG_V1=n kernel, memcg_vmstats's
non-hierarchical arrays are written to on every rstat flush, despite their
values never being read / accessed.
The same holds when the kernel does support v1 but the controller has been
blocked from v1 hierarchies with the boot param cgroup_no_v1={memory,all}.
A v1 mount is refused in that case, so the legacy memory.stat can never
exist and the arrays are just as unread / unaccessed.
Compile out the non-hierarchical memcg-wide arrays if CONFIG_MEMCG_V1 is
not set. If it is set but cgroup_no_v1= has blocked the controller, skip
updates on the arrays.
This makes flushes cheaper. mem_cgroup_stat_aggregate() can now skip the
read-modify-write of ac->local[i]. Nothing else accesses state_local or
events_local, so those cachelines were getting pulled in solely for the
writes and they are separate from the ones the loop is already walking.
On an 80-cpu x86_64 machine with 500 cgroups each running a workload that
dirties anon, file, dirty/writeback, slab, kmem, mlock, and reclaim
counters, timing mem_cgroup_css_rstat_flush() in-kernel in TSC ticks per
flush showed roughly
before after delta
memcg-wide aggregation 1231 1180 -4.1%
overall flush function 2452 2397 -2.2%
These numbers are from taking the median of 70 samples, one per 20s window
on each kernel. The 95% intervals observed on the two deltas are [-5.41%,
-1.76%] and [-4.53%, -0.14%]. The values above include the timing
overhead itself, so only the delta is meaningful here.
Counting the items that actually changed, a median of 1.5 of the 77
memcg-wide items (57 state + 20 events) had a non-zero per-cpu delta at
each flush, which means the benchmarks above are with one or two fewer
cachelines pulled in per flush. The count is low because the benchmark
reads memory.stat in a loop to keep the flush rate up. For cases where
flushes are triggered only by the 2s periodic worker, more changes will
have accumulated between flushes, so more cachelines are skipped and the
per-flush saving should be larger.
Link: https://lore.kernel.org/20260903215616.1456239-1-joannelkoong@gmail.com
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
Memcg charging can be done from any context, but calling schedule_work()
isn't safe from an NMI. If memory.high is breached from a context where
spinning isn't allowed, use irq_work to schedule the reclaim work.
Found this via code inspection. I spent a little bit trying to trigger it
for real, but the only way I managed was by writing a hacky driver absuing
alloc_pages_nolock().
Link: https://lore.kernel.org/20260904173145.2028377-1-stevensd@google.com
Fixes: 3ac4638a734a ("memcg: make memcg_rstat_updated nmi safe")
Signed-off-by: David Stevens <stevensd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
Replace the two ad-hoc CACHELINE_PADDING members with named cache line
groups:
memcg_pn_read_mostly memcg, lruvec_stats_percpu, lruvec_stats,
shrinker_info, objcg
memcg_pn_lruvec lruvec
memcg_pn_write_hot lru_zone_size, iter, nmi slab stats
memcg_pn_cold orig_objcg, objcg_list
The group markers give the same isolation the padding did, but they are
named and the build now checks them.
lruvec still gets its own lines. Commit f59adcf59332 ("mm: memcg: add
cacheline padding after lruvec in mem_cgroup_per_node") showed why that
matters: lru_zone_size[] is written under lru_lock but read without it by
lruvec_lru_size(), so it must not share a line with lruvec.
Splitting the cold fields out costs one extra cache line per node per
memcg.
No functional change.
Link: https://lore.kernel.org/20260905030522.1887837-7-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
These fields are set when the memcg is created and only read after
that:
vmstats_percpu read on every stat update
vmstats
zswap_max, zswap_writeback
private_id read on every eviction and refault
kmemcg_id read on every list_lru lookup
oom_group
Put them in one cache line group at the end of the struct, right before
nodeinfo[]. nodeinfo[] is read-mostly too but it is a flexible array, so
it cannot sit inside a group. The group ends without padding so the two
share a line.
This also drops the ____cacheline_aligned_in_smp on vmstats_percpu added
by commit c1afbd5de131 ("mm/memcontrol: avoid false sharing between
vmstats and events"). That only aligned the start of the field.
cgwb_domain followed it on the same line and is written on every writeout
completion. A group boundary covers both sides.
No functional change.
Link: https://lore.kernel.org/20260905030522.1887837-6-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
These fields are only touched by the cgroup control paths:
memory_peaks, swap_peaks, peaks_lock memory.peak open/read/release
events_file, events_local_file,
swap_events_file cgroup_file_notify()
cgwb_list, cgwb_domain, cgwb_frn writeback setup and the
foreign dirty slow path
mm_list MGLRU mm list
They sit in the middle of the struct today. The three cgroup_file members
alone are 192 bytes of notify state next to the vmstats pointer. Put them
in one cache line group.
No functional change.
Link: https://lore.kernel.org/20260905030522.1887837-5-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
These fields are written on the charge, reclaim and socket paths:
socket_pressure written by reclaim, read on every socket charge
memory_events bumped for this memcg and every ancestor, so a
busy child dirties the whole chain
memory_events_local
vmpressure written on every reclaim iteration
private_id_ref written on every swap charge and uncharge
kmem_stat
high_irq_work, high_work
They are spread over the struct today and share cache lines with
read-mostly fields. Put them in one cache line group.
socket_pressure is kept next to memory_events because
mem_cgroup_sk_under_memory_pressure() reads one and bumps the other.
Add memcg_struct_check() so the build fails if a field lands outside
its group.
No functional change.
Link: https://lore.kernel.org/20260905030522.1887837-4-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
The two members of struct mem_cgroup_private_id have different access
patterns. The id is read on every eviction and refault through
mem_cgroup_private_id(), and is only written when the memcg is created and
destroyed. The ref is written on every swap charge and uncharge.
Split them into private_id and private_id_ref so a later patch can put
them into different cache line groups. A struct member cannot be split
across two groups.
No functional change.
Link: https://lore.kernel.org/20260905030522.1887837-3-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
Patch series "memcg: group struct fields by access pattern".
Every so often we get a memcg performance regression caused by nothing
more than a field moving. Someone adds a field, removes one, or puts a
few behind a config option. The layout shifts, fields with different
access patterns land on the same cache line, and a bot reports a
regression.
Two examples. commit 98c9daf5ae6b ("mm: memcg: guard memcg1-specific
members of struct mem_cgroup_per_node") moved lruvec next to
lru_zone_size[] and needed commit f59adcf59332 ("mm: memcg: add cacheline
padding after lruvec in mem_cgroup_per_node") to fix it. commit
c1afbd5de131 ("mm/memcontrol: avoid false sharing between vmstats and
events") had to add ____cacheline_aligned_in_smp for the same reason.
Each fix was correct but nothing stops the next field addition from
undoing it.
This series makes the layout a contract the compiler checks, the same
way struct net_device does it. Fields are sorted into named cache line
groups by access pattern, and memcg_struct_check() verifies at build
time that every field sits in its group. A field added in the wrong
place now breaks the build instead of quietly costing a few percent.
struct mem_cgroup gets three groups:
memcg_write_hot written on the charge, reclaim and socket paths
memcg_cold only the cgroup control paths touch these
memcg_read_mostly set when the memcg is created, then only read
Testing
=======
The cgroup selftests give identical results with and without the series.
For performance, two identical 30 core Xeon machines each ran both
kernels, with the boot order swapped between them so that machine and
order effects cancel. The useful tests run two workloads at once in one
cgroup, because false sharing only shows up when one side reads a field
that the other side writes.
slab allocs + page faults, slab side +1.2%
page faults + memory.stat readers, fault side +1.3%
page faults + memory.stat readers, reader side +0.8%
everything else no change
No test regressed. The gains are small but the point of the series is the
build time contract.
This patch (of 6):
current_obj_cgroup() reads memcg->nodeinfo[nid]->objcg on every accounted
allocation. The field sits at the end of struct mem_cgroup_per_node, on
the same cache line as lru_zone_size[] and iter. lru_zone_size[] is
written on every LRU add and remove, and iter is written on every reclaim
iteration.
Move objcg next to the other read-mostly pointers at the start of the
struct.
No functional change.
Link: https://lore.kernel.org/20260905030522.1887837-1-shakeel.butt@linux.dev
Link: https://lore.kernel.org/20260905030522.1887837-2-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
Each generation of MGLRU accounts anon and file folio numbers separately,
and the page table walker updates each generation's counters in batch once
the walk is done. The walker promotes a folio's generation with a cmpxchg
on folio->flags, and update_batch_size() then reads the live flags again
to pick the anon/file column to charge. The walk holds neither the lruvec
lock nor the folio lock, so the type can flip between the cmpxchg and that
read: the lazyfree path clears PG_swapbacked, and reclaim sets it back on
a dirty lazyfree folio. The batched delta pair is then recorded in the
wrong type column. Nothing reconciles it afterwards, permanently skewing
lrugen->nr_pages and the reclaim budgets derived from it.
Fix it by capturing the type from the flags snapshot the cmpxchg
linearized against: folio_update_gen() returns the type of the state it
transitioned from, and update_batch_size() accounts with that.
A folio's type only changes while it is off the LRU list, inside a del/add
pair under the lruvec lock, with the gen bits cleared in between. The
generation and PG_swapbacked sit in the same folio->flags word, so the
cmpxchg snapshot captures them together. Let G be the generation that
snapshot captured (old_gen) and G' the one it wrote (new_gen); the CAS can
land in only three places:
- before the del: the folio is anon at G; the batch records anon
G -> G', and the del later removes the folio from the anon
counters;
- between del and add: gen == -1, so folio_update_gen() returns -1
without touching the flags and no batch is recorded; the del/add
pair accounts for the move alone;
- after the add: the folio is file at the fresh generation the add
charged; the batch records file, that gen -> G', matching that
charge.
Unlike the drift of lazy promotions, which sort_folio() repairs under the
lruvec lock, the phantom deltas from before this fix land in a column the
folio never occupies again, so nothing ever repairs them.
Link: https://lore.kernel.org/20260906-mglru-flags-cleanup-v6-6-9aacbd77d4ca@tencent.com
Fixes: bd74fdaea146 ("mm: multi-gen LRU: support page table walks")
Signed-off-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lian Wang <lianux.mm@gmail.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Ridong Chen <ridong.chen@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Yu Zhao <yuzhao@google.com>
Cc: Zi Yan <ziy@nvidia.com>
|
|
read_ctrl_pos() encodes the tier range in a single "tier" parameter via
"tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)" as the
end. This is hard to follow, maintain, or extend. Tier values 0..3
select a single tier, while tier == MAX_NR_TIERS selects the full range.
Replace it with explicit (tier_min, tier_max) parameters using a closed
[tier_min, tier_max] interval, and add LRU_TIER_MIN and LRU_TIER_MAX for
the tier bounds. The call sites now become self-documenting:
- get_tier_idx: (LRU_TIER_MIN, LRU_TIER_MIN) for the first tier,
(tier, tier) for each subsequent tier
- get_type_to_scan: (LRU_TIER_MIN, LRU_TIER_MAX) for the full range
No functional change.
Link: https://lore.kernel.org/20260906-mglru-flags-cleanup-v6-5-9aacbd77d4ca@tencent.com
Signed-off-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lian Wang <lianux.mm@gmail.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Yu Zhao <yuzhao@google.com>
Cc: Zi Yan <ziy@nvidia.com>
|
|
folio_migrate_flags() copies PG_referenced separately from the MGLRU refs
counter, which folio_migrate_refs() transfers. Yet under MGLRU,
PG_referenced and the refs counter bits together describe the referenced
status of a folio.
Consolidate the two: rename folio_migrate_refs() to
folio_migrate_lru_refs() and let it copy the complete referenced status,
i.e., the MGLRU refs count including PG_referenced, or just PG_referenced
for the active/inactive LRU. Drop the open-coded PG_referenced copy so
the referenced status is transferred in one place. No behavior change is
intended: under the active/inactive LRU the extra bits are unused, so
operating on them is a noop.
Transfer the reference state first, before the destination folio is marked
up to date, as a best effort to retain referenced status.
Link: https://lore.kernel.org/20260906-mglru-flags-cleanup-v6-3-9aacbd77d4ca@tencent.com
Signed-off-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lian Wang <lianux.mm@gmail.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Yu Zhao <yuzhao@google.com>
Cc: Zi Yan <ziy@nvidia.com>
|
|
Instead of doing bit ops on folio->flags.f, introduce helpers for
adjusting a folio's refs and generation info, making the code easier to
debug and understand.
No functional change is intended: some combined atomic operations are
split into two, which only creates harmless transient states. There is no
measurable performance impact, and some paths even look slightly better in
the generated assembly.
Link: https://lore.kernel.org/20260906-mglru-flags-cleanup-v6-2-9aacbd77d4ca@tencent.com
Signed-off-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lian Wang <lianux.mm@gmail.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Ridong Chen <ridong.chen@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Yu Zhao <yuzhao@google.com>
Cc: Zi Yan <ziy@nvidia.com>
|
|
Patch series "mm/mglru: clean up folio counters and flag usage", v6.
This is a cleanup series separated out from the MGLRU-FG series [1]. As
that series is getting too long in following updates, separate out the
clean up part for easier review and merge.
No feature change is intended, except one bugfix. It mostly replaces the
open-coded bit operations scattered throughout the MGLRU code with new
helpers, with proper kdocs, sanity debug checks, and hardens a few MGLRU
functions.
A subtle generation counter leak is also found during the refactoring and
the fix is included.
Also collected review feedbacks on the cleanup part from the posted
series.
This patch (of 6):
Instead of using an unsigned long and checking the counter value at the
updater side, turn the counter into a signed long and check at the reader
side. This reduces overhead and simplifies the code.
commit ca707239e8a7 ("mm: update_lru_size warn and reset bad lru_size")
added a sanity check for memcg counter underflow: lru_zone_size is
unsigned, so an underflow wraps it around and returns an enormously large
number, then the memcg shrinker loops almost forever as the calculated
number of folios to shrink is huge. It also checked if a zero value
matches the empty LRU list, so the positive and negative deltas had to be
handled separately. However that emptiness check was already removed by
commit b4536f0c829c ("mm, memcg: fix the active list aging for lowmem
requests when memcg is enabled"), so handling the deltas separately is no
longer needed.
The remaining update-side check is costly and cannot really catch the leak
it is after anyway. It runs on every LRU folio, and if a folio was
removed without updating the counter while other folios remain on the LRU,
the WARN only triggers much later, from a likely innocent callsite. While
readers are much rarer than writers, only the reclaim and reparenting
paths read it, once per batch.
Checking at the reader side instead leaves the update path a plain
addition, and puts the warning where the value is actually consumed.
Note this changes the behavior on underflow: the correction is removed and
a negative value is kept. A massive leak of the LRU size counter would
indicate that something else has gone very wrong, and one should fix that
leaking site instead. Besides, the original behavior might cause false
positives, or make things worse if the accounting happens after the actual
insertion: the value is not leaked, just delayed, so force-fixing it would
cause a bigger problem. The warning now only kicks in when a consumer
actually uses it, in which case the reader gets zero.
Link: https://lore.kernel.org/20260906-mglru-flags-cleanup-v6-0-9aacbd77d4ca@tencent.com
Link: https://lore.kernel.org/20260906-mglru-flags-cleanup-v6-1-9aacbd77d4ca@tencent.com
Link: https://lore.kernel.org/linux-mm/20260804-mglru-fg-v1-0-4d8dad39dad6@tencent.com/ [1]
Signed-off-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
Reviewed-by: Barry Song <baohua@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Yu Zhao <yuzhao@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Lian Wang <lianux.mm@gmail.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
|
|
The last caller of PageWriteback() was removed in commit f75987e543c2
("libceph: remove pinning assertion in ceph_msg_data_iter_next()"). This
flag is now only used on folios, so we can remove all the page accessors.
folio_test_clear_writeback() is not used, so don't add
FOLIO_TEST_CLEAR_FLAG() for it.
Link: https://lore.kernel.org/20260905-remove-pagewriteback-v1-1-06f41c00db03@columbia.edu
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
Page flag helpers for configuration-dependent flags provide false or no-op
variants so that their users can be independent of the configuration.
The HWPoison helpers do not fully follow this pattern. When
CONFIG_MEMORY_FAILURE is enabled, PAGEFLAG() and TESTSCFLAG() provide the
regular, test-and-set, and test-and-clear operations. When it is
disabled, only PAGEFLAG_FALSE() is instantiated, leaving
TestSetPageHWPoison() and TestClearPageHWPoison() undefined.
Use TESTSCFLAG_FALSE() to provide the missing accessors when memory
failure handling is disabled. Both accessors return false, which is
consistent with HWPoison state being unavailable, and makes the accessor
interface consistent across configurations.
Also remove now-unneeded test_and_clear_pmem_poison() from nvdimm/pmem.
Link: https://lore.kernel.org/20260903053535.17611-1-kaitao.cheng@linux.dev
Link: https://lore.kernel.org/20260903053535.17611-2-kaitao.cheng@linux.dev
Link: https://lore.kernel.org/20260903053535.17611-3-kaitao.cheng@linux.dev
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Oscar Salvador <osalvador@suse.de>
Cc: Alison Schofield <alison.schofield@intel.com>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Ira Weiny <iweiny@kernel.org>
|
|
HugeTLB folios are currently charged to the memcg of the allocating task.
This gives the wrong result when a userfaultfd handler populates a HugeTLB
VMA that belongs to another process. The UFFDIO_COPY ioctl operates on
the userfaultfd context's mm, but get_mem_cgroup_from_current() charges
the folio to the handler's memcg instead.
This can be reproduced by placing the faulting process and its userfaultfd
handler in different memory cgroups. Have the target process register a
HugeTLB mapping with userfaultfd, trigger a missing fault, and let the
handler resolve it with UFFDIO_COPY. The hugepage usage is then reported
in the handler's memory.current instead of the target's.
The generic userfaultfd population path avoids this problem by charging
folios to dst_vma->vm_mm.
Pass the target mm through hugetlb_alloc_folio() and charge the folio by
using get_mem_cgroup_from_mm(). This preserves the existing charge timing
and error handling while making HugeTLB userfaultfd population consistent
with the generic path.
Link: https://lore.kernel.org/20260903075048.3316-1-zhoujinmeng@bytedance.com
Fixes: 8cba9576df60 ("hugetlb: memcg: account hugetlb-backed memory in memory controller")
Signed-off-by: Jinmeng Zhou <zhoujinmeng@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Hongfu Li <lihongfu@kylinos.cn>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: <stable@vger.kernel.org>
|
|
Protection keys are encoded in architecture-specific HIGH_ARCH_* VMA flag
bits. show_vma_flags(), which is used by VMA tracepoints and %pGv, does
not name those bits, leaving them as raw hexadecimal values.
Name the protection-key encoding bits pkey_bit0 through pkey_bit4 under
CONFIG_ARCH_HAS_PKEYS. The names make clear that these are bits of one
protection-key value rather than independent protection keys. For
example, protection key 3 is represented as:
pkey_bit0|pkey_bit1
Honor CONFIG_ARCH_PKEY_BITS when exposing bit 3 and bit 4 so only bits
provided by the architecture are included.
Link: https://lore.kernel.org/1871da32001243b37147bac5af02d2f2ec64304e.1788330432.git.zhaomeijing@lixiang.com
Signed-off-by: Meijing Zhao <zhaomeijing@lixiang.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
|
|
show_vma_flags(), which is used by VMA tracepoints and %pGv, leaves
architecture-specific HIGH_ARCH_* bits unnamed. Arm64 MTE flags and user
shadow stack flags are therefore printed as raw hexadecimal values.
These bit positions are shared between architectures. For example, bit 37
represents VM_MTE_ALLOWED on arm64 but VM_SHADOW_STACK on x86, so a shared
HIGH_ARCH_* bit cannot be given an unconditional name.
Add conditionally compiled names for VM_MTE, VM_MTE_ALLOWED and
VM_SHADOW_STACK. Keep the configuration guards aligned with the
definitions of these aliases so each shared bit position is decoded
according to the target architecture.
For example, an arm64 VMA containing VM_MTE_ALLOWED is now printed as:
...|account|mte_allowed|...
instead of:
...|account|0x2000000000
Link: https://lore.kernel.org/1c7f7003cff8d00209baa8498f1efee113f5b5a4.1788330432.git.zhaomeijing@lixiang.com
Signed-off-by: Meijing Zhao <zhaomeijing@lixiang.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
|
|
Patch series "mm: trace: decode architecture-specific VMA flags", v2.
show_vma_flags(), which is used by VMA tracepoints and %pGv, names generic
VM_* bits but does not fully decode architecture-specific flag positions.
As a result, trace output and VMA dumps can show a generic "arch_1" name
or a raw hexadecimal value instead of the meaning assigned by the target
architecture.
This series adds symbolic names for the arm64 and sparc64 meanings of
VM_ARCH_1, arm64 MTE flags, user shadow stack flags, and protection-key
encoding bits under their corresponding configuration guards.
This patch (of 3):
The VM_ARCH_1 bit has architecture-specific meanings. show_vma_flags(),
which is used by VMA tracepoints and %pGv, already reports the powerpc,
parisc and no-MMU meanings, but falls back to the generic "arch_1" name on
arm64 and sparc64.
Report VM_ARM64_BTI as "bti" and VM_SPARC_ADI as "adi" so trace output and
%pGv dumps expose the actual architecture-specific state.
Link: https://lore.kernel.org/cover.1788330432.git.zhaomeijing@lixiang.com
Link: https://lore.kernel.org/bcf0fea58240c4b6daa253739683ed7709e42fbb.1788330432.git.zhaomeijing@lixiang.com
Signed-off-by: Meijing Zhao <zhaomeijing@lixiang.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
|
|
Cross-merge networking fixes after downstream PR (net-7.3-rc7).
Conflicts:
include/net/strparser.h
0984ebc631792 ("strparser: make sure __strp_recv isn't running before tearing down the parser")
02fd0a2111374 ("net: strparser: removed the aborted bit and aborts counter")
https://lore.kernel.org/asYfmLsfy-SCxXjr@sirena.co.uk
Adjacent changes:
Documentation/networking/strparser.rst
5f193cdabf25 ("docs: networking: strparser: remove general mode")
0984ebc63179 ("strparser: make sure __strp_recv isn't running before tearing down the parser")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
Pull networking fixes from Jakub Kicinski:
"Including fixes from wireless, wireguard, CAN and Bluetooth.
We have one known regression to wrap up in VLAN handling.
Current release - regressions:
- Bluetooth: RFCOMM: fix deadlock on rfcomm_mutex
Previous releases - regressions:
- can: fix regression in handling RPS after migrating metadata to skb_ext
- eth:
- iavf: fix regressions in reconfig impacting bonding
- mana: fix packet forwarding performance regression
- stmmac: remove buggy VLAN acceleration support
Previous releases - always broken:
- a few high prio fixes for tun, and af_packet
- amt: fix a UaF on tunnel teardown
- eth:
- bnxt: fix PCIe AER recovery and FLR handling issues
- macb: don't modify Tx skbs before taking ownership
- axienet: don't leak Tx skbs on interface stop
- wifi:
- nxpwifi: number of LLM-ish fixes
- assorted mt76 fixes"
* tag 'net-7.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net: (128 commits)
net: macb: copy shared skbs before appending the FCS
net: macb: check TX ring before modifying skb
vsock: Fix memory leak in vmci_transport_recv_dgram_cb()
wireguard: noise: reject response consumption after intermediate initiation
wireguard: queueing: preserve tstamp_type when encapsulating packet
net: openvswitch: validate transport header presence in set_ipv6_addr
net/smc: protect clcsock lifetime in smc_getname
ipv6: do not warn on route notification size race
ipv4: do not warn on route notification size race
ipv4: validate checksum_start before completing checksum
ptp: ocp: fix PCIe delay estimation calculation
xen/netfront: don't leak the skb when xennet_fill_frags() fails
net/packet: call packet_parse_headers after virtio_net_hdr_to_skb
xen/netfront: drop RX packets with a short Ethernet header
net: skbuff: don't leave stale bytes in skb_copy_and_csum_bits()
net: sparx5: free the matchall entry on destroy
selftests: mlxsw: Test port range occupancy on template create
mlxsw: spectrum_flower: Fix port range register leak in tmplt_create()
net: dsa: microchip: fix KSZ8765 fiber detection
net/mlx5e: Order ICOSQ cc update after CQ doorbell
...
|
|
The sm_tx_state field of struct port and tx_states_t enum
were introduced in the initial 802.3ad implementation but
have never had any users.
The TX machine uses ntt and sm_tx_timer_counter instead.
Remove them.
Signed-off-by: Timur Stukan <timur.stukan.linux@gmail.com>
Reviewed-by: Fernando Fernandez Mancera <fmancera@suse.de>
Link: https://patch.msgid.link/20261006103207.68974-1-timur.stukan.linux@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
USB tethering on the OnePlus Nord CE 2 (IV2201, USB ID 22d9:2766)
running Android 13 has download speeds of approximately 2mbps
with the default 2048-byte receive limit.
The phone's published RNDIS code only enables packet batching
when the computer reports a receive limit above 2048 bytes.
Testing shows that increasing this limit to 2560 bytes
restores normal speeds.
Add a quirk for USB device 22d9:2766 to use and advertise a
receive buffer of at least 16 KiB. This fits the phone code's default
batch of ten packets at a 1500-byte MTU.
Tested on 7.3.0-rc5 with the affected phone, including reconnecting
USB tethering. Download speeds with the change exceed 100mpbs.
Link: https://github.com/OnePlusOSS/android_kernel_oneplus_mt6877/blob/3c1e3d432d0c35afa7fded974b50fa6617219e0f/drivers/usb/gadget/function/f_rndis.c#L538-L551
Link: https://github.com/OnePlusOSS/android_kernel_oneplus_mt6877/blob/3c1e3d432d0c35afa7fded974b50fa6617219e0f/drivers/usb/gadget/function/u_ether.c#L869-L874
Signed-off-by: Rokas Juras <asrokasjuras@gmail.com>
Link: https://patch.msgid.link/20261005074852.76818-1-asrokasjuras@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
Instead of using ".cpu generic+lse", use ".arch_extension lse" to
enable LSE.
The ".cpu generic+lse" directive replaces the assembler's current feature
set with generic+lse, dropping other previously enabled features. In
contrast, ".arch_extension lse" enables LSE while preserving the current
feature set.
For example, the following code is rejected with "selected processor does
not support retaa", because ".cpu generic+lse" drops PAuth even though it
was enabled by armv9-a:
.arch armv9-a
.text
.cpu generic+lse
retaa
We encountered this issue while building the kernel with GCC configured
using --with-arch=armv9-a. Since Armv9-A includes pointer authentication,
GCC can assume that PAuth instructions are available and may use retaa
when generating PAuth-protected returns.
Signed-off-by: Emanuele Rocca <emanuele.rocca@arm.com>
Link: https://patch.msgid.link/20261005162137.71397-1-emanuele.rocca@arm.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
If a packet with bad checksum metadata gets into the ipv4 stack,
skb_checksum_help can corrupt the network header and cause a bunch of
mischief.
This was discovered and reported by Paulos, and has been reporoduced
by others independently since.
We really shouldn't allow such packets in, but as a defence
in depth measure, let's also check before we complete the checksum.
A more complete validation at input is forthcoming, but needs more work.
Cc: stable@vger.kernel.org
Fixes: f43798c27684 ("tun: Allow GSO using virtio_net_hdr")
Fixes: bfd5f4a3d605 ("packet: Add GSO/csum offload support.")
Reported-by: Paulos Yibelo <habte.yibelo@gmail.com>
Closes: https://lore.kernel.org/netdev/20260922030310.8684-2-habte.yibelo@gmail.com/
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Link: https://patch.msgid.link/0a012b4923e189c4c593ef4f471e5ff0edbe9030.1791412497.git.mst@redhat.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
gro_cells allocates a per-CPU struct gro_cell with per-CPU queues
(cell->napi_skbs), per-CPU local_lock_t (cell->bh_lock), and per-CPU
napi_struct (cell->napi). On !CONFIG_PREEMPT_RT, local_lock_t only
provides lockdep annotation (and is a complete no-op when
CONFIG_DEBUG_LOCK_ALLOC is disabled), relying on per-CPU locality and
disabled BH context for mutual exclusion; it does not provide cross-CPU
synchronization.
When threaded NAPI is enabled on a device backed by gro_cells (e.g.
gre0, vxlan, geneve) via sysfs, unbound kernel threads are created for
each per-CPU NAPI. When gro_cells_receive() on CPU A enqueues a packet
to cell_A and calls napi_schedule(&cell_A->napi) while holding
cell_A->bh_lock, the woken kthread can run concurrently on remote CPU B.
When CPU B runs gro_cell_poll(&cell_A->napi) and calls
__local_lock_nested_bh(&cell_A->bh_lock), lockdep detects that the lock
is already held by CPU A's task and triggers a warning:
WARNING: CPU: 1 PID: 377 at net/core/gro_cells.c:66 gro_cell_poll
DEBUG_LOCKS_WARN_ON(l->owner)
CPU: 1 UID: 0 PID: 377 Comm: napi/gre0-0 Not tainted 7.3.0-rc2
Call Trace:
<TASK>
__local_lock_nested_bh include/linux/local_lock_internal.h:92
gro_cell_poll+0x5aa/0x6c0 net/core/gro_cells.c:66
napi_threaded_poll+0x39b/0x600 net/core/dev.c:7048
kthread+0x71e/0x870 kernel/kthread.c:464
ret_from_fork+0x5d/0x90 arch/x86/kernel/process.c:167
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:264
</TASK>
On !CONFIG_PREEMPT_RT kernels without lockdep, concurrent execution of
__skb_queue_tail() on CPU A and __skb_dequeue() on CPU B without
cross-CPU locks leads to silent queue corruption.
Virtual per-CPU NAPIs cannot support unbound threaded NAPI. While
gro_cells also sets NAPI_STATE_NO_BUSY_POLL (which skips napi_hash_add()
and prevents configuring them via Netlink), NAPI_STATE_NO_BUSY_POLL
cannot be used to reject threaded NAPI in sysfs because drivers such as
WireGuard set NAPI_STATE_NO_BUSY_POLL on per-peer NAPIs while running
them in threaded mode by default.
Introduce NAPI_STATE_PERCPU to mark per-CPU NAPIs that cannot run in
unbound threaded mode. Set NAPI_STATE_PERCPU in gro_cells_init(), and
fail early with -EOPNOTSUPP in modify_napi_threaded() and
netdev_nl_napi_set_config() when trying to enable threaded NAPI on
NAPI_STATE_PERCPU NAPIs, while still allowing disabling (writing 0) as a
no-op.
Fixes: 25718fdcbdd2 ("net: gro_cells: Use nested-BH locking for gro_cell")
Fixes: 29863d41bb6e ("net: implement threaded-able napi poll loop support")
Reported-by: syzbot+f0661448aa9511ce744a@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/netdev/6aad6d7b.0c43d342.320d00.0001.GAE@google.com
Signed-off-by: Naman Gulati <namangulati@google.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Link: https://patch.msgid.link/20261006235113.2394030-1-namangulati@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
Commit 730ff06d3f5c ("net: mana: Use page pool fragments for RX buffers
instead of full pages to improve memory efficiency.") started handing
out RX buffers with zero headroom so that two buffers fit into one page
at the default MTU.
The MANA TX path, however, stores the per scatter-gather entry DMA
mappings in `struct mana_skb_head` at skb->head, and mana_start_xmit()
therefore calls skb_cow_head(skb, MANA_HEADROOM). The port advertises
this requirement as ndev->needed_headroom = MANA_HEADROOM.
As a result every packet that is received and then forwarded out of a
MANA port fails the skb_cow() in ip_forward() and gets reallocated and
copied by pskb_expand_head(). This is invisible to a plain RX or TX
workload, but it puts a full skb reallocation plus memcpy on the hot
path of every single forwarded packet, which is exactly what a
router/NVA workload does.
Restore the headroom. Note that reserving MANA_HEADROOM (232) is not
enough: ip_forward() asks for LL_RESERVED_SPACE(dev), which rounds
hard_header_len + needed_headroom up to HH_DATA_MOD and is 256 bytes on
ethernet. Use LL_RESERVED_SPACE() directly so the value keeps tracking
both constants. Also, since LL_RESERVED_SPACE() tracks MANA_HEADROOM,
it grows with MAX_SKB_FRAGS and for MAX_SKB_FRAGS >= 19 it is greater
than 256, so we have to account for that by using the headroom the RX
queue actually uses (instead of assuming XDP_PACKET_HEADROOM) and
turning MANA_XDP_MTU_MAX into MANA_XDP_MTU_MAX(ndev) (note that at the
default CONFIG_MAX_SKB_FRAGS=17 they are equivalent).
At the default MTU on a 4K page this means a buffer no longer fits twice
into a page (SKB_DATA_ALIGN(1500 + MANA_RXBUF_PAD + 256) = 2112), so the
frag-vs-single decision is now made by computing the real buffer size
instead of comparing the MTU against PAGE_SIZE / 2. The page_pool
fragment path is still used wherever at least two buffers genuinely fit,
e.g. on 16K and 64K page sizes.
Measured on an Azure VM with a MANA NIC acting as a forwarding NVA (UDP,
1400 byte payload, 4 streams, 8 Gbps offered, only the forwarding
node's kernel differs), 8 runs each, median:
forwarded pps throughput
before 272,830 3.06 Gbps
after 390,560 4.37 Gbps (+43%)
perf on the forwarding node, same workload:
memset_orig __pi_memcpy pskb_expand_head
before 10.07% 3.96% present
after 0.94% 0.64% gone
Cc: stable@vger.kernel.org
Fixes: 730ff06d3f5c ("net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency.")
Signed-off-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
Reviewed-by: Simon Horman <horms@kernel.org>
Link: https://patch.msgid.link/20261003013647.2051416-1-hamzamahfooz@linux.microsoft.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
Add option to expose additional fixed-sized partition starting
from the beginning of storage.
Xe driver can request this partition exposure if firmware or hardware
have detected failure that may involve corrupted partition table.
Fixed-sized partition allows full storage re-write in this situation.
Acked-by: Miquel Raynal <miquel.raynal@bootlin.com>
Signed-off-by: Alexander Usyskin <alexander.usyskin@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/20260930-cri_surviv-v3-1-357aeed5fc60@intel.com
Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
|
|
Add tracepoints to track the disksize-grow-pending lifecycle in the
writeback path and the block-zero-EOF entry point:
- ext4_iomap_wb_disksize_pending_submit: ioend is marked as
DISKSIZE_GROW_IO in ext4_iomap_writeback_submit().
- ext4_iomap_wb_disksize_pending_complete: ioend of type
DISKSIZE_GROW_IO completes in ext4_iomap_finish_ioend().
- ext4_iomap_wb_disksize_pending_wait: ioend worker waits for the
pending zeroed EOF block to complete.
- ext4_iomap_wb_update_disksize: i_disksize is advanced in
ext4_iomap_wb_update_disksize, including the new value and
whether the update is a disksize-grow completion.
- ext4_block_zero_eof: ext4_block_zero_eof is called with the range
and zeroing outcome, capturing the producer-side entry point.
Together with the previous mark/clear/wait tracepoints, these cover the
full lifetime of the disksize-grow-pending state.
Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Link: https://patch.msgid.link/20260903124017.2325538-1-yi.zhang@huaweicloud.com
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
|
