| Age | Commit message (Collapse) | Author |
|---|
|
Patch series "mm/damon: introduce data access-as-a-data attribute", v1.1.
TL;DR: extend DAMON's data attributes monitoring system to support page
table accessed bit and PG_idle based access monitoring.
DAMON was initially introduced as a data access monitor. Users found data
access pattern becomes more useful when it is combined with other data
attributes such as belonging cgroups and backing page types. For such
cases, DAMON has extended to support such data attributes monitoring in
addition to the original data access monitoring. The DAMON probes system
was introduced for this purpose. Users set probes for filtering data
attributes of their interest. For use cases where the primary interests
are the attributes but the access pattern, probe weights system has been
introduced. When it is used, DAMON applies its adaptive regions
adjustment based on the monitored data attributes.
However, DAMON stops access monitoring when the probe weights are used.
DAMON cannot optimally help users who have interests in both data
attributes and access patterns. Data access can also be thought of as
another data attribute, though. Extend the probe system to support data
access as a data attribute.
Introduce a new probe filter type, pgidle_unset. It shows if the page is
not set as idle. Specifically, it shows the page table accessed bit and
the PG_idle flag. It can inform if the region is ever accessed. But it
cannot say when it is accessed. To answer the second question, introduce
a new probe feature, prep actions. Using the features, Users can specify
what preparation actions should be made to each region for each probe.
DAMON executes the preparation actions for each sampling interval, like it
is doing the preparation for access check in the access monitoring mode.
To help 'pgidle_unset' probe action use case, 'set_pgidle' preparation
action is introduced together. The action does exactly what the access
monitoring was doing: clearing the page table accessed bits and setting
the PG_idle flags.
Tests
=====
I compared the access pattern monitoring results from the classic way and
the probe based way. As expected, the probe based way shows the results
similar to that of the classic way. More detailed test methods and
results are below.
First, do the access monitoring using the DAMON user-space tool [1] in the
classic way. The system is idle. It shows no access as expected.
$ sudo ./damo/damo start
$ sudo ./damo/damo report access
heatmap:
00000000000000000000000000000000000000008999999811111110000000000000000000000000
# min/max temperatures: -640,000,000, -100,000,000, column size: 99.800
MiB
intervals: sample 5 ms aggr 100 ms (max access hz 200)
0 addr 4.000 KiB size 3.898 GiB access 0 hz age 6.400 s
1 addr 3.898 GiB size 787.301 MiB access 0 hz age 1 s
2 addr 4.667 GiB size 773.457 MiB access 0 hz age 5.800 s
3 addr 5.423 GiB size 2.374 GiB access 0 hz age 6.400 s
memory bw estimate: 0 B per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
Start an artificial memory access generator (masim) [2] in another window.
$ ./masim/masim.py run --config_file ./masim/configs/zigzag.cfg
Show the monitoring results. As expected, it captures accesses.
$ sudo ./damo/damo report access
heatmap:
00000000000000000000000000000000000000011111111111111118888888833378988888888888
# min/max temperatures: -1,470,000,000, -12,482,536, column size: 99.800
MiB
intervals: sample 5 ms aggr 100 ms (max access hz 200)
0 addr 4.000 KiB size 1.542 GiB access 0 hz age 14.700 s
1 addr 1.542 GiB size 788.324 MiB access 0 hz age 14.600 s
2 addr 2.311 GiB size 772.844 MiB access 0 hz age 14.200 s
[...]
50 addr 6.839 GiB size 1.742 MiB access 0 hz age 1.300 s
51 addr 6.840 GiB size 8.000 KiB access 100 hz age 0 ns
52 addr 6.840 GiB size 1.496 MiB access 160 hz age 0 ns
[...]
97 addr 7.162 GiB size 1.199 MiB access 40 hz age 400 ms
98 addr 7.163 GiB size 1.199 MiB access 20 hz age 400 ms
99 addr 7.164 GiB size 2.004 MiB access 160 hz age 0 ns
100 addr 7.166 GiB size 646.004 MiB access 0 hz age 400 ms
memory bw estimate: 26.148 GiB per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
After the artificial memory access generator (masim) is terminated,
restart DAMON with the probe-based access monitoring. As expected, it
shows no access since the system is idle again.
$ sudo ./damo/damo stop
$ sudo ./damo/damo start --probe_prep set_pgidle \
--probe_filter allow pgidle_unset --probe_weight 1
$ sudo ./damo/damo report attrs
heatmap:
00000000000000000000000000000000000000008999999711111100000000000000000000000000
# min/max temperatures: -600,000,000, -430,000,000, column size: 99.800
MiB
probe prep: set_pgidle, filter: allow pgidle_unset (weight: 1)
intervals: sample 5 ms aggr 100 ms (max probe hits 20)
# addr size age probe_hits
0 4.000 KiB 3.898 GiB 6 s 0
1 5.285 GiB 2.512 GiB 6 s 0
2 4.659 GiB 641.816 MiB 5.700 s 0
3 3.898 GiB 778.375 MiB 4.300 s 0
memory bw estimate: 0 B per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
Start the artificial memory access generator [2] again.
$ ./masim/masim.py run --config_file ./masim/configs/zigzag.cfg
Show the monitoring results. As expected, it captures accesses similar to
the classic monitoring mode.
$ sudo ./damo/damo report attrs
heatmap:
00000000000000000000000000000000000000011111110000000177777777777777878798887777
# min/max temperatures: -1,330,000,000, 84,847,514, column size: 99.800
MiB
probe prep: set_pgidle, filter: allow pgidle_unset (weight: 1)
intervals: sample 5 ms aggr 100 ms (max probe hits 20)
# addr size age probe_hits
0 4.000 KiB 1.508 GiB 13.300 s 0
1 1.508 GiB 794.684 MiB 13.200 s 0
2 2.284 GiB 764.555 MiB 13 s 0
[...]
50 6.711 GiB 4.625 MiB 0 ns 17
51 6.716 GiB 2.504 MiB 0 ns 17
52 6.732 GiB 796.000 KiB 0 ns 17
[...]
90 7.162 GiB 512.000 KiB 2.200 s 2
91 7.061 GiB 700.000 KiB 2.300 s 1
92 6.935 GiB 316.000 KiB 2.300 s 4
memory bw estimate: 0 B per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
Patches Sequence
================
First four patches (patches 1-4) introduce the new probe filter type for
knowing if a region is accessed. Patch 1 defines the new type in the
core. Patch 2 implements the execution of the new filter in the physical
address space DAMON operation set. Patch 3 implements a user interface on
DAMON sysfs interface. Patch 4 updates the documentation for the new
filter type.
Following 13 patches (patches 5-17) introduce the probe preparation
actions feature. Patch 5 defines the data structure for specifying the
preparation actions. Patch 6 completes setup of the API parameter for the
prep. Patch 7 extends the DAMON operation set callback list to connect
the parameter with the underlying operation set. Patch 8 implements the
execution of the prep in the physical address space DAMON operation set.
Following five patches (patches 9-13) extends DAMON sysfs interface for
the new prep feature. Patch 14 adds simple selftest for basic file
operations of the new sysfs files. Final three patches (patches 15-17)
respectively update design, usage and ABI documents for the new feature
and its interface.
This patch (of 17):
Introduce a new DAMON filter type, pgidle_unset. It will match pages that
have their PG_Idle flag unset, or the page table accessed bit set. In
other words, it says if the page is accessed.
Link: https://lore.kernel.org/20260901132506.99243-1-sj@kernel.org
Link: https://lore.kernel.org/20260901132506.99243-2-sj@kernel.org
Link: https://github.com/damonitor/damo [1]
Link: https://github.com/sjp38/masim [2]
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
folio_alloc_swap() reports most failures with generic negative error
codes. Reclaim callers consequently cannot tell whether splitting a large
folio could make progress, or whether no swap space is available for even
a single page.
Classify failures using both the global free swap count and the remaining
capacity in the folio's memcg swap hierarchy. Return -ENOSPC when global
swap space is exhausted, -ENOMEM when splitting cannot overcome the
failure, and -E2BIG for a large folio when allocating or charging a
smaller folio might still succeed.
Use this classification for all folio_alloc_swap() failure paths,
including capability rejection, swap slot allocation failure, and memcg
swap charge failure. Callers are updated separately to split large folios
only on -E2BIG.
Link: https://lore.kernel.org/20260830042920.2280454-3-xueyuan.chen21@gmail.com
Signed-off-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Kairui Song <ryncsn@gmail.com>
Suggested-by: Barry Song <baohua@kernel.org>
Suggested-by: Youngjun Park <youngjun.park@lge.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nanzhe Zhao <zhaonanzhe@xiaomi.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
|
|
Patch series "mm: avoid large folio splits when swap is unavailable", v7.
This is v7 of Barry's original RFC patch, "mm: Avoiding split large folios
if swap has no space":
https://lore.kernel.org/r/20260618221720.71768-1-baohua@kernel.org
Barry's RFC showed the no-swap case with MADV_PAGEOUT on 16KB mTHP: the
large-folio split counter increased by 1024 even though no swapout
progress was possible. Skipping the split in that case kept the counter
at 0.
This series makes folio_alloc_swap() classify failures according to
whether splitting a large folio might allow swapout to make progress.
Callers can then avoid destroying the large folio when neither global swap
availability nor the folio's memcg swap hierarchy has capacity for even a
smaller folio.
Patch #1 adds page_counter_margin(), a small helper that computes the
minimum remaining chargeable space across a page_counter hierarchy.
Patch #2 establishes the folio_alloc_swap() return-value contract:
- -E2BIG: splitting may let smaller folios make progress
- -ENOSPC: no global swap space is available
- -ENOMEM: splitting is not expected to help, including when the
folio's memcg swap hierarchy has no remaining capacity
Patch #3 makes vmscan split a large folio only when folio_alloc_swap()
returns -E2BIG. Other failures keep the existing activation path and
avoid destroying the large folio when no smaller part can be backed by
swap either.
Patch #4 applies the same contract to shmem_writeout(), which currently
splits a large folio on every folio_alloc_swap() failure. It now enters
the split fallback only on -E2BIG; other failures redirty and reactivate
the folio as before.
Testing:
With a 1GB anonymous mapping backed by 16KB mTHPs and memory.swap.max=0,
the patch reduced the median latency of 30 process_madvise(MADV_PAGEOUT)
runs from 743.8 ms to 181.7 ms, while the number of large-folio splits per
run dropped from 65536 to 0. Neither kernel swapped out any pages.
I also ran DaCapo h2 under swap pressure and found no statistically
significant change in wall time or CPU time. The overall benefit appears
minor and workload-dependent.
This patch (of 4):
mem_cgroup_get_nr_swap_pages() open-codes the remaining capacity across
the memcg swap counter hierarchy.
Add page_counter_margin() to return the minimum usable space from a page
counter to the root, and use it in mem_cgroup_get_nr_swap_pages(). This
is a pure refactoring with no intended behavior change.
Link: https://lore.kernel.org/20260830042920.2280454-1-xueyuan.chen21@gmail.com
Link: https://lore.kernel.org/20260830042920.2280454-2-xueyuan.chen21@gmail.com
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Kairui Song <ryncsn@gmail.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Nanzhe Zhao <zhaonanzhe@xiaomi.com>
Cc: Youngjun Park <youngjun.park@lge.com>
|
|
There are no more users left for any pxd_ERROR() either in generic MM or
in the platform MM. Hence all these platform macros along with their
generic fallback could be dropped across the tree.
Link: https://lore.kernel.org/20260831054331.625505-9-anshuman.khandual@arm.com
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> # m68k
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Helge Deller <deller@gmx.de>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: James Bottomley <james.bottomley@HansenPartnership.com>
Cc: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Rich Felker <dalias@libc.org>
Cc: Samuel Holland <samuel.holland@sifive.com>
Cc: WANG Xuerui <kernel@xen0n.name>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
|
|
Patch series "mm: Drop pxd_ERROR()".
pxd_ERROR() macros have been provided by all platforms, which are very
much identical and can be dropped off completely if these pgtable printing
could be moved to callers in generic MM aka all pxd_clear_bad().
But first cleanups and re-organizations are required in some platforms
that are using these macros internally. Afterwards
[pte|pmd|pud|p4d|pgd]_ERROR() macros have been completely dropped from the
entire tree.
This patch (of 8):
Move ptval_to_str() inside a header thus making the helper more generally
available for new users which are being added later. While here, also
move another related string size macro PTVAL_STR_MAX inside the header as
well.
Link: https://lore.kernel.org/20260831054331.625505-1-anshuman.khandual@arm.com
Link: https://lore.kernel.org/20260831054331.625505-2-anshuman.khandual@arm.com
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Helge Deller <deller@gmx.de>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: James Bottomley <james.bottomley@HansenPartnership.com>
Cc: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Rich Felker <dalias@libc.org>
Cc: Samuel Holland <samuel.holland@sifive.com>
Cc: WANG Xuerui <kernel@xen0n.name>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
|
|
Introduce memcpy_nontemporal() for write-once copy sites that want a named
non-temporal copy primitive.
On x86_64, override the helper in arch/x86/include/asm/string_64.h using
the usual self-macro pattern, next to the existing memcpy_flushcache()
backend that memcpy_nontemporal() wraps.
include/linux/string.h provides the generic memcpy_nontemporal() fallback
as
#define memcpy_nontemporal(dst, src, len) \
((void)memcpy(dst, src, len))
instead of an inline wrapper, so architectures without a specialized
backend keep the usual memcpy() FORTIFY coverage when the compiler can
still see object sizes at the original call site. It also makes the
memcpy_nontemporal() API uniformly void, matching memcpy_flushcache() and
the x86 backend, so callers cannot accidentally depend on a return value
on fallback architectures.
memcpy_nontemporal() is only a copy primitive. It does not imply a drain
or a publication barrier. Callers that use it before a producer-consumer
or device-visible handoff must provide the required ordering at that
handoff point.
The immediate user is the ZONE_DEVICE template-copy path. It populates
struct page descriptors in a write-once pattern, so a regular cached
memcpy() can incur avoidable write-allocate traffic and cache pollution
for data with little near-term reuse.
Link: https://lore.kernel.org/20260831111638.76012-6-lizhe.67@bytedance.com
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
|
|
Callers that want to update section bits from a PFN currently need to
open-code:
set_page_section(page, pfn_to_section_nr(pfn));
and guard that sequence with #ifdef SECTION_IN_PAGE_FLAGS.
Add set_page_section_from_pfn() to wrap that update in one place. When
section bits are stored in page flags, the helper derives the section
number from the PFN and updates the page flags. Otherwise keep it as a
no-op so callers can use one helper without open-coding
SECTION_IN_PAGE_FLAGS.
Convert set_page_links() to use the new helper so later ZONE_DEVICE
fast-path patches can also update section bits without open-coding
SECTION_IN_PAGE_FLAGS at each callsite.
This keeps the PFN-to-section translation local to the configurations that
actually store section bits in struct page flags, and avoids exposing that
detail to generic callers.
No functional change intended.
Link: https://lore.kernel.org/20260831111638.76012-3-lizhe.67@bytedance.com
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Balbir Singh <balbirs@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Kees Cook <kees@kernel.org>
|
|
The logic that finds the struct pid for a given pid number and assigns it
to a damon_target is duplicated in multiple places. Including
damon_sysfs_add_target() of mm/damon/sysfs.c and the start functions of
the two sample modules, samples/damon/wsse.c and samples/damon/prcl.c.
Add a function that does the work, and replace the duplicated code in the
places with calls to the function.
Link: https://lore.kernel.org/20260831142611.77572-3-sj@kernel.org
Signed-off-by: Enze Li <lienze@kylinos.cn>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Hari Mishal <harimishal1@gmail.com>
Cc: Jaeyeon Lee <jaeyeon.lee.dev@gmail.com>
Cc: Li Youhong <liyouhong@kylinos.cn>
Cc: Shuah Khan <shuah@kernel.org>
Cc: "Zenghui Yu (Huawei)" <zenghui.yu@linux.dev>
Cc: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
|
|
Patch series "mm/damon: Introduce a huge page collapsing mechanism using
auto tuning", v4.
Overview
========
This patchset introduces a new autotuning which allows to collapse hot
regions into hugepages.
Motivation
==========
Since TLB is a bottleneck for many systems[1], a way to optimize TLB
misses (or hits) is to use huge pages. Unfortunately, using "always" in
THP leads to memory fragmentation and memory waste. For this reason, most
application guides and system administrators suggest to disable THP.
Selective huge page collapse per process is possible using prctl and a
launcher. However, this does not solve the issue with hot region
detection. Additionally, it the sysadmin should create a launcher that
uses PRCTL to enable THP for a particular process.
We can use the DAMON support for DAMOS_HUGEPAGE and DAMOS_COLLAPSE, to
target a certain process. DAMOS_COLLAPSE can also target the hot regions
in that process.
Still, there is an issue with the amount of huge page consumption. Since
huge pages can lead to memory fragmentation and waste, there should be a
way to limit the amount of huge page consumption. There is hugetlbfs, but
it requires changes to the application code or the use of libhugetlbfs.
DAMON has now a way to autotune some of the variables and adjust quotas
automatically, so that DAMON is fired only under the right circumstances.
It would be nice to have something similar, but for huge pages.
Solution
========
A new autotuning quota goal[2], damos_hugepage_mem_bp, is introduced,
which checks the huge page consumption to total memory consumption. This
new quota mechanism reuses current autotuning architecture.
In order to test this new mechanism, a sample module[3] was created, but
not included in this patch series. To demonstrate the tool, damo user
space tool was modified[4], which sets up huge pages collapse autotuning.
Benchmarks
==========
Setup: physical server with arm64 processor with 4 NUMA nodes, 1 TB RAM
and running mariaDB 10.5.29. Sysbench was used for the benchmark, with 20
tables and 3 million rows per table. The database was pinned to one of
the nodes, and the benchmark framework to a different node. No network
traffic involved in the benchmark.
Damo user space tool was forked and hugepage_mem_bp support added[4].
DAMON was lauched using this command line:
sudo ./damo start $(pidof mariadbd) \
--monitoring_nr_regions_range 10 1000 \
--monitoring_intervals 5000 100000 60000000 \
--damos_quota_time 0 --damos_quota_space 128000000 \
--damos_quota_interval 1000 \
--damos_quota_weights 0 1 1 \
--damos_quota_goal hugepage_mem_bp <target> \
--damos_quota_goal_tuner temporal \
--damos_apply_interval 50000 \
--damos_access_rate 0 max --damos_age 50 max \
--damos_action collapse --debug_damon
<target> was 1000 to taget 10% hugepage to total memory ratio, or 2500 to
target 25%. Tuner was also tested with consistent and temporal.
Results
=======
After the last timestamp, there was no change in huge page use, and the
total huge page to memory consumption ratio barely moved.
hugepage_mem_bp: 1000
goal tuner: temporal
+-----------+----------------+----------------+----------------------+
| timestamp | total mem used | huge page used | percentage hugepage |
+-----------+----------------+----------------+----------------------+
| 0 | 16945.04297 | 0 | 0 |
| 7 | 17008.69531 | 74 | 0.435071583 |
| 8 | 17036.40234 | 194 | 1.138738074 |
| 9 | 17017.01563 | 314 | 1.845211916 |
| 10 | 17029.67969 | 434 | 2.548491856 |
| 61 | 17111.30859 | 584 | 3.412947623 |
| 120 | 17071.05859 | 694 | 4.065360072 |
| 180 | 17133.88281 | 804 | 4.692456513 |
| 203 | 17088.16406 | 916 | 5.360435426 |
| 204 | 17126.34766 | 1046 | 6.107548562 |
| 205 | 17093.84375 | 1176 | 6.879669764 |
| 206 | 17142.77734 | 1298 | 7.571701913 |
| 209 | 17149.17969 | 1686 | 9.831374041 |
| 210 | 17097.30859 | 1754 | 10.25892462 |
+-----------+----------------+----------------+----------------------+
hugepage_mem_bp: 1000
goal tuner: consistent
+-----------+----------------+----------------+----------------------+
| timestamp | total mem used | huge page used | percentage hugepage |
+-----------+----------------+----------------+----------------------+
| 0 | 16955.24609 | 0 | 0 |
| 34 | 17039.71875 | 106 | 0.622075995 |
| 78 | 17009.47656 | 554 | 3.257007927 |
| 90 | 17048.92188 | 596 | 3.495822225 |
| 150 | 17092.90625 | 706 | 4.130368409 |
| 180 | 17053.08984 | 764 | 4.480126517 |
| 233 | 17100.50391 | 1496 | 8.748280216 |
| 239 | 17098.89063 | 2216 | 12.95990511 |
| 240 | 17135.44531 | 2334 | 13.62088908 |
| 245 | 17132.55078 | 2932 | 17.11362212 |
| 246 | 17117.95313 | 3052 | 17.82923448 |
| 250 | 17163.12109 | 3532 | 20.57900763 |
+-----------+----------------+----------------+----------------------+
hugepage_mem_bp: 2500
goal tuner: temporal
+-----------+----------------+----------------+----------------------+
| timestamp | total mem used | huge page used | percentage hugepage |
+-----------+----------------+----------------+----------------------+
| 0 | 17010.31641 | 0 | 0 |
| 9 | 17063.6875 | 50 | 0.2930199 |
| 10 | 17051.75781 | 170 | 0.996964664 |
| 60 | 17133.85547 | 572 | 3.338419663 |
| 90 | 17192.07813 | 626 | 3.641211932 |
| 120 | 17221.44531 | 682 | 3.960178647 |
| 181 | 17199.76172 | 790 | 4.593086886 |
| 208 | 17222.77734 | 1206 | 7.002354939 |
| 214 | 17245.17969 | 1904 | 11.04076637 |
| 215 | 17240.45703 | 2024 | 11.73982799 |
| 220 | 17234.79688 | 2624 | 15.22501262 |
| 228 | 17222.83594 | 3584 | 20.80958103 |
| 231 | 17247.55469 | 3944 | 22.86700968 |
| 235 | 17229.37109 | 4424 | 25.67708349 |
+-----------+----------------+----------------+----------------------+
hugepage_mem_bp: 1000
goal tuner: consist
+-----------+----------------+----------------+----------------------+
| timestamp | total mem used | huge page used | percentage hugepage |
+-----------+----------------+----------------+----------------------+
| 0 | 17125.85156 | 0 | 0 |
| 38 | 17081.23438 | 76 | 0.444932716 |
| 39 | 17133.11719 | 196 | 1.143983304 |
| 40 | 17119.83984 | 316 | 1.84581166 |
| 60 | 17109.72656 | 554 | 3.237924335 |
| 90 | 17164.11328 | 628 | 3.65879664 |
| 180 | 17177.66016 | 792 | 4.610639591 |
| 220 | 17180.86719 | 1378 | 8.020549749 |
| 226 | 17187.82031 | 1980 | 11.51978531 |
| 233 | 17143.48438 | 2818 | 16.4377319 |
| 240 | 17137.38281 | 3656 | 21.33347921 |
| 250 | 17175.5 | 4856 | 28.27283049 |
| 260 | 17199.66406 | 6056 | 35.20999002 |
| 270 | 17203.98438 | 7254 | 42.16465118 |
| 275 | 17207.21875 | 7762 | 45.10897498 |
+-----------+----------------+----------------+----------------------+
More detailed tables are provided here[5]
From this, we can conclude that the huge page autotuner works fine,
achieving the target. When using consistent autotuner, it actually
over-achieves the target, which is expected, since quota esz_bp is not set
to 0 to cap the DAMOS policy.
Patches Sequence
================
Patch 1 -> Introduce DAMOS_QUOTA_HUGEPAGE_MEM_BP and autotuning
Patch 2 -> sysfs support for the new quota goal
Patch 3 -> Document hugepage_mem_bp parameter
This patch (of 3):
Introduce DAMOS_QUOTA_HUGEPAGE_MEM_BP auto tuning. Add a new DAMOS quota
goal metric to measure the amount of huge page consumption to total memory
consumption ratio.
Vmstat may lag, which in some cases may lead to NR_FREE_PAGES being
greater than or equal to the amount of RAM in the system. A guard is
added to avoid the extremely unlikely case [6]. In the case, return 100%
(10000 bp).
Link: https://lore.kernel.org/20260831144732.80910-1-sj@kernel.org
Link: https://lore.kernel.org/20260831144732.80910-2-sj@kernel.org
Link: https://dl.acm.org/doi/pdf/10.1145/3307650.3322227 [1]
Link: https://lore.kernel.org/e67f05ad-dbb9-45e6-ba30-b167a99ac67d@huawei-partners.com [2]
Link: https://lore.kernel.org/20260616150316.580819-3-gutierrez.asier@huawei-partners.com [3]
Link: https://github.com/asierHuawei/damo/commit/79ae1a4ab1c012a7161db85a000d14f08fa36736 [4]
Link: https://lore.kernel.org/all/03f678dd-9ef3-4b97-b753-c2e4554c5159@huawei-partners.com/ [5]
Link: https://lore.kernel.org/all/20260715151615.99767-1-sj@kernel.org/ [6]
Signed-off-by: Asier Gutierrez <gutierrez.asier@huawei-partners.com>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
damon_nr_running_ctxs() has had no in-tree users since commit 5ec4333b1967
("mm/damon: remove DAMON debugfs interface") removed all of its callers.
Remove the unused declaration and definition.
Link: https://lore.kernel.org/20260831145724.82387-3-sj@kernel.org
Signed-off-by: Cheng-Han Wu <hank20010209@gmail.com>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
|
|
Patch series "mm/damon/core: remove unused helper functions", v2.
Both damon_targets_empty() and damon_nr_running_ctxs() have had no in-tree
users since commit 5ec4333b1967 ("mm/damon: remove DAMON debugfs
interface") removed their remaining callers.
Remove the unused declarations and definitions.
This patch (of 2):
damon_targets_empty() has had no in-tree users since commit 5ec4333b1967
("mm/damon: remove DAMON debugfs interface") removed its last caller.
Remove the unused declaration and definition.
Link: https://lore.kernel.org/20260831145724.82387-1-sj@kernel.org
Link: https://lore.kernel.org/20260831145724.82387-2-sj@kernel.org
Signed-off-by: Cheng-Han Wu <hank20010209@gmail.com>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
|
|
Fix a typo (nr_snapshots -> max_nr_snapshots) and corrects a grammar
error.
Link: https://lore.kernel.org/20260831150227.83416-4-sj@kernel.org
Signed-off-by: Liew Rui Yan <aethernet65535@gmail.com>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
Patch series "docs/mm/damon/design: add explanation of nr_snapshots", v3.
Add an explanation of nr_snapshots to avoid misunderstandings.
This patch (of 3):
Change "tried to be applied" -> "completely tried to be applied" to
maintain consistency between the documentation and the code.
Link: https://lore.kernel.org/20260831150227.83416-1-sj@kernel.org
Link: https://lore.kernel.org/20260831150227.83416-2-sj@kernel.org
Signed-off-by: Liew Rui Yan <aethernet65535@gmail.com>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
There are basically two parallel ways to look up a VMA: the traditional
way, which is protected by mmap_read_lock, and the RCU-based per-VMA lock
way which is based on RCU and refcounts. However, per-VMA locks will fail
if the lock is help by a writer and therefore never waits. In a number of
places we need to wait for the lock and it's done by falling back to
mmap_read_lock, locking the VMA and releasing the mmap_lock once VMA is
locked.
Add vma_start_read_unlocked() - a variant of the RCU-based lookup that
waits for writers. This is basically the same as the existing RCU-based
lookup, but on a failure to lock it temporarily takes mmap_lock for read
and waits for writers to finish before locking the VMA, dropping the
mmap_read_lock and returning the locked VMA. This has some advantages:
1. Callers do not need to have a fallback path for when they collide
with writers.
2. Its fast path does not require taking mmap_lock for read.
Basically, when applied correctly, this approach results in faster *and*
simpler code.
While at it, fix the comments for vma_start_read_locked(),
vma_start_read_locked_nested(), and uffd_lock_vma().
Link: https://lore.kernel.org/20260831203056.838265-4-surenb@google.com
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Todd Kjos <tkjos@android.com>
Cc: Christian Brauner <christian@brauner.io>
Cc: Carlos Llamas <cmllamas@google.com>
Cc: Alice Ryhl <aliceryhl@google.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: David Ahern <dsahern@kernel.org>
Cc: Arve Hjønnevåg <arve@android.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
|
|
Patch series "mm: Unconditional per-VMA locks and cleanups", v7.
tl;dr: Make per-VMA locks available in all configs. Simplify some of the
per-VMA lock users now that they can rely on them being always available.
Binder and networking folks: Your code is the target of the cleanups. I'm
cc'ing you now on v2 because there's emerging consensus on the mm side
that the approach here is sane. I'm not quite sure how this pile would
get merged, but ack/review tags would be appreciated if this looks good to
you.
Longer version:
When working on some x86 shadow stack code, it was a real pain to avoid
causing recursive locking problems with mmap_lock. One way to avoid those
was to avoid mmap_lock and use per-VMA locks instead. They are great, but
they are not available in all configs which makes them unusable in generic
code, or if you want to completely avoid mmap_lock.
Make per-VMA locks available in all configs. Right now, they are only
available on select architectures when SMP and MMU are enabled. But all
of the primitives that per-VMA locks are built on (RCU, maple trees,
refcounts) work just fine without SMP or MMU.
The only real downside is that making VMAs a wee bit bigger on !MMU and
!SMP builds.
The upside is much cleaner code, lower complexity and less #ifdeffery.
Clean up a binder VMA locking site now that it can rely on per-VMA locks.
Building on top of universally-available per-VMA locks, introduce a new
helper. Since the new API does not require callers to have a fallback to
mmap_lock, it's much easier to use. Callers can potentially replace this
very common kernel idiom:
mmap_read_lock(mm);
vma = vma_lookup()
// fiddle with vma
mmap_read_unlock(mm);
with:
vma = vma_start_read_unlocked(mm, address);
// fiddle with vma
vma_end_read(vma);
Which avoids mmap_lock entirely in the fast path.
Use that new API for another binder site and one in the TCP code.
This patch (of 7):
The per-VMA locks have been around for several years. They've had some
bugs worked out of them and have seen quite wide use. However, they are
still only available when architectures explicitly enable them. Remove
the conditional compilation around the per-VMA locks, making them
available on all architectures and configs.
The approach up to now seemed to be to add ARCH_SUPPORTS_PER_VMA_LOCK when
the architecture started using per-VMA locks in the fault handler. But,
contrary to the naming, the Kconfig option does not really indicate
whether the architecture supports per-VMA locks or not. It is more of a
marker for whether the architecture is likely to benefit from per-VMA
locks.
To me, the most important thing side-effect of universal availability is
letting per-VMA locks be used in SMP=n configs. This lets us use
per-VMA locking in all x86 code without fallbacks.
Overall, this just generally makes the kernel simpler. Just look at the
diffstat. It also opens the door to users that want to use the per-VMA
locks in common code. Doing *that* brings additional simplifications.
The downside of this is adding some fields to vm_area_struct and
mm_struct. There are likely ways to optimize this, especially for things
like SMP=n configs. For now, do the simplest thing: use the same
implementation everywhere.
== Considerations for NOMMU config ==
NOMMU systems do not write-lock VMAs, therefore read-locking a VMA would
always succeed unless VMA is detached. Therefore for NOMMU config we make
vma_mark_attached() a NOOP, which keeps VMAs always in detached state.
This causes VMA read-locking to always fail and the caller falls back to
locking mmap_lock.
The following functions will have a different implementation in NOMMU
config:
- vma_mark_attached(), vma_mark_detached() are made NOOPs, keeping VMAs
always in a detached state and preventing assertions and refcount
underflows;
- vma_start_write(), vma_start_write_killable() are made NOOPs to avoid
warnings in __vma_start_write() due to VMAs being detached. These
functions are not used in NOMMU code but __vma_start_write() is an
exported function, therefore might be used by drivers.
- vma_assert_attached() is made NOOP because it's reachable from NOMMU
code via split_vma()->vma_iter_store_new()->vma_iter_store_overwrite();
- vma_assert_write_locked() is asserting vma->vm_mm is write-locked, as
was done before this change;
- vma_assert_locked() is asserting vma->vm_mm is locked, as was done
before this change;
The following functions work for both MMU and NOMMU configs:
- vma_lock_init() performs the same initialization as for MMU config;
- mm_lock_seqcount_init(), mm_lock_seqcount_begin(),
mm_lock_seqcount_end() are called from mmap_write_{lock|unlock} and
update mm_lock_seq correctly.
- mmap_lock_speculate_try_begin(), mmap_lock_speculate_retry() work as
is because mm_lock_seq is updated correctly;
- vma_start_read(), vma_start_read_locked() will always fail because
VMAs are always detached;
- vma_end_read() will never be called because vma_start_read() never
succeeds;
- vma_is_attached() always return false because VMAs are always
detached;
- vma_assert_detached() will never trigger because VMAs are never
attached;
- vma_start_read_locked() always return false because VMAs are always
detached;
- lock_vma_under_rcu() will be safe as the attempted read lock will bail;
Changes in the following files are not affecting NOMMU config:
task_mmu.c - not compiled when CONFIG_MMU=n;
pagewalk.c - not compiled when CONFIG_MMU=n;
userfaultfd.c - not compiled when CONFIG_MMU=n (CONFIG_USERFAULTFD depends
on CONFIG_MMU);
The following changes in the BPF code are made to keep NOMMU config
working like before:
stack_map_lock_vma() - keeps mmap_lock in NOMMU config;
bpf_iter_task_vma_new() - bails out in NOMMU config;
Link: https://lore.kernel.org/20260831203056.838265-1-surenb@google.com
Link: https://lore.kernel.org/20260831203056.838265-2-surenb@google.com
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Todd Kjos <tkjos@android.com>
Cc: Christian Brauner <christian@brauner.io>
Cc: Carlos Llamas <cmllamas@google.com>
Cc: Alice Ryhl <aliceryhl@google.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: David Ahern <dsahern@kernel.org>
Cc: Arve Hjønnevåg <arve@android.com>
|
|
Commit ee040cbd6e48 ("mm/page_alloc: don't warn about large allocations
with __GFP_NOFAIL") remove a warning on allocating large folio with
__GFP_NOFAIL, which was placed there by commit 903edea6c53f ("mm: warn
about illegal __GFP_NOFAIL usage in a more appropriate location and
manner").
While in that commit, it also documented this behavior which is out-dated
now.
Adjust the document to align to current code, and adjust the comment while
at it.
Link: https://lore.kernel.org/20260829025847.26779-1-richard.weiyang@gmail.com
Signed-off-by: Wei Yang <richard.weiyang@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
|
|
This flag is now only used on folios, so we can remove all the page
accessors. folio_test_clear_reclaim() is not used, so don't add
FOLIO_TEST_CLEAR_FLAG() for it.
Link: https://lore.kernel.org/20260829-b4-page_io-folios-v2-2-649728091117@columbia.edu
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
|
|
Nothing reads it anymore, so the field and the helper that reset it on
css alloc and css reset can go.
Link: https://lore.kernel.org/20260902174311.1772372-8-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: T.J. Mercier <tjmercier@google.com>
|
|
tree_node, usage_in_excess and on_tree only existed for the soft limit
rbtree. They also doubled as the buffer between the read-mostly head of
struct mem_cgroup_per_node and its update-often tail, so replace them
with the explicit padding that CONFIG_MEMCG_V1=n already used.
Link: https://lore.kernel.org/20260902174311.1772372-7-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: T.J. Mercier <tjmercier@google.com>
|
|
The soft limit rbtree was the only caller. Dropping it leaves
MEMCG_LRU_HEAD unreachable, since nothing else ever rotates a memcg with
that op, so remove the op too and update the memcg LRU comment.
Link: https://lore.kernel.org/20260902174311.1772372-6-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: T.J. Mercier <tjmercier@google.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
mm_vmscan_memcg_softlimit_reclaim_begin and
mm_vmscan_memcg_softlimit_reclaim_end were only emitted by
mem_cgroup_shrink_node(), which is gone, so they can never fire again.
Link: https://lore.kernel.org/20260902174311.1772372-4-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: T.J. Mercier <tjmercier@google.com>
|
|
Patch series "memcg: remove the v1 soft limit", v2.
The v1 soft limit was deprecated in v6.12 by commit 569c4f62d84a ("memcg:
initiate deprecation of v1 soft limit") and nobody has reported depending
on it in the ~21 months since. memory.low and memory.min in v2 have
covered the same ground for far longer.
The knob has since been made inert by "memcg: make the v1 soft limit knob
inert", already queued in mm-hotfixes as a backportable fix for a syzbot
report [1]. Nothing can enter the soft limit rbtree anymore, so this
series just deletes the machinery that is now dead: the reclaim pass in
kswapd and direct reclaim, mem_cgroup_shrink_node() and its tracepoints,
the per-node rbtree, lru_gen_soft_reclaim() and the MEMCG_LRU_HEAD op, the
per-node tree fields, mem_cgroup->soft_limit, and finally the v1 event
ratelimiting which is now down to a single target.
memory.soft_limit_in_bytes itself is untouched: writes stay ignored and
reads keep returning the maximum value.
This patch (of 8):
Nothing can put a cgroup on the soft limit rbtree anymore, so the tree is
always empty and both callers of memcg1_soft_limit_reclaim() are
guaranteed no-ops. Remove the reclaim pass from direct reclaim and from
kswapd, along with its implementation.
In shrink_zones() this leaves the global reclaim branch with a last_pgdat
check that is now redundant with the identical check right below it, so
drop it and move the explaining comment down to the check that remains.
That check could only ever fire once last_pgdat was set, which implies
first_pgdat had already been assigned, so skipping it does not change
which node consider_reclaim_throttle() gets.
Link: https://lore.kernel.org/20260902174311.1772372-1-shakeel.butt@linux.dev
Link: https://lore.kernel.org/20260902174311.1772372-2-shakeel.butt@linux.dev
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: T.J. Mercier <tjmercier@google.com>
|
|
The shmem unused huge shrinker keeps a per-superblock list of inodes whose
tail huge folio extends beyond i_size. Since that list is not memcg
aware, reclaim triggered by one memcg can scan inodes from the whole
superblock and split shmem huge folios charged to unrelated memcgs.
Convert the shrink list to a memcg-aware list_lru. Queue each inode on
the list_lru sublist matching the memcg and node of the current tail huge
folio, so non-root memcg reclaim only walks candidates charged to the
reclaiming memcg. Global reclaim, root memcg reclaim and shmem quota
reclaim keep global semantics.
Rather than pinning a struct mem_cgroup reference in shmem_inode_info,
store a struct obj_cgroup reference instead. The list_lru add and delete
paths resolve the current memcg from the objcg under RCU, so that memcg
offline and list_lru entry migration remain consistent: list_lru migrates
entries to the parent memcg sublist on offline, and obj_cgroup_memcg()
follows the same reparenting, ensuring the correct sublist is always found
at delete time. This avoids pinning a dying memcg through a long-lived
CSS reference.
The list_lru still tracks inodes while the actual split target is the
current tail huge folio, so validate the folio memcg/node during scan. If
the folio no longer matches the reclaim context or splitting cannot
proceed, requeue the inode according to the current tail folio; if the
inode is no longer shrinkable, drop the scan entry.
This can be tested with the shrinker debugfs interface by allocating 32
tmpfs tail THPs in each of two memcgs, then scanning the sb-tmpfs shrinker
with memcg A's cgroup id:
before A scan after A scan
base A=64M, B=64M A=64M, B=64M (per-memcg count is skipped)
patched A=64M, B=64M A=0, B=64M
Link: https://lore.kernel.org/94cc7fe1fd645254ae90effc1c0687678438d940.1786955972.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Qinyun Tan <qinyuntan@linux.alibaba.com>
|
|
Patch series "make unused huge shrinker memcg aware", v4.
The shmem unused huge shrinker maintains a per-superblock list of inodes
whose tail huge folio extends beyond i_size. Because this list is not
memcg aware, reclaim triggered by memcg A can scan inodes across the
entire superblock and split huge folios charged to unrelated memcg B,
causing unexpected impact on it.
In the worst case, memcg A has no reclaimable shmem at all, making the
reclaim entirely useless and incurring unnecessary latency. We observed
this in production, where page lock contention during split caused
multi-hundred-millisecond stalls:
tid 11340 comm scanner locked a page for 182264 us! kstack:
unlock_page+1
split_huge_page_to_list+3135
shmem_unused_huge_shrink+767
super_cache_scan+329
do_shrink_slab+291
shrink_slab+533
shrink_node+400
do_try_to_free_pages+206
try_to_free_mem_cgroup_pages+262
try_charge_memcg+591
mem_cgroup_charge+136
__handle_mm_fault+2431
handle_mm_fault+194
do_user_addr_fault+462
__do_page_fault+176
do_page_fault+48
page_fault+62
Usama's recent patch [1] prevents the shmem unused shrinker from being
invoked during memcg-level reclaim altogether, but this is overly
conservative: we can do better by reclaiming only the shmem charged to the
reclaiming memcg.
This series converts the shrinker list to a memcg-aware list_lru, so that
non-root memcg reclaim walks only candidates charged to the reclaiming
memcg. Global reclaim, root memcg reclaim and shmem quota reclaim retain
their existing global semantics.
To avoid pinning a dying memcg through a long-lived CSS reference, each
inode stores an obj_cgroup reference instead of a mem_cgroup reference.
The list_lru add/delete paths resolve the current memcg from the objcg
under RCU, staying consistent with list_lru's own memcg migration on
offline.
This patch (of 3):
obj_cgroup_memcg() currently requires a non-NULL objcg, so callers that
may hold a NULL objcg must guard the call with an explicit NULL check.
This pattern is duplicated in folio_memcg(), folio_memcg_check(),
mm/page_owner.c, and mm/zswap.c.
Teach obj_cgroup_memcg() to accept NULL and return NULL in that case, then
remove the redundant NULL checks at the call sites. Also remove the
mem_cgroup_from_entry() wrapper in zswap, which existed solely to provide
this NULL-safe behaviour, and replace its two callers with direct
obj_cgroup_memcg() calls.
No functional change intended.
Link: https://lore.kernel.org/cover.1786955972.git.zhengqi.arch@bytedance.com
Link: https://lore.kernel.org/09bcf74312246a6e4146be8a0cb9787f8beddb28.1786955972.git.zhengqi.arch@bytedance.com
Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
|
|
On the rollback path, folio_put() has already dropped the last reference
of new_folio. On the success path, new_folio is already unlocked and can
be freed concurrently. The trace_mm_khugepaged_collapse_file() is left
with a dangling folio pointer.
So using the folio_pfn() before dropping the reference, closing
use-after-free window.
Link: https://lore.kernel.org/20260909025804.3233645-4-vernon2gm@gmail.com
Fixes: 4c9473e87e75 ("mm/khugepaged: add tracepoint to collapse_file()")
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Zach O'Keefe <zokeefe@google.com>
Cc: <stable@vger.kernel.org>
|
|
After the page table lock has dropped, the folio can be freed
concurrently. The trace_mm_khugepaged_scan_pmd() is left with a dangling
folio pointer.
So using the folio_pfn() before dropping the page table lock, closing
use-after-free window.
And other pre-existing bug, When the `for (i = 0; i < HPAGE_PMD_NR; i++)`
iteration to terminate and the folio operation preceding is normal, but
pfn will be incorrect. so we really only trace the PFN if it really was
problematic.
Link: https://lore.kernel.org/20260909025804.3233645-3-vernon2gm@gmail.com
Fixes: 7d2eba0557c1 ("mm: add tracepoint for scanning pages")
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Zach O'Keefe <zokeefe@google.com>
Cc: <stable@vger.kernel.org>
|
|
Patch series "mm: khugepaged: fix tracepoint UAF", v5.
The khugepaged tracepoints take a folio pointer and call folio_pfn(), but
by then the folio may no longer be valid: freed after folio_put(),
folio_unlock() or pte_unmap_unlock(), or not a folio at all but an
xarray-encoded swap entry. On classic SPARSEMEM, dereferencing it oopses
khugepaged as soon as the trace event is enabled; on other memory models
it merely prints a bogus pfn.
Pass the pfn to the tracepoints directly, captured while the folio is
still pinned, closing the use-after-free windows in
mm_khugepaged_scan_file(), mm_khugepaged_scan_pmd() and
mm_khugepaged_collapse_file().
This patch (of 3):
When the swap entries found exceed max_ptes_swap, the loop is left via
break with folio still holding the xarray value that encodes the swap
entry, not valid folio pointer.
That value is passed to trace_mm_khugepaged_scan_file(), which feeds it to
folio_pfn(). On FLATMEM and SPARSEMEM_VMEMMAP, the page_to_pfn() is plain
pointer arithmetic, so the trace event merely prints bogus scan_pfn. On
classic SPARSEMEM, the page_to_pfn() reads page->flags, dereferencing the
tiny encoded integer and oopsing khugepaged whenever the trace event is
enabled.
So when folio is the swap entry value, simply set pfn to -1, just like
exhausted scan naturally.
And the folio_put() has maybe dropped the last reference of folio. The
trace_mm_khugepaged_scan_file() is left with a dangling folio pointer. so
using the folio_pfn() before dropping the reference, closing
use-after-free window.
Link: https://lore.kernel.org/20260909025804.3233645-1-vernon2gm@gmail.com
Link: https://lore.kernel.org/20260909025804.3233645-2-vernon2gm@gmail.com
Fixes: d41fd2016ed0 ("mm/khugepaged: add tracepoint to hpage_collapse_scan_file()")
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Zach O'Keefe <zokeefe@google.com>
Cc: <stable@vger.kernel.org>
|
|
Commit 0c6378a71574 ("arch: introduce set_direct_map_valid_noflush()")
added set_direct_map_valid_noflush() to allow updating the direct map for
a physically contiguous range in execmem.
As Brendan recently pointed out [1], this API is confusing because on
arm64 it means that is sets VALID bit in ptes, while on other
architectures it is an analog of set_direct_map_default_noflush().
The only user of set_direct_map_valid_noflush() was execmem's ROX cache
freeing path and it was switched to utilize VM_FLUSH_RESET_PERMS for
resetting permissions of the direct map alias.
With the last user gone and with
set_direct_map_{invalid,default}_noflush() accepting number of pages as a
parameter, set_direct_map_valid_noflush() become a copy of
set_memory_valid() on arm64 and a duplicate of
set_direct_map_{invalid,default}_noflush() on other architecture, it is
safe to remove set_direct_map_valid_noflush().
Also drop a stale comment in arm64::__kernel_map_pages() that Linus
bothered to add when merging changes containing
set_direct_map_valid_noflush() to his tree.
This reverts commit 0c6378a71574daa6cd1534ad42a956e3262756c7.
Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-6-949b64a9f755@kernel.org
Link: https://lore.kernel.org/all/DJ69RCVRBO0Y.3JCYSW50IC4RC@linux.dev [1]
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Brendan Jackman <brendan.jackman@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Len Brown <lenb@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: WANG Xuerui <kernel@xen0n.name>
Cc: Will Deacon <will@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
|
|
Patch series "arch, mm/execmem: resolve confusion about
set_direct_map_valid_noflush()", v3.
Recent discussion about implementation of execmem's ROX caches on arm64
revealed a confusion about how set_direct_map_valid_noflush() implemented
on different architectures.
On arm64 it sets or clears the PTE_VALID bit marking a PTE as present or
not present.
On other architectures it's a range version of
set_direct_map_invalid_noflush() and set_direct_map_default_noflush()
Unlike arm64::set_direct_map_valid_noflush(),
set_direct_map_default_noflush() not only marks PTE as present, but also
sets its default protection mode.
Other than that, initial design of execmem ROX caches didn't rely on
restoration of large mappings that's now available on x86, but completely
removed the memory allocated for the ROX cache from the direct map to
ensure that large mappings are not split. This precluded usage of
VM_FLUSH_RESET_PERMS for the ROX cache allocations and required execmem to
implement manipulation of the direct map alias.
Current implementation of ROX caches does not remove the direct map alias
but simply calls set_memory_rox() that updates the permissions in both
vmalloc address space and the direct map and relies on
collapse_large_pages() in x86 CPA to keep large mappings.
This allow using VM_FLUSH_RESET_PERMS for execmem ROX cache allocations
with small adjustments to set_direct_map APIs and vmalloc::reset_perms()
behaviour: adding number of pages parameter to set_direct_map APIs and
making resetting of the direct map permissions in vmalloc VMAP_HUGE
friendly.
Implement these adjustments, make execmem always use VM_FLUSH_RESET_PERMS
and revert set_direct_map_valid_noflush() changes.
This patch (of 6):
When set_direct_map APIs were introduced by the commit d253ca0c3865
("x86/mm/cpa: Add set_direct_map_*() functions") the single page parameter
made sense because the initial callers (vmalloc and hibernation) had sets
of unsorted struct pages that required changes of their mappings in the
direct map.
Since there is an increasing demand for direct map manipulation and it is
also desirable to be able to update larger physically contiguous mappings,
for example an entire large folio, extend set_direct_map APIs to receive
number of pages parameter.
As there is still only a handful of callers, change the existing functions
directly and update all the call sites rather than adding wrappers for
single page case.
Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-0-949b64a9f755@kernel.org
Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-1-949b64a9f755@kernel.org
Link: https://lore.kernel.org/all/20260611130144.1385343-4-abarnas@google.com [1]
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: "Borislav Petkov (AMD)" <bp@alien8.de>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Len Brown <lenb@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: WANG Xuerui <kernel@xen0n.name>
Cc: Will Deacon <will@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
|
|
HUGE_BOOTMEM_ZONES_VALID is only used by the huge_bootmem_page flag
handling in mm/hugetlb.c. Keep the definition next to that private data
structure instead of exposing it through the public hugetlb header.
No functional change is intended.
Link: https://lore.kernel.org/20260910063256.64386-18-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
Track early CMA hugetlb pages from the hstate instead of storing a
redundant bootmem flag. This removes the unused helper and keeps the
bootmem metadata minimal.
Link: https://lore.kernel.org/20260910063256.64386-16-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
The HUGE_BOOTMEM_HVO flag tracked whether a bootmem huge page had already
gone through the old early vmemmap optimization path.
Now that HugeTLB uses section-based vmemmap optimization, that state is
already reflected in the compound page order stored in section metadata.
Remove HUGE_BOOTMEM_HVO and its helper, and use the section state directly
when deciding whether to mark a folio as vmemmap-optimized.
Link: https://lore.kernel.org/20260910063256.64386-15-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
section_map_size() no longer provides any shared logic.
After the sparse-vmemmap changes, its only remaining user is the
!CONFIG_SPARSEMEM_VMEMMAP path in __populate_section_memmap(), which can
compute the size inline with PAGE_ALIGN(sizeof(struct page) *
PAGES_PER_SECTION).
Remove section_map_size() and inline the remaining calculation.
Link: https://lore.kernel.org/20260910063256.64386-14-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
After removing SPARSEMEM_VMEMMAP_PREINIT, sparse_init_nid() no longer
needs the transient sparse_usagebuf state and its helper wrappers.
Allocate the usemap buffer directly in sparse_init_nid(), pass it to
sparse_init_one_section(), and drop sparse_usage_init(),
sparse_usage_fini(), and sparse_init_early_section().
Link: https://lore.kernel.org/20260910063256.64386-13-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
SPARSEMEM_VMEMMAP_PREINIT existed only to support HugeTLB's early vmemmap
optimization setup.
Now that HugeTLB bootmem vmemmap optimization uses the common
section-based sparse-vmemmap path, the sparse initialization code no
longer needs a separate pre-initialization mechanism for vmemmap
population.
Remove the related Kconfig symbols, section flag, and empty early hook, so
present sections always go through the normal sparse setup path.
Link: https://lore.kernel.org/20260910063256.64386-12-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
HugeTLB bootmem vmemmap optimization still carries its own early setup
path, including pre-populating optimized mappings before the generic
sparse-vmemmap code runs.
Now that the section-based vmemmap optimization can derive HugeTLB vmemmap
deduplication from section metadata, HugeTLB only needs to mark the
bootmem huge page range with the appropriate order. The generic
sparse-vmemmap population path can then allocate and map the shared tail
vmemmap pages without any HugeTLB-specific early population code.
Do that by recording the compound page order when a bootmem huge page is
allocated and dropping the dedicated pre-HVO helpers and related
special-casing.
This removes duplicate early setup logic and switches HugeTLB to the
section-based vmemmap optimization path.
Link: https://lore.kernel.org/20260910063256.64386-11-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
section_nr_vmemmap_pages() can account ordinary sections and DAX sections,
but section-based vmemmap optimization stores the compound page order in
struct mem_section and retains a different number of vmemmap pages.
Teach section_nr_vmemmap_pages() to recognize section-based optimized
sections and calculate their vmemmap page count from the stored compound
page order and the HVO retained page count.
Link: https://lore.kernel.org/20260910063256.64386-6-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
memmap_init_range() initializes every struct page in the target range.
For compound pages with vmemmap optimization, the tail struct pages are
backed by a shared vmemmap page.
Initializing those tail struct pages would overwrite the shared vmemmap
page contents, requiring users such as HugeTLB to restore the metadata
afterwards.
Track the compound page order for HVO-backed sections and use that
metadata to detect struct pages that fall into the shared tail vmemmap
range. Skip those shared tail pages in memmap_init_range(), then
initialize pageblock migratetypes for the processed range with a helper
after the per-page initialization loop.
Keep direct mem_section access inside sparse helpers. Expose
pfn_to_section_compound_order() for callers that only need the order
associated with a PFN. This lets memmap_init_range() skip shared tail
vmemmap pages without exposing __pfn_to_section() to !SPARSEMEM builds.
This is a preparatory change for consolidating handling across users of
vmemmap optimization, and it also avoids redundant initialization of
shared tail vmemmap pages during early boot. That early-boot benefit
appears only once HugeTLB is switched to this common handling, since
HugeTLB is the early-boot user that creates those shared tail vmemmap
pages.
Link: https://lore.kernel.org/20260910063256.64386-4-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
The macros VMEMMAP_TAIL_MIN_ORDER and NR_VMEMMAP_TAILS describe the order
range where HVO can be applied, but their names tie that range to the
tail-page cache implementation.
Rename them with a VMEMMAP_OPTIMIZATION prefix and use the new names in
the HVO paths. This makes the code describe the optimization requirements
rather than the tail-page cache implementation detail.
No functional change intended.
Link: https://lore.kernel.org/20260910063256.64386-3-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
Patch series "mm: Introduce section-based vmemmap optimization for
HugeTLB", v6.
This series is split out from the earlier, larger series "mm: Generalize
HVO for HugeTLB and device DAX" [1]. While the parent series generalizes
vmemmap optimization across HugeTLB and device DAX, this subset addresses
a single, self-contained step: making the generic sparse-vmemmap code
section-based optimization aware and switching HugeTLB bootmem pages to
this path.
HugeTLB vmemmap optimization currently has its own early boot setup path.
It pre-populates optimized vmemmap mappings before the normal
sparse-vmemmap population code runs, and sparsemem carries
SPARSEMEM_VMEMMAP_PREINIT only to support that special case.
That makes the HugeTLB vmemmap optimization path harder to share with
other users of sparse-vmemmap optimization and leaves a fair amount of
HugeTLB-specific boot-time state in the generic memory initialization
flow.
This series introduces section-based vmemmap optimization support in the
sparse-vmemmap code and switches HugeTLB bootmem pages over to it.
Instead of having HugeTLB pre-populate optimized vmemmap mappings itself,
HugeTLB now records the compound page order in the corresponding memory
sections. The generic sparse-vmemmap population path can then allocate or
reuse shared tail vmemmap pages based on section metadata.
The patches are organized as follows:
- patches 1-2 prepare sparsemem and vmemmap optimization metadata
- patches 3-8 teach the common sparse-vmemmap paths to use that state
- patches 9-10 switch HugeTLB bootmem optimization to the
section-based path
- patches 11-17 clean up sparsemem and HugeTLB bootmem code that is no
longer needed after the conversion
This is intended to be the second smaller step toward the broader HVO
generalization. The device DAX conversion and the wider HVO consolidation
are left for follow-up series.
This patch (of 17):
struct mem_section is currently forced to a power-of-2 size so the
section-to-root lookup can use a mask instead of a modulo.
That requirement makes future extensions harder than necessary: adding a
small field can require configuration-dependent padding or layout checks
just to preserve the lookup scheme. Keep the lookup correct for any
struct mem_section size by using a plain modulo instead.
Do not leave the layout entirely unconstrained, though. Keep struct
mem_section double-word aligned so modest size changes, such as adding
another word-sized field on 64-bit systems, still keep a compact and
efficient layout. If future fields grow the structure beyond that sweet
spot, the lookup remains correct; only the exact layout efficiency
changes.
Link: https://lore.kernel.org/20260910063256.64386-2-songmuchun@bytedance.com
Link: https://lore.kernel.org/all/20260513130542.35604-1-songmuchun@bytedance.com/ [1]
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
|
|
The treewide rename in commit 5e0a760b4441 ("mm, treewide: rename
MAX_ORDER to MAX_PAGE_ORDER") left a few spots still using the old name:
- two comments in include/net/mana/mana.h and mm/page_alloc.c;
- the gdb helper scripts/gdb/linux/mm.py, where self.MAX_ORDER is
a local mirror of the kernel's MAX_ORDER define.
Rename the leftover instances to MAX_PAGE_ORDER so the tree is consistent.
No functional changes.
Link: https://lore.kernel.org/20260819082052.3338603-1-xiqi2@huawei.com
Signed-off-by: Qi Xi <xiqi2@huawei.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Jan Kiszka <jan.kiszka@siemens.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Kieran Bingham <kbingham@kernel.org>
Cc: Konstantin Taranov <kotaranov@microsoft.com>
Cc: Long Li <longli@microsoft.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Nanyong Sun <sunnanyong@huawei.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth
Luiz Augusto von Dentz says:
====================
bluetooth pull request for net:
Core:
- hci_sync: Fix command skb lifetime during scan setup
- hci_sock: Serialize dead-device detachment
- RFCOMM: free the skb when the DLC has no owner
- RFCOMM: connect the session socket without rfcomm_mutex
- RFCOMM: Fix NULL tty_dev dereference in rfcomm_dev_shutdown
- SCO: serialise sco_conn lifetime against sco_recv_scodata()
- ISO: Reject concurrent BIS listener setup
- ISO: Serialize concurrent connect calls
- MGMT: Fix status of pending commands flushed on power off
Driver:
- btintel_pcie: fix TX descriptor bounds check
- btintel_pcie: use managed IRQ teardown
- btintel_pcie: Add shared HW reset for clean start
* tag 'for-net-2026-10-05' of git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth:
Bluetooth: ISO: Serialize concurrent connect calls
Bluetooth: ISO: Reject concurrent BIS listener setup
Bluetooth: hci_sync: Fix command skb lifetime during scan setup
Bluetooth: hci_sock: Serialize dead-device detachment
Bluetooth: RFCOMM: Fix NULL tty_dev dereference in rfcomm_dev_shutdown
Bluetooth: btintel_pcie: use managed IRQ teardown
Bluetooth: MGMT: Fix status of pending commands flushed on power off
Bluetooth: btintel_pcie: Add shared HW reset for clean start
Bluetooth: RFCOMM: connect the session socket without rfcomm_mutex
Bluetooth: btintel_pcie: fix TX descriptor bounds check
Bluetooth: SCO: serialise sco_conn lifetime against sco_recv_scodata()
Bluetooth: RFCOMM: free the skb when the DLC has no owner
====================
Link: https://patch.msgid.link/20261005194436.3594085-1-luiz.dentz@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- Taking a CPU offline could hang, or stall until the watchdog ejected
the BPF scheduler, when tasks on the dying CPU were still held by the
scheduler or sitting on a user dispatch queue. Re-enqueue them onto
the local queue when the runqueue goes offline so that the CPU pushes
them off like the other sched classes.
- The sequence number guarding against stale dispatches was per
runqueue, so a task re-enqueued on another CPU could get the same
number and a dispatch meant for its earlier instance was applied to
the new one. Use a per-task counter.
- A task dispatched to another CPU's local queue got its ops.dequeue()
only when picked to run and flagged as a core-sched pick. Call it at
insertion like for same-CPU dispatches.
* tag 'sched_ext-for-7.3-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Generate qseq from a per-task counter
selftests/sched_ext: Add a test for ops.dequeue() on remote local DSQ moves
sched_ext: Call ops.dequeue() when a task arrives on a remote local DSQ
sched_ext: Fix CPU hotplug hang when a dying CPU's tasks sit in the BPF scheduler
|
|
IPv6 uncached routes are linked to the global per-cpu lists,
rt6_uncached_list.
When unregistering a netdev, rt6_uncached_list_flush_dev() iterates
over the potentially long lists to find uncached routes tied to
the device and swap it with blackhole_netdev.
Since it is called for every device in dying netns under RTNL,
it adds O(N_dev x (N_cpu + N_route)) costs to any batched device
unregistration.
Let's call it once per batched device unregistration without RTNL.
Because it now runs without RTNL, rt->dst.dev and rt->rt6i_idev->dev
may transition to NETREG_UNREGISTERED at different times. Thus, we
must check that both devices are blackhole_netdev before unlinking
the route; otherwise, the remaining device reference would not be
flushed when the other device is unregistered.
In addition, since netdev_run_todo() runs even when IPv6 is disabled,
rt6_uncached_list_flush_dev() must check ipv6_mod_enabled() before
touching rt6_uncached_list.
Note also that netdev_run_todo() cannot be called before
rt6_uncached_list is initialised because:
1. unregister_netdevice_many_notify() panics when
dev_boot_phase == 1 (net_dev_init() is subsys_initcall()).
2. the only device registered before fs_initcall_sync() is
loopback and loopback_net_init() panics in case of failure.
Reported-by: Chris J Arges <carges@cloudflare.com>
Closes: https://lore.kernel.org/netdev/20260917-hash-bucket-route-lists-v3-0-30493a37b6eb@cloudflare.com/
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
Reviewed-by: Ido Schimmel <idosch@nvidia.com>
Tested-by: Chris J Arges <carges@cloudflare.com>
Link: https://patch.msgid.link/20261001204752.2572265-7-kuniyu@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
IPv4 uncached routes are linked to the global per-cpu lists,
rt_uncached_list.
When unregistering a netdev, rt_flush_dev() iterates over the
potentially long lists to find uncached routes tied to the device
and swap it with blackhole_netdev.
Since it is called for every device in dying netns under RTNL,
it adds O(N_dev x (N_cpu + N_route)) costs to any batched device
unregistration.
Let's call it once per batched device unregistration without RTNL.
Note that rt_flush_dev() must be called after setting dev->reg_state
to NETREG_UNREGISTERED. Otherwise, because rt_flush_dev(NULL) runs
without RTNL, it could race with unregister_netdevice_many_notify()
and prematurely purge routes for NETREG_UNREGISTERING dev, for
which flush_all_backlogs() has not been called yet.
Reported-by: Chris J Arges <carges@cloudflare.com>
Closes: https://lore.kernel.org/netdev/20260917-hash-bucket-route-lists-v3-0-30493a37b6eb@cloudflare.com/
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
Reviewed-by: Ido Schimmel <idosch@nvidia.com>
Tested-by: Chris J Arges <carges@cloudflare.com>
Link: https://patch.msgid.link/20261001204752.2572265-6-kuniyu@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
The next patch needs to use dev_isalive() in xfrm.
We should try to avoid using the dev_ prefix on APIs
we export via include/linux/.
Let's move it to include/linux/netdevice.h and rename
to netif_is_alive().
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
Reviewed-by: Ido Schimmel <idosch@nvidia.com>
Tested-by: Chris J Arges <carges@cloudflare.com>
Link: https://patch.msgid.link/20261001204752.2572265-4-kuniyu@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
Commit 9e8db5913264 ("net: avoid false positives in untrusted gso
validation") added a '&& skb->network_header' check before flow-dissecting
GSO packets without VIRTIO_NET_HDR_F_NEEDS_CSUM in
__virtio_net_hdr_to_skb(), because some callers (such as tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header()) called
virtio_net_hdr_*_to_skb() before initializing skb->network_header and
skb->dev.
Because __alloc_skb() and __build_skb_around() zero-initialize
skb->network_header to 0 (unlike mac_header and transport_header which
are initialized to ~0U), those four callers always had
skb->network_header == 0 and bypassed flow dissection in
__virtio_net_hdr_to_skb(). More generally, skb->network_header is an
offset from skb->head (where 0 is also a valid offset whenever
skb_headroom(skb) is 0), not a boolean flag.
Whenever the 'if (gso_type && skb->network_header)' branch was skipped,
the fallback 'else if (gso_type)' only pulled nh_min_len + thlen (40 bytes
for TCPv4) without dissecting the packet, without validating ip_proto or
n_proto, and without setting skb->transport_header.
If the packet has a malformed network header, it is not rejected and a
subsequent skb_probe_transport_header() also fails, leaving
skb->transport_header at ~0U (0xffff). Similarly, if an IPv4 packet
carries IP options (ihl > 5) or an IPv6 packet carries extension headers,
pulling only nh_min_len + thlen can leave the TCP header outside
skb->head. In both cases, tcp_hdrlen(skb) in skb_gso_transport_seglen()
reads out-of-bounds:
BUG: KASAN: slab-out-of-bounds in skb_gso_transport_seglen
Read of size 2 by task poc/133
skb_gso_transport_seglen (net/core/gso.c:155)
skb_gso_validate_mac_len (net/core/gso.c:270)
tbf_enqueue (net/sched/sch_tbf.c:260)
dev_qdisc_enqueue (net/core/dev.c:4227)
__dev_queue_xmit (net/core/dev.c:4884)
In addition, checking virtio_net_hdr_match_proto() only inside
'if (!skb->protocol)' before flow dissection both skipped validation when
skb->protocol was pre-set by the caller and rejected VLAN-tagged frames
whose outer L2 protocol is ETH_P_8021Q or ETH_P_8021AD.
Fix this by:
1. Initializing skb->dev and skb->network_header (plus skb->protocol for
IFF_TUN) before virtio_net_hdr_*_to_skb() in tun_get_user(),
tun_xdp_one(), virtnet_receive_done(), and raw_verify_header(). In
tun_get_user(), drop the redundant skb_reset_mac_header(skb) in the
IFF_TUN case since __virtio_net_hdr_to_skb() unconditionally resets
mac_header.
2. Removing '&& skb->network_header' and the unvalidated
'else if (gso_type)' fallback in __virtio_net_hdr_to_skb() so all GSO
packets without VIRTIO_NET_HDR_F_NEEDS_CSUM are flow-dissected, have
their transport header pulled into linear data, and have
skb->transport_header set.
3. Moving the virtio_net_hdr_match_proto() check to after
skb_flow_dissect_flow_keys_basic(), validating keys.basic.n_proto
against hdr_gso_type.
Fixes: 9e8db5913264 ("net: avoid false positives in untrusted gso validation")
Fixes: d5be7f632bad ("net: validate untrusted gso packets without csum offload")
Fixes: 924a9bc362a5 ("net: check if protocol extracted by virtio_net_hdr_set_proto is correct")
Reported-by: Weiming Shi <bestswngs@gmail.com>
Closes: https://lore.kernel.org/netdev/20260927163117.746432-2-bestswngs@gmail.com/
Signed-off-by: Eric Dumazet <edumazet@kernel.org>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Link: https://patch.msgid.link/20261001191140.2818991-3-edumazet@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
|
The RCU dereference macros end with a cast that is meant to hand back a
plain kernel pointer from a __rcu pointer.
For that it uses:
((typeof(*p) __force __kernel *)(local))
The problem is that typeof preserves every qualifier, including the
address space qualifiers (__rcu).
Recent versions of smatch[1] care about this and throw tens of warnings
like this one:
./include/trace/events/vb2.h:46:1: warning: incorrect type in assignment (different address spaces)
./include/trace/events/vb2.h:46:1: expected struct tracepoint_func *it_func_ptr
./include/trace/events/vb2.h:46:1: got struct tracepoint_func __rcu *
Use a new macro TYPEOF_NO_ADDRESS_SPACE() for the result type. This new
macro strips all the qualifiers when running with sparse (so const and
volatile are gone). But keeps all the qualifiers when running with the
compiler, caring about const/volatile mismatch.
[1] https://github.com/error27/smatch/commit/e53027a4e816a772403baafa83c09e4a94c1cb8f
Suggested-by: Dan Carpenter <dan.carpenter@linaro.org>
Link: https://lore.kernel.org/r/20260825-unqual-v1-1-7024fb81b4f9@chromium.org
Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
The kernel-doc comment of rcuref_read() refers to rcuread_is_dead, which
does not exist. The name is rcuref_is_dead. Say rcuref_is_dead.
Fixes: 3efa66ce6ee1 ("rcuref: Provide rcuref_is_dead()")
Assisted-by: LLM
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
cmpxchg_emu_u8() emulates one-byte cmpxchg() in terms of four-byte
cmpxchg() for the architectures lacking native one-byte atomics.
The same architectures also lack native two-byte cmpxchg(), where
such an operation is not supported and either fails to compile via
BUILD_BUG() or fails to link, because the bad pointer sentinels
these architectures declare are never defined.
Add cmpxchg_emu_u16(), the two-byte sibling. It reads the enclosing
word with READ_ONCE(), splices the two target bytes through a union
and loops on cmpxchg() of the full word until the compare succeeds.
Like cmpxchg_emu_u8() it is fully ordered.
Unlike cmpxchg_emu_u8() it casts the old and new values to u16
internally and returns unsigned long, taking the old and new values
as unsigned long, per the suggestion from David Laight. The switch
statements in the architecture macros instantiate every size case,
so a cmpxchg() on a pointer type checks the two-byte case as well,
and a u16 parameter or return would make the macro casts and return
conversions warn there. With unsigned long parameters and return the
call sites need no narrowing casts, pointer exchanges compile warning
free, and the function still compares and returns exactly the 16 bits
the caller asked for, which matches the hardware cmpxchg r16
behaviour where a 16-bit compare only looks at the low 16 bits of
the register.
cmpxchg_emu_u8() keeps returning the old value unmasked, so a caller
that passes a wider old than 8 bits still gets it back as passed,
matching the behaviour the one-byte emulator always had.
The Kconfig symbol gating this file is renamed from
ARCH_NEED_CMPXCHG_1_EMU to ARCH_NEED_CMPXCHG_1_2_EMU, as it now
selects both the one-byte and the two-byte emulation.
[ paulmck: Apply kernel test robot feedback. ]
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Suggested-by: David Laight <david.laight.linux@gmail.com>
Signed-off-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: David Laight <david.laight.linux@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Arnd Bergmann <arnd@kernel.org>
|
