| Age | Commit message (Collapse) | Author |
|---|
|
Patch series "Open HugeTLB allocation routine for more generic use", v4.
The motivation for this patch series is guest_memfd, which would like to
use HugeTLB as a generic source of huge pages but not adopt HugeTLB's
reservation at mmap() time.
By refactoring alloc_hugetlb_folio() and some dependent functions, there
is now an option to allocate HugeTLB folios without providing a VMA.
Specifically, HugeTLB allocation used to be dependent on the VMA to
1. Look up reservations in the resv_map
2. Get mpol, stored at vma->vm_policy
This refactoring provides hugetlb_alloc_folio(), which focuses on just the
allocation itself, and associated memory and HugeTLB charging (cgroups).
alloc_hugetlb_folio() still handles reservations in the resv_map and
subpools.
Regarding naming, I'm definitely open to alternative names :) I chose
hugetlb_alloc_folio() because I'm seeing this function as a general
allocation function that is provided by the HugeTLB subsystem (hence the
hugetlb_ prefix). I'm intending for alloc_hugetlb_folio() to be later
refactored as a static function for use just by HugeTLB, and HugeTLBfs
should probably use hugetlb_alloc_folio() directly.
To see how hugetlb_alloc_folio() is used by guest_memfd, the most recent
patch series that uses this more generic HugeTLB allocation routine is at
[1], and a newer revision of that patch series is at [2].
Independently of guest_memfd, I believe this change is useful in
simplifying alloc_hugetlb_folio(). alloc_hugetlb_folio() was so coupled
to a VMA that even HugeTLBfs allocates HugeTLB folios using a pseudo-VMA.
This patch (of 6):
The dequeue_hugetlb_folio_vma() function currently handles the gbl_chg
parameter to determine if a folio can be dequeued based on global page
availability. This leaks reservation-specific logic into the dequeueing
path.
Relocate this logic to alloc_hugetlb_folio() so that
dequeue_hugetlb_folio_vma() focuses solely on selecting and dequeuing a
folio. In alloc_hugetlb_folio(), only attempt to dequeue a folio if a
reservation exists (gbl_chg == 0) or if there are available huge pages in
the global pool.
No functional change intended.
Link: https://lore.kernel.org/20260702-hugetlb-open-up-v4-0-d53cefcccf34@google.com
Link: https://lore.kernel.org/20260702-hugetlb-open-up-v4-1-d53cefcccf34@google.com
Link: https://lore.kernel.org/all/cover.1747264138.git.ackerleytng@google.com/T/ [1]
Link: https://github.com/googleprodkernel/linux-cc/tree/wip-gmem-conversions-hugetlb-restructuring-12-08-25 [2]
Link: https://lore.kernel.org/all/agqaUcVp_hwH-VXr@localhost.localdomain/ [3]
Link: https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8@google.com [4]
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: James Houghton <jthoughton@google.com>
Acked-by: Oscar Salvador <osalvador@suse.de>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Jiaqi Yan <jiaqiyan@google.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Michael Roth <michael.roth@amd.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Pratyush Yadav <pratyush@kernel.org>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Shivank Garg <shivankg@amd.com>
Cc: Vishal Annapurve <vannapurve@google.com>
Cc: Yan Zhao <yan.y.zhao@intel.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
zswap_shrinker_count() calls mem_cgroup_flush_stats(), which takes the
global cgroup rstat lock synchronously. On machines with many CPUs and
NUMA nodes, this creates severe lock contention in the kswapd reclaim
path:
- Multiple kswapd threads (one per NUMA node) run concurrently.
- do_shrink_slab() invokes zswap_shrinker_count() for each
memcg-aware shrinker pass.
- Each call flushes the full cgroup rstat hierarchy under the global
lock.
On AMD EPYC 9684X machines (96 cores, 192 threads, 12 NUMA nodes) running
production workloads with zswap enabled, perf shows 2.88% of kernel cycles
in osq_lock contention from this path:
2.88% [k] osq_lock
--__mutex_lock.constprop.0
--__cgroup_rstat_lock
--cgroup_rstat_flush_locked
--cgroup_rstat_flush
--zswap_shrinker_count
do_shrink_slab
shrink_slab
shrink_node
balance_pgdat
kswapd
84% of kswapd kernel cycles are spent in
shrink_slab -> zswap_shrinker_count -> cgroup_rstat_flush, not in actual
page reclaim (shrink_lruvec).
Controlled A/B on identical hardware and workload:
shrinker=Y: 2.88% osq_lock, memory PSI 1.58%
shrinker=N: 0.00% osq_lock, memory PSI 0.57%
eBPF-based rstat lock wait measurement across 8 production metals
confirms the contention splits cleanly along shrinker enablement:
shrinker=Y: 50-250x more contended lock acquisitions (248/s vs 1.1/s)
shrinker=N: baseline lock wait (0.0017 s/s vs 1.04 s/s)
zswap_shrinker_count() only produces a heuristic estimate, scaled by
compression ratio via mult_frac(). The actual writeback happens in
zswap_shrinker_scan(). Slightly stale stats are acceptable here.
Switch to mem_cgroup_flush_stats_ratelimited(), which only flushes if
the periodic 2-second flusher is one full cycle late. This matches the
approach already used in prepare_scan_control() (mm/vmscan.c) for the
same reclaim path.
After applying this patch, rstat flush latency and lock wait time on
shrinker=Y machines dropped to the same level as shrinker=N controls,
while the zswap shrinker continues to function (pool size remains
bounded under the max_pool_percent cap).
Previously discussed:
- Chengming Zhou (Dec 2023): rstat contention from
zswap_shrinker_count [1]
- Shakeel Butt (Aug 2024): zswap_shrinker_count still uses sync
flush [2]
- Yosry Ahmed (Aug 2024): suggested eliminating in-kernel
flushers [3]
- Jesper Dangaard Brouer (Sep 2024): cgroup/rstat V11 patch [4]
Link: https://lore.kernel.org/20260702180908.150136-1-yunzhao@cloudflare.com
Link: https://lore.kernel.org/linux-mm/20231206103935.3440502-1-zhouchengming@bytedance.com/ [1]
Link: https://lore.kernel.org/linux-mm/CALvZod7LFxLCxVpOFH8b2Ppm8T40HPGMKQwX_=NPCWB_mFW+oQ@mail.gmail.com/ [2]
Link: https://lore.kernel.org/linux-mm/CAJD7tkYvFyOSX+rP_FKGBhxvZiCDxtpsNp-c5CGOA-4Bq9oXSg@mail.gmail.com/ [3]
Link: https://lore.kernel.org/linux-mm/172616070094.2055617.17676042522679701515.stgit@firesoul/ [4]
Suggested-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Yunzhao Li <yunzhao@cloudflare.com>
Tested-by: Yunzhao Li <yunzhao@cloudflare.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Jesper Dangaard Brouer <hawk@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Yosry Ahmed <yosry@kernel.org>
Cc: Yunzhao Li <yunzhao@cloudflare.com>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
A local unprivileged process can create a shmem/tmpfs file with i_size ==
LLONG_MAX using memfd_create() and fallocate(). If the last page is
present in the page cache, lseek(SEEK_HOLE) on that page returns
0x8000000000000000 as a successful offset, which is LLONG_MIN when stored
in loff_t.
The same file has readable data at the last byte, but SEEK_DATA from that
offset returns ENXIO.
The overflow is in mapping_seek_hole_data():
pos = round_up((u64)pos + 1, seek_size);
For the final page below LLONG_MAX, the next page boundary is
0x8000000000000000, which is then used as a signed file offset. When
assigned to the loff_t pos, this overflows to LLONG_MIN, so a subsequent
"pos > end" comparison does not catch it.
Keep mapping_seek_hole_data() inside its documented [start, end) search
range: compute round_up() into a u64 variable and compare against (u64)end
so the overflow is detected, then clamp pos to end when the rounded-up
value goes past the search limit.
Link: https://lore.kernel.org/20260630125047.703170-1-yanzhen20011121@163.com
Signed-off-by: Zhen Yan <yanzhen20011121@163.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
No code touches damon_region->nr_accesses_bp field. Remove it.
Link: https://lore.kernel.org/20260630040812.149729-19-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_moving_sum() is no longer being called for real purpose but its unit
test. Testing a function that is not being used for real users makes no
sense. Remove the test and the function.
Link: https://lore.kernel.org/20260630040812.149729-18-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The function is not using attrs parameter. Remove it.
Link: https://lore.kernel.org/20260630040812.149729-17-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The function is not using the parameter. Remove it.
Link: https://lore.kernel.org/20260630040812.149729-16-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_update_region_access_rate() is not using attrs parameter. Remove
it.
Link: https://lore.kernel.org/20260630040812.149729-15-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
DAMON core sets and updates nr_accesses_bp in multiple places. It
explains how delicate it is. The field is no more being used for any real
purpose, and replaced by a simpler function. Remove the setups and
updates.
Link: https://lore.kernel.org/20260630040812.149729-14-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
drgn_dump_damon_status is dumping nr_accesses_bp field for future use
case. nr_accesses_bp is not being used for a real purpose, though. Hence
there will be no future test for it. Do not dump it.
Link: https://lore.kernel.org/20260630040812.149729-13-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
DAMON core unit tests set up nr_accesses_bp for representing realistic
damon_region, and also test the field. nr_acceses_bp is no longer being
used for a real use case. Remove the setup and tests.
Link: https://lore.kernel.org/20260630040812.149729-12-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_verify_merge_regions_of() is only for nr_accesses_bp validation.
But nr_accesses_bp is no more being used for a real purpose. Remove the
validation.
Link: https://lore.kernel.org/20260630040812.149729-11-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
nr_accesses_bp is no longer being used in real use cases. Remove its
validation function.
Link: https://lore.kernel.org/20260630040812.149729-10-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
nr_accesses_bp is delicate. Once it is corrupted, the consequence is the
bad madness of DAMON monitoring results. From developments of features of
size, we historically found nr_accesses_bp can be corrupted by complicated
bugs that are not easy to debug. Hence we added a function for finding
the corruption and fixing it right away.
There are no more uses of nr_accesses_bp. Hence the function for
corruption detection and fix is no more needed. Rip it out.
Link: https://lore.kernel.org/20260630040812.149729-9-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_nr_accesses_mvsum() returns a value same to nr_accesses_bp. Also
the function is more simple and therefore more tolerant to errors.
Execution of the function would be more expensive than the simple read of
the field, but because the function is quite simple, the overhead should
be negligible. Use it in the DAMON sysfs interface for scheme-tried
regions, instead of the nr_accesses_bp.
Link: https://lore.kernel.org/20260630040812.149729-8-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_nr_accesses_mvsum() returns a value same to nr_accesses_bp. Also
the function is more simple and therefore more tolerant to errors.
Execution of the function would be more expensive than the simple read of
the field, but because the function is quite simple, the overhead should
be negligible. Use it in the DAMON region exporting trace points instead
of the nr_accesses_bp.
Link: https://lore.kernel.org/20260630040812.149729-7-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_nr_accesses_mvsum() returns a value same to nr_accesses_bp. Also
the function is more simple and therefore more tolerant to errors.
Execution of the function would be more expensive than the simple read of
the field, but because the function is quite simple, the overhead should
be negligible. Use it in __damos_valid_target() instead of the
nr_accesses_bp.
Link: https://lore.kernel.org/20260630040812.149729-6-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_set_attrs() works like reverting aggregations that were made so far
for this aggregation window. If this is the end of the aggregation,
however, kdamond_fn() will do the operations at the end of the aggregation
interval, using cached timestamps. For such operations that rely on
damon_region->nr_accesses, damon_update_monitoring_results() doesn't reset
the nr_accesses if it is called at the end of the aggregation window.
damon_nr_accesses_mvsum() works with fresh timestamps, though. The
nr_accesses that are not reset in this case can make the logic to
unnecessarily count nr_accesses, resulting in returning
higher-than-expected pseudo moving sum nr_accesses. No code is using
damon_nr_accesses_mvsum() yet, so this is not causing a real problem.
Following commits will add usage of the function, though. For safe
usages, calculate the pseudo moving sum without nr_accesses if the
remaining window is full.
Link: https://lore.kernel.org/20260630040812.149729-5-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Each iteration of kdamond_fn() main loop caches and use the next
aggregation time (next_aggregation_sis) because it can be updated in the
middle, inside kdamond_call(). If that happens,
damon_update_monitoring_result() is called for scaling the access
frequency information of each region according to the changed intervals.
The function does not update damon_region->last_nr_accesses when it is at
the end of the aggregation, because it will anyway be reset after the
function is executed, in kdamond_reset_aggregated().
Let's suppose damon_nr_accesses_mvsum() is called with the not yet updated
last_nr_accesses. It will use the fresh next_aggregation_sis in the
context instead of the cached one, unlike kdamond_fn(). As a result, use
of not updated last_nr_acceses with the updated next_aggregation_sis
result in returning wrong value.
There is no such damon_nr_accesses_nvsum() call at the moment, so this is
no problem. It is planned to add such calls, though. Prevent the issue
by updating last_nr_accesses always. This adds overhead, but that's fine
because the overhead is not big, and it is anyway not a fast path.
Link: https://lore.kernel.org/20260630040812.149729-4-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add a simple unit test for damon_nr_accesses_mvsum()'s internal core
logic, damon_mvsum(). The test contains cases for just-started windows,
partially completed windows, and just-completed windows.
Link: https://lore.kernel.org/20260630040812.149729-3-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/damon: optimize out nr_accesses_bp".
TLDR: Replace damon_region->nr_accesses_bp, which is easy to be wrong,
with a simpler on-demand moving sum function, damon_nr_accesses_mvsum().
Background
==========
DAMON's monitoring output (access pattern snapshot, or more technically
speaking, damon_region->nr_accesses) is completed once per aggregation
interval, which is 100 ms by default. Users can arbitrarily increase the
interval for demand. Under the suggested intervals auto-tuning setup, it
can span up to 200 seconds. If the aggregation interval is too long, the
snapshot users cannot use it in reasonable time. To mitigate this, we
introduced a new field of damon_region, namely nr_accesses_bp. It
contains a pseudo moving sum of nr_accesses in bp units and is updated for
each sampling interval.
It turned out keeping it correctly updated every sampling interval is not
that easy. From online parameter update feature development and more
experimental hacks, we found it is easy to be corrupted. Once it is
corrupted, DAMON's monitoring outputs become quite insane. Hence we added
a few validation checks. It is easy to be corrupted because it requires
every update per sampling interval to be correct.
Solution
========
There is no real reason to keep it updated every sampling interval. Due
to the simple pseudo-moving sum mechanism and existing helper field
(last_nr_accesses), we can also calculate the pseudo moving sum on demand
in a much simpler way.
Implement a function for getting the pseudo moving sum on demand, and
replace nr_accessses_bp uses with the new function. Also remove no more
needed tests for nr_accesses_bp and the per-sampling interval update
functions. Finally, remove the nr_accesses_bp. The new function is quite
simple.
Discussion
==========
Depending on the use case, multiple nr_accesses readers could be executed
in the same kdamond_fn() main loop iteration, which is executed once per
sampling interval. Such readers include DAMON region exporting
tracepoints (damon_[region_]aggregated and damos_before_apply), DAMOS, and
DAMON sysfs interface logic for update_schemes_tried_regions command. In
this case, the new function will be called multiple times and this could
be overhead compared to the old logic, which simply reads the field
without any additional work. Nonetheless, the new function is quite
simple. And the new approach does nothing while there is no need to read.
The old approach had to execute its update function for each region for
every sampling interval. Hence the new approach is believed to be even
more lightweight in common case, and the overhead is anyway negligible.
One more advantage of this change is that one field from the damon_region
struct is removed. On setups that uses a high number of DAMON regions,
this could be a potential memory space benefit.
Patches Sequence
================
Patch 1 introduces the new function for getting the pseudo moving sum of
nr_accesses on demands. Patch 2 implements a unit test for the new
function's internal logic. Patch 3 and 4 update monitoring logic and the
new function to ready for safe use on the existing logic. Patches 5-7
replace uses of nr_accesses_bp in DAMOS, tracepoints and DAMON sysfs
interface with the new function, respectively. Patches 8-10 removes
nr_accesses_bp validation functions in DAMON core, one by one. Patches 11
and 12 further remove tests and test helper for nr_accesses_bp,
respectively. Patches 13 removes the setups and updates or nr_accesses_bp
field. Patches 14-16 cleans up function parameters that are no more being
used due to the previous patch. Patch 17 removes the function that was
used for updating nr_accesses_bp field with its unit test, which is the
single remaining caller of the function. Finally, patch 18 removes
damon_region->nr_accesses_bp field.
This patch (of 18):
Introduce a new DAMON core function, damon_nr_accesses_mvsum(). It
returns a pseudo moving sum value of a given region's nr_accesses for the
last aggregation interval. The internal logic is the same to
nr_accesses_bp. The difference is that nr_accesses_bp is updated for each
sampling interval, while the new function needs to be executed only when
requested. Hence the function's return value is the same as the value of
nr_accesses_bp.
Link: https://lore.kernel.org/20260630040812.149729-1-sj@kernel.org
Link: https://lore.kernel.org/20260630040812.149729-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Gregory points out that these descriptions are cursed and confusing,
considering what these flags actually do. This is mostly due to historic
implementation choices and cgroup1 baggage. Improve the description of
their actual effects.
Link: https://lore.kernel.org/20260701182102.1586784-1-hannes@cmpxchg.org
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reported-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Acked-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Non-cooperarive uffd events are inherently racy and can happen in parallel
with other userfaultfd operations.
During event tests in uffd-unit-tests, the uffd monitor calls
UFFDIO_UNREGISTER upon receiving UFFD_EVENT_REMOVE.
In parallel, the faulting_process() verifies that the removed memory is
actually zeroed.
If a verification read wins the race with UFFDIO_UNREGISTER, it causes a
missing fault that uffd monitor would receive after UFFDIO_UNREGISTER is
complete. The monitor resolves the fault using UFFDIO_COPY that fails
with -ENOENT which means that VMA has been changed (see commit
27d02568f529 ("userfaultfd: mcopy_atomic: return -ENOENT when no
compatible VMA found")).
Treat -ENOENT returned by UFFDIO_COPY as non-fatal, the same way -EEXIST
is treated for concurrent faults, and don't fail the test.
Link: https://lore.kernel.org/20260701200932.1470525-1-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
dmirror_fault() is called from the dmirror_read() and dmirror_write()
retry loops after dmirror_do_read() or dmirror_do_write() finds a missing
device page table entry.
If the mirrored mm has already exited, mmget_not_zero() fails. The
current code returns 0 in that case, which tells the caller that faulting
succeeded even though no page was faulted and no device page table entry
was installed. The caller then retries the same address, hits -ENOENT
again, and can loop forever without making progress.
Return -EFAULT instead, so the ioctl fails when the mirrored mm is no
longer faultable.
Link: https://lore.kernel.org/178294308408.327222.3319445682023999403.stgit@skinsburskii
Fixes: b2ef9f5a5cb37 ("mm/hmm/test: add selftest driver for HMM")
Signed-off-by: Stanislav Kinsburskii <skinsburskii@linux.microsoft.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Ralph Campbell <rcampbell@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
We miss a failed allocation check for pgdat->per_cpu_nodestats, which
results in a NULL deref when we offset into the per-cpu area.
Propagate -ENOMEM up the stack and leave per_cpu_nodestats pointing at
boot_nodestats so a later online can retry the allocation.
hotadd_init_pgdat() returns NULL on failure, which __try_online_node()
already maps to -ENOMEM.
On failure nothing needs to be unwound:
- the node is never marked online
- per_cpu_nodestats is left pointing at boot_nodestats
- __add_memory_resource() cleans up pending memblock resources
- later online attempts retry the per_cpu_nodestats allocation
Link: https://lore.kernel.org/20260701221613.2818148-1-gourry@gourry.net
Fixes: 75ef71840539 ("mm, vmstat: add infrastructure for per-node vmstats")
Signed-off-by: Gregory Price <gourry@gourry.net>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260627202243.758289-1-gourry%40gourry.net
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
per various review comments
Link: https://lore.kernel.org/87bbf335-648f-4065-abc8-3eaab5a3beeb@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
migrate_vma_collect_pmd() can detect a large folio while holding the PTE
lock, then drop the PTE lock before calling migrate_vma_split_folio().
The split helper took its own reference, but only after the lock had
already been dropped.
One way to hit this is device migration over a range that contains a large
folio. The walker reads the PTE while holding the PTE lock and derives
the folio either from a present PTE via vm_normal_page(), or from a
non-present PTE that encodes a device-private softleaf entry. It then has
to drop the PTE lock because split_folio() can block. Before
migrate_vma_split_folio() gets a folio reference, concurrent reclaim,
migration, or truncation can replace or clear the entry and drop the last
reference to the folio. The split helper would then take a reference and
lock on a stale folio pointer.
Take a temporary reference before dropping the PTE lock and pass that
reference into migrate_vma_split_folio(). The helper consumes the
reference, so split_folio() still sees only the expected caller pin
instead of an extra pin that could make the split fail.
Link: https://lore.kernel.org/20260701140638.840773-1-usama.arif@linux.dev
Fixes: 022a12deda53 ("mm/migrate_device: handle partially mapped folios during collection")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reported-by: sashiko-bot <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260630164143.1595669-1-usama.arif%40linux.dev
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
build fix
Link: https://lore.kernel.org/DJSGLP9VFLRI.355AAWR2VHIEL@nvidia.com
Signed-off-by: "Zi Yan" <ziy@nvidia.com>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Zhen Ni <zhen.ni@easystack.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
v11:
According to the comments of David, simplify page_to_nid() to pass
&(PF_POISONED_CHECK(page)->flags) directly.
v10:
According to the comments of David, drop the redundant CONFIG_NUMA split
in page_to_nid()/folio_nid() and remove the SECTIONS_WIDTH != 0 guard
around ASSERT_EXCLUSIVE_BITS() in memdesc_section().
Link: https://lore.kernel.org/20260708083308.747930-1-hui.zhu@linux.dev
Co-developed-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
KCSAN reports a data race between page_to_nid()/folio_pgdat() reading
page->flags and folio_trylock()/folio_lock() concurrently doing
test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:
BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp
The race is benign: nid/zone bits are set once at page init and never
overlap with PG_locked. However, ASSERT_EXCLUSIVE_BITS() inside
memdesc_nid/zonenum() was checking a by-value copy of the flags word, not
the live page->flags, so it failed to annotate the real access.
Change memdesc_nid(), memdesc_zonenum(), memdesc_section(), and
memdesc_is_zone_device() to take a const memdesc_flags_t * and update all
callers to pass &page->flags / &folio->flags, so ASSERT_EXCLUSIVE_BITS()
operates on the actual shared word.
Guard the ASSERT_EXCLUSIVE_BITS() call in memdesc_zonenum() under
ZONES_WIDTH != 0 to avoid a zero-mask check on configs where the zone
field is absent. memdesc_section() needs no such guard, since
SECTIONS_WIDTH is never 0 wherever SECTION_IN_PAGE_FLAGS is defined.
Under CONFIG_NUMA=n, memdesc_nid() itself is stubbed to "return 0" instead
of reading page->flags, since NODES_MASK is 0 and the check can never
fire; page_to_nid()/folio_nid() now just call memdesc_nid()
unconditionally and rely on that stub, instead of duplicating the
CONFIG_NUMA split at each call site.
Link: https://lore.kernel.org/20260630070810.470763-1-hui.zhu@linux.dev
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Co-developed-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Passing a stack-local child_pid to clone() with CLONE_CHILD_CLEARTID is
unsafe: the kernel clears that address when the child exits, which may
happen after the test function has returned and the stack slot has been
reused.
Neither testcase uses the settid/cleartid pointers for synchronization.
Drop CLONE_PARENT_SETTID and CLONE_CHILD_CLEARTID and pass NULL for the
clone tid arguments. Wait for the clone child to exit via tkill in
test_sigsegv_handler_with_different_pkey_for_stack(), matching
test_pkru_sigreturn(), so the detached thread cannot overlap with the next
testcase.
Link: https://lore.kernel.org/20260706081600.3570203-7-lihongfu@kylinos.cn
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Cc: David Hildenbrand <david@kernel.org>
Cc: Joey Gouly <joey.gouly@arm.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Keith Lucas <keith.lucas@oracle.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>
Cc: Ross Zwisler <zwisler@google.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Yury Khrustalev <yury.khrustalev@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add missing pthread_create() return checks in pkey sighandler tests to
avoid hanging in pthread_cond_wait() when thread creation fails.
Link: https://lore.kernel.org/20260706081600.3570203-6-lihongfu@kylinos.cn
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Reviewed-by: Kevin Brodsky <kevin.brodsky@arm.com>
Tested-by: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Joey Gouly <joey.gouly@arm.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Keith Lucas <keith.lucas@oracle.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>
Cc: Ross Zwisler <zwisler@google.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Yury Khrustalev <yury.khrustalev@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add missing checks against mmap() return value, replace (void *)-1 with
MAP_FAILED for better readability and consistency.
Link: https://lore.kernel.org/20260706081600.3570203-5-lihongfu@kylinos.cn
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Reviewed-by: Kevin Brodsky <kevin.brodsky@arm.com>
Tested-by: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Joey Gouly <joey.gouly@arm.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Keith Lucas <keith.lucas@oracle.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>
Cc: Ross Zwisler <zwisler@google.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Yury Khrustalev <yury.khrustalev@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Use pkey_assert(0) instead of perror("clone") when clone_raw() fails. The
old path only printed an error and continued; the test now exits via
pkey_assert() on failure so it does not hang or proceed with an invalid
child.
Link: https://lore.kernel.org/20260706081600.3570203-4-lihongfu@kylinos.cn
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Reviewed-by: Kevin Brodsky <kevin.brodsky@arm.com>
Tested-by: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Joey Gouly <joey.gouly@arm.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Keith Lucas <keith.lucas@oracle.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>
Cc: Ross Zwisler <zwisler@google.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Yury Khrustalev <yury.khrustalev@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add per-test tracing to the pkey signal-handler selftest and use
pkey_assert() for error handling. Each test enables tracing at start and
disables it at end; on failure, pkey_assert() calls abort_hooks() to turn
tracing off so ftrace is not left enabled.
Link: https://lore.kernel.org/20260706081600.3570203-3-lihongfu@kylinos.cn
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Reviewed-by: Kevin Brodsky <kevin.brodsky@arm.com>
Tested-by: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Joey Gouly <joey.gouly@arm.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Keith Lucas <keith.lucas@oracle.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>
Cc: Ross Zwisler <zwisler@google.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Yury Khrustalev <yury.khrustalev@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "selftests/mm: refactor pkey helpers and fix mmap error
handling", v10.
The main changes in this series are to refactor shared tracing and
assertion helpers into a common file, unify both pkey selftests on
pkey_assert() and per-test tracing for consistent diagnostics, and add
missing mmap() return checks with MAP_FAILED used throughout for
readability and consistency.
This patch (of 10):
Move pkey selftest debugging helpers into shared code so both pkey
selftests can use the same tracing and abort-hook logic. Also fix
cat_into_file() to print file, not str, in the open() failure message.
Link: https://lore.kernel.org/20260706081600.3570203-1-lihongfu@kylinos.cn
Link: https://lore.kernel.org/20260706081600.3570203-2-lihongfu@kylinos.cn
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Reviewed-by: Kevin Brodsky <kevin.brodsky@arm.com>
Tested-by: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Joey Gouly <joey.gouly@arm.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Keith Lucas <keith.lucas@oracle.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>
Cc: Ross Zwisler <zwisler@google.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Yury Khrustalev <yury.khrustalev@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
shmem_fault and shmem_writeout access inode->i_private without holding a
lock, while shmem_fallocate is modifying it while holding a lock, thus a
data-race is created.
Fix this by using READ_ONCE and WRITE_ONCE, which provides an atomic,
lockless read and write of inode->i_private which prevents compiler
optimizations such as caching in registers and add writing to
inode->i_private with WRITE_ONCE to prevent the compiler from writing in
registers.
Link: https://lore.kernel.org/20260630120222.11562-1-yahia.a.abdrabou@gmail.com
Signed-off-by: yahia ahmed <yahia.a.abdrabou@gmail.com>
Reported-by: syzbot+76cc716982cf0254f302@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=76cc716982cf0254f302
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Clean up mm/vmpressure.c by separating the cgroup v1 userspace eventfd
interface from the shared and v2 in-kernel code.
Currently, almost half of mm/vmpressure.c exists to serve tree=true:
struct vmpressure_event, the events list and its mutex, the work_struct
and vmpressure_work_fn that drains tree_scanned/tree_reclaimed, the parent
walk, vmpressure_event(), vmpressure_register_event(),
vmpressure_unregister_event(), and vmpressure_prio() (which always calls
vmpressure() with tree=true).
Move it all into mm/memcontrol-v1.c (built only when CONFIG_MEMCG_V1=y) as
a single contiguous block, following the per-component layout already used
by that file. Keeping the v1 vmpressure code with the rest of the
deprecated cgroup v1 memory controller makes the full footprint of the
CONFIG_MEMCG_V1 option easy to see in one place, which matters more than
component-level file separation for code that has no active development.
vmpressure.c keeps the shared bits (constants, vmpressure_calc_level, the
runtime hierarchy check, the tree=false body, init/cleanup plumbing) and
calls into three small v1 hooks for the tree=true accumulator and the v1
portions of init/cleanup. The hooks have static-inline no-op stubs in
include/linux/vmpressure.h for the !MEMCG_V1 case, so callers don't need
ifdefs. vmpressure_prio() gets the same treatment, which means vmscan.c's
call site disappears at compile time on v2-only kernels.
The only #ifdef CONFIG_MEMCG_V1 in source remains around the v1-only
fields inside struct vmpressure itself.
Memory savings on CONFIG_MEMCG_V1=n (measured with pahole):
struct vmpressure : 112B -> 24B
struct mem_cgroup : 1664B -> 1536B
This split is the first step toward eventually making vmpressure
CONFIG_MEMCG_V1 only. The v2 in-kernel socket pressure path (tree=false)
cannot be removed today immediately: PSI is not an exact replacement for
vmpressure, and switching networking socket-buffer back-off to PSI may
regress networking performance or increase memory pressure in workloads
that today rely on vmpressure's hysteresis. The medium-term plan is to
introduce a PSI-based socket-pressure path, keep vmpressure available for
v2 behind a defconfig as an opt-out for several releases, and only then
drop the tree=false path entirely, at which point everything that remains
of the vmpressure block in mm/memcontrol-v1.c is the whole subsystem.
Link: https://lore.kernel.org/20260630112617.1198623-3-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Simplify the guard. Both cgroup_subsys_on_dfl() and tree are bool, so
the two combinations that have no consumer (v1 + tree=false, v2 +
tree=true) are exactly the cases where dfl == tree.
Link: https://lore.kernel.org/e8e1a409-48d8-4fa7-ae98-49485a1607f6@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/vmpressure: reduce CPU, memory and code overhead on
cgroup v2", v3.
The vmpressure subsystem has two distinct consumers, gated by the @tree
argument:
tree=false : in-kernel socket pressure, consumed by TCP/SCTP. This
is cgroup v2 only; v1 sockets read memcg->tcpmem_pressure
instead.
tree=true : cgroup v1 userspace eventfd notifications via the
memory.pressure_level / cgroup.event_control interface.
v2 has no equivalent (userspace gets reclaim signals
through memory.pressure / PSI, which doesn't touch
vmpressure).
So of the four (hierarchy, tree) combinations, only two carry data that
anyone reads. The existing early return in vmpressure() covered v1 +
tree=false; the symmetric v2 + tree=true case was falling through and
doing the full lock / accumulate / schedule_work / parent-walk dance, even
though the events list it eventually iterates is empty on cgroup v2
(vmpressure_register_event() is wired up only through the v1 cftype
"memory.pressure_level" and can't be reached from a v2 memcg).
Patch 1 extends the existing early return to also skip v2 + tree=true. On
a v2-only host this eliminates a contended path where reclaimers can
serialize on a single global sr_lock. bpftrace on a 176-core production
host (cgroup v2, 285 memcgs, sustained reclaim) showed ~16,200 such calls
per minute with tree = true.
Patch 2 follows up with a cleanup: it splits the v1 userspace eventfd
interface (struct vmpressure_event, the events list and its mutex, the
work_struct and its handler, the parent walk, vmpressure_register_event /
unregister_event, and vmpressure_prio) into a new mm/memcontrol-v1.c built
only when CONFIG_MEMCG_V1=y, behind small no-op stubs in the header.
mm/vmpressure.c keeps the shared bits and the tree=false socket-pressure
path. The size of vmpressure.c goes down to half and the code is much
more simpler. The only #ifdef CONFIG_MEMCG_V1 remaining in source is
around the v1-only fields inside struct vmpressure itself. Memory savings
on CONFIG_MEMCG_V1=n:
struct vmpressure : 112B -> 24B
struct mem_cgroup : 1664B -> 1536B
This split is the first step toward eventually making vmpressure
CONFIG_MEMCG_V1 only. The v2 in-kernel socket pressure path (tree=false)
cannot be removed today immediately: PSI is not an exact replacement for
vmpressure, and switching networking socket-buffer back-off to PSI may
regress networking performance or increase memory pressure in workloads
that today rely on vmpressure's hysteresis. The medium-term plan is to
introduce a PSI-based socket-pressure path, keep vmpressure available for
v2 behind a defconfig as an opt-out for several releases, and only then
drop the tree=false path entirely, at which point everything that remains
in mm/memcontrol-v1.c is the whole subsystem.
This patch (of 2):
vmpressure() has two outputs gated by the @tree argument:
@tree=false drives in-kernel socket pressure (mem_cgroup_set_
socket_pressure), consumed by TCP/SCTP. This only
applies on cgroup v2; on v1 socket memory is charged
separately via tcpmem and the consumer reads
memcg->tcpmem_pressure instead.
@tree=true drives userspace eventfd notifications via the v1
memory.pressure_level / cgroup.event_control interface.
v2 has no equivalent: userspace gets reclaim signals
through memory.pressure (PSI), which does not touch
vmpressure.
The existing early return covered v1 + @tree=false. The symmetric v2 +
@tree=true case was falling through and doing the full lock / accumulate /
schedule_work / parent-walk dance for an events list that can never be
populated. bpftrace on a 176-core production host (cgroup v2,
CONFIG_MEMCG_V1=n, 285 memcgs, sustained reclaim) showed ~16,200
@tree=true vmpressure() calls per minute. Add an early return that skips
cgroup v2 + tree = true which avoids us doing all this work. On a v2-only
host this also eliminates a lock contention path that can serialise
reclaimers on a single global sr_lock.
Link: https://lore.kernel.org/20260630112617.1198623-1-usama.arif@linux.dev
Link: https://lore.kernel.org/20260630112617.1198623-2-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Since 79359d6d24df ("hugetlb: perform vmemmap optimization on a list of
pages") __hugetlb_vmemmap_optimize_folios() unconditionally issues a final
flush_tlb_all() in its out path. However, a TLB flush must be paired with
an actual page table modification.
When the input folio list is empty, neither PMD splitting nor PTE
remapping takes place, so no page tables are modified and the flush is
pure overhead. An empty list is reached in common paths such as
gather_bootmem_prealloc_node() on nodes without bootmem gigantic pages,
hugetlb_pages_alloc_boot_node() when no pages were allocated, and runtime
allocation failure paths in set_max_huge_pages().
Add an early return for empty lists. This restores the basic invariant
that TLB flushes are only issued when page tables have been modified, and
it also makes the NULL hstate passed by gather_bootmem_prealloc_node() on
an empty list harmless.
Assisted-by: kimi-cli:kimi-k2.7 code
Assisted-by: Github Copilot:gpt-5.2 #Reported-by
Link: https://lore.kernel.org/20260701053422.3664813-1-guanwentao@uniontech.com
Link: https://lore.kernel.org/20260630113857.3319612-1-guanwentao@uniontech.com
Fixes: 79359d6d24df ("hugetlb: perform vmemmap optimization on a list of pages")
Signed-off-by: Wentao Guan <guanwentao@uniontech.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Guan Wentao <guanwentao@uniontech.com>
Cc: Oscar Salvador <osalvador@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
A check in mas_walk() was incorrect and caused inefficient use of the
maple state. The same issue existed in mas_erase(), but was left unfixed.
Making a helper function is the obvious answer.
Link: https://lore.kernel.org/20260630190843.3563858-20-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Prior to ending the ascension loop of larger operations like split,
rebalance, and spanning store the gap in the node had been calculated.
Once the node is inserted into the tree, the gap is recalculated in
mas_update_gap(). This can be avoided by creating a helper for
mas_update_gap() that accepts the known gap value, which reduces the
operations required for gap updating path.
Link: https://lore.kernel.org/20260630190843.3563858-19-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The mas_prev_range() function takes a min and not a max.
Link: https://lore.kernel.org/20260630190843.3563858-18-liam@infradead.org
Fixes: 6b9e93e01020 ("maple_tree: add mas_prev_range() and mas_find_range_rev interface")
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The GFP flags in two tests are obviously incorrect. Make the tests
correctly run by updating the GFP flags.
Link: https://lore.kernel.org/all/d9cbb89faa5bdb71d451781d214a51ce8923a83e.camel@perches.com/
Link: https://lore.kernel.org/20260630190843.3563858-17-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Reported-by: Joe Perches <joe@perches.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
During a discussion on the maple tree erase process and GFP flags, Jason
suggested there be an amendment to the documentation to clarify the
situation on allocations within the tree.
The added text is an attempt to better explain that the tree may allocate,
even when erasing, and provide some guidance on how to work around such
issues.
Link: https://lore.kernel.org/all/20260617180419.GA231643@ziepe.ca/
Link: https://lore.kernel.org/20260630190843.3563858-16-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Suggested-by: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Rik van Riel <riel@surriel.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Allocations should never fail in the circumstances that are expected to
occur. Add checks in the code to ensure the circumstances are correctly
set up by the user and warn if they are not.
Also add a warning on failure to allocate, which should never happen.
Link: https://lore.kernel.org/20260630190843.3563858-15-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
State that the mas_erase() and mtree_erase() functions may use GFP_KERNEL
on allocation retry. Don't just depend on people reading the
documentation by adding a check that will warn of the use.
Link: https://lore.kernel.org/20260630190843.3563858-14-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Reviewed-by: Rik van Riel <riel@surriel.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
If mas_alloc_cyclic() is called during a low memory situation, it is
possible the lock may be dropped so reclaim can occur. There is a window
where some other task may allocate the same id and cause the mas_insert()
to fail with -EEXIST. In this scenario the function will return -EEXIST,
which is not expected.
Modifying the retry on mas_nomem() to re-search for a slot means that any
race with other writes will not matter as the lock will be held between
finding the index and writing the index.
Moving the flag logic avoids cases where the flag is modified on drop
lock/reacquire or when the write fails after clearing the flag.
No existing users are exposed to this issue.
Link: https://lore.kernel.org/20260630190843.3563858-13-liam@infradead.org
Fixes: 9b6713cc7522 ("maple_tree: Add mtree_alloc_cyclic()")
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Reported-by: Chris Mason <clm@meta.com>
Reviewed-by: Chuck Lever <cel@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Instead of calculating the parent pointer each time for a child, cache the
majority of the parent pointer and only change the slot per child.
Drop the mas_set_parent() function since the last user has been removed.
Testing on a tree containing 2048 entries of height 4 had an increased
gain of 3.51% on nodes tracking gaps.
Link: https://lore.kernel.org/20260630190843.3563858-12-liam@infradead.org
Signed-off-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Chris Mason <clm@meta.com>
Cc: Chuck Lever <cel@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joe Perches <joe@perches.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
