diff options
Diffstat (limited to 'Documentation')
25 files changed, 685 insertions, 88 deletions
diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI/testing/sysfs-block-zram index 64c03010e951..1ea9018a627f 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -152,15 +152,15 @@ Description: compression algorithm parameters. What: /sys/block/zram<id>/compressed_writeback -Date: Decemeber 2025 +Date: December 2025 Contact: Richard Chang <richardycc@google.com> Description: - The compressed_writeback device atrribute toggles compressed + The compressed_writeback device attribute toggles compressed writeback feature. What: /sys/block/zram<id>/writeback_batch_size Date: November 2025 Contact: Sergey Senozhatsky <senozhatsky@chromium.org> Description: - The writeback_batch_size device atrribute sets the maximum + The writeback_batch_size device attribute sets the maximum number of in-flight writeback operations. diff --git a/Documentation/ABI/testing/sysfs-bus-dax b/Documentation/ABI/testing/sysfs-bus-dax index b34266bfae49..60703fffa99e 100644 --- a/Documentation/ABI/testing/sysfs-bus-dax +++ b/Documentation/ABI/testing/sysfs-bus-dax @@ -151,3 +151,27 @@ Description: memmap_on_memory parameter for memory_hotplug. This is typically set on the kernel command line - memory_hotplug.memmap_on_memory set to 'true' or 'force'." + +What: /sys/bus/dax/devices/daxX.Y/state +Contact: nvdimm@lists.linux.dev +Description: + (RW) Controls the state of the memory region. + Applies to all memory blocks associated with the device. + Only applies to dax_kmem devices. + + Reading returns the current state; the writable states mirror + the per-block /sys/devices/system/memory/memoryX/state ABI:: + + "unplugged": memory blocks are not present + "online": memory is online, zone chosen by the kernel + "online_kernel": memory is online in ZONE_NORMAL + "online_movable": memory is online in ZONE_MOVABLE + + "offline" (memory blocks are present but offline) may also be + reported - this happens when the device is bound while the + auto_online_blocks policy is "offline". It cannot be written, + as it's not useful and creates device destruction races. + + A device can only be onlined from the "unplugged" state, so a + device must be returned to "unplugged" before it can be onlined + into a different state. diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-damon b/Documentation/ABI/testing/sysfs-kernel-mm-damon index 4fdec63a47d4..e675a57145e3 100644 --- a/Documentation/ABI/testing/sysfs-kernel-mm-damon +++ b/Documentation/ABI/testing/sysfs-kernel-mm-damon @@ -36,7 +36,10 @@ Description: Writing 'on' or 'off' to this file makes the kdamond starts or kdamond. Writing 'clear_schemes_tried_regions' to the file removes contents of the 'tried_regions' directory. Writing 'update_schemes_effective_quotas' to the file updates - '.../quotas/effective_bytes' files of this kdamond. + '.../quotas/effective_bytes' files of this kdamond. Writing + 'update_tuned_intervals' to the file updates 'sample_us' and + 'aggr_us' files under 'monitoring_attrs/intervals' with the + auto-tuned sampling and aggregation intervals. What: /sys/kernel/mm/damon/admin/kdamonds/<K>/pid Date: Mar 2022 @@ -112,7 +115,7 @@ Description: Writing a value to this file sets the update interval of the DAMON context in microseconds as the value. Reading this file returns the value. -What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/access_bp +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intervals_goal/access_bp Date: Feb 2025 Contact: SJ Park <sj@kernel.org> Description: Writing a value to this file sets the monitoring intervals @@ -120,7 +123,7 @@ Description: Writing a value to this file sets the monitoring intervals the given time interval (aggrs in same directory), in bp (1/10,000). Reading this file returns the value. -What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/aggrs +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intervals_goal/aggrs Date: Feb 2025 Contact: SJ Park <sj@kernel.org> Description: Writing a value to this file sets the time interval to achieve @@ -128,14 +131,14 @@ Description: Writing a value to this file sets the time interval to achieve access events ratio (access_bp in same directory) within. Reading this file returns the value. -What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/min_sample_us +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intervals_goal/min_sample_us Date: Feb 2025 Contact: SJ Park <sj@kernel.org> Description: Writing a value to this file sets the minimum value of auto-tuned sampling interval in microseconds. Reading this file returns the value. -What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/max_sample_us +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intervals_goal/max_sample_us Date: Feb 2025 Contact: SJ Park <sj@kernel.org> Description: Writing a value to this file sets the maximum value of @@ -144,7 +147,7 @@ Description: Writing a value to this file sets the maximum value of What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/nr_regions/min -WDate: Mar 2022 +Date: Mar 2022 Contact: SJ Park <sj@kernel.org> Description: Writing a value to this file sets the minimum number of monitoring regions of the DAMON context as the value. Reading @@ -157,6 +160,52 @@ Description: Writing a value to this file sets the maximum number of monitoring regions of the DAMON context as the value. Reading this file returns the value. +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/nr_probes +Date: May 2026 +Contact: SJ Park <sj@kernel.org> +Description: Writing a number 'N' to this file creates the number of + directories for each DAMON probe named '0' to 'N-1' under the + probes/ directory. + +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/weight +Date: Jul 2026 +Contact: SJ Park <sj@kernel.org> +Description: Writing to and reading from this file sets and gets the + per-probe attribute weight. + +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/nr_filters +Date: May 2026 +Contact: SJ Park <sj@kernel.org> +Description: Writing a number 'N' to this file creates the number of + directories for each DAMON probe filter named '0' to 'N-1' + under the filters/ directory. + +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/type +Date: May 2026 +Contact: SJ Park <sj@kernel.org> +Description: Writing to and reading from this file sets and gets the type of + the memory of the interest. + +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/path +Date: May 2026 +Contact: SJ Park <sj@kernel.org> +Description: If 'memcg' is written to the 'type' file, writing to and + reading from this file sets and gets the path to the memory + cgroup of the interest. + +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/matching +Date: May 2026 +Contact: SJ Park <sj@kernel.org> +Description: Writing 'Y' or 'N' to this file sets whether the filter is for + the memory of the 'type', or all except the 'type'. + +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/allow +Date: May 2026 +Contact: SJ Park <sj@kernel.org> +Description: Writing 'Y' or 'N' to this file sets whether to allow or reject + hitting the probe for the memory that satisfies the 'type' and + the 'matching' of the directory. + What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/nr_targets Date: Mar 2022 Contact: SJ Park <sj@kernel.org> @@ -241,7 +290,7 @@ Description: Writing to and reading from this file sets and gets the maximum What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/nr_accesses/min Date: Mar 2022 Contact: SJ Park <sj@kernel.org> -Description: Writing to and reading from this file sets and gets the manimum +Description: Writing to and reading from this file sets and gets the minimum 'nr_accesses' of the scheme's target regions. What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/nr_accesses/max @@ -601,3 +650,15 @@ Contact: SJ Park <sj@kernel.org> Description: Reading this file returns the size of the memory in the region that passed DAMON operations layer-handled filters of the scheme in bytes. + +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/<R>/probes/ +Date: May 2026 +Contact: SJ Park <sj@kernel.org> +Description: Directory for DAMON data-attribute monitoring probes of the + region. + +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/<R>/probes/<P>/hits +Date: May 2026 +Contact: SJ Park <sj@kernel.org> +Description: Reading this file returns the number of data-attribute + monitoring probe-hit positive samples of the region. diff --git a/Documentation/admin-guide/cgroup-v1/cpusets.rst b/Documentation/admin-guide/cgroup-v1/cpusets.rst index c7909e5ac136..52a213aff04e 100644 --- a/Documentation/admin-guide/cgroup-v1/cpusets.rst +++ b/Documentation/admin-guide/cgroup-v1/cpusets.rst @@ -284,7 +284,7 @@ take action. ==> Unless this feature is enabled by writing "1" to the special file /dev/cpuset/memory_pressure_enabled, the hook in the rebalance - code of __alloc_pages() for this metric reduces to simply noticing + code of the page allocator for this metric reduces to simply noticing that the cpuset_memory_pressure_enabled flag is zero. So only systems that enable this feature will compute the metric. diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index b5493a7f8f22..53f08950a630 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2064,8 +2064,11 @@ Kernel parameters hugetlb_cma= [HW,CMA,EARLY] The size of a CMA area used for allocation of gigantic hugepages. Or using node format, the size of a CMA area per node can be specified. - Format: nn[KMGTPE] or (node format) - <node>:nn[KMGTPE][,<node>:nn[KMGTPE]] + The size can be an absolute value (e.g., 2G) or a + percentage of the total memory or node memory (e.g., 20%). + Format: nn[KMGTPE] or nn% or (node format) + <node>:nn[KMGTPE][,<node>:nn[KMGTPE]] or + <node>:nn%[,<node>:nn%] The size must be a multiple of the gigantic page size. When using node format, this applies to each per-node size. diff --git a/Documentation/admin-guide/mm/damon/usage.rst b/Documentation/admin-guide/mm/damon/usage.rst index 011296f1e7c2..da5f9afd08ae 100644 --- a/Documentation/admin-guide/mm/damon/usage.rst +++ b/Documentation/admin-guide/mm/damon/usage.rst @@ -73,9 +73,10 @@ comma (","). │ │ │ │ │ │ │ intervals_goal/access_bp,aggrs,min_sample_us,max_sample_us │ │ │ │ │ │ nr_regions/min,max │ │ │ │ │ │ :ref:`probes <damon_usage_sysfs_probes>`/nr_probes - │ │ │ │ │ │ │ 0/filters/nr_filters - │ │ │ │ │ │ │ │ 0/type,matching,allow,path - │ │ │ │ │ │ │ │ ... + │ │ │ │ │ │ │ 0/weight + │ │ │ │ │ │ │ │ filters/nr_filters + │ │ │ │ │ │ │ │ │ 0/type,matching,allow,path + │ │ │ │ │ │ │ │ │ ... │ │ │ │ │ │ │ ... │ │ │ │ │ :ref:`targets <sysfs_targets>`/nr_targets │ │ │ │ │ │ :ref:`0 <sysfs_target>`/pid_target,obsolete_target @@ -246,7 +247,7 @@ writing to and reading from the files. Under ``nr_regions`` directory, two files for the lower-bound and upper-bound of DAMON's monitoring regions (``min`` and ``max``, respectively), which controls the monitoring overhead, exist. You can set and get the values by -writing to and rading from the files. +writing to and reading from the files. For more details about the intervals and monitoring regions range, please refer to the Design document (:doc:`/mm/damon/design`). @@ -264,7 +265,7 @@ Please refer to the :ref:`design document of the feature <damon_design_monitoring_intervals_autotuning>` for the internal of the tuning mechanism. Reading and writing the four files under ``intervals_goal`` directory shows and updates the tuning parameters that described in the -:ref:design doc <damon_design_monitoring_intervals_autotuning>` with the same +:ref:`design doc <damon_design_monitoring_intervals_autotuning>` with the same names. The tuning starts with the user-set ``sample_us`` and ``aggr_us``. The tuning-applied current values of the two intervals can be read from the ``sample_us`` and ``aggr_us`` files after writing ``update_tuned_intervals`` to @@ -286,6 +287,10 @@ In each probe directory, one directory, ``filters`` exists. The directory contains files for installing filters for the probe, that is used to determine the data attribute for the probe. +Each probe directory also contains ``weight`` file. Reading from and writing +to the file gets and sets the :ref:`attributes-only monitoring +<damon_design_attrs_only_monitoring>` weight for the attribute of the probe. + In the beginning, ``filters`` directory has only one file, ``nr_filters``. Writing a number (``N``) to the file creates the number of child directories named ``0`` to ``N-1``. Each directory represents each filter and works in a @@ -377,7 +382,7 @@ schemes/<N>/ In each scheme directory, nine directories (``access_pattern``, ``quotas``, ``watermarks``, ``core_filters``, ``ops_filters``, ``filters``, ``dests``, ``stats``, and ``tried_regions``) and three files (``action``, ``target_nid`` -and ``apply_interval``) exist. +and ``apply_interval_us``) exist. The ``action`` file is for setting and getting the scheme's :ref:`action <damon_design_damos_action>`. The keywords that can be written to and read @@ -743,7 +748,7 @@ counter). Finally the tenth field (``X``) shows the ``age`` of the region (refer to :ref:`design <damon_design_age_tracking>` for more details of the counter). -If the event was ``damon:damos_beofre_apply``, the ``perf script`` output would +If the event was ``damon:damos_before_apply``, the ``perf script`` output would be somewhat like below:: kdamond.0 47293 [000] 80801.060214: damon:damos_before_apply: ctx_idx=0 scheme_idx=0 target_idx=0 nr_regions=11 121932607488-135128711168: 0 136 diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index c57e61b5d8aa..ffa690a171c8 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -19,8 +19,11 @@ There are four components to pagemap: * Bit 55 pte is soft-dirty (see Documentation/admin-guide/mm/soft-dirty.rst) * Bit 56 page exclusively mapped (since 4.2) - * Bit 57 pte is uffd-wp write-protected (since 5.13) (see - Documentation/admin-guide/mm/userfaultfd.rst) + * Bit 57 pte is tracked by userfaultfd (since 5.13) — in a + ``VM_UFFD_WP`` VMA this indicates a write-protected PTE; in a + ``VM_UFFD_RWP`` VMA it indicates an RWP-protected PTE. WP and + RWP are mutually exclusive per VMA, so the meaning is + unambiguous. See Documentation/admin-guide/mm/userfaultfd.rst. * Bit 58 pte is a guard region (since 6.15) (see madvise (2) man page) * Bits 59-60 zero * Bit 61 page is file-page or shared-anon (since 3.5) @@ -244,7 +247,8 @@ in this IOCTL: Following flags about pages are currently supported: - ``PAGE_IS_WPALLOWED`` - Page has async-write-protection enabled -- ``PAGE_IS_WRITTEN`` - Page has been written to from the time it was write protected +- ``PAGE_IS_WRITTEN`` - Page in a ``UFFDIO_REGISTER_MODE_WP`` VMA has been + written to since it was write-protected. Only reported inside such VMAs. - ``PAGE_IS_FILE`` - Page is file backed - ``PAGE_IS_PRESENT`` - Page is present in the memory - ``PAGE_IS_SWAPPED`` - Page is in swapped @@ -252,6 +256,9 @@ Following flags about pages are currently supported: - ``PAGE_IS_HUGE`` - Page is PMD-mapped THP or Hugetlb backed - ``PAGE_IS_SOFT_DIRTY`` - Page is soft-dirty - ``PAGE_IS_GUARD`` - Page is a part of a guard region +- ``PAGE_IS_ACCESSED`` - Page in a ``UFFDIO_REGISTER_MODE_RWP`` VMA has been + accessed since RWP was applied. Only reported inside such VMAs. See + Documentation/admin-guide/mm/userfaultfd.rst for the RWP workflow. The ``struct pm_scan_arg`` is used as the argument of the IOCTL. diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst index 23f8d13c2629..16f37135ed80 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -761,7 +761,7 @@ compact_fail but failed. It is possible to establish how long the stalls were using the function -tracer to record how long was spent in __alloc_pages() and +tracer to record how long was spent in the page allocator and using the mm_page_alloc tracepoint to identify which allocations were for huge pages. diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst index e5cc8848dcb3..783d969f0e28 100644 --- a/Documentation/admin-guide/mm/userfaultfd.rst +++ b/Documentation/admin-guide/mm/userfaultfd.rst @@ -131,6 +131,16 @@ userfaults on the range registered. Not all ioctls will necessarily be supported for all memory types (e.g. anonymous memory vs. shmem vs. hugetlbfs), or all types of intercepted faults. +.. note:: + + Re-registering an already-registered range must not drop any of the + modes that install per-PTE markers — currently + ``UFFDIO_REGISTER_MODE_WP`` and ``UFFDIO_REGISTER_MODE_RWP``. Doing + so would strand markers with no flag to describe them, so the call + is rejected with ``-EBUSY``; userspace must issue + ``UFFDIO_UNREGISTER`` first. This differs from older kernels, which + silently replaced the mode bits on re-registration. + Userland can use the ``uffdio_register.ioctls`` to manage the virtual address space in the background (to add or potentially also remove memory from the ``userfaultfd`` registered range). This means a userfault @@ -265,16 +275,16 @@ tracking and it can be different in a few ways: - Dirty information will not get lost if the pte was zapped due to various reasons (e.g. during split of a shmem transparent huge page). - - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit - set; dirty when uffd-wp bit cleared), it has different semantics on - some of the memory operations. For example: ``MADV_DONTNEED`` on + - Due to a reverted meaning of soft-dirty (page clean when the uffd bit + is set; dirty when the uffd bit is cleared), it has different semantics + on some of the memory operations. For example: ``MADV_DONTNEED`` on anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as - dirtying of memory by dropping uffd-wp bit during the procedure. + dirtying of memory by dropping the uffd bit during the procedure. The user app can collect the "written/dirty" status by looking up the -uffd-wp bit for the pages being interested in /proc/pagemap. +uffd bit for the pages being interested in /proc/pagemap. -The page will not be under track of uffd-wp async mode until the page is +The page will not be under track of userfaultfd-wp async mode until the page is explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault that was tracked by async mode userfaultfd-wp is invalid. @@ -297,6 +307,253 @@ transparent to the guest, we want that same address range to act as if it was still poisoned, even though it's on a new physical host which ostensibly doesn't have a memory error in the exact same spot. +Read-Write Protection +--------------------- + +``UFFDIO_REGISTER_MODE_RWP`` enables read-write protection tracking on a +memory range. It is similar to (but faster than) ``mprotect(PROT_NONE)`` +combined with a signal handler; unlike ``mprotect(PROT_NONE)``, RWP only +traps accesses to *present* PTEs, so accesses to unpopulated addresses in a +protected range fall through to the normal missing-page path. It uses the +PROT_NONE hinting mechanism (same as NUMA balancing) to make pages +inaccessible while keeping them resident in memory. Works on anonymous, +shmem, and hugetlbfs memory. + +RWP is designed for VM memory managers that need to track the working set +of guest memory for cold page eviction to tiered or remote storage. + +**Setup:** + +1. Open a userfaultfd and enable ``UFFD_FEATURE_RWP`` via ``UFFDIO_API``. + Optionally request ``UFFD_FEATURE_RWP_ASYNC`` as well — it requires + ``UFFD_FEATURE_RWP`` to be set in the same ``UFFDIO_API`` call. + +2. Register the guest memory range with ``UFFDIO_REGISTER_MODE_RWP`` + (and ``UFFDIO_REGISTER_MODE_MISSING`` if evicted pages will need to be + fetched back from storage). + +**Feature availability:** + +RWP is built on top of two kernel primitives: a spare PTE bit owned by +userfaultfd (``CONFIG_HAVE_ARCH_USERFAULTFD_WP``) and architecture support +for present-but-inaccessible PTEs (``CONFIG_ARCH_HAS_PTE_PROTNONE``). When both +are available on a 64-bit kernel, the build selects +``CONFIG_USERFAULTFD_RWP=y`` and the ``VM_UFFD_RWP`` VMA flag becomes +available. + +``UFFD_FEATURE_RWP`` and ``UFFD_FEATURE_RWP_ASYNC`` are unavailable when +the running kernel or architecture does not support them — for example +32-bit kernels (where ``VM_UFFD_RWP`` is unavailable), kernels built +without ``CONFIG_USERFAULTFD_RWP``, and architectures whose ptes cannot +carry the uffd bit at runtime (e.g. riscv without the ``SVRSW60T59B`` +extension). Requesting an unsupported feature in +``uffdio_api.features`` makes ``UFFDIO_API`` fail with ``EINVAL`` and +leaves the userfaultfd context uninitialized; the structure is returned +zeroed, so the error path cannot be used to discover what the kernel +supports. The recommended probe sequence is therefore to open a +throwaway userfaultfd, call ``UFFDIO_API`` once with ``features = 0``, +inspect the returned bitmask, close that fd, then open the real one +and call ``UFFDIO_API`` again with only the supported features set. + +**Protecting and Unprotecting:** + +Use ``UFFDIO_RWPROTECT`` to protect or unprotect a range, mirroring the +``UFFDIO_WRITEPROTECT`` interface:: + + struct uffdio_rwprotect rwp = { + .range = { .start = addr, .len = len }, + .mode = UFFDIO_RWPROTECT_MODE_RWP, /* protect */ + }; + ioctl(uffd, UFFDIO_RWPROTECT, &rwp); + +Setting ``UFFDIO_RWPROTECT_MODE_RWP`` sets PROT_NONE on present PTEs in the +range. Pages stay resident and their physical frames are preserved — only +access permissions are removed. + +Clearing ``UFFDIO_RWPROTECT_MODE_RWP`` restores normal VMA permissions and +wakes any faulting threads (unless ``UFFDIO_RWPROTECT_MODE_DONTWAKE`` is set). + +**Scope of protection:** + +RWP protection is a property of *present* PTEs. ``UFFDIO_RWPROTECT`` only +affects entries that are already populated. Unpopulated addresses within +the range remain unpopulated; when first accessed they fault through the +normal missing path (``do_anonymous_page()``, ``do_swap_page()``, +``finish_fault()``) and the resulting PTE is not RWP-protected. To observe +the population itself, co-register the range with +``UFFDIO_REGISTER_MODE_MISSING``. + +Protection is preserved across page reclaim: a page swapped out while +RWP-protected carries the marker on its swap entry, and swap-in restores +the PROT_NONE state so the first access after swap-in still faults. The +same applies to pages temporarily replaced by migration entries. + +Operations that drop the PTE entirely — ``MADV_DONTNEED`` on anonymous +memory, hole-punch on shmem, truncation of a file mapping — also drop the +RWP marker: the next access re-populates the range without protection. +Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``), there is no +persistent RWP marker today. The user needs to re-arm the range with +``UFFDIO_RWPROTECT`` after any operation that explicitly frees PTEs. + +**Fault Handling:** + +When a protected page is accessed: + +- **Sync mode** (default): The faulting thread blocks and a + ``UFFD_PAGEFAULT_FLAG_RWP`` message is delivered to the userfaultfd + handler. The handler resolves the fault with ``UFFDIO_RWPROTECT`` + (clearing ``MODE_RWP``), which restores the PTE permissions and wakes + the faulting thread. + +- **Async mode** (``UFFD_FEATURE_RWP_ASYNC``): The kernel automatically + restores PTE permissions and the thread continues without blocking. No + message is delivered to the handler. + +**Runtime Mode Switching:** + +``UFFDIO_SET_MODE`` toggles ``UFFD_FEATURE_RWP_ASYNC`` at runtime, allowing +the VMM to switch between lightweight async detection and safe sync +eviction without re-registering. The toggle takes ``mmap_write_lock()`` +and calls ``vma_start_write()`` on each UFFD-armed VMA, draining +in-flight per-VMA-locked faults before the new mode takes effect. + +**Working-set detection with PAGEMAP_SCAN:** + +RWP-protected PTEs carry the uffd PTE bit; an access (and, in async mode, its +auto-resolution) clears it. ``PAGEMAP_SCAN`` reports ``PAGE_IS_ACCESSED`` once +the bit is clear on a ``VM_UFFD_RWP`` VMA, so a *non-inverted* scan reports the +pages that were touched during the interval -- the hot set:: + + struct pm_scan_arg arg = { + .size = sizeof(arg), + .start = guest_mem_start, + .end = guest_mem_end, + .vec = (uint64_t)regions, + .vec_len = regions_len, + .category_mask = PAGE_IS_ACCESSED, + .return_mask = PAGE_IS_ACCESSED, + }; + long n = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg); + +The returned ``page_region`` array lists the hot ranges. ``PAGE_IS_ACCESSED`` +is set on an accessed page whether it is still present or has since been +swapped out, so the hot scan needs no ``PAGE_IS_PRESENT`` filter -- unpopulated +holes carry neither bit and are excluded on their own. + +Track the hot set and reclaim everything else from the backing file (see the +workflow below). Do **not** invert the scan to enumerate "cold" pages +directly: an inverted scan reports only the ``VM_UFFD_RWP`` PTEs that are still +protected, i.e. the resident portion of *this* VMA. For a file mapping the +working set spans the whole file -- pages that live in the page cache but are +not mapped into this VMA (a pre-populated tmpfs file, or memory populated +through another mapping) are ``pte_none`` here, never appear in the scan, and +would never be considered for eviction even though they occupy memory. Driving +eviction from "file offsets minus the hot set" avoids that blind spot; a cold +PTE scan cannot. To additionally record the *first* access to a cached but +unmapped page (e.g. pre-populated content) as hot, co-register the range with +``UFFDIO_REGISTER_MODE_MINOR``: such accesses then fault as minor faults +instead of mapping the page silently. + +**Cleanup:** + +When the userfaultfd is closed or the range is unregistered, all PROT_NONE +PTEs are automatically restored to their normal VMA permissions. This +prevents pages from becoming permanently inaccessible. + +**VMM Working Set Tracking Workflow:** + +A typical VMM lifecycle for cold page eviction to tiered storage. Two +mappings of the same shmem (or hugetlbfs) file are used: ``guest_mem`` is +the RWP-registered mapping that vCPUs access through, and ``io_mem`` is a +private mapping for VMM-side I/O. Reading ``io_mem`` does not go through +the RWP-protected PTEs of ``guest_mem``, so the VMM's own ``pwrite()`` +never traps on its own :: + + /* One-time setup */ + fd = memfd_create("guest", MFD_CLOEXEC); + ftruncate(fd, guest_size); + guest_mem = mmap(NULL, guest_size, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); /* vCPU view, RWP-registered */ + io_mem = mmap(NULL, guest_size, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); /* VMM I/O view, unprotected */ + + uffd = userfaultfd(O_CLOEXEC | O_NONBLOCK); + struct uffdio_api api = { + .api = UFFD_API, + .features = UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC, + }; + ioctl(uffd, UFFDIO_API, &api); + if (!(api.features & UFFD_FEATURE_RWP)) + /* RWP unavailable on this kernel/arch -- fall back. */ + ioctl(uffd, UFFDIO_REGISTER, &(struct uffdio_register){ + .range = { guest_mem, guest_size }, + .mode = UFFDIO_REGISTER_MODE_RWP | + UFFDIO_REGISTER_MODE_MISSING, + }); + + /* Tracking loop */ + while (vm_running) { + /* 1. Detection phase (async -- no vCPU stalls) */ + ioctl(uffd, UFFDIO_RWPROTECT, &(struct uffdio_rwprotect){ + .range = full_range, + .mode = UFFDIO_RWPROTECT_MODE_RWP }); + sleep(tracking_interval); + + /* + * 2. Switch to sync BEFORE scanning. In async mode a vCPU + * access races eviction: it would auto-resolve and mark the + * page hot just as the VMM writes it out and punches it, + * losing the update. Sync mode makes such accesses block and + * be delivered, freezing the hot snapshot for the rest of the + * iteration. + */ + ioctl(uffd, UFFDIO_SET_MODE, + &(struct uffdio_set_mode){ + .disable = UFFD_FEATURE_RWP_ASYNC }); + + /* 3. Read the hot set: pages touched this interval. */ + ioctl(pagemap_fd, PAGEMAP_SCAN, &(struct pm_scan_arg){ + .category_mask = PAGE_IS_ACCESSED, + .return_mask = PAGE_IS_ACCESSED, + ... + }); + + /* + * 4. Reclaim the file offsets that are NOT in the hot set. + * Driving this from the file's offset space (rather than from a + * cold PTE scan) also reclaims pages that are cached but not + * mapped into guest_mem, e.g. pre-populated content. + */ + for each non-hot offset range: + /* Read from io_mem -- bypasses RWP, no fault. */ + pwrite(storage_fd, (char *)io_mem + off, len, off); + /* Drop the page from the shared file. */ + fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, + off, len); + /* + * Wake any vCPU blocked on the RWP fault for this range: + * fallocate() does not iterate ctx->fault_pending_wqh. + */ + ioctl(uffd, UFFDIO_WAKE, &(struct uffdio_range){ + .start = (uintptr_t)guest_mem + off, .len = len }); + + /* 5. Resume async tracking */ + ioctl(uffd, UFFDIO_SET_MODE, + &(struct uffdio_set_mode){ + .enable = UFFD_FEATURE_RWP_ASYNC }); + } + +During step 4, a vCPU that accesses a ``guest_mem`` offset being evicted +blocks with a ``UFFD_PAGEFAULT_FLAG_RWP`` fault while the eviction is in +progress. After ``fallocate()`` punches the page out and ``UFFDIO_WAKE`` +fires, the vCPU retries the access, faults as ``MISSING``, and the +handler resolves it with ``UFFDIO_COPY`` from storage. + +This workflow targets shmem and hugetlbfs (both support a private +``io_mem`` mapping over the same fd). Anonymous-memory backings need a +different inner-loop strategy because the VMM has no way to read the +page without going through the RWP-protected mapping. + QEMU/KVM ======== diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index b9b0c218bfb4..5b318d17aa4b 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -19,9 +19,6 @@ The files in this directory can be used to tune the operation of the virtual memory (VM) subsystem of the Linux kernel and the writeout of dirty data to disk. -Default values and initialization routines for most of these -files can be found in mm/swap.c. - Currently, these files are in /proc/sys/vm: - admin_reserve_kbytes @@ -67,6 +64,7 @@ Currently, these files are in /proc/sys/vm: - page-cluster - page_lock_unfairness - panic_on_oom +- panic_on_unrecoverable_memory_failure - percpu_pagelist_high_fraction - stat_interval - stat_refresh @@ -925,6 +923,85 @@ panic_on_oom=2+kdump gives you very strong tool to investigate why oom happens. You can get snapshot. +panic_on_unrecoverable_memory_failure +====================================== + +When a hardware memory error (e.g. multi-bit ECC) hits a kernel page +that cannot be recovered by the memory failure handler, the default +behaviour is to ignore the error and continue operation. This is +dangerous because the corrupted data remains accessible to the kernel, +risking silent data corruption or a delayed crash when the poisoned +memory is next accessed. + +When enabled, this sysctl triggers a panic on memory failure events +hitting kernel-owned pages that the handler cannot recover: +``PageReserved`` (firmware reservations, kernel image, vDSO, zero +page, and similar memblock-reserved regions), ``PageSlab``, +``PageTable``, and ``PageLargeKmalloc``. These are owned by the +kernel and the memory failure handler cannot reliably evict their +contents. + +Other unrecoverable kernel-owned populations (vmalloc allocations, +kernel stack pages, ...) are not currently covered because the +handler has no page-type signal that distinguishes them from a +userspace folio temporarily off the LRU during migration or +compaction. Such pages still go through the standard +MF_MSG_GET_HWPOISON path: ``PG_hwpoison`` is set on them and a +delayed crash on the next access remains possible. Coverage may +grow as the handler gains stronger kernel-ownership signals. + +Recoverable failure paths are also intentionally left out: in-flight +buddy allocations and other transient races with the page allocator +can reach the same diagnostic, and panicking on them would risk +killing the box for a page destined for userspace where the standard +SIGBUS recovery path applies. Pages whose state could not be +classified at all are not covered either, since an unknown state is +not a sound basis for a panic decision. + +For many environments it is preferable to panic immediately with a clean +crash dump that captures the original error context, rather than to +continue and face a random crash later whose cause is difficult to +diagnose. + +Use cases +--------- + +This option is most useful in environments where unattributed crashes +are expensive to debug or where data integrity must take precedence +over availability: + +* Large fleets, where multi-bit ECC errors on kernel pages are observed + regularly and post-mortem analysis of an unrelated downstream crash + (often seconds to minutes after the original error) consumes + significant engineering effort. + +* Systems configured with kdump, where panicking at the moment of the + hardware error produces a vmcore that still contains the faulting + address, the affected page state, and the originating MCE/GHES + record — context that is typically lost by the time a delayed crash + occurs. + +* High-availability clusters that rely on fast, deterministic node + failure for failover, and prefer an immediate panic over silent data + corruption propagating to replicas or persistent storage. + +* Kernel and platform developers reproducing hwpoison issues with + tools such as ``mce-inject`` or error-injection debugfs interfaces, + where panicking on the unrecoverable path makes regressions + immediately visible instead of surfacing as later, unrelated + failures. + += ===================================================================== +0 Try to continue operation (default). +1 Panic immediately. If the ``panic`` sysctl is also non-zero then the + machine will be rebooted. += ===================================================================== + +Example:: + + echo 1 > /proc/sys/vm/panic_on_unrecoverable_memory_failure + + percpu_pagelist_high_fraction ============================= diff --git a/Documentation/core-api/maple_tree.rst b/Documentation/core-api/maple_tree.rst index ccdd1615cf97..1eae24a9c18b 100644 --- a/Documentation/core-api/maple_tree.rst +++ b/Documentation/core-api/maple_tree.rst @@ -17,7 +17,8 @@ supports iterating over a range of entries and going to the previous or next entry in a cache-efficient manner. The tree can also be put into an RCU-safe mode of operation which allows reading and writing concurrently. Writers must synchronize on a lock, which can be the default spinlock, or the user can set -the lock to an external lock of a different type. +the lock to an external lock of a different type. Note that external locks may +interfere with allocations in a low memory situation. The Maple Tree maintains a small memory footprint and was designed to use modern processor cache efficiently. The majority of the users will be able to @@ -42,6 +43,15 @@ successful store operation within a given code segment when allocating cannot be done. Allocations of nodes are relatively small at around 256 bytes. +Since the maple tree uses internal nodes that are allocated and has rules on +data density, erasing an entry may cause allocations to occur. That is, +erasing an entry may consume memory. Users must take care to ensure that they +do not violate the larger system constraints on when and how memory is +allocated. Most situations are fine to allocate, but the pre-allocation +support is provided as a mechanism to avoid trickier situations. There is also +the possibility of using special entries and clean up the tree later, in +extreme circumstances. + .. _maple-tree-normal-api: Normal API @@ -63,7 +73,8 @@ success or an error code otherwise. mtree_store_range() works in the same way but takes a range. mtree_load() is used to retrieve the entry stored at a given index. You can use mtree_erase() to erase an entire range by only knowing one value within that range, or mtree_store() call with an entry of -NULL may be used to partially erase a range or many ranges at once. +NULL may be used to partially erase a range or many ranges at once. Note that +mtree_erase() may use GFP_KERNEL on allocations. If you want to only store a new entry to a range (or index) if that range is currently ``NULL``, you can use mtree_insert_range() or mtree_insert() which @@ -163,7 +174,9 @@ You can use mas_erase() to erase an entire range by setting index and last of the maple state to the desired range to erase. This will erase the first range that is found in that range, set the maple state index and last as the range that was erased and return the entry that existed -at that location. +at that location. Note that mas_erase() may allocate with the GFP_KERNEL flag. +If this is not okay, consider using mas_store_gfp() and pass it a ``NULL``, +after setting up the correct range by walking to the entry. You can walk each entry within a range by using mas_for_each(). If you want to walk each element of the tree then ``0`` and ``ULONG_MAX`` may be used as @@ -211,7 +224,7 @@ Advanced Locking The maple tree uses a spinlock by default, but external locks can be used for tree updates as well. To use an external lock, the tree must be initialized -with the ``MT_FLAGS_LOCK_EXTERN flag``, this is usually done with the +with the ``MT_FLAGS_LOCK_EXTERN`` flag, this is usually done with the MTREE_INIT_EXT() #define, which takes an external lock as an argument. Functions and structures diff --git a/Documentation/core-api/mm-api.rst b/Documentation/core-api/mm-api.rst index aabdd3cba58e..c1d03a5a2a19 100644 --- a/Documentation/core-api/mm-api.rst +++ b/Documentation/core-api/mm-api.rst @@ -73,6 +73,7 @@ Readahead Writeback --------- +.. kernel-doc:: include/linux/writeback.h .. kernel-doc:: mm/page-writeback.c :export: @@ -117,7 +118,7 @@ More Memory Management Functions .. #kernel-doc:: mm/hmm.c (build warnings) .. kernel-doc:: mm/memremap.c .. kernel-doc:: mm/hugetlb.c -.. kernel-doc:: mm/swap.c +.. kernel-doc:: mm/folio.c .. kernel-doc:: mm/memcontrol.c .. #kernel-doc:: mm/memory-tiers.c (build warnings) .. kernel-doc:: mm/shmem.c diff --git a/Documentation/dev-tools/kmemleak.rst b/Documentation/dev-tools/kmemleak.rst index 7d784e03f3f9..a8a83bc69ceb 100644 --- a/Documentation/dev-tools/kmemleak.rst +++ b/Documentation/dev-tools/kmemleak.rst @@ -198,6 +198,14 @@ systems, because of pointers temporarily stored in CPU registers or stacks. Kmemleak defines MSECS_MIN_AGE (defaulting to 1000) representing the minimum age of an object to be reported as a memory leak. +The ``min_unref_scans`` module parameter (default 1) requires an object to +be seen unreferenced in that many consecutive scans before it is reported. +Keeping it at 1 preserves the historical behaviour; higher values filter +the transient false positives described above, at the cost of delaying +genuine reports by up to that many scans. It can be set at boot with +``kmemleak.min_unref_scans=<n>`` or at run-time via +``/sys/module/kmemleak/parameters/min_unref_scans``. + Limitations and Drawbacks ------------------------- diff --git a/Documentation/driver-api/basics.rst b/Documentation/driver-api/basics.rst index 8b6a5888cb11..3b182cfdf135 100644 --- a/Documentation/driver-api/basics.rst +++ b/Documentation/driver-api/basics.rst @@ -90,6 +90,9 @@ Reference counting .. kernel-doc:: lib/refcount.c :export: +.. kernel-doc:: include/linux/percpu-refcount.h +.. kernel-doc:: lib/percpu-refcount.c + Atomics ------- diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst index 08d01bc62c31..1a50d41a39a1 100644 --- a/Documentation/filesystems/locking.rst +++ b/Documentation/filesystems/locking.rst @@ -355,13 +355,14 @@ should perform any validation and preparation necessary to ensure that writes can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted through -->swap_rw(), it should set SWP_FS_OPS, otherwise IO will be submitted +->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted directly to the block device ``sis->bdev``. ->swap_deactivate() will be called in the sys_swapoff() path after ->swap_activate() returned success. -->swap_rw will be called for swap IO if SWP_FS_OPS was set by ->swap_activate(). +->swap_rw will be called for swap IO if swap_fs_activate was called by +->swap_activate(). file_lock_operations ==================== diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 2ccd5b2dfdd3..27189f6e004e 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -608,6 +608,7 @@ encoded manner. The codes are the following: um userfaultfd missing tracking uw userfaultfd wr-protect tracking ui userfaultfd minor fault + ur userfaultfd read-write-protect tracking ss shadow/guarded control stack page sl sealed lf lock on fault pages diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst index 7c753148af88..e7677423a20f 100644 --- a/Documentation/filesystems/vfs.rst +++ b/Documentation/filesystems/vfs.rst @@ -977,7 +977,7 @@ cache in your filesystem. The following members are defined: can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted - through ->swap_rw(), it should set SWP_FS_OPS, otherwise IO will + through ->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted directly to the block device ``sis->bdev``. ``swap_deactivate`` @@ -985,7 +985,7 @@ cache in your filesystem. The following members are defined: successful. ``swap_rw`` - Called to read or write swap pages when SWP_FS_OPS is set. + Called to read or write swap pages when swap_fs_activate was called. The File Object =============== diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst index 5389d241176a..c3a28467955f 100644 --- a/Documentation/mm/allocation-profiling.rst +++ b/Documentation/mm/allocation-profiling.rst @@ -46,6 +46,11 @@ sysctl: Runtime info: /proc/allocinfo + Profiling data can be retrieved either by reading `/proc/allocinfo` directly as + text or programmatically via `ioctl()` calls defined in `<uapi/linux/alloc_tag.h>`. + The ioctl interface supports structured binary data extraction as well as filtering + by module name, function, file, line number, accuracy, or allocation size limits. + Example output:: root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec diff --git a/Documentation/mm/damon/design.rst b/Documentation/mm/damon/design.rst index 2da7ca0d3d17..aed6cb1cf483 100644 --- a/Documentation/mm/damon/design.rst +++ b/Documentation/mm/damon/design.rst @@ -86,7 +86,7 @@ To know how user-space can do the configuration via :ref:`DAMON sysfs interface documentation. - .. _damon_design_vaddr_target_regions_construction: +.. _damon_design_vaddr_target_regions_construction: VMA-based Target Address Range Construction ------------------------------------------- @@ -316,6 +316,31 @@ Another way to do this for higher accuracy is using :ref:`DAMOS filter information in page level. But, because it is operated in page level, the overhead is proportional to the size of the memory. +.. _damon_design_attrs_only_monitoring: + +Data Attributes-only Monitoring +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Data access is the primary monitoring information for DAMON. Hence it +:ref:`adjusts <damon_design_data_attrs_monitoring>` regions using the access +:ref:`counter <damon_design_region_based_sampling>` (``nr_accesses``). In some +use cases, however, users may want some of :ref:`attributes +<damon_design_data_attrs_monitoring>` to be the primary information. + +Data attributes-only monitoring mode supports this use case. For the mode, +each attribute probe has their priority weight value. Users can describe by +what combination of the attributes the primary information is decided, by +setting the priority weight value. If the total sum of the weights is not +zero, the mode is enabled. The regions adjustment mechanism uses the weighted +sum of the :ref:`probe hit counts <damon_design_data_attrs_monitoring>` instead +of ``nr_accesses`` in the case. When the mode is enabled, access monitoring is +automatically turned off. The access counter (``nr_accesses``) will always be +zero and not updated. Hence the mode is called Data Attributes "only" +monitoring. + +Refer to the :ref:`admin guide <damon_usage_sysfs_probes>` to know how users +can use the mode. + Dynamic Target Space Updates Handling ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -686,9 +711,11 @@ mechanism tries to make ``current_value`` of ``target_metric`` be same to (1/10,000). - ``inactive_mem_bp``: Inactive to active + inactive (LRU) memory size ratio in bp (1/10,000). +- ``node_eligible_mem_bp``: Scheme target access pattern-eligible memory ratio + of a node in bp (1/10,000). -``nid`` is optionally required for only ``node_mem_used_bp``, -``node_mem_free_bp``, ``node_memcg_used_bp`` and ``node_memcg_free_bp`` to +``nid`` is optionally required for ``node_mem_used_bp``, ``node_mem_free_bp``, +``node_memcg_used_bp``, ``node_memcg_free_bp`` and ``node_eligible_mem_bp`` to point the specific NUMA node. ``path`` is optionally required for only ``node_memcg_used_bp`` and @@ -930,11 +957,11 @@ control parameters for the usage would also need to be optimized for the purpose. To support such cases, yet more DAMON API user kernel modules that provide more -simple and optimized user space interfaces are available. Currently, two -modules for proactive reclamation and LRU lists manipulation are provided. For -more detail, please read the usage documents for those -(:doc:`/admin-guide/mm/damon/stat`, :doc:`/admin-guide/mm/damon/reclaim` and -:doc:`/admin-guide/mm/damon/lru_sort`). +simple and optimized user space interfaces are available. Currently, three +modules for access monitoring statistics, proactive reclamation, and LRU lists +manipulation are provided. For more detail, please read the usage documents for +those (:doc:`/admin-guide/mm/damon/stat`, :doc:`/admin-guide/mm/damon/reclaim` +and :doc:`/admin-guide/mm/damon/lru_sort`). .. _damon_design_special_purpose_modules_exclusivity: diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst index 7d61b7a8b65b..fc1b8dc19825 100644 --- a/Documentation/mm/hmm.rst +++ b/Documentation/mm/hmm.rst @@ -156,42 +156,57 @@ During the ops->invalidate() callback the device driver must perform the update action to the range (mark range read only, or fully unmap, etc.). The device must complete the update before the driver callback returns. -When the device driver wants to populate a range of virtual addresses, it can -use:: +When the device driver wants to populate a range of virtual addresses, the +normal interface is:: - int hmm_range_fault(struct hmm_range *range); + int hmm_range_fault_unlocked_timeout(struct hmm_range *range, + unsigned long timeout); It will trigger a page fault on missing or read-only entries if write access is requested (see below). Page faults use the generic mm page fault code path just -like a CPU page fault. The usage pattern is:: +like a CPU page fault. + +The caller must not hold ``mmap_read_lock`` before the call. +``hmm_range_fault_unlocked_timeout()`` takes the mmap read lock internally and +allows ``handle_mm_fault()`` to drop it during fault handling. This is required +for VMAs whose fault handlers may release the mmap lock, for example regions +managed by ``userfaultfd``. + +If the mmap lock is dropped or the range is invalidated, the function refreshes +``range->notifier_seq`` and restarts the walk internally. ``-EINTR`` is returned +if mmap lock acquisition is interrupted or a fatal signal is pending during +retry handling. + +The timeout is specified in jiffies; passing ``0`` means retry indefinitely. The +timeout exists to preserve caller policy for repeated mmu-notifier invalidation +and is checked between retry attempts. HMM does not interrupt page fault +handling when the timeout expires, but returns ``-EBUSY`` if the retry budget is +exhausted before a stable range is obtained. + +The usage pattern is:: int driver_populate_range(...) { struct hmm_range range; + unsigned long timeout; ... + timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); range.notifier = &interval_sub; range.start = ...; range.end = ...; range.hmm_pfns = ...; - if (!mmget_not_zero(interval_sub->notifier.mm)) + if (!mmget_not_zero(interval_sub.mm)) return -EFAULT; again: - range.notifier_seq = mmu_interval_read_begin(&interval_sub); - mmap_read_lock(mm); - ret = hmm_range_fault(&range); - if (ret) { - mmap_read_unlock(mm); - if (ret == -EBUSY) - goto again; - return ret; - } - mmap_read_unlock(mm); + ret = hmm_range_fault_unlocked_timeout(&range, timeout); + if (ret) + goto out_put; take_lock(driver->update); - if (mmu_interval_read_retry(&ni, range.notifier_seq) { + if (mmu_interval_read_retry(range.notifier, range.notifier_seq)) { release_lock(driver->update); goto again; } @@ -200,13 +215,31 @@ like a CPU page fault. The usage pattern is:: * under the update lock */ release_lock(driver->update); - return 0; + ret = 0; + + out_put: + mmput(interval_sub.mm); + return ret; } The driver->update lock is the same lock that the driver takes inside its invalidate() callback. That lock must be held before calling mmu_interval_read_retry() to avoid any race with a concurrent CPU page table -update. +update. The retry check must use the same notifier and sequence number stored +in ``range`` by ``hmm_range_fault_unlocked_timeout()``. + +Holding the mmap lock across HMM faults +======================================= + +Most callers should use ``hmm_range_fault_unlocked_timeout()``. If a driver +really needs to hold the mmap lock across work outside HMM, it can use:: + + int hmm_range_fault(struct hmm_range *range); + +The mmap lock must be held by the caller and will remain held on return. This +interface cannot support VMAs whose fault handlers need to drop the mmap lock. +New callers should prefer ``hmm_range_fault_unlocked_timeout()`` unless they +have a specific requirement to keep the mmap lock held across the call. Leverage default_flags and pfn_flags_mask ========================================= @@ -221,8 +254,8 @@ permission, it sets:: range->default_flags = HMM_PFN_REQ_FAULT; range->pfn_flags_mask = 0; -and calls hmm_range_fault() as described above. This will fill fault all pages -in the range with at least read permission. +and calls the HMM range fault helper as described above. This will fault +all pages in the range with at least read permission. Now let's say the driver wants to do the same except for one page in the range for which it wants to have write permission. Now driver set:: @@ -236,9 +269,9 @@ address == range->start + (index_of_write << PAGE_SHIFT) it will fault with write permission i.e., if the CPU pte does not have write permission set then HMM will call handle_mm_fault(). -After hmm_range_fault completes the flag bits are set to the current state of -the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be set if the page is -writable. +After the HMM range fault helper completes the flag bits are set to the +current state of the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be +set if the page is writable. Represent and manage device memory from core kernel point of view @@ -316,7 +349,7 @@ between device driver specific code and shared common code: system memory and device private memory. One of the first steps migrate_vma_setup() does is to invalidate other - device's MMUs with the ``mmu_notifier_invalidate_range_start(()`` and + device's MMUs with the ``mmu_notifier_invalidate_range_start()`` and ``mmu_notifier_invalidate_range_end()`` calls around the page table walks to fill in the ``args->src`` array with PFNs to be migrated. The ``invalidate_range_start()`` callback is passed a diff --git a/Documentation/mm/page_owner.rst b/Documentation/mm/page_owner.rst index 6b12f3b007ec..a6bd3fe6423a 100644 --- a/Documentation/mm/page_owner.rst +++ b/Documentation/mm/page_owner.rst @@ -65,7 +65,15 @@ un-tracking state. Usage ===== -1) Build user-space helper:: +1) Build user-space helpers: +:: + + To filter page_owner output: + + cd tools/mm + make page_owner_filter + + To sort and analyze page_owner output: cd tools/mm make page_owner_sort @@ -74,7 +82,11 @@ Usage 3) Do the job that you want to debug. -4) Analyze information from page owner:: +4) (Optional) Filter page_owner output:: + + ./page_owner_filter -m handle -n 0,1,2 > filtered_page_owner.txt + +5) Analyze information from page owner:: cat /sys/kernel/debug/page_owner_stacks/show_stacks > stacks.txt cat stacks.txt @@ -263,3 +275,65 @@ STANDARD FORMAT SPECIFIERS f free whether the page has been released or not st stacktrace stack trace of the page allocation ator allocator memory allocator for pages + +Filtering page_owner output +============================ + +page_owner supports filtering output at the kernel level before reading, +which reduces the amount of data that needs to be processed in userspace. + +The page_owner_filter tool provides a convenient interface for this filtering +capability. It supports two types of filters: + +1. **print_mode filter**: Control what information is printed for each page + - ``stack``: Print full stack traces (default, compatible with existing usage) + - ``handle``: Print only stack handle numbers (much faster, smaller output) + - ``stack_handle``: Print both stack traces and handle numbers + + The ``handle`` mode uses numeric identifiers instead of full stack traces. + The mapping from handles to actual stack traces can be obtained via the + show_stacks_handles interface. + +2. **NUMA node filter**: Filter pages by NUMA node ID + - Supports single node: ``-n 0`` + - Multiple nodes: ``-n 0,1,2`` + - Ranges: ``-n 0-3`` + - Mixed format: ``-n 0,2-3,5`` + +Usage examples:: + + # Filter by print mode + ./page_owner_filter -m handle + ./page_owner_filter -m stack_handle + + # Filter by NUMA node + ./page_owner_filter -n 0 + ./page_owner_filter -n 0-3 + + # Combined filters + ./page_owner_filter -m stack -n 0,1,2 + ./page_owner_filter -m handle -n 0,2-3 + + # Save to file + ./page_owner_filter -m handle -o filtered_output.txt + +The handle mode is particularly useful for monitoring and performance-critical +scenarios as it dramatically reduces output size. Testing shows handle mode can +reduce output size by ~66% (84MB vs 244MB) and improve read performance by ~4.4x +compared to full stack output. + +The NUMA node filter is useful for NUMA-aware memory allocation analysis and debugging. + +Behind the scenes, page_owner_filter opens /sys/kernel/debug/page_owner and +writes filter commands before reading the filtered output. The filtering uses +per-file-descriptor state, allowing each open() to have independent filter settings. + +Each file descriptor maintains its own filter state, so you can have multiple +independent filtering operations running concurrently. For example, in different +terminals you can run different filters simultaneously:: + + # Terminal 1: Filter node 0 + ./page_owner_filter -n 0 > node0_output.txt + + # Terminal 2: Filter node 1 (runs concurrently) + ./page_owner_filter -n 1 > node1_output.txt diff --git a/Documentation/mm/physical_memory.rst b/Documentation/mm/physical_memory.rst index b76183545e5b..a09407d72973 100644 --- a/Documentation/mm/physical_memory.rst +++ b/Documentation/mm/physical_memory.rst @@ -262,9 +262,6 @@ General PFN that needs to be initialized. Defined only when ``CONFIG_DEFERRED_STRUCT_PAGE_INIT`` is enabled -``deferred_split_queue`` - Per-node queue of huge pages that their split was deferred. Defined only when ``CONFIG_TRANSPARENT_HUGEPAGE`` is enabled. - ``__lruvec`` Per-node lruvec holding LRU lists and related parameters. Used only when memory cgroups are disabled. It should not be accessed directly, use diff --git a/Documentation/mm/process_addrs.rst b/Documentation/mm/process_addrs.rst index 042d64d72421..a7296f251799 100644 --- a/Documentation/mm/process_addrs.rst +++ b/Documentation/mm/process_addrs.rst @@ -724,7 +724,7 @@ the zap and the invocation of :c:func:`!free_pgtables`. Since it is assumed that all such steps have been taken, page table entries are cleared without page table locks (in the :c:func:`!pgd_clear`, :c:func:`!p4d_clear`, -:c:func:`!pud_clear`, and :c:func:`!pmd_clear` functions. +:c:func:`!pud_clear`, and :c:func:`!pmd_clear` functions). .. note:: It is possible for leaf page tables to be torn down independent of the page tables above it as is done by diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index 3f0ef1e27eb0..2fc53093752d 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -346,6 +346,8 @@ Code Seq# Include File Comments <mailto:luzmaximilian@gmail.com> 0xA5 20-2F linux/surface_aggregator/dtx.h Microsoft Surface DTX driver <mailto:luzmaximilian@gmail.com> +0xA6 00-0F uapi/linux/alloc_tag.h Memory allocation profiling + <mailto:surenb@google.com> 0xAA 00-3F linux/uapi/linux/userfaultfd.h 0xAB 00-1F linux/nbd.h 0xAC 00-1F linux/raw.h diff --git a/Documentation/userspace-api/mseal.rst b/Documentation/userspace-api/mseal.rst index ea9b11a0bd89..c039d782f9f5 100644 --- a/Documentation/userspace-api/mseal.rst +++ b/Documentation/userspace-api/mseal.rst @@ -50,8 +50,8 @@ mseal syscall signature * The start address (``addr``) is not allocated. * The end address (``addr`` + ``len``) is not allocated. * A gap (unallocated memory) between start and end address. - - **-EPERM**: - * sealing is supported only on 64-bit CPUs, 32-bit is not supported. + - **-ENOSYS**: + * The kernel does not implement ``mseal()``. **Note about error return**: - For above error cases, users can expect the given memory range is @@ -62,7 +62,8 @@ mseal syscall signature memory range could happen. However, those cases should be rare. **Architecture support**: - mseal only works on 64-bit CPUs, not 32-bit CPUs. + mseal is built only for 64-bit kernels. 32-bit kernels return + ``-ENOSYS``. **Idempotent**: users can call mseal multiple times. mseal on an already sealed memory @@ -131,11 +132,11 @@ Use cases - Chrome browser: protect some security sensitive data structures. - System mappings: - The system mappings are created by the kernel and includes vdso, vvar, + The system mappings are created by the kernel and include vdso, vvar, vvar_vclock, vectors (arm compat-mode), sigpage (arm compat-mode), uprobes. Those system mappings are readonly only or execute only, memory sealing can - protect them from ever changing to writable or unmmap/remapped as different + protect them from ever changing to writable or unmapped/remapped as different attributes. This is useful to mitigate memory corruption issues where a corrupted pointer is passed to a memory management system. @@ -143,9 +144,6 @@ Use cases the CONFIG_MSEAL_SYSTEM_MAPPINGS seals all system mappings of this architecture. - The following architectures currently support this feature: x86-64, arm64, - loongarch and s390. - WARNING: This feature breaks programs which rely on relocating or unmapping system mappings. Known broken software at the time of writing includes CHECKPOINT_RESTORE, UML, gVisor, rr. Therefore |