From 88f0adba8aa1cf87a50dd0b55e42e27e6a77db35 Mon Sep 17 00:00:00 2001 From: Zi Yan Date: Tue, 4 Aug 2026 17:04:28 -0400 Subject: xarray: honor XA_FLAGS_ACCOUNT in xas_split_alloc() XArray operations that allocate xa_nodes, such as xas_nomem() and xas_alloc(), add __GFP_ACCOUNT when the array has XA_FLAGS_ACCOUNT set. This charges the allocated memory and avoids the workingset convergence issue described by commit 7b785645e8f13 ("mm: fix page cache convergence regression"). xas_split_alloc() does not add _GFP_ACCOUNT when XA_FLAGS_ACCOUNT is present. Fix it. Link: https://lore.kernel.org/20260804-add-gfp_account-to-xas_split_alloc-v3-2-38cb3ff325c5@nvidia.com Fixes: 6b24ca4a1a8d ("mm: Use multi-index entries in the page cache") Signed-off-by: Zi Yan Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Johannes Weiner Cc: Baolin Wang Cc: Barry Song Cc: David Hildenbrand Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Matthew Wilcox (Oracle) Cc: Ryan Roberts Cc: William Kucharski Cc: Signed-off-by: Andrew Morton --- lib/xarray.c | 3 +++ 1 file changed, 3 insertions(+) (limited to 'lib') diff --git a/lib/xarray.c b/lib/xarray.c index 9a8b4916540c..bfe7bef80f34 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1053,6 +1053,9 @@ void xas_split_alloc(struct xa_state *xas, void *entry, unsigned int order, if (xas->xa_shift + XA_CHUNK_SHIFT > order) return; + if (xas->xa->xa_flags & XA_FLAGS_ACCOUNT) + gfp |= __GFP_ACCOUNT; + do { struct xa_node *node; -- cgit v1.2.3 From 58ef4551474fb1304e8fb7659144547e731913e5 Mon Sep 17 00:00:00 2001 From: Zhan Xusheng Date: Wed, 5 Aug 2026 15:05:29 +0800 Subject: maple_tree: remove unused mas_is_root_limits() The last callers of mas_is_root_limits() were removed by commit b8852ef30c67 ("maple_tree: remove maple big node and subtree structs"), together with the maple subtree state (mast_*) code that used it. As a static inline it does not trigger -Wunused-function, so it went unnoticed. Remove it. No functional change. Link: https://lore.kernel.org/20260805070529.4118794-1-zhanxusheng@xiaomi.com Signed-off-by: Zhan Xusheng Reviewed-by: Liam R. Howlett (Oracle) Cc: Alice Ryhl Cc: Andrew Ballance Signed-off-by: Andrew Morton --- lib/maple_tree.c | 5 ----- 1 file changed, 5 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 529acc056e55..b48bc4064ad2 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -356,11 +356,6 @@ static __always_inline bool mte_is_root(const struct maple_enode *node) return ma_is_root(mte_to_node(node)); } -static inline bool mas_is_root_limits(const struct ma_state *mas) -{ - return !mas->min && mas->max == ULONG_MAX; -} - static __always_inline bool mt_is_alloc(struct maple_tree *mt) { return (mt->ma_flags & MT_FLAGS_ALLOC_RANGE); -- cgit v1.2.3 From ebe4894fe16aeda6f8d991b43ccd4be70bfc9240 Mon Sep 17 00:00:00 2001 From: Suren Baghdasaryan Date: Wed, 8 Jul 2026 18:01:23 +0000 Subject: alloc_tag: add ioctl to /proc/allocinfo Patch series "alloc_tag: introduce IOCTL-based filtering for MAP", v8. Currently, memory allocation profiling data is primarily exposed through /proc/allocinfo. While useful for manual inspection, this text-based interface poses challenges for production monitoring and large-scale analysis: 1. Userspace must parse large amounts of text to extract specific fields. 2. To find specific tags, userspace must read the entire dataset, requiring many context switches and high data copying. 3. The kernel currently aggregates per-CPU counters for every allocation size, even those the user intends to filter out immediately. This series introduces a new IOCTL-based binary interface for allocinfo that supports kernel-side filtering. By allowing the user to specify a filter mask, we significantly reduce the work performed in-kernel and the amount of data transferred to userspace. The IOCTL mechanism was chosen for allocinfo to address the per-CPU counter aggregation bottleneck. A traditional read() operation must report the total allocation count and sizes for every code tag in the system. Doing so requires iterating across all CPUs to sum their per-CPU counters for thousands of tags, which introduces substantial runtime overhead. The IOCTL interface allows userspace to push selective filtering criteria directly into the kernel before the per-CPU counter aggregation. The kernel aggregates per-CPU counters only for a small subset of tags that match the filter. This results in significant performance improvement. Beyond fast filtered retrieval, the IOCTL foundation allows introducing a context capture mechanism in the future to capture the context for specific allocations. Performance measurements were conducted on an Intel Xeon Platinum 8481C (224 CPUs) with caches dropped before each run. The IOCTL mechanism shows a ~20x performance improvement for filtered queries. The kernel avoids the expensive per-CPU counter aggregation (alloc_tag_read) for any tags that fail the initial string or location filters. Scenario 1: Specific File Filtering (arch/x86/events/rapl.c) 1. Traditional (cat /proc/allocinfo | grep): 22ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 2: Compound Filtering (Filename + Size) 1. Traditional: (cat ... | grep | awk): 21ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 3: Size-Based Filtering (min_size = 1MB) 1. Traditional: (cat ... | awk): 21ms (sys) 2. IOCTL Interface: 14ms (sys) This patch (of 6): Add the following ioctl commands for /proc/allocinfo file: ALLOCINFO_IOC_CONTENT_ID - gets content identifier which can be used to check whether the file content has changed specifically due to module load/unload. Every time a module is loaded / unloaded, the returned value will be different. By comparing the identifier value at the beginning and at the end of the content retrieval operation, users can validate retrieved information for consistency. ALLOCINFO_IOC_GET_AT - gets the record at the specified position. This is the position of a record in /proc/allocinfo. ALLOCINFO_IOC_GET_NEXT - gets the record next to the last retrieved one. If no records were previously retrieved, returns the first record. Note, function file and module names often have the same prefixes, therefore when filtering for them, we compare the last 64 characters to minimize the chances of name collisions. [akpm@linux-foundation.org: include compat.h, per Suren] Closes: https://lore.kernel.org/oe-kbuild-all/202607091820.qbjlGhKK-lkp@intel.com/ Link: https://lore.kernel.org/cover.1783532853.git.abhishekbapat@google.com Link: https://lore.kernel.org/15596de2607ef13e7c77c6d74763f4ae992ec475.1783532853.git.abhishekbapat@google.com Signed-off-by: Suren Baghdasaryan Signed-off-by: Abhishek Bapat Acked-by: Hao Ge Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Signed-off-by: Andrew Morton --- Documentation/mm/allocation-profiling.rst | 5 + Documentation/userspace-api/ioctl/ioctl-number.rst | 2 + MAINTAINERS | 1 + include/linux/codetag.h | 2 + include/uapi/linux/alloc_tag.h | 65 ++++++ lib/codetag.c | 18 ++ mm/alloc_tag.c | 239 ++++++++++++++++++++- 7 files changed, 330 insertions(+), 2 deletions(-) create mode 100644 include/uapi/linux/alloc_tag.h (limited to 'lib') diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst index e928aa3e4e1e..b2ebcef8af6f 100644 --- a/Documentation/mm/allocation-profiling.rst +++ b/Documentation/mm/allocation-profiling.rst @@ -57,6 +57,11 @@ sysctl: Runtime info: /proc/allocinfo + Profiling data can be retrieved either by reading `/proc/allocinfo` directly as + text or programmatically via `ioctl()` calls defined in ``. + The ioctl interface supports structured binary data extraction as well as filtering + by module name, function, file, line number, accuracy, or allocation size limits. + Example output:: root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index 3f0ef1e27eb0..2fc53093752d 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -346,6 +346,8 @@ Code Seq# Include File Comments 0xA5 20-2F linux/surface_aggregator/dtx.h Microsoft Surface DTX driver +0xA6 00-0F uapi/linux/alloc_tag.h Memory allocation profiling + 0xAA 00-3F linux/uapi/linux/userfaultfd.h 0xAB 00-1F linux/nbd.h 0xAC 00-1F linux/raw.h diff --git a/MAINTAINERS b/MAINTAINERS index 449e89dd5194..bf429cd01d30 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -16940,6 +16940,7 @@ S: Maintained F: Documentation/mm/allocation-profiling.rst F: include/linux/alloc_tag.h F: include/linux/pgalloc_tag.h +F: include/uapi/linux/alloc_tag.h F: mm/alloc_tag.c MEMORY MANAGEMENT - BALLOON diff --git a/include/linux/codetag.h b/include/linux/codetag.h index ddae7484ca45..a25a085c2df1 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -77,6 +77,8 @@ struct codetag_iterator { void codetag_lock_module_list(struct codetag_type *cttype); bool codetag_trylock_module_list(struct codetag_type *cttype); void codetag_unlock_module_list(struct codetag_type *cttype); +unsigned long codetag_get_content_id(struct codetag_type *cttype); +unsigned int codetag_get_count(struct codetag_type *cttype); struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); diff --git a/include/uapi/linux/alloc_tag.h b/include/uapi/linux/alloc_tag.h new file mode 100644 index 000000000000..ee6a023cbaf4 --- /dev/null +++ b/include/uapi/linux/alloc_tag.h @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * alloc_tag IOCTL API definition + * + * Copyright (C) 2026 Google, LLC. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _UAPI_ALLOC_TAG_H +#define _UAPI_ALLOC_TAG_H + +#include + +/* + * Function, file and module names often have the same prefixes, therefore + * when filtering by these criteria, we compare the last 64 characters to + * minimize the chances of name collisions + */ +#define ALLOCINFO_STR_SIZE 64 + +struct allocinfo_content_id { + __u64 id; +}; + +struct allocinfo_tag { + /* Longer names are trimmed */ + char modname[ALLOCINFO_STR_SIZE]; + char function[ALLOCINFO_STR_SIZE]; + char filename[ALLOCINFO_STR_SIZE]; + __u64 lineno; +}; + +/* The alignment ensures 32-bit compatible interfaces are not broken */ +struct allocinfo_counter { + __u64 bytes; + __u64 calls; + __u8 accurate; +} __attribute__((aligned(8))); + +struct allocinfo_tag_data { + struct allocinfo_tag tag; + struct allocinfo_counter counter; +}; + +struct allocinfo_get_at { + __u64 pos; /* input */ + struct allocinfo_tag_data data; +}; + +#define _ALLOCINFO_IOC_CONTENT_ID 0 +#define _ALLOCINFO_IOC_GET_AT 1 +#define _ALLOCINFO_IOC_GET_NEXT 2 + +#define ALLOCINFO_IOC_BASE 0xA6 +#define ALLOCINFO_IOC_CONTENT_ID _IOR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_CONTENT_ID, \ + struct allocinfo_content_id) +#define ALLOCINFO_IOC_GET_AT _IOWR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_GET_AT, \ + struct allocinfo_get_at) +#define ALLOCINFO_IOC_GET_NEXT _IOR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_GET_NEXT, \ + struct allocinfo_tag_data) + +#endif /* _UAPI_ALLOC_TAG_H */ diff --git a/lib/codetag.c b/lib/codetag.c index 4001a7ea6675..a9cda4c962a3 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -19,6 +19,8 @@ struct codetag_type { struct codetag_type_desc desc; /* generates unique sequence number for module load */ unsigned long next_mod_seq; + /* bumped on every module load and unload */ + unsigned long content_id; }; struct codetag_range { @@ -50,6 +52,20 @@ void codetag_unlock_module_list(struct codetag_type *cttype) up_read(&cttype->mod_lock); } +unsigned long codetag_get_content_id(struct codetag_type *cttype) +{ + lockdep_assert_held(&cttype->mod_lock); + + return cttype->content_id; +} + +unsigned int codetag_get_count(struct codetag_type *cttype) +{ + lockdep_assert_held(&cttype->mod_lock); + + return cttype->count; +} + struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) { struct codetag_iterator iter = { @@ -204,6 +220,7 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) down_write(&cttype->mod_lock); cmod->mod_seq = ++cttype->next_mod_seq; + ++cttype->content_id; mod_id = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); if (mod_id >= 0) { if (cttype->desc.module_load) { @@ -368,6 +385,7 @@ void codetag_unload_module(struct module *mod) cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); kfree(cmod); + ++cttype->content_id; } up_write(&cttype->mod_lock); if (found && cttype->desc.free_section_mem) diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index b60ee89704cc..b2ac166880ac 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -5,6 +5,8 @@ #include #include #include +#include +#include #include #include #include @@ -14,6 +16,7 @@ #include #include #include +#include #include "internal.h" #include "page_alloc.h" @@ -59,6 +62,10 @@ struct allocinfo_private { struct codetag_iterator iter; struct codetag_iterator reported_iter; bool print_header; + /* ioctl uses a separate iterator not to interfere with reads */ + struct codetag_iterator ioctl_iter; + bool positioned; /* seq_open_private() sets to 0 */ + struct mutex ioctl_lock; }; static void *allocinfo_start(struct seq_file *m, loff_t *pos) @@ -142,6 +149,235 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +/* + * Initializes seq_file operations and allocates private state when opening + * the /proc/allocinfo procfs entry. + */ +static int allocinfo_open(struct inode *inode, struct file *file) +{ + int ret; + + ret = seq_open_private(file, &allocinfo_seq_op, + sizeof(struct allocinfo_private)); + if (!ret) { + struct seq_file *m = file->private_data; + struct allocinfo_private *priv = m->private; + + mutex_init(&priv->ioctl_lock); + } + return ret; +} + +/* + * Cleans up the seq_file state and frees up the private state allocated in + * allocinfo_open() when closing the /proc/allocinfo file descriptor. + */ +static int allocinfo_release(struct inode *inode, struct file *file) +{ + struct seq_file *m = file->private_data; + struct allocinfo_private *priv = m->private; + + mutex_destroy(&priv->ioctl_lock); + return seq_release_private(inode, file); +} + +/* + * Returns a pointer to the suffix of a string so that its length fits within + * ALLOCINFO_STR_SIZE, preserving the trailing characters. + * Function, file and module names often have the same prefixes, therefore + * when filtering by these criteria, we compare the last 64 characters to + * minimize the chances of name collisions + */ +static const char *allocinfo_str(const char *str) +{ + size_t len = strlen(str); + + /* Keep an extra space for the trailing NULL. */ + if (len >= ALLOCINFO_STR_SIZE) + str += (len - ALLOCINFO_STR_SIZE) + 1; + return str; +} + +/* Copy a string and trim from the beginning if it's too long */ +static void allocinfo_copy_str(char *dest, const char *src) +{ + strscpy_pad(dest, allocinfo_str(src), ALLOCINFO_STR_SIZE); +} + +/* + * Populates the UAPI allocinfo_tag_data structure with active runtime + * profiling counters extracted from the given kernel codetag. + */ +static void allocinfo_to_params(struct codetag *ct, + struct allocinfo_tag_data *data) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + + if (ct->modname) + allocinfo_copy_str(data->tag.modname, ct->modname); + else + data->tag.modname[0] = '\0'; + allocinfo_copy_str(data->tag.function, ct->function); + allocinfo_copy_str(data->tag.filename, ct->filename); + data->tag.lineno = ct->lineno; + data->counter.bytes = counter.bytes; + data->counter.calls = counter.calls; + data->counter.accurate = !alloc_tag_is_inaccurate(tag); +} + +/* + * Retrieves the unique content ID representing the current allocation tag module + * layout, allowing userspace to detect if modules were loaded / unloaded. + */ +static int allocinfo_ioctl_get_content_id(struct seq_file *m, void __user *arg) +{ + struct allocinfo_content_id params; + + codetag_lock_module_list(alloc_tag_cttype); + params.id = codetag_get_content_id(alloc_tag_cttype); + codetag_unlock_module_list(alloc_tag_cttype); + if (copy_to_user(arg, ¶ms, sizeof(params))) + return -EFAULT; + + return 0; +} + +/* + * Seeks the ioctl iterator to the specified 0-indexed tag position, reads its + * profiling data and returns it to userspace. + */ +static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) +{ + struct allocinfo_private *priv; + struct codetag *ct; + __u64 pos; + struct allocinfo_get_at params = {0}; + + if (copy_from_user(¶ms, arg, sizeof(params))) + return -EFAULT; + + priv = m->private; + pos = params.pos; + + mutex_lock(&priv->ioctl_lock); + codetag_lock_module_list(alloc_tag_cttype); + + if (pos >= codetag_get_count(alloc_tag_cttype)) { + codetag_unlock_module_list(alloc_tag_cttype); + mutex_unlock(&priv->ioctl_lock); + return -ENOENT; + } + + /* Find the codetag */ + priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype); + ct = codetag_next_ct(&priv->ioctl_iter); + while (ct && pos--) + ct = codetag_next_ct(&priv->ioctl_iter); + if (ct) { + allocinfo_to_params(ct, ¶ms.data); + priv->positioned = true; + } + + codetag_unlock_module_list(alloc_tag_cttype); + mutex_unlock(&priv->ioctl_lock); + + if (!ct) + return -ENOENT; + + if (copy_to_user(arg, ¶ms, sizeof(params))) + return -EFAULT; + + return 0; +} + +/* + * Advances the ioctl iterator to the next allocation tag in the sequence and + * returns its profiling data to userspace. + */ +static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg) +{ + struct allocinfo_private *priv; + struct codetag *ct; + struct allocinfo_tag_data params; + int ret = 0; + + memset(¶ms, 0, sizeof(params)); + priv = m->private; + + mutex_lock(&priv->ioctl_lock); + codetag_lock_module_list(alloc_tag_cttype); + + if (!priv->positioned) { + priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype); + priv->positioned = true; + } + + ct = codetag_next_ct(&priv->ioctl_iter); + if (ct) + allocinfo_to_params(ct, ¶ms); + + if (!ct) { + priv->positioned = false; + ret = -ENOENT; + } + codetag_unlock_module_list(alloc_tag_cttype); + mutex_unlock(&priv->ioctl_lock); + + if (ret == 0) { + if (copy_to_user(arg, ¶ms, sizeof(params))) + return -EFAULT; + } + return ret; +} + +/* + * Entry point ioctl function for /proc/allocinfo routing requests to fetch the + * layout content ID, seek to a specific tag, or read sequential tags. + */ +static long allocinfo_ioctl(struct file *file, unsigned int cmd, + unsigned long __arg) +{ + void __user *arg = (void __user *)__arg; + int ret; + + switch (cmd) { + case ALLOCINFO_IOC_CONTENT_ID: + ret = allocinfo_ioctl_get_content_id(file->private_data, arg); + break; + case ALLOCINFO_IOC_GET_AT: + ret = allocinfo_ioctl_get_at(file->private_data, arg); + break; + case ALLOCINFO_IOC_GET_NEXT: + ret = allocinfo_ioctl_get_next(file->private_data, arg); + break; + default: + ret = -ENOIOCTLCMD; + break; + } + + return ret; +} + +#ifdef CONFIG_COMPAT +static long allocinfo_compat_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + return allocinfo_ioctl(file, cmd, (unsigned long)compat_ptr(arg)); +} +#endif + +static const struct proc_ops allocinfo_proc_ops = { + .proc_open = allocinfo_open, + .proc_read_iter = seq_read_iter, + .proc_lseek = seq_lseek, + .proc_release = allocinfo_release, + .proc_ioctl = allocinfo_ioctl, +#ifdef CONFIG_COMPAT + .proc_compat_ioctl = allocinfo_compat_ioctl, +#endif +}; + size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep) { struct codetag_iterator iter; @@ -999,8 +1235,7 @@ static int __init alloc_tag_init(void) return 0; } - if (!proc_create_seq_private(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_seq_op, - sizeof(struct allocinfo_private), NULL)) { + if (!proc_create(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_proc_ops)) { pr_err("Failed to create %s file\n", ALLOCINFO_FILE_NAME); shutdown_mem_profiling(false); return -ENOMEM; -- cgit v1.2.3 From d9c2fc1ffa40eb13a1384de5077a9cd65dda3881 Mon Sep 17 00:00:00 2001 From: Mark Sercombe Date: Thu, 13 Aug 2026 20:38:47 +0200 Subject: maple_tree: fix comment typo Fix a spelling mistake n a code comment. This is a comment only change with no functional impact. Link: https://lore.kernel.org/20260813183847.474357-1-sercombe.joel.mark@gmail.com Signed-off-by: Mark Sercombe Reviewed-by: Andrew Morton Reviewed-by: Liam R. Howlett (Oracle) Cc: Alice Ryhl Cc: Andrew Ballance Signed-off-by: Andrew Morton --- lib/maple_tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index b48bc4064ad2..a0542b491bc2 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3590,7 +3590,7 @@ set_content: /** * mas_prealloc_calc() - Calculate number of nodes needed for a - * given store oepration + * given store operation * @wr_mas: The maple write state * @entry: The entry to store into the tree * -- cgit v1.2.3 From f73be537f4b3f639defa2cf60f692dd2b64badbc Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Wed, 12 Aug 2026 09:47:43 -1000 Subject: percpu: drop CONFIG_DEBUG_FORCE_WEAK_PER_CPU alpha requires percpu variables in modules to be defined as weak so that the compiler generates GOT based external references for them. This puts two extra restrictions on percpu variable definitions. The symbol must be globally unique even when static and a static percpu variable can't be defined inside a function. DEBUG_FORCE_WEAK_PER_CPU exists to give generic code build coverage for these restrictions without building for alpha. MEM_ALLOC_PROFILING defines a static percpu counter at each allocation call site and thus can't be built with weak percpu definitions, so it depends on !DEBUG_FORCE_WEAK_PER_CPU. As allmodconfig enables DEBUG_FORCE_WEAK_PER_CPU, this knocks MEM_ALLOC_PROFILING out of allmodconfig build coverage. allmodconfig coverage for MEM_ALLOC_PROFILING is worth more than build coverage for restrictions which only matter to alpha module builds. Drop DEBUG_FORCE_WEAK_PER_CPU. Restriction violations will now show up only on alpha builds. Link: https://lore.kernel.org/178656406317.2437052.7257990869957704195@slm.duckdns.org Signed-off-by: Tejun Heo Reported-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Acked-by: Gabriele Monaco [include/rv/da_monitor.h] Cc: Dennis Zhou Cc: Kent Overstreet Cc: Steven Rostedt Signed-off-by: Andrew Morton --- include/linux/percpu-defs.h | 7 +------ include/rv/da_monitor.h | 2 +- lib/Kconfig.debug | 15 --------------- mm/Kconfig.debug | 1 - 4 files changed, 2 insertions(+), 23 deletions(-) (limited to 'lib') diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index 2cba7cc2b01f..dbe3267a0a13 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -65,13 +65,8 @@ * * Archs which need weak percpu definitions should set * CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU when necessary. - * - * To ensure that the generic code observes the above two - * restrictions, if CONFIG_DEBUG_FORCE_WEAK_PER_CPU is set weak - * definition is used for all cases. */ -#if (defined(CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU) && defined(MODULE)) || \ - defined(CONFIG_DEBUG_FORCE_WEAK_PER_CPU) +#if defined(CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU) && defined(MODULE) /* * __pcpu_scope_* dummy variable is used to enforce scope. It * receives the static modifier when it's used in front of diff --git a/include/rv/da_monitor.h b/include/rv/da_monitor.h index 34b8fba9ecd4..6b641697106b 100644 --- a/include/rv/da_monitor.h +++ b/include/rv/da_monitor.h @@ -24,7 +24,7 @@ /* * Per-cpu variables require a unique name although static in some - * configurations (e.g. CONFIG_DEBUG_FORCE_WEAK_PER_CPU or alpha modules). + * configurations (e.g. alpha modules). */ #define DA_MON_NAME CONCATENATE(da_mon_, MONITOR_NAME) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index b82515cde538..00921b1676e8 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -613,21 +613,6 @@ config BUILTIN_MODULE_RANGES It also records an anchor symbol to determine the load address of the section. -config DEBUG_FORCE_WEAK_PER_CPU - bool "Force weak per-cpu definitions" - depends on DEBUG_KERNEL - help - s390 and alpha require percpu variables in modules to be - defined weak to work around addressing range issue which - puts the following two restrictions on percpu variable - definitions. - - 1. percpu symbols must be unique whether static or not - 2. percpu variables can't be defined inside a function - - To ensure that generic code follows the above rules, this - option forces all percpu variables to be defined as weak. - config WARN_CONTEXT_ANALYSIS bool "Compiler context-analysis warnings" depends on CC_IS_CLANG && CLANG_VERSION >= 230000 diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 5737a504efbb..15dca19dd07d 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -326,7 +326,6 @@ config MEM_ALLOC_PROFILING default n depends on MMU depends on PROC_FS - depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION select SLAB_OBJ_EXT -- cgit v1.2.3 From 0f90a0d0c7c2893e1d6878b5de99dfa4e8f0c12d Mon Sep 17 00:00:00 2001 From: Qiang Liu Date: Wed, 12 Aug 2026 17:28:56 +0800 Subject: lib/test_hmm: fix garbage pfn and wrong direction in devmem fault debug Move pr_debug() inside the `if (dpage)` block to avoid printing garbage pfn for NULL dpage, and correct the direction label from "sys to dev" to "dev to sys". Link: https://lore.kernel.org/20260812092856.55296-1-liuqiangneo@163.com Signed-off-by: Qiang Liu Assisted-by: Qoder:Qwen-3.8-MAX-Preview Cc: Jason Gunthorpe Cc: Leon Romanovsky Signed-off-by: Andrew Morton --- lib/test_hmm.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) (limited to 'lib') diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 6205fb313bd0..7c4d10eae6fe 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -1151,10 +1151,9 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, if (!dpage && !order) return VM_FAULT_OOM; - pr_debug("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n", - page_to_pfn(spage), page_to_pfn(dpage)); - if (dpage) { + pr_debug("migrating from dev to sys pfn src: 0x%lx pfn dst: 0x%lx\n", + page_to_pfn(spage), page_to_pfn(dpage)); lock_page(dpage); *dst |= migrate_pfn(page_to_pfn(dpage)); } -- cgit v1.2.3 From b41e934a2b6a9cfca5a52ad5feb804cac8bf012a Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:25 -0400 Subject: maple_tree: add rcu locking check when LOCKDEP is enabled Patch series "maple_tree: lock checking and clean ups", v2. The goals of this series are: 1. Lock issue detection A number of syzbot reports are incorrectly pointing to the mm exit as a source of the locking error. The first three patches attempt to help users detect errors in their locking - but they still have to use LOCKDEP. I guess it's still down to hope and prayers. 2. Documentation fixes The documentation was lacking clarity, there are updates to try and help the users, especially around the erase() cases. 3. Two benign issues The cyclic allocator may have a race, although no in-kernel user can hit it. The erase functions may cause allocation issues if used with the incorrect locking type, but none are present in-tree. Beyond these goals there are some test fixes, some general speed-up patches targeting extra work and cycles, and dropping dead code. This patch (of 19): When CONFIG_LOCKDEP and CONFIG_RCU_STRICT_GRACE_PERIOD is enabled, check for rcu locking issues by recording the grace period in the maple state and checking the rcu window is still valid whenever the maple state is reused with a state that is not MA_START or MA_PAUSED. Link: https://lore.kernel.org/20260630190843.3563858-1-liam@infradead.org Link: https://lore.kernel.org/20260630190843.3563858-2-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 3 +++ lib/maple_tree.c | 50 +++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletion(-) (limited to 'lib') diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 1b3014377105..1acf932fcd33 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -484,6 +484,9 @@ struct ma_state { unsigned char mas_flags; unsigned char end; /* The end of the node */ enum store_type store_type; /* The type of store needed for this operation */ +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + unsigned long rcu_gp; +#endif }; struct ma_wr_state { diff --git a/lib/maple_tree.c b/lib/maple_tree.c index a0542b491bc2..6d805521bedd 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1148,6 +1148,42 @@ static inline void mas_free(struct ma_state *mas, struct maple_enode *used) ma_free_rcu(mte_to_node(used)); } +void mas_lock_check(struct ma_state *mas) +{ + +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + if (!mas_is_active(mas)) + return; + + if (!mt_locked(mas->tree)) { + if (mt_in_rcu(mas->tree)) + WARN_ON_ONCE(poll_state_synchronize_rcu(mas->rcu_gp)); + } +#endif + +} + +void mas_init_lock_check(struct ma_state *mas) +{ +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + if (!mt_locked(mas->tree)) { + if (mt_in_rcu(mas->tree)) + mas->rcu_gp = get_state_synchronize_rcu(); + } +#endif + +} + +static void mas_may_init_lock_check(struct ma_state *mas) +{ +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + if (mas_is_start(mas) || mas_is_paused(mas)) + mas_init_lock_check(mas); + else + mas_lock_check(mas); +#endif +} + /* * mas_start() - Sets up maple state for operations. * @mas: The maple state. @@ -1166,6 +1202,7 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) if (likely(mas_is_start(mas))) { struct maple_enode *root; + mas_init_lock_check(mas); mas->min = 0; mas->max = ULONG_MAX; @@ -4355,6 +4392,7 @@ void *mas_walk(struct ma_state *mas) { void *entry; + mas_may_init_lock_check(mas); if (!mas_is_active(mas) && !mas_is_start(mas)) mas->status = ma_start; retry: @@ -4992,6 +5030,7 @@ static void mas_may_activate(struct ma_state *mas) mas->status = ma_start; } else { mas->status = ma_active; + mas_lock_check(mas); } } @@ -5069,6 +5108,7 @@ void *mas_next(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_next_setup(mas, max, &entry)) return entry; @@ -5092,6 +5132,7 @@ void *mas_next_range(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_next_setup(mas, max, &entry)) return entry; @@ -5200,6 +5241,7 @@ void *mas_prev(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_prev_setup(mas, min, &entry)) return entry; @@ -5223,6 +5265,7 @@ void *mas_prev_range(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_prev_setup(mas, min, &entry)) return entry; @@ -5269,6 +5312,7 @@ EXPORT_SYMBOL_GPL(mt_prev); */ void mas_pause(struct ma_state *mas) { + mas_lock_check(mas); mas->status = ma_pause; mas->node = NULL; } @@ -5377,6 +5421,7 @@ void *mas_find(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_setup(mas, max, &entry)) return entry; @@ -5404,6 +5449,7 @@ void *mas_find_range(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_setup(mas, max, &entry)) return entry; @@ -5516,6 +5562,7 @@ void *mas_find_rev(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_rev_setup(mas, min, &entry)) return entry; @@ -5542,6 +5589,7 @@ void *mas_find_range_rev(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_rev_setup(mas, min, &entry)) return entry; @@ -5618,7 +5666,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) if (!mas->sheaf && !mas->alloc) return false; - mas->status = ma_start; + mas_reset(mas); return true; } -- cgit v1.2.3 From 19db85d08379909f5371310c37ba8d53d47fc11f Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:27 -0400 Subject: maple_tree: add write lock checking with lockdep sequence numbers Use the lockdep sequence numbers to ensure the write lock is not dropped between write operations. The lockdep sequence is recorded on any walk that starts from the top of the tree and re-checked prior to any operation using an active node. Link: https://lore.kernel.org/20260630190843.3563858-4-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Cc: Breno Leitao Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 7 ++++-- lib/maple_tree.c | 58 ++++++++++++++++++++++++++++++++++++---------- 2 files changed, 51 insertions(+), 14 deletions(-) (limited to 'lib') diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 1acf932fcd33..d63ac92208d0 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -484,9 +484,12 @@ struct ma_state { unsigned char mas_flags; unsigned char end; /* The end of the node */ enum store_type store_type; /* The type of store needed for this operation */ -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) +#ifdef CONFIG_LOCKDEP + u32 ld_seq; +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD unsigned long rcu_gp; -#endif +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ +#endif /* CONFIG_LOCKDEP */ }; struct ma_wr_state { diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 6d805521bedd..ca25609cbe61 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1148,40 +1148,71 @@ static inline void mas_free(struct ma_state *mas, struct maple_enode *used) ma_free_rcu(mte_to_node(used)); } -void mas_lock_check(struct ma_state *mas) + +#ifdef CONFIG_LOCKDEP +static struct lockdep_map *mas_lockdep_map(struct ma_state *mas) { + struct maple_tree *mt = mas->tree; + + if (mt_external_lock(mt)) + return mt->ma_external_lock; + + return &(mt->ma_lock).dep_map; +} + +#endif -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) +static void mas_lock_check(struct ma_state *mas) +{ +#ifdef CONFIG_LOCKDEP + struct lockdep_map *map; if (!mas_is_active(mas)) return; +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD if (!mt_locked(mas->tree)) { if (mt_in_rcu(mas->tree)) WARN_ON_ONCE(poll_state_synchronize_rcu(mas->rcu_gp)); } -#endif +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ + + map = mas_lockdep_map(mas); + if (map && lock_is_held(map)) + WARN_ON_ONCE(mas->ld_seq != lock_sequence(map)); +#endif /* CONFIG_LOCKDEP */ } -void mas_init_lock_check(struct ma_state *mas) +static void mas_init_lock_check(struct ma_state *mas) { -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) +#ifdef CONFIG_LOCKDEP + struct lockdep_map *map; +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD if (!mt_locked(mas->tree)) { if (mt_in_rcu(mas->tree)) mas->rcu_gp = get_state_synchronize_rcu(); + return; } -#endif +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ + + map = mas_lockdep_map(mas); + if (map && lock_is_held(map)) + mas->ld_seq = lock_sequence(mas_lockdep_map(mas)); +#endif /* CONFIG_LOCKDEP */ } static void mas_may_init_lock_check(struct ma_state *mas) { -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) - if (mas_is_start(mas) || mas_is_paused(mas)) +#ifdef CONFIG_LOCKDEP +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD + if (mas_is_start(mas) || mas_is_paused(mas)) { mas_init_lock_check(mas); - else - mas_lock_check(mas); -#endif + return; + } +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ + mas_lock_check(mas); +#endif /* CONFIG_LOCKDEP */ } /* @@ -4864,6 +4895,7 @@ void *mas_store(struct ma_state *mas, void *entry) { MA_WR_STATE(wr_mas, mas, entry); + mas_may_init_lock_check(mas); trace_ma_write(TP_FCT, mas, 0, entry); #ifdef CONFIG_DEBUG_MAPLE_TREE if (MAS_WARN_ON(mas, mas->index > mas->last)) @@ -4922,6 +4954,7 @@ int mas_store_gfp(struct ma_state *mas, void *entry, gfp_t gfp) MA_WR_STATE(wr_mas, mas, entry); int ret = 0; + mas_may_init_lock_check(mas); retry: mas_wr_preallocate(&wr_mas, entry); if (unlikely(mas_nomem(mas, gfp))) { @@ -4952,6 +4985,7 @@ void mas_store_prealloc(struct ma_state *mas, void *entry) { MA_WR_STATE(wr_mas, mas, entry); + mas_lock_check(mas); if (mas->store_type == wr_store_root) { mas_wr_prealloc_setup(&wr_mas); goto store; @@ -4984,6 +5018,7 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) { MA_WR_STATE(wr_mas, mas, entry); + mas_may_init_lock_check(mas); mas_wr_prealloc_setup(&wr_mas); mas->store_type = mas_wr_store_type(&wr_mas); mas_prealloc_calc(&wr_mas, entry); @@ -5469,7 +5504,6 @@ EXPORT_SYMBOL_GPL(mas_find_range); static bool mas_find_rev_setup(struct ma_state *mas, unsigned long min, void **entry) { - switch (mas->status) { case ma_active: goto active; -- cgit v1.2.3 From 7b5bacfcc2cbbd78860c628a079172d067bee25b Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Mon, 20 Jul 2026 17:30:38 -0400 Subject: maple_tree-stop-flooding-logs-when-debug_locks-is-set-to-zero-fix stop flooding logs when debug_locks is set to zero When lockdep detects an issue, it sets debug_locks to 0 disabling further reports. The newly added maple tree sequence counting on locks to detect errors is not taking this nor the fact that lock_is_held() can return -1 (unknown state) into account. The resulting action is a flood of logging of not matching sequence numbers being reported on failure. __lock_sequnece() will return u32 ~0 when debug_locks is zero, and the real sequnece count cannot return such a high value as it is less than 32bits. By always updating the sequence number, regardless of lock state and by ignoring ~0 value in the sequence number will avoid ever printing a WARN_ON when this condition is hit. Link: https://lore.kernel.org/fo5r2xl3fql6gyvjasfvfu4ftugjti2vxg7zskdw7hjeo3xvqm@kk66qmsf762s Signed-off-by: Liam R. Howlett (Oracle) Tested-by: Breno Leitao Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index ca25609cbe61..d8c826e1ca0c 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1166,6 +1166,8 @@ static void mas_lock_check(struct ma_state *mas) { #ifdef CONFIG_LOCKDEP struct lockdep_map *map; + u32 seq; + if (!mas_is_active(mas)) return; @@ -1177,8 +1179,12 @@ static void mas_lock_check(struct ma_state *mas) #endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ map = mas_lockdep_map(mas); - if (map && lock_is_held(map)) - WARN_ON_ONCE(mas->ld_seq != lock_sequence(map)); + if (!map) + return; + + seq = lock_sequence(map); + if (seq != UINT_MAX && mas->ld_seq != UINT_MAX) + WARN_ON_ONCE(mas->ld_seq != seq); #endif /* CONFIG_LOCKDEP */ } @@ -1196,8 +1202,8 @@ static void mas_init_lock_check(struct ma_state *mas) #endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ map = mas_lockdep_map(mas); - if (map && lock_is_held(map)) - mas->ld_seq = lock_sequence(mas_lockdep_map(mas)); + if (map) /* Update regardless of lock state */ + mas->ld_seq = lock_sequence(map); #endif /* CONFIG_LOCKDEP */ } -- cgit v1.2.3 From 37527a73d58f1a791b172adca8ad01d3cb4ddea6 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:29 -0400 Subject: maple_tree: drop dead code from mas_extend_spanning_null() mas_extend_spanning_null() had a clause if the end of the range being written (mas->last) is the same as the end of the existing range it is overwriting (wr_mas->r_max), action will be taken. This code path is not possible because the only calling function increments mas->last (unless it's ULONG_MAX) to walk to one beyond the write and then resets the value back to the initial value. In the case of mas->last == ULONG_MAX, then the second part of the statement will always be false - mas->last cannot be less than the node max. This code never executed and is flawed anyways (the arguments are incorrectly ordered), so removing it is the safest action. Since the code never executes, it is not fixing any issue so Fixes tag is not given. Link: https://lore.kernel.org/20260630190843.3563858-6-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 7 ------- 1 file changed, 7 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index d8c826e1ca0c..3509e293c84f 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -2997,13 +2997,6 @@ static inline void mas_extend_spanning_null(struct ma_wr_state *l_wr_mas, if (r_mas->last < r_wr_mas->r_max) r_mas->last = r_wr_mas->r_max; r_mas->offset++; - } else if ((r_mas->last == r_wr_mas->r_max) && - (r_mas->last < r_mas->max) && - !mas_slot_locked(r_mas, r_wr_mas->slots, r_mas->offset + 1)) { - r_mas->last = mas_safe_pivot(r_mas, r_wr_mas->pivots, - r_wr_mas->type, r_mas->offset + 1); - r_mas->offset++; - r_wr_mas->r_max = r_mas->last; } } -- cgit v1.2.3 From 0286fe4b2d1affcd5a8fb12bea8cba7382c28d43 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:31 -0400 Subject: maple_tree: clarify comments on mas_nomem() When an allocation completely fails, the return is false. If the allocation succeeds or partially succeeds, return true to indicate a retry of the operation. Note that since the lock may have been dropped, the operation is retried from the start - including potentially allocating more memory. Link: https://lore.kernel.org/20260630190843.3563858-8-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 3509e293c84f..baaaa128594c 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5677,10 +5677,11 @@ EXPORT_SYMBOL_GPL(mas_erase); /** * mas_nomem() - Check if there was an error allocating and do the allocation - * if necessary If there are allocations, then free them. + * if necessary. + * * @mas: The maple state * @gfp: The GFP_FLAGS to use for allocations - * Return: true on allocation, false otherwise. + * Return: False on no memory. True otherwise (partial success as well) */ bool mas_nomem(struct ma_state *mas, gfp_t gfp) __must_hold(mas->tree->ma_lock) @@ -5696,6 +5697,10 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) mas_alloc_nodes(mas, gfp); } + /* + * Return false on zero forward progress. Partial allocations are kept + * so the retry path will attempt to get the rest. + */ if (!mas->sheaf && !mas->alloc) return false; -- cgit v1.2.3 From 07f58b70ae0352fad3327da895daaffa88a055f2 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:32 -0400 Subject: maple_tree: use prefetched value in mas_wr_store_type() The slot contents exist in wr_mas->content, which has less overhead than reading the slot again. Link: https://lore.kernel.org/20260630190843.3563858-9-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index baaaa128594c..d5fa85bc7aab 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3280,7 +3280,7 @@ static inline void mas_wr_slot_store(struct ma_wr_state *wr_mas) void __rcu **slots = wr_mas->slots; bool gap = false; - gap |= !mt_slot_locked(mas->tree, slots, offset); + gap |= !wr_mas->content; gap |= !mt_slot_locked(mas->tree, slots, offset + 1); if (wr_mas->offset_end - offset == 1) { -- cgit v1.2.3 From 9af2387e82f5e74a1f1a0ff36d8dec21c44d852d Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:33 -0400 Subject: maple_tree: optimise mas_wr_node_store() when not in rcu mode Clearing the entire node on the stack is unnecessary since most of the node will be overwritten anyways. Just clear what isn't used after the data is in place. Benchmarking shows a speedup of 0.67% on a height 4 tree with 2048 entries. Link: https://lore.kernel.org/20260630190843.3563858-10-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index d5fa85bc7aab..56812db8b0ef 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3187,7 +3187,7 @@ static void mas_wr_spanning_store(struct ma_wr_state *wr_mas) static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) { unsigned char dst_offset, offset_end; - unsigned char copy_size, node_pivots; + unsigned char copy_size, node_pivots, node_slots; struct maple_node reuse, *newnode; unsigned long *dst_pivots; void __rcu **dst_slots; @@ -3200,6 +3200,7 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) in_rcu = mt_in_rcu(mas->tree); offset_end = wr_mas->offset_end; node_pivots = mt_pivots[wr_mas->type]; + node_slots = mt_slots[wr_mas->type]; /* Assume last adds an entry */ new_end = mas->end + 1 - offset_end + mas->offset; if (mas->last == wr_mas->end_piv) { @@ -3211,7 +3212,6 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) if (in_rcu) { newnode = mas_pop_node(mas); } else { - memset(&reuse, 0, sizeof(struct maple_node)); newnode = &reuse; } @@ -3255,7 +3255,21 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) dst_pivots[new_end] = mas->max; done: - mas_leaf_set_meta(newnode, maple_leaf_64, new_end); + if (!in_rcu && new_end + 2 < node_slots) { + unsigned char clear_from = new_end + 1; + + /* + * Note that the last slot is never cleared, since the metadata + * will be stored there or it has a value. + */ + memset(dst_slots + clear_from, 0, + sizeof(void __rcu *) * (node_slots - clear_from)); + if (clear_from < node_pivots) + memset(dst_pivots + clear_from, 0, + sizeof(unsigned long) * (node_pivots - clear_from)); + } + + mas_leaf_set_meta(newnode, wr_mas->type, new_end); if (in_rcu) { struct maple_enode *old_enode = mas->node; -- cgit v1.2.3 From a3b6c7806af820e77a47c4b92903cc49eff3e978 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:34 -0400 Subject: maple_tree: micro optimisation of mas_wr_store_type() Use three new local booleans instead of reading other structures. This has shown an increase of 0.62% on a 2048 entry tree of height 4. Link: https://lore.kernel.org/20260630190843.3563858-11-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 56812db8b0ef..e2c780a64c9c 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3737,6 +3737,9 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas) { struct ma_state *mas = wr_mas->mas; unsigned char new_end; + bool appending; + bool one_slot; + bool in_rcu; if (unlikely(mas_is_none(mas) || mas_is_ptr(mas))) return wr_store_root; @@ -3756,21 +3759,30 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas) return wr_new_root; new_end = mas_wr_new_end(wr_mas); + in_rcu = mt_in_rcu(mas->tree); + appending = mas->offset == mas->end; + one_slot = wr_mas->offset_end - mas->offset == 1; + /* Potential spanning rebalance collapsing a node */ if (new_end < mt_min_slots[wr_mas->type]) { if (!mte_is_root(mas->node)) return wr_rebalance; + if (!in_rcu) { + if (appending) + return wr_append; + else if (mas->end == new_end && one_slot) + return wr_slot_store; + } return wr_node_store; } if (new_end >= mt_slots[wr_mas->type]) return wr_split_store; - if (!mt_in_rcu(mas->tree) && (mas->offset == mas->end)) + if (!in_rcu && appending) return wr_append; - if ((new_end == mas->end) && (!mt_in_rcu(mas->tree) || - (wr_mas->offset_end - mas->offset == 1))) + if (new_end == mas->end && (!in_rcu || one_slot)) return wr_slot_store; return wr_node_store; -- cgit v1.2.3 From 52ae6211e83f980c6cb1738edfacd8bce062a94f Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:35 -0400 Subject: maple_tree: add bulk parent set helper Instead of calculating the parent pointer each time for a child, cache the majority of the parent pointer and only change the slot per child. Drop the mas_set_parent() function since the last user has been removed. Testing on a tree containing 2048 entries of height 4 had an increased gain of 3.51% on nodes tracking gaps. Link: https://lore.kernel.org/20260630190843.3563858-12-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 97 ++++++++++++++++++++++++-------------------------------- 1 file changed, 42 insertions(+), 55 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index e2c780a64c9c..c968e25bea0a 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -450,46 +450,6 @@ enum maple_type mas_parent_type(struct ma_state *mas, struct maple_enode *enode) return 0; } -/* - * mas_set_parent() - Set the parent node and encode the slot - * @mas: The maple state - * @enode: The encoded maple node. - * @parent: The encoded maple node that is the parent of @enode. - * @slot: The slot that @enode resides in @parent. - * - * Slot number is encoded in the enode->parent bit 3-6 or 2-6, depending on the - * parent type. - */ -static inline -void mas_set_parent(struct ma_state *mas, struct maple_enode *enode, - const struct maple_enode *parent, unsigned char slot) -{ - unsigned long val = (unsigned long)parent; - unsigned long shift; - unsigned long type; - enum maple_type p_type = mte_node_type(parent); - - MAS_BUG_ON(mas, p_type == maple_dense); - MAS_BUG_ON(mas, p_type == maple_leaf_64); - - switch (p_type) { - case maple_range_64: - case maple_arange_64: - shift = MAPLE_PARENT_SLOT_SHIFT; - type = MAPLE_PARENT_RANGE64; - break; - default: - case maple_dense: - case maple_leaf_64: - shift = type = 0; - break; - } - - val &= ~MAPLE_NODE_MASK; /* Clear all node metadata in parent */ - val |= (slot << shift) | type; - mte_to_node(enode)->parent = ma_parent_ptr(val); -} - /* * mte_parent_slot() - get the parent slot of @enode. * @enode: The encoded maple node. @@ -871,6 +831,42 @@ static inline void ma_set_meta_gap(struct maple_node *mn, enum maple_type mt, meta->gap = offset; } +/* + * mas_set_parent_slots() - Bulk operation to set many slot parent pointers + * @mas: The maple state + * @parent: The encoded maple node that is the parent of @enode. + * @slot: The slot that of the @enode. + * @start_slot: The offset into @slot + * @count: The number of slots to set (eg: exclusive) + */ +static inline +void mas_set_parent_slots(struct ma_state *mas, struct maple_enode *parent, + void __rcu **slots, unsigned char start_slot, unsigned char count) +{ + unsigned long val; + unsigned long shift; + unsigned long type; + enum maple_type p_type = mte_node_type(parent); + unsigned char i; + + MAS_BUG_ON(mas, p_type != maple_range_64 && + p_type != maple_arange_64); + + shift = MAPLE_PARENT_SLOT_SHIFT; + type = MAPLE_PARENT_RANGE64; + + val = (unsigned long)parent; + val &= ~MAPLE_NODE_MASK; + + for (i = 0; i < count; i++) { + unsigned long pval = val | ((start_slot + i) << shift) | type; + struct maple_enode *child; + + child = mt_slot_locked(mas->tree, slots, i); + mte_to_node(child)->parent = ma_parent_ptr(pval); + } +} + /* * mat_add() - Add a @dead_enode to the ma_topiary of a list of dead nodes. * @mat: the ma_topiary, a linked list of dead nodes. @@ -1609,14 +1605,10 @@ static inline void mas_adopt_children(struct ma_state *mas, struct maple_node *node = mte_to_node(parent); void __rcu **slots = ma_slots(node, type); unsigned long *pivots = ma_pivots(node, type); - struct maple_enode *child; - unsigned char offset; + unsigned char end; - offset = ma_data_end(node, type, pivots, mas->max); - do { - child = mas_slot_locked(mas, slots, offset); - mas_set_parent(mas, child, parent, offset); - } while (offset--); + end = ma_data_end(node, type, pivots, mas->max); + mas_set_parent_slots(mas, parent, slots, 0, end + 1); } /* @@ -1998,15 +1990,10 @@ unsigned long node_copy(struct ma_state *mas, struct maple_node *src, s_slots = ma_slots(src, s_mt) + start; s_pivots = ma_pivots(src, s_mt) + start; memcpy(d_slots, s_slots, size * sizeof(void __rcu *)); - if (!ma_is_leaf(d_mt) && s_mt == maple_copy) { - struct maple_enode *edst = mt_mk_node(dst, d_mt); - - for (int i = 0; i < size; i++) - mas_set_parent(mas, - mt_slot_locked(mas->tree, d_slots, i), - edst, d_start + i); - } + if (!ma_is_leaf(d_mt) && s_mt == maple_copy) + mas_set_parent_slots(mas, mt_mk_node(dst, d_mt), + d_slots, d_start, size); d_gaps = ma_gaps(dst, d_mt); if (d_gaps) { -- cgit v1.2.3 From 0da436138d7727bf170829781b187729ffb82476 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:36 -0400 Subject: maple_tree: catch race in mas_alloc_cyclic() If mas_alloc_cyclic() is called during a low memory situation, it is possible the lock may be dropped so reclaim can occur. There is a window where some other task may allocate the same id and cause the mas_insert() to fail with -EEXIST. In this scenario the function will return -EEXIST, which is not expected. Modifying the retry on mas_nomem() to re-search for a slot means that any race with other writes will not matter as the lock will be held between finding the index and writing the index. Moving the flag logic avoids cases where the flag is modified on drop lock/reacquire or when the write fails after clearing the flag. No existing users are exposed to this issue. Link: https://lore.kernel.org/20260630190843.3563858-13-liam@infradead.org Fixes: 9b6713cc7522 ("maple_tree: Add mtree_alloc_cyclic()") Signed-off-by: Liam R. Howlett (Oracle) Reported-by: Chris Mason Reviewed-by: Chuck Lever Cc: Boqun Feng Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 43 ++++++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 19 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index c968e25bea0a..190f480d6850 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3868,35 +3868,40 @@ int mas_alloc_cyclic(struct ma_state *mas, unsigned long *startp, void *entry, unsigned long range_lo, unsigned long range_hi, unsigned long *next, gfp_t gfp) { - unsigned long min = range_lo; - int ret = 0; - - range_lo = max(min, *next); - ret = mas_empty_area(mas, range_lo, range_hi, 1); - if ((mas->tree->ma_flags & MT_FLAGS_ALLOC_WRAPPED) && ret == 0) { - mas->tree->ma_flags &= ~MT_FLAGS_ALLOC_WRAPPED; - ret = 1; - } - if (ret < 0 && range_lo > min) { - mas_reset(mas); - ret = mas_empty_area(mas, min, range_hi, 1); - if (ret == 0) - ret = 1; - } - if (ret < 0) - return ret; + int ret; + unsigned long min; + min = range_lo; do { + range_lo = max(min, *next); + ret = mas_empty_area(mas, range_lo, range_hi, 1); + if (ret < 0 && range_lo > min) { + mas_reset(mas); + ret = mas_empty_area(mas, min, range_hi, 1); + if (ret == 0) + ret = 1; + } + if (ret < 0) + goto out; + mas_insert(mas, entry); } while (mas_nomem(mas, gfp)); - if (mas_is_err(mas)) - return xa_err(mas->node); + if (mas_is_err(mas)) { + ret = xa_err(mas->node); + goto out; + } + + if ((mas->tree->ma_flags & MT_FLAGS_ALLOC_WRAPPED) && ret == 0) { + mas->tree->ma_flags &= ~MT_FLAGS_ALLOC_WRAPPED; + ret = 1; + } *startp = mas->index; *next = *startp + 1; if (*next == 0) mas->tree->ma_flags |= MT_FLAGS_ALLOC_WRAPPED; +out: mas_destroy(mas); return ret; } -- cgit v1.2.3 From 1f24a2525a1dd7d5c5e98bddd247bf326a224706 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:37 -0400 Subject: maple_tree: document that erase may use GFP_KERNEL for allocations State that the mas_erase() and mtree_erase() functions may use GFP_KERNEL on allocation retry. Don't just depend on people reading the documentation by adding a check that will warn of the use. Link: https://lore.kernel.org/20260630190843.3563858-14-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Reviewed-by: Rik van Riel Cc: Jason Gunthorpe Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Joe Perches Cc: Peter Zijlstra Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 190f480d6850..440863bdea26 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5658,6 +5658,10 @@ EXPORT_SYMBOL_GPL(mas_find_range_rev); * Searches for @mas->index, sets @mas->index and @mas->last to the range and * erases that range. * + * Note that erase requires allocations and will use GFP_KERNEL to do so if + * necessary. If the allocation fails, the internal lock will be dropped to + * retry. + * * Return: the entry that was erased or %NULL, @mas->index and @mas->last are updated. */ void *mas_erase(struct ma_state *mas) @@ -5666,13 +5670,21 @@ void *mas_erase(struct ma_state *mas) unsigned long index = mas->index; MA_WR_STATE(wr_mas, mas, NULL); + /* + * In low memory situations, the allocation is retried with the gfp flag + * GFP_KERNEL. The internal spinlock is dropped in mas_nomem(), however + * the external lock is not dropped. + */ + if (mt_external_lock(mas->tree)) + might_alloc(GFP_KERNEL); + if (!mas_is_active(mas) || !mas_is_start(mas)) mas->status = ma_start; write_retry: entry = mas_state_walk(mas); if (!entry) - return NULL; + goto out; /* Must reset to ensure spanning writes of last slot are detected */ mas_reset(mas); @@ -5683,8 +5695,10 @@ write_retry: goto write_retry; } - if (mas_is_err(mas)) + if (mas_is_err(mas)) { + entry = NULL; goto out; + } mas_wr_store_entry(&wr_mas); out: @@ -6012,6 +6026,10 @@ EXPORT_SYMBOL(mtree_alloc_rrange); * Erasing is the same as a walk to an entry then a store of a NULL to that * ENTIRE range. In fact, it is implemented as such using the advanced API. * + * Note that erase requires allocations and will use GFP_KERNEL to do so if + * necessary. If the allocation fails, the internal lock will be dropped to + * retry. + * * Return: The entry stored at the @index or %NULL */ void *mtree_erase(struct maple_tree *mt, unsigned long index) @@ -6021,6 +6039,7 @@ void *mtree_erase(struct maple_tree *mt, unsigned long index) MA_STATE(mas, mt, index, index); trace_ma_op(TP_FCT, &mas); + might_alloc(GFP_KERNEL); mtree_lock(mt); entry = mas_erase(&mas); mtree_unlock(mt); -- cgit v1.2.3 From 1a2842ea6f8bad2ecfc5f6ace6d61dfdee1c6a9f Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:38 -0400 Subject: maple_tree: WARN_ON_ONCE when allocations fail Allocations should never fail in the circumstances that are expected to occur. Add checks in the code to ensure the circumstances are correctly set up by the user and warn if they are not. Also add a warning on failure to allocate, which should never happen. Link: https://lore.kernel.org/20260630190843.3563858-15-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Rik van Riel Cc: Jason Gunthorpe Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Joe Perches Cc: Peter Zijlstra Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 440863bdea26..054de396c873 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5721,6 +5721,10 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) if (likely(mas->node != MA_ERROR(-ENOMEM))) return false; + /* Allocations can fail, don't do this. */ + WARN_ON_ONCE(!gfpflags_allow_blocking(gfp) && + mt_external_lock(mas->tree)); + if (gfpflags_allow_blocking(gfp) && !mt_external_lock(mas->tree)) { mtree_unlock(mas->tree); mas_alloc_nodes(mas, gfp); @@ -5731,9 +5735,12 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) /* * Return false on zero forward progress. Partial allocations are kept - * so the retry path will attempt to get the rest. + * so the retry path will attempt to get the rest. The failure should + * not happen as we try our best to reclaim. The user would need an + * external lock with a non-blocking gfp in a low memory situation - + * which would have triggered the first warning in this function. */ - if (!mas->sheaf && !mas->alloc) + if (WARN_ON_ONCE(!mas->sheaf && !mas->alloc)) return false; mas_reset(mas); -- cgit v1.2.3 From 658205e0a5f8fffbae718f56b38220c96f41a1c1 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:42 -0400 Subject: maple_tree: avoid extra gap calculation Prior to ending the ascension loop of larger operations like split, rebalance, and spanning store the gap in the node had been calculated. Once the node is inserted into the tree, the gap is recalculated in mas_update_gap(). This can be avoided by creating a helper for mas_update_gap() that accepts the known gap value, which reduces the operations required for gap updating path. Link: https://lore.kernel.org/20260630190843.3563858-19-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 054de396c873..ccda926a67e3 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1566,14 +1566,26 @@ ascend: goto ascend; } +static __always_inline void mas_update_gap_known(struct ma_state *mas, + unsigned long gap) +{ + unsigned char pslot; + unsigned long p_gap; + + pslot = mte_parent_slot(mas->node); + p_gap = ma_gaps(mte_parent(mas->node), + mas_parent_type(mas, mas->node))[pslot]; + + if (p_gap != gap) + mas_parent_gap(mas, pslot, gap); +} + /* * mas_update_gap() - Update a nodes gaps and propagate up if necessary. * @mas: the maple state. */ static inline void mas_update_gap(struct ma_state *mas) { - unsigned char pslot; - unsigned long p_gap; unsigned long max_gap; if (!mt_is_alloc(mas->tree)) @@ -1583,13 +1595,7 @@ static inline void mas_update_gap(struct ma_state *mas) return; max_gap = mas_max_gap(mas); - - pslot = mte_parent_slot(mas->node); - p_gap = ma_gaps(mte_parent(mas->node), - mas_parent_type(mas, mas->node))[pslot]; - - if (p_gap != max_gap) - mas_parent_gap(mas, pslot, max_gap); + mas_update_gap_known(mas, max_gap); } /* @@ -2137,8 +2143,8 @@ static inline void mas_wmb_replace(struct ma_state *mas, struct maple_copy *cp) mas->node = mt_slot_locked(mas->tree, cp->slot, 0); /* Insert the new data in the tree */ mas_topiary_replace(mas, old_enode, cp->height); - if (!mte_is_leaf(mas->node)) - mas_update_gap(mas); + if (mt_is_alloc(mas->tree) && !mte_is_root(mas->node)) + mas_update_gap_known(mas, cp->gap[0]); mtree_range_walk(mas); } -- cgit v1.2.3 From efecab401cb15fd3bb9bc05990609acb6b267ff2 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Tue, 30 Jun 2026 15:08:43 -0400 Subject: maple_tree: add helper mas_make_walkable() A check in mas_walk() was incorrect and caused inefficient use of the maple state. The same issue existed in mas_erase(), but was left unfixed. Making a helper function is the obvious answer. Link: https://lore.kernel.org/20260630190843.3563858-20-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) (limited to 'lib') diff --git a/lib/maple_tree.c b/lib/maple_tree.c index ccda926a67e3..5ee1175af26e 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -261,6 +261,12 @@ static inline bool mas_is_underflow(struct ma_state *mas) return mas->status == ma_underflow; } +static inline void mas_make_walkable(struct ma_state *mas) +{ + if (!mas_is_active(mas) && !mas_is_start(mas)) + mas->status = ma_start; +} + static __always_inline struct maple_node *mte_to_node( const struct maple_enode *entry) { @@ -4447,8 +4453,7 @@ void *mas_walk(struct ma_state *mas) void *entry; mas_may_init_lock_check(mas); - if (!mas_is_active(mas) && !mas_is_start(mas)) - mas->status = ma_start; + mas_make_walkable(mas); retry: entry = mas_state_walk(mas); if (mas_is_start(mas)) { @@ -5684,9 +5689,7 @@ void *mas_erase(struct ma_state *mas) if (mt_external_lock(mas->tree)) might_alloc(GFP_KERNEL); - if (!mas_is_active(mas) || !mas_is_start(mas)) - mas->status = ma_start; - + mas_make_walkable(mas); write_retry: entry = mas_state_walk(mas); if (!entry) -- cgit v1.2.3