From 051f5d223dfc1806e216f60e4b29c1bf35f5c2d3 Mon Sep 17 00:00:00 2001 From: Ingo Molnar Date: Sun, 5 Jul 2026 11:05:17 +0200 Subject: lockdep: Enable the printing of held locks of remote running tasks and print task CPU MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Background: ========== Currently lockdep does not print out the held locks of non-current tasks that are running on some other CPU, due to the fact that the held locks array is in flux and may be unreliable to print. Syzkaller on the other hand found it that the analysis of locking bugs is easier if we print this information too, because the more locking information the merrier. In particular races are bound to have multiple tasks running on different CPUs, and the exclusion of their held locks information is unnecessarily limiting. So while it's still true that printing out their held locks array is racy, it's not as bad as it seems. There's 16 internal callers to lockdep_print_held_locks(): - 14 callers call it with the current task, which should be safe out of box. - 1 caller, debug_show_all_locks(), calls it with RCU held, which should guarantee that 'p' cannot go away under us. - 1 caller, debug_show_held_locks(), exposes the internal API with the constraint that it should only be called by drivers or platform code if the task isn't actively running - we can assume that if it nevertheless does, it will be Their Problem™. As for held locks being changed from under debug_show_held_locks(), while the task cannot go away, so the held-locks array itself is safe (although potentially non-stable), AFAICS the worst-case race can be garbage printed out by print_lock(), not any actual crashes. In particular: unsigned int class_idx = hlock->class_idx; may be stale (belong to a lock that already got released on another CPU), but it should still be a valid class index bound by MAX_LOCKDEP_KEYS, and thus the lock_classes_in_use bitmap use should be safe. The other two accesses are ::acquire_ip and ::instance: printk(KERN_CONT "%px", hlock->instance); print_lock_name(hlock, lock); printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip); But both are printed out as pointers, so no risk of dereference of a dangling pointer. We may print a garbage pointer. Also note that the check itself doesn't protect debug_show_held_locks() from printing garbage, as there's nothing that keeps a task from becoming runnable a nanosecond after we've run the task_is_running() check. In fact I'd argue that it's better to make this function *more* racy, for the simple robustness reason that we absolutely do not want it to crash even in the racy case. TL;DR: it should be fine to print the held locks of running tasks too, as long as we print out the information as well that a task is running, so that users are aware of any racy output. Implementation: ============== Implement that change. Also re-flow the function and streamline the printout into a single statement for all cases, which changes the 'no locks held by' / '%d lock[s] held by' phrasing that had a dependency on English spelling of plurals, to a uniform: locks held by bash/1234: %d Which spells correctly for 0, 1 and higher values, and should also be easier to parse both for humans and for scripts. Finally, print out the last CPU a task has ran on. This is very useful information for races and for locking bugs in particular. This basically extends the 'on CPU#%d' message we print for running tasks to all tasks we print. Reported-by: Tetsuo Handa Suggested-by: Tetsuo Handa Tested-by: Tetsuo Handa Signed-off-by: Ingo Molnar Cc: Boqun Feng Cc: Gary Guo Cc: Mark Brown Cc: Theodore Tso Cc: Miguel Ojeda Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Will Deacon Cc: Greg Kroah-Hartman Cc: Waiman Long Link: https://patch.msgid.link/akoeSIQGwqd9cZwd@gmail.com --- kernel/locking/lockdep.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) (limited to 'kernel/locking') diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index 2d4c5bab5af8..ff4bbf14665e 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -787,17 +787,33 @@ static void lockdep_print_held_locks(struct task_struct *p) { int i, depth = READ_ONCE(p->lockdep_depth); - if (!depth) - printk("no locks held by %s/%d.\n", p->comm, task_pid_nr(p)); - else - printk("%d lock%s held by %s/%d:\n", depth, - str_plural(depth), p->comm, task_pid_nr(p)); /* - * It's not reliable to print a task's held locks if it's not sleeping - * and it's not the current task. + * Note that it's always somewhat unreliable to print held locks + * of a task that is running on another CPU, but we cannot guarantee + * the stability of ->held_locks without actually stopping all active + * remote CPUs, which we absolutely do not want to do because it's + * very intrusive and thus slow. + * + * So we do the next best thing here: we print out the held lock + * array on a best-effort basis, without crashing even if the + * fields are being modified on another CPU. Note the careful + * construction of print_lock() so that it never crashes. + * + * We also print out the CPU the task is or was last running on, with + * the message saying 'on CPU...' if the task is running, and + * 'last CPU' if it's not. + * + * Also note that the task_is_running(p) information is fundamentally + * racy: even if the message says the task is 'on CPU', the task may + * have scheduled out already, or if it says 'last CPU', it may just + * have scheduled in on another CPU. But even with these limitations + * it's still useful debuggining information. */ - if (p != current && task_is_running(p)) - return; + printk("locks held by %s/%d: %d, %s CPU#%d%s\n", + p->comm, task_pid_nr(p), depth, + task_is_running(p) ? "last" : "on", task_cpu(p), + depth > 0 ? ":" : ""); + for (i = 0; i < depth; i++) { printk(" #%d: ", i); print_lock(p->held_locks + i); -- cgit v1.2.3 From 7577e00b9ab506202b9f1a33de3cc8cc6413a4db Mon Sep 17 00:00:00 2001 From: Naveen Kumar Chaudhary Date: Thu, 11 Jun 2026 23:08:17 +0530 Subject: locking/lockdep: Fix NULL pointer dereference in __lock_set_class() register_lock_class() can return NULL when the lock class pool is exhausted, graph_lock() fails, or key validation fails. However, __lock_set_class() uses the return value directly in pointer arithmetic without a NULL check: class = register_lock_class(lock, subclass, 0); hlock->class_idx = class - lock_classes; If class is NULL, this computes a wild offset that corrupts hlock->class_idx. The subsequent reacquire_held_locks() call will invoke hlock_class() with this corrupted index, leading to a NULL or out-of-bounds pointer dereference. Add the missing NULL check, consistent with how __lock_acquire() already handles this case at the same call site. Fixes: 64aa348edc61 ("lockdep: lock_set_subclass - reset a held lock's subclass") Signed-off-by: Naveen Kumar Chaudhary Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Waiman Long Reviewed-by: Dmitry Ilvokhin Link: https://patch.msgid.link/h2kfw43n4527x6mgi2lwpz2rieqnfzgictpv4wr5nyfjkc47co@2r5vz4uz44db --- kernel/locking/lockdep.c | 2 ++ 1 file changed, 2 insertions(+) (limited to 'kernel/locking') diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index ff4bbf14665e..25d77d4a1061 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -5453,6 +5453,8 @@ __lock_set_class(struct lockdep_map *lock, const char *name, lock->wait_type_outer, lock->lock_type); class = register_lock_class(lock, subclass, 0); + if (!class) + return 0; hlock->class_idx = class - lock_classes; curr->lockdep_depth = i; -- cgit v1.2.3 From c9c8578ad58e66a64ca887731e2b6ebf9d71174b Mon Sep 17 00:00:00 2001 From: Sun Shaojie Date: Tue, 23 Jun 2026 18:41:32 +0800 Subject: locking/percpu-rwsem: Annotate intentional data race in readers_active_check() KCSAN reports a data race between readers_active_check() and a concurrently executing reader: BUG: KCSAN: data-race in readers_active_check / percpu_down_write race at unknown origin, with read to 0xffff9f3eb5bf5f30 of 4 bytes by task 1271 on cpu 14: readers_active_check+0x... percpu_down_write+0x152/0x1f0 value changed: 0xfffffff9 -> 0xfffffff8 readers_active_check() calls per_cpu_sum(*sem->read_count), which iterates over all CPUs and reads each CPU's per-CPU read_count variable. Concurrently, a reader on a remote CPU is modifying its own CPU's read_count via this_cpu_inc() / this_cpu_dec() as it enters and exits the critical section. These are plain reads and writes to the same per-CPU storage, hence KCSAN flags a data race. This race is benign. readers_active_check() is called from the percpu_down_write() wait loop (rcuwait_wait_event) after sem->block is already set. At this point: - New readers must immediately back out (they see block set, decrement their counter, and wake the writer), so counters can only decrease. - If the sum catches a reader's increment before its decrement, readers_active_check() sees a non-zero sum and returns false. The writer merely iterates the wait loop again -- a harmless retry. - A false zero (observing sum == 0 while a reader is still active) cannot happen: per_cpu_sum() reads each CPU's counter, and each per-CPU int read is atomic on all architectures, so an active reader's counter is always seen as non-zero. Annotate the read with data_race() to suppress the KCSAN warning and document the intentional nature of this unlocked access. Signed-off-by: Sun Shaojie Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/20260623104132.505117-1-sunshaojie@kylinos.cn --- kernel/locking/percpu-rwsem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'kernel/locking') diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c index f7e152c40d6d..6c78961fe753 100644 --- a/kernel/locking/percpu-rwsem.c +++ b/kernel/locking/percpu-rwsem.c @@ -211,7 +211,7 @@ EXPORT_SYMBOL_GPL(percpu_is_read_locked); */ static bool readers_active_check(struct percpu_rw_semaphore *sem) { - if (per_cpu_sum(*sem->read_count) != 0) + if (data_race(per_cpu_sum(*sem->read_count)) != 0) return false; /* -- cgit v1.2.3 From f7e2cb6d495aa1a88368650970a230b45870859b Mon Sep 17 00:00:00 2001 From: Dmitry Ilvokhin Date: Tue, 4 Aug 2026 07:15:43 +0000 Subject: locking/qspinlock: Add contended_release tracepoint Unlike mutex and rw_semaphore, qspinlock has no owner field, so "perf lock contention --lock-owner" cannot attribute a contended spinlock to its holder. The waiter-side contention_begin event records that a spinlock is contended, but not by whom. Firing contended_release in the holder's context at unlock is the only way to capture the holder of a contended spinlock. Combine the contention check, trace call and release in an out-of-line queued_spin_release_traced() so the compiler need not preserve the lock pointer in a callee-saved register across the call. The check in queued_spin_unlock() is paid on every unlock, even while the tracepoint is disabled: a static-branch NOP on x86_64, and a few more instructions to manage a stack frame elsewhere. Gate it behind CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE (default n) so nobody pays for a tracepoint they do not use. Sleeping locks fire contended_release regardless. On x86 this generic path is used only with PARAVIRT_SPINLOCKS=n (e.g. defconfig). PARAVIRT_SPINLOCKS=y kernels keep the paravirt static_call unlock and are wired up separately. All below are with the QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE option enabled. _raw_spin_unlock(), x86_64 defconfig, GCC 11, tracepoint compiled in but disabled. The unlock is the single 'movb'. The only instruction added to the executed path is the 2-byte static-branch NOP. The CALL to the traced helper and the JMP back are emitted out of line and are reached only once the static branch is patched on: endbr64 ; 4 bytes xchg %ax,%ax ; 2 static-branch NOP ; (added) movb $0x0,(%rdi) ; 3 unlock (single store) A: decl %gs:__preempt_count ; 7 je B ; 2 jmp __x86_return_thunk ; 5 call queued_spin_release_traced ; 5 out of line, reached ; only when the ; tracepoint is on jmp A ; 2 (added) B: call __SCT__preempt_schedule ; 5 jmp __x86_return_thunk ; 5 Baseline is the same stream without the NOP and the out-of-line CALL/JMP: 31 bytes vs 40 (+9 bytes). Binary size impact on x86_64, defconfig: +680 bytes (+0.00%), since all standard configs out-of-line unlock. Architectures with inlined unlock (s390 (always), csky and loongarch (both when !PREEMPTION)) will see a bigger increase in binary size. On the same path (x86_64, PARAVIRT_SPINLOCKS=n) with the tracepoint disabled, a _raw_spin_unlock()-heavy nginx workload [1] shows no measurable difference between baseline and patched kernels in throughput, latency, cycles, instructions, IPC, or L1 instruction-cache misses (kernel and total): all deltas stay within run-to-run noise. Unlike x86, on arm64 the frame setup code (STP, MOV and LDP) lands on the executed path in addition to static-branch NOP. Binary size impact on arm64, defconfig: +932 bytes (+0.00%). The _raw_spin_unlock()-heavy nginx workload reflects the larger hot path: L1 instruction-cache misses rise ~1.4% (kernel and total) and instruction count ~0.4%, consistent with the per-unlock frame. cpu_cycles, throughput and latency show no measurable change and are within run-to-run noise. Architectures with fully custom qspinlock implementations (e.g. PowerPC) are not covered by this change. [1]: https://lore.kernel.org/all/aiphFXe_TPNPxZ_n@shell.ilvokhin.com/ Signed-off-by: Dmitry Ilvokhin Signed-off-by: Peter Zijlstra (Intel) Acked-by: Juergen Gross Link: https://patch.msgid.link/0d998e22a0c595f670cfc6725bb683323aced5cb.1785778551.git.d@ilvokhin.com --- include/asm-generic/qspinlock.h | 21 +++++++++++++++++++++ kernel/Kconfig.locks | 20 ++++++++++++++++++++ kernel/locking/qspinlock.c | 22 ++++++++++++++++++++++ 3 files changed, 63 insertions(+) (limited to 'kernel/locking') diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h index ae45289e8ec7..2ca94e41823b 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -41,6 +41,7 @@ #include #include +#include #ifndef queued_spin_is_locked /** @@ -130,12 +131,32 @@ static __always_inline void queued_spin_release(struct qspinlock *lock) #endif #ifndef queued_spin_unlock + +DECLARE_TRACEPOINT(contended_release); + +extern void queued_spin_release_traced(struct qspinlock *lock); + /** * queued_spin_unlock - unlock a queued spinlock * @lock : Pointer to queued spinlock structure + * + * Generic tracing wrapper around the arch-overridable + * queued_spin_release(). */ static __always_inline void queued_spin_unlock(struct qspinlock *lock) { + /* + * Trace and release are combined in queued_spin_release_traced() so + * the compiler does not need to preserve the lock pointer across the + * function call, avoiding callee-saved register save/restore on the + * hot path. queued_spin_release() is therefore called both here and in + * queued_spin_release_traced(). Keep the two in sync. + */ + if (IS_ENABLED(CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE) && + tracepoint_enabled(contended_release)) { + queued_spin_release_traced(lock); + return; + } queued_spin_release(lock); } #endif diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks index 4198f0273ecd..1c6423aafcd4 100644 --- a/kernel/Kconfig.locks +++ b/kernel/Kconfig.locks @@ -243,6 +243,26 @@ config QUEUED_SPINLOCKS def_bool y if ARCH_USE_QUEUED_SPINLOCKS depends on SMP +config QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE + bool "Trace contended_release on queued spinlocks" + depends on QUEUED_SPINLOCKS && TRACEPOINTS + help + Fire the lock:contended_release tracepoint when a contended queued + spinlock is released, so it is possible to attribute a contended + spinlock to its holder. + + Architectures that can patch the unlock site do this at no cost and + do not need this option. + + Everywhere else the check is compiled into queued_spin_unlock() and + a small cost is paid on every unlock even when the tracepoint is + disabled: a static-branch NOP and possibly a few more instructions + to manage a stack frame. + + Sleeping locks fire lock:contended_release regardless of this option. + + If unsure, say N. + config BPF_ARCH_SPINLOCK bool diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index af8d122bb649..33fe6d437c8f 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -104,6 +104,28 @@ static __always_inline u32 __pv_wait_head_or_lock(struct qspinlock *lock, #define queued_spin_lock_slowpath native_queued_spin_lock_slowpath #endif +#if !defined(queued_spin_unlock) && \ + IS_ENABLED(CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE) +/* + * Out-of-line trace-and-release path for queued_spin_unlock(), used when + * the contended_release tracepoint is enabled. + * + * queued_spin_release() is duplicated here on purpose: doing the release + * in this function (rather than tracing here and releasing in the caller) + * lets queued_spin_unlock() return right after the call, so the + * tracepoint-disabled hot path never has to keep lock live across a call + * in a callee-saved register. Keep this release in sync with the one in + * queued_spin_unlock(). + */ +void __lockfunc queued_spin_release_traced(struct qspinlock *lock) +{ + if (queued_spin_is_contended(lock)) + trace_call__contended_release(lock); + queued_spin_release(lock); +} +EXPORT_SYMBOL(queued_spin_release_traced); +#endif + #endif /* _GEN_PV_LOCK_SLOWPATH */ /** -- cgit v1.2.3 From b359800c6970cb653d41ed2af18fd8e95dbb822f Mon Sep 17 00:00:00 2001 From: Dmitry Ilvokhin Date: Tue, 4 Aug 2026 07:15:44 +0000 Subject: tracing/lock: Use TRACE_EVENT_FN() for contended_release queued_spin_unlock() gates its contended_release trace call behind a static branch, so a NOP sits on the unlock path even while the tracepoint is disabled. Removing that requires replacing the unlock implementation only while contended_release is enabled, which needs a callback when the tracepoint is toggled. Convert contended_release to TRACE_EVENT_FN() and add weak no-op arch_contended_release_trace_reg()/arch_contended_release_trace_unreg() hooks. The default hooks are empty, so this is a no-op until an architecture overrides them. No functional change intended. Signed-off-by: Dmitry Ilvokhin Signed-off-by: Peter Zijlstra (Intel) Acked-by: Juergen Gross Link: https://patch.msgid.link/1c2fcccfb584c075c02890c484f22c76a1948bf1.1785778551.git.d@ilvokhin.com --- include/trace/events/lock.h | 10 ++++++++-- kernel/locking/mutex.c | 4 ++++ 2 files changed, 12 insertions(+), 2 deletions(-) (limited to 'kernel/locking') diff --git a/include/trace/events/lock.h b/include/trace/events/lock.h index 1ded869cd619..b1d5b18c4514 100644 --- a/include/trace/events/lock.h +++ b/include/trace/events/lock.h @@ -137,7 +137,11 @@ TRACE_EVENT(contention_end, TP_printk("%p (ret=%d)", __entry->lock_addr, __entry->ret) ); -TRACE_EVENT(contended_release, +/* kernel/locking/mutex.c */ +int arch_contended_release_trace_reg(void); +void arch_contended_release_trace_unreg(void); + +TRACE_EVENT_FN(contended_release, TP_PROTO(void *lock), @@ -151,7 +155,9 @@ TRACE_EVENT(contended_release, __entry->lock_addr = lock; ), - TP_printk("%p", __entry->lock_addr) + TP_printk("%p", __entry->lock_addr), + + arch_contended_release_trace_reg, arch_contended_release_trace_unreg ); #endif /* _TRACE_LOCK_H */ diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 8a85912d7ee6..942a939cee95 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -1272,6 +1272,10 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(contention_begin); EXPORT_TRACEPOINT_SYMBOL_GPL(contention_end); EXPORT_TRACEPOINT_SYMBOL_GPL(contended_release); +__weak int arch_contended_release_trace_reg(void) { return 0; } + +__weak void arch_contended_release_trace_unreg(void) { } + /** * atomic_dec_and_mutex_lock - return holding mutex if we dec to 0 * @cnt: the atomic which we are to dec -- cgit v1.2.3 From e901c1510e24726dcbd6340ee927b3ac8b992043 Mon Sep 17 00:00:00 2001 From: Boqun Feng Date: Tue, 4 Aug 2026 11:26:57 -0700 Subject: irq,spin_lock: Add counted interrupt disabling/enabling Currently the nested interrupt disabling and enabling is represented by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example: spin_lock_irqsave(l1, flag1); spin_lock_irqsave(l2, flag2); spin_unlock_irqrestore(l1, flags1); // accesses to interrupt-disable protected data will cause races This is even easier to trigger with guard facilities: unsigned long flag2; scoped_guard(spin_lock_irqsave, l1) { spin_lock_irqsave(l2, flag2); } // l2 locked but interrupts are enabled. spin_unlock_irqrestore(l2, flag2); (Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design) And because of this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design. To resolve this, introduce a new set of interrupt disabling APIs: * local_interrupt_disable(); * local_interrupt_enable(); They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call saves the interrupt state into a per-CPU variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a per-CPU counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section. Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follows: spin_lock_irq_disable(l1); spin_lock_irq_disable(l2); spin_unlock_irq_enable(l1); // Interrupts are still disabled. spin_unlock_irq_enable(l2); doesn't have the issue that interrupts are accidentally enabled. This also makes the wrapper of interrupt-disabling locks on Rust easier to design. [boqun: Apply Peter's feedback and fix spell errors reported by Ingo] [boqun: Address the duplicate spin_acquire() spotted by sashiko] Co-developed-by: Lyude Paul Signed-off-by: Lyude Paul Signed-off-by: Boqun Feng Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/20260804182657.87716-1-boqun@kernel.org --- include/linux/interrupt_rc.h | 82 ++++++++++++++++++++++++++++++++++++++++ include/linux/preempt.h | 4 ++ include/linux/spinlock.h | 23 +++++++++++ include/linux/spinlock_api_smp.h | 41 ++++++++++++++++++++ include/linux/spinlock_api_up.h | 15 ++++++++ include/linux/spinlock_rt.h | 18 +++++++++ kernel/locking/spinlock.c | 31 +++++++++++++++ kernel/softirq.c | 28 +++++++++++++- 8 files changed, 240 insertions(+), 2 deletions(-) create mode 100644 include/linux/interrupt_rc.h (limited to 'kernel/locking') diff --git a/include/linux/interrupt_rc.h b/include/linux/interrupt_rc.h new file mode 100644 index 000000000000..b9a7f05ecf42 --- /dev/null +++ b/include/linux/interrupt_rc.h @@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __LINUX_INTERRUPT_RC_H +#define __LINUX_INTERRUPT_RC_H + +/* + * include/linux/interrupt_rc.h - refcounted local processor interrupt + * management. + * + * Since the implementation of this API currently depends on + * local_irq_save()/local_irq_restore(), we split this into its own header to + * make it easier to include without hitting circular header dependencies. + */ + +#include +#include +#include +#include + +#ifndef MODULE +/* Per-CPU interrupt disabling state for local_interrupt_{disable,enable}(). */ +DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state); + +static __always_inline void __local_interrupt_disable(void) +{ + unsigned long flags; + + local_irq_save(flags); + raw_cpu_write(local_interrupt_disable_state, flags); +} + +static __always_inline void __local_interrupt_enable(void) +{ + unsigned long flags = raw_cpu_read(local_interrupt_disable_state); + + local_irq_restore(flags); +} + +#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE +static __always_inline void _local_interrupt_disable(void) +{ + __local_interrupt_disable(); +} + +static __always_inline void _local_interrupt_enable(void) +{ + __local_interrupt_enable(); +} +#else +extern void _local_interrupt_disable(void); +extern void _local_interrupt_enable(void); +#endif + +#else /* !MODULE */ +extern void _local_interrupt_disable(void); +extern void _local_interrupt_enable(void); +#endif /* !MODULE */ + +static inline void local_interrupt_disable(void) +{ + int new_count; + + WARN_ON_ONCE(in_nmi()); + + new_count = hardirq_disable_enter(); + + /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ + + if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) + _local_interrupt_disable(); +} + +static inline void local_interrupt_enable(void) +{ + int new_count; + + new_count = hardirq_disable_exit(); + + if ((new_count & HARDIRQ_DISABLE_MASK) == 0) + _local_interrupt_enable(); +} + +#endif /* !__LINUX_INTERRUPT_RC_H */ diff --git a/include/linux/preempt.h b/include/linux/preempt.h index e2d3079d3f5f..33fc4c814a9f 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -151,6 +151,10 @@ static __always_inline unsigned char interrupt_context_level(void) #define in_softirq() (softirq_count()) #define in_interrupt() (irq_count()) +#define hardirq_disable_count() ((preempt_count() & HARDIRQ_DISABLE_MASK) >> HARDIRQ_DISABLE_SHIFT) +#define hardirq_disable_enter() __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET) +#define hardirq_disable_exit() __preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) + /* * The preempt_count offset after preempt_disable(); */ diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h index 241277cd34cf..3d405cc4c121 100644 --- a/include/linux/spinlock.h +++ b/include/linux/spinlock.h @@ -57,6 +57,7 @@ #include #include #include +#include #include #include #include @@ -273,9 +274,11 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock) #endif #define raw_spin_lock_irq(lock) _raw_spin_lock_irq(lock) +#define raw_spin_lock_irq_disable(lock) _raw_spin_lock_irq_disable(lock) #define raw_spin_lock_bh(lock) _raw_spin_lock_bh(lock) #define raw_spin_unlock(lock) _raw_spin_unlock(lock) #define raw_spin_unlock_irq(lock) _raw_spin_unlock_irq(lock) +#define raw_spin_unlock_irq_enable(lock) _raw_spin_unlock_irq_enable(lock) #define raw_spin_unlock_irqrestore(lock, flags) \ do { \ @@ -290,6 +293,8 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock) #define raw_spin_trylock_irqsave(lock, flags) _raw_spin_trylock_irqsave(lock, &(flags)) +#define raw_spin_trylock_irq_disable(lock) _raw_spin_trylock_irq_disable(lock) + #ifndef CONFIG_PREEMPT_RT /* Include rwlock functions for !RT */ #include @@ -372,6 +377,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock) raw_spin_lock_irq(&lock->rlock); } +static __always_inline void spin_lock_irq_disable(spinlock_t *lock) + __acquires(lock) __no_context_analysis +{ + raw_spin_lock_irq_disable(&lock->rlock); +} + #define spin_lock_irqsave(lock, flags) \ do { \ raw_spin_lock_irqsave(spinlock_check(lock), flags); \ @@ -402,6 +413,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock) raw_spin_unlock_irq(&lock->rlock); } +static __always_inline void spin_unlock_irq_enable(spinlock_t *lock) + __releases(lock) __no_context_analysis +{ + raw_spin_unlock_irq_enable(&lock->rlock); +} + static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags) __releases(lock) __no_context_analysis { @@ -427,6 +444,12 @@ static __always_inline bool _spin_trylock_irqsave(spinlock_t *lock, unsigned lon } #define spin_trylock_irqsave(lock, flags) _spin_trylock_irqsave(lock, &(flags)) +static __always_inline int spin_trylock_irq_disable(spinlock_t *lock) + __cond_acquires(true, lock) __no_context_analysis +{ + return raw_spin_trylock_irq_disable(&lock->rlock); +} + /** * spin_is_locked() - Check whether a spinlock is locked. * @lock: Pointer to the spinlock. diff --git a/include/linux/spinlock_api_smp.h b/include/linux/spinlock_api_smp.h index bda5e7a390cd..90909d933dab 100644 --- a/include/linux/spinlock_api_smp.h +++ b/include/linux/spinlock_api_smp.h @@ -28,6 +28,8 @@ _raw_spin_lock_nest_lock(raw_spinlock_t *lock, struct lockdep_map *map) void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock) __acquires(lock); void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock) __acquires(lock); +void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock) + __acquires(lock); unsigned long __lockfunc _raw_spin_lock_irqsave(raw_spinlock_t *lock) __acquires(lock); @@ -39,6 +41,7 @@ int __lockfunc _raw_spin_trylock_bh(raw_spinlock_t *lock) __cond_acquires(true, void __lockfunc _raw_spin_unlock(raw_spinlock_t *lock) __releases(lock); void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock) __releases(lock); void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock) __releases(lock); +void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock) __releases(lock); void __lockfunc _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags) __releases(lock); @@ -55,6 +58,11 @@ _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags) #define _raw_spin_lock_irq(lock) __raw_spin_lock_irq(lock) #endif +/* Use the same config as spin_lock_irq() temporarily. */ +#ifdef CONFIG_INLINE_SPIN_LOCK_IRQ +#define _raw_spin_lock_irq_disable(lock) __raw_spin_lock_irq_disable(lock) +#endif + #ifdef CONFIG_INLINE_SPIN_LOCK_IRQSAVE #define _raw_spin_lock_irqsave(lock) __raw_spin_lock_irqsave(lock) #endif @@ -79,6 +87,11 @@ _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags) #define _raw_spin_unlock_irq(lock) __raw_spin_unlock_irq(lock) #endif +/* Use the same config as spin_unlock_irq() temporarily. */ +#ifdef CONFIG_INLINE_SPIN_UNLOCK_IRQ +#define _raw_spin_unlock_irq_enable(lock) __raw_spin_unlock_irq_enable(lock) +#endif + #ifdef CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE #define _raw_spin_unlock_irqrestore(lock, flags) __raw_spin_unlock_irqrestore(lock, flags) #endif @@ -105,6 +118,16 @@ static __always_inline bool _raw_spin_trylock_irq(raw_spinlock_t *lock) return false; } +static __always_inline bool _raw_spin_trylock_irq_disable(raw_spinlock_t *lock) + __cond_acquires(true, lock) +{ + local_interrupt_disable(); + if (_raw_spin_trylock(lock)) + return true; + local_interrupt_enable(); + return false; +} + static __always_inline bool _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags) __cond_acquires(true, lock) { @@ -143,6 +166,15 @@ static inline void __raw_spin_lock_irq(raw_spinlock_t *lock) LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock); } +static inline void __raw_spin_lock_irq_disable(raw_spinlock_t *lock) + __acquires(lock) __no_context_analysis +{ + local_interrupt_disable(); + preempt_disable(); + spin_acquire(&lock->dep_map, 0, 0, _RET_IP_); + LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock); +} + static inline void __raw_spin_lock_bh(raw_spinlock_t *lock) __acquires(lock) __no_context_analysis { @@ -188,6 +220,15 @@ static inline void __raw_spin_unlock_irq(raw_spinlock_t *lock) preempt_enable(); } +static inline void __raw_spin_unlock_irq_enable(raw_spinlock_t *lock) + __releases(lock) +{ + spin_release(&lock->dep_map, _RET_IP_); + do_raw_spin_unlock(lock); + local_interrupt_enable(); + preempt_enable(); +} + static inline void __raw_spin_unlock_bh(raw_spinlock_t *lock) __releases(lock) { diff --git a/include/linux/spinlock_api_up.h b/include/linux/spinlock_api_up.h index a9d5c7c66e03..d03d3065ee04 100644 --- a/include/linux/spinlock_api_up.h +++ b/include/linux/spinlock_api_up.h @@ -42,6 +42,9 @@ #define __LOCK_IRQSAVE(lock, flags, ...) \ do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0) +#define __LOCK_IRQ_DISABLE(lock, ...) \ + do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0) + #define ___UNLOCK_(lock) \ do { __release(lock); (void)(lock); } while (0) @@ -61,6 +64,9 @@ #define __UNLOCK_IRQRESTORE(lock, flags, ...) \ do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0) +#define __UNLOCK_IRQ_ENABLE(lock, ...) \ + do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0) + #define _raw_spin_lock(lock) __LOCK(lock) #define _raw_spin_lock_nested(lock, subclass) __LOCK(lock) #define _raw_read_lock(lock) __LOCK(lock, shared) @@ -70,6 +76,7 @@ #define _raw_read_lock_bh(lock) __LOCK_BH(lock, shared) #define _raw_write_lock_bh(lock) __LOCK_BH(lock) #define _raw_spin_lock_irq(lock) __LOCK_IRQ(lock) +#define _raw_spin_lock_irq_disable(lock) __LOCK_IRQ_DISABLE(lock) #define _raw_read_lock_irq(lock) __LOCK_IRQ(lock, shared) #define _raw_write_lock_irq(lock) __LOCK_IRQ(lock) #define _raw_spin_lock_irqsave(lock, flags) __LOCK_IRQSAVE(lock, flags) @@ -97,6 +104,13 @@ static __always_inline int _raw_spin_trylock_irq(raw_spinlock_t *lock) return 1; } +static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock) + __cond_acquires(true, lock) +{ + __LOCK_IRQ_DISABLE(lock); + return 1; +} + static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags) __cond_acquires(true, lock) { @@ -132,6 +146,7 @@ static __always_inline int _raw_write_trylock_irqsave(rwlock_t *lock, unsigned l #define _raw_write_unlock_bh(lock) __UNLOCK_BH(lock) #define _raw_read_unlock_bh(lock) __UNLOCK_BH(lock, shared) #define _raw_spin_unlock_irq(lock) __UNLOCK_IRQ(lock) +#define _raw_spin_unlock_irq_enable(lock) __UNLOCK_IRQ_ENABLE(lock) #define _raw_read_unlock_irq(lock) __UNLOCK_IRQ(lock, shared) #define _raw_write_unlock_irq(lock) __UNLOCK_IRQ(lock) #define _raw_spin_unlock_irqrestore(lock, flags) \ diff --git a/include/linux/spinlock_rt.h b/include/linux/spinlock_rt.h index 373618a4243c..560d06384e0c 100644 --- a/include/linux/spinlock_rt.h +++ b/include/linux/spinlock_rt.h @@ -96,6 +96,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock) rt_spin_lock(lock); } +static __always_inline void spin_lock_irq_disable(spinlock_t *lock) + __acquires(lock) +{ + rt_spin_lock(lock); +} + #define spin_lock_irqsave(lock, flags) \ do { \ typecheck(unsigned long, flags); \ @@ -122,6 +128,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock) rt_spin_unlock(lock); } +static __always_inline void spin_unlock_irq_enable(spinlock_t *lock) + __releases(lock) +{ + rt_spin_unlock(lock); +} + static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags) __releases(lock) @@ -131,6 +143,12 @@ static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, #define spin_trylock(lock) rt_spin_trylock(lock) +static __always_inline int spin_trylock_irq_disable(spinlock_t *lock) + __cond_acquires(true, lock) +{ + return rt_spin_trylock(lock); +} + #define spin_trylock_bh(lock) rt_spin_trylock_bh(lock) #define spin_trylock_irq(lock) rt_spin_trylock(lock) diff --git a/kernel/locking/spinlock.c b/kernel/locking/spinlock.c index b42d293da38b..83a17eaf5717 100644 --- a/kernel/locking/spinlock.c +++ b/kernel/locking/spinlock.c @@ -129,6 +129,21 @@ static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock) \ */ BUILD_LOCK_OPS(spin, raw_spinlock, __acquires); +/* No rwlock_t variants for now, so just build this function by hand */ +static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock) +{ + for (;;) { + preempt_disable(); + local_interrupt_disable(); + if (likely(do_raw_spin_trylock(lock))) + break; + local_interrupt_enable(); + preempt_enable(); + + arch_spin_relax(&lock->raw_lock); + } +} + #ifndef CONFIG_PREEMPT_RT BUILD_LOCK_OPS(read, rwlock, __acquires_shared); BUILD_LOCK_OPS(write, rwlock, __acquires); @@ -176,6 +191,14 @@ noinline void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock) EXPORT_SYMBOL(_raw_spin_lock_irq); #endif +#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ +noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock) +{ + __raw_spin_lock_irq_disable(lock); +} +EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable); +#endif + #ifndef CONFIG_INLINE_SPIN_LOCK_BH noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock) { @@ -208,6 +231,14 @@ noinline void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock) EXPORT_SYMBOL(_raw_spin_unlock_irq); #endif +#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ +noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock) +{ + __raw_spin_unlock_irq_enable(lock); +} +EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable); +#endif + #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock) { diff --git a/kernel/softirq.c b/kernel/softirq.c index 10af5ed859e7..0c9b2269a8d6 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -9,6 +9,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE #include #include #include @@ -88,6 +89,20 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context); #endif +DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state); + +void _local_interrupt_disable(void) +{ + __local_interrupt_disable(); +} +EXPORT_SYMBOL(_local_interrupt_disable); + +void _local_interrupt_enable(void) +{ + __local_interrupt_enable(); +} +EXPORT_SYMBOL(_local_interrupt_enable); + DEFINE_PER_CPU(unsigned int, nmi_nesting); /* @@ -728,10 +743,19 @@ static inline void __irq_exit_rcu(void) #endif account_hardirq_exit(current); preempt_count_sub(HARDIRQ_OFFSET); - if (!in_interrupt() && local_softirq_pending()) { + /* + * Interrupts may happen between hardirq_disable_enter() and + * local_irq_save() in local_interrupt_disable(), if irq_exit() invokes + * softirq here, we may have a softirq handler calling + * local_interrupt_disable() but it won't disable the IRQ because + * hardirq disabling count is already 1, hence we need to prevent + * invoking softirq when a local_interrupt_disable() is ongoing. + */ + if (!in_interrupt() && !hardirq_disable_count() && + local_softirq_pending()) { /* * If we left hrtimers unarmed, make sure to arm them now, - * before enabling interrupts to run SoftIRQ. + * before enabling interrupts to run softirq. */ hrtimer_rearm_deferred(); invoke_softirq(); -- cgit v1.2.3 From 8f2109843137da8068f19a120d79e58ef3838429 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:10 -0400 Subject: locking/lockdep: add sequence counter to held_lock Add an 8 bit small sequence counter to the held_lock struct to detect if the lock as been dropped and reacquired. This is useful when a data structure depends on a constant locking context, but is not able to detect locking and unlocking of the lock through its own API. Since the __lock_unpin_lock() will no longer detect underflow by casting the unsigned int to a signed int, update the casting code to use a temp variable for calculations using a signed int. Link: https://lore.kernel.org/20260821192627.4085470-3-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Suggested-by: Peter Zijlstra Cc: Ingo Molnar Cc: Will Deacon Cc: Boqun Feng Cc: Waiman Long Link: https://lore.kernel.org/all/h3tpnj5kzcrxms5picmimtkpg4aypcpip5wbd6bt2rpdj5k7eb@nhtzs3lefrkq/ Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Jason Gunthorpe Cc: Joe Perches Cc: Rik van Riel Signed-off-by: Andrew Morton --- include/linux/lockdep.h | 3 +++ include/linux/lockdep_types.h | 3 ++- include/linux/sched.h | 1 + kernel/locking/lockdep.c | 58 +++++++++++++++++++++++++++++++++++++------ 4 files changed, 57 insertions(+), 8 deletions(-) (limited to 'kernel/locking') diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index 621566345406..a6451ecbbe9a 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -273,6 +273,9 @@ extern struct pin_cookie lock_pin_lock(struct lockdep_map *lock); extern void lock_repin_lock(struct lockdep_map *lock, struct pin_cookie); extern void lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie); +extern u32 lock_sequence(struct lockdep_map *lock); +#define lockdep_sequence(lock) lock_sequence(&(lock)->dep_map) + #define lockdep_depth(tsk) (debug_locks ? (tsk)->lockdep_depth : 0) #define lockdep_assert(cond) \ diff --git a/include/linux/lockdep_types.h b/include/linux/lockdep_types.h index eae115a26488..55c4b152fedf 100644 --- a/include/linux/lockdep_types.h +++ b/include/linux/lockdep_types.h @@ -253,7 +253,8 @@ struct held_lock { unsigned int hardirqs_off:1; unsigned int sync:1; unsigned int references:11; /* 32 bits */ - unsigned int pin_count; + unsigned int pin_count:24; + unsigned int seq_count:8; }; #else /* !CONFIG_LOCKDEP */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d1..14d5ce8dd613 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1288,6 +1288,7 @@ struct task_struct { u64 curr_chain_key; int lockdep_depth; unsigned int lockdep_recursion; + unsigned int lockdep_seq; struct held_lock held_locks[MAX_LOCK_DEPTH]; #endif diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index 2d4c5bab5af8..a69567bdd791 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -5077,7 +5077,7 @@ static int __lock_is_held(const struct lockdep_map *lock, int read); static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass, int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, - int references, int pin_count, int sync) + int references, int pin_count, int sync, int seq) { struct task_struct *curr = current; struct lock_class *class = NULL; @@ -5183,6 +5183,7 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass, hlock->holdtime_stamp = lockstat_clock(); #endif hlock->pin_count = pin_count; + hlock->seq_count = seq; if (check_wait_context(curr, hlock)) return 0; @@ -5388,7 +5389,7 @@ static int reacquire_held_locks(struct task_struct *curr, unsigned int depth, hlock->read, hlock->check, hlock->hardirqs_off, hlock->nest_lock, hlock->acquire_ip, - hlock->references, hlock->pin_count, 0)) { + hlock->references, hlock->pin_count, 0, hlock->seq_count)) { case 0: return 1; case 1: @@ -5669,14 +5670,17 @@ static void __lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie struct held_lock *hlock = curr->held_locks + i; if (match_held_lock(hlock, lock)) { + int pin_count; + if (WARN(!hlock->pin_count, "unpinning an unpinned lock\n")) return; - hlock->pin_count -= cookie.val; + pin_count = hlock->pin_count - cookie.val; - if (WARN((int)hlock->pin_count < 0, "pin count corrupted\n")) - hlock->pin_count = 0; + if (WARN(pin_count < 0, "pin count corrupted\n")) + pin_count = 0; + hlock->pin_count = pin_count; return; } } @@ -5684,6 +5688,24 @@ static void __lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie WARN(1, "unpinning an unheld lock\n"); } +static u32 __lock_sequence(struct lockdep_map *lock) +{ + struct task_struct *curr = current; + int i; + + if (unlikely(!debug_locks)) + return ~0; + + for (i = 0; i < curr->lockdep_depth; i++) { + struct held_lock *hlock = curr->held_locks + i; + + if (match_held_lock(hlock, lock)) + return hlock->seq_count; + } + + return ~0; +} + /* * Check whether we follow the irq-flags state precisely: */ @@ -5866,7 +5888,8 @@ void lock_acquire(struct lockdep_map *lock, unsigned int subclass, lockdep_recursion_inc(); __lock_acquire(lock, subclass, trylock, read, check, - irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 0); + irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 0, + ++current->lockdep_seq); lockdep_recursion_finish(); raw_local_irq_restore(flags); } @@ -5914,7 +5937,8 @@ void lock_sync(struct lockdep_map *lock, unsigned subclass, int read, lockdep_recursion_inc(); __lock_acquire(lock, subclass, 0, read, check, - irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 1); + irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 1, + ++current->lockdep_seq); check_chain_key(current); lockdep_recursion_finish(); raw_local_irq_restore(flags); @@ -6000,6 +6024,26 @@ void lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie) } EXPORT_SYMBOL_GPL(lock_unpin_lock); +u32 lock_sequence(struct lockdep_map *lock) +{ + unsigned long flags; + u32 seq = ~0; + + if (unlikely(!lockdep_enabled())) + return seq; + + raw_local_irq_save(flags); + check_flags(flags); + + lockdep_recursion_inc(); + seq = __lock_sequence(lock); + lockdep_recursion_finish(); + raw_local_irq_restore(flags); + + return seq; +} +EXPORT_SYMBOL_GPL(lock_sequence); + #ifdef CONFIG_LOCK_STAT static void print_lock_contention_bug(struct task_struct *curr, struct lockdep_map *lock, -- cgit v1.2.3 From 02c6be7d675b21d81f0ba3a524346850a8c0e3bf Mon Sep 17 00:00:00 2001 From: Eric Dumazet Date: Mon, 24 Aug 2026 15:51:29 +0000 Subject: locking/lockdep: Invalidate stale class_cache entries for zapped classes syzbot reported a lockdep splat hitting DEBUG_LOCKS_WARN_ON(1) in hlock_class() due to an invalid class_idx: WARNING: kernel/locking/lockdep.c:238 at __lock_acquire+0x382/0x2cf0 kernel/locking/lockdep.c:5203 Workqueue: wg-crypt-wg0 wg_packet_tx_worker RIP: 0010:hlock_class kernel/locking/lockdep.c:238 [inline] RIP: 0010:check_wait_context kernel/locking/lockdep.c:4870 [inline] RIP: 0010:__lock_acquire+0x389/0x2cf0 kernel/locking/lockdep.c:5203 Call Trace: lock_acquire+0x106/0x350 kernel/locking/lockdep.c:5886 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:173 tcp_tsq_handler+0x29/0x200 net/ipv4/tcp_output.c:1291 tcp_tsq_workfn+0x384/0x410 net/ipv4/tcp_output.c:1325 ... When a lock class is zapped (e.g. during module unload or key unregistration), zap_class() clears the class's bit in lock_classes_in_use and removes it from the class hash table. However, existing lockdep_map instances embedded in data structures may still retain a pointer to the zapped class in their class_cache[] array. When __lock_acquire() subsequently runs on such a lock, it finds lock->class_cache[subclass] != NULL, skipping register_lock_class() and assigning hlock->class_idx to the index of the zapped class. When check_wait_context() or hlock_class() inspects the held_lock, it finds !test_bit(class_idx, lock_classes_in_use) and warns. Furthermore, if the zapped slot is subsequently re-allocated to an unrelated lock key, the stale class_cache entry would erroneously match the unrelated class (ABA issue). Add lock_class_cache_is_valid() to validate that the cached class is within lock_classes bounds, still allocated in lock_classes_in_use (using uninstrumented arch_test_bit() in __always_inline context so it is safe in noinstr contexts like match_held_lock()), and that class->key matches the expected subkey (taking lockdep_set_subclass() overrides into account). Also use READ_ONCE()/WRITE_ONCE() when accessing class_cache[]. If the entry is invalid or stale, fall back to register_lock_class() / look_up_lock_class(). Fixes: a0b0fd53e1e6 ("locking/lockdep: Free lock classes that are no longer in use") Closes: https://lore.kernel.org/netdev/6a8c66dc.4d75e56a.c9a88.0050.GAE@google.com/T/#u Reported-by: syzbot+2d770620059281e225a4@syzkaller.appspotmail.com Assisted-by: Gemini:gemini-3.1-pro Signed-off-by: Eric Dumazet Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/20260824155129.676096-1-edumazet@google.com --- kernel/locking/lockdep.c | 50 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 8 deletions(-) (limited to 'kernel/locking') diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index 25d77d4a1061..763f79806bd8 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -963,6 +963,34 @@ look_up_lock_class(const struct lockdep_map *lock, unsigned int subclass) return NULL; } +static __always_inline bool lock_class_cache_is_valid(const struct lockdep_map *lock, + const struct lock_class *class, + unsigned int subclass) +{ + unsigned int class_subclass; + + if (!class) + return false; + + if (unlikely(class < lock_classes || class >= lock_classes + MAX_LOCKDEP_KEYS)) + return false; + + if (unlikely(!arch_test_bit(class - lock_classes, lock_classes_in_use))) + return false; + + if (unlikely(!lock->key)) + return false; + + class_subclass = subclass ? subclass : class->subclass; + if (unlikely(class_subclass >= MAX_LOCKDEP_SUBCLASSES)) + return false; + + if (unlikely(READ_ONCE(class->key) != lock->key->subkeys + class_subclass)) + return false; + + return true; +} + /* * Static locks do not have their class-keys yet - for them the key is * the lock object itself. If the lock is in the per cpu area, the @@ -1395,9 +1423,9 @@ out_unlock_set: out_set_class_cache: if (!subclass || force) - lock->class_cache[0] = class; + WRITE_ONCE(lock->class_cache[0], class); else if (subclass < NR_LOCKDEP_CACHING_CLASSES) - lock->class_cache[subclass] = class; + WRITE_ONCE(lock->class_cache[subclass], class); /* * Hash collision, did we smoke some? We found a class with a matching @@ -4957,7 +4985,7 @@ void lockdep_init_map_type(struct lockdep_map *lock, const char *name, int i; for (i = 0; i < NR_LOCKDEP_CACHING_CLASSES; i++) - lock->class_cache[i] = NULL; + WRITE_ONCE(lock->class_cache[i], NULL); #ifdef CONFIG_LOCK_STAT lock->cpu = raw_smp_processor_id(); @@ -5022,12 +5050,15 @@ EXPORT_SYMBOL_GPL(__lockdep_no_track__); void lockdep_set_lock_cmp_fn(struct lockdep_map *lock, lock_cmp_fn cmp_fn, lock_print_fn print_fn) { - struct lock_class *class = lock->class_cache[0]; + struct lock_class *class = READ_ONCE(lock->class_cache[0]); unsigned long flags; raw_local_irq_save(flags); lockdep_recursion_inc(); + if (!lock_class_cache_is_valid(lock, class, 0)) + class = NULL; + if (!class) class = register_lock_class(lock, 0, 0); @@ -5119,8 +5150,11 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass, if (DEBUG_LOCKS_WARN_ON(subclass >= MAX_LOCKDEP_SUBCLASSES)) return 0; - if (subclass < NR_LOCKDEP_CACHING_CLASSES) - class = lock->class_cache[subclass]; + if (subclass < NR_LOCKDEP_CACHING_CLASSES) { + class = READ_ONCE(lock->class_cache[subclass]); + if (!lock_class_cache_is_valid(lock, class, subclass)) + class = NULL; + } /* * Not cached? */ @@ -5323,9 +5357,9 @@ static noinstr int match_held_lock(const struct held_lock *hlock, return 1; if (hlock->references) { - const struct lock_class *class = lock->class_cache[0]; + const struct lock_class *class = READ_ONCE(lock->class_cache[0]); - if (!class) + if (!lock_class_cache_is_valid(lock, class, 0)) class = look_up_lock_class(lock, 0); /* -- cgit v1.2.3 From 912edebe8501a36c6bedcef03bd238ab90a7e060 Mon Sep 17 00:00:00 2001 From: Sebastian Andrzej Siewior Date: Tue, 1 Sep 2026 15:54:51 +0200 Subject: futex: Provide rt_mutex_.*_schedule() equivalents for futex scheduling There is rt_mutex_{pre|post}_schedule() around rt_mutex_wait_proxy_lock() to ensure that sched_submit_work()/ sched_update_worker() is invoked before we schedule out and block on rt_mutex while waiting for it become available. The reason is that blocking on rt_mutex assigns a pi_waiter for the PI chain and sched_submit_work() will also assign a pi_waiter if it blocks on lock but a this point we already have a waiter assigned. We can't skip sched_submit_work() entirely because I/O relies on the fact that I/O queue is flushed while it blocks on a sleeping lock. Therefore sched_submit_work() is moved before we block on the lock. Sleeping lock in this context means mutex or rw_semaphore not spinlock_t on PREEMPT_RT. Because the mutex abstraction on PREEMPT_RT uses the same abstraction as the futex proxy lock, the futex code ended up using rt_mutex_{pre|post}_schedule(), too. Using it is/ was just to keep the task_struct::sched_rt_mutex assertion happy. Futex proxy lock is used only in the syscall context of a task. At this point it never got any I/O that needs to be flushed and it can't be a workqueue that needs to notify that it will be scheduled out. Therefore sched_submit_work() does nothing here. By mistake futex_wait_requeue_pi() -> rt_mutex_wait_proxy_lock() did not get the rt_mutex_{pre|post}_schedule() annotation. This was not noticed because in this callchain the lock is (usually) not contended and so rt_mutex_slowlock_block() does not schedule, triggering the assert. Adding rt_mutex_pre_schedule() here looks wrong (as noted by PeterZ) because at this point there is a pi_waiter recorded and invoking sched_submit_work() with a possible lock contention would be wrong. Add rt_mutex_futex_{pre|post}_schedule() which toggles the sched_rt_mutex assert and does not involve sched_submit_work(). Add asserts here to ensure that sched_submit_work() would do nothing. Use it only in futex proxy lock case which is rt_mutex_wait_proxy_lock(). Remove it from futex_lock_pi(). Fixes: d14f9e930b90 ("locking/rtmutex: Use rt_mutex specific scheduler helpers") Reported-by: Yao Kai Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Thomas Gleixner Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260901135453.3121948-2-bigeasy@linutronix.de Closes: https://lore.kernel.org/all/20260717084922.4153317-2-yaokai34@huawei.com --- include/linux/sched/rt.h | 2 ++ kernel/futex/pi.c | 16 +++------------- kernel/locking/rtmutex_api.c | 2 ++ kernel/sched/core.c | 16 ++++++++++++++++ 4 files changed, 23 insertions(+), 13 deletions(-) (limited to 'kernel/locking') diff --git a/include/linux/sched/rt.h b/include/linux/sched/rt.h index 4e3338103654..922935cc3383 100644 --- a/include/linux/sched/rt.h +++ b/include/linux/sched/rt.h @@ -52,8 +52,10 @@ static inline bool rt_or_dl_task_policy(struct task_struct *tsk) #ifdef CONFIG_RT_MUTEXES extern void rt_mutex_pre_schedule(void); +extern void rt_mutex_futex_pre_schedule(void); extern void rt_mutex_schedule(void); extern void rt_mutex_post_schedule(void); +extern void rt_mutex_futex_post_schedule(void); /* * Must hold either p->pi_lock or task_rq(p)->lock. diff --git a/kernel/futex/pi.c b/kernel/futex/pi.c index 88788e584ec8..98f1b962e59a 100644 --- a/kernel/futex/pi.c +++ b/kernel/futex/pi.c @@ -1070,17 +1070,11 @@ retry_private: * Caution; releasing @hb in-scope. The hb->lock is still locked * while the reference is dropped. The reference can not be dropped * after the unlock because if a user initiated resize is in progress - * then we might need to wake him. This can not be done after the - * rt_mutex_pre_schedule() invocation. The hb will remain valid because - * the thread, performing resize, will block on hb->lock during - * the requeue. + * then we might need to wake him. The hb will remain valid + * because the thread, performing resize, will block on + * hb->lock during the requeue. */ futex_private_hash_put(no_free_ptr(hbr.fph)); - /* - * Must be done before we enqueue the waiter, here is unfortunately - * under the hb lock, but that *should* work because it does nothing. - */ - rt_mutex_pre_schedule(); rt_mutex_init_waiter(&rt_waiter); @@ -1146,10 +1140,6 @@ cleanup: * the */ futex_q_lockptr_lock(&q); - /* - * Waiter is unqueued. - */ - rt_mutex_post_schedule(); no_block: /* * Fixup the pi_state owner and possibly acquire the lock if we diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c index 5d48d64725b1..eb18b094473c 100644 --- a/kernel/locking/rtmutex_api.c +++ b/kernel/locking/rtmutex_api.c @@ -423,6 +423,7 @@ int __sched rt_mutex_wait_proxy_lock(struct rt_mutex_base *lock, { int ret; + rt_mutex_futex_pre_schedule(); raw_spin_lock_irq(&lock->wait_lock); /* sleep on the mutex */ set_current_state(TASK_INTERRUPTIBLE); @@ -433,6 +434,7 @@ int __sched rt_mutex_wait_proxy_lock(struct rt_mutex_base *lock, */ fixup_rt_mutex_waiters(lock, true); raw_spin_unlock_irq(&lock->wait_lock); + rt_mutex_futex_post_schedule(); return ret; } diff --git a/kernel/sched/core.c b/kernel/sched/core.c index f78275192036..449ccd871be8 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7637,6 +7637,17 @@ void rt_mutex_pre_schedule(void) sched_submit_work(current); } +/* + * Used within the futex syscall context, skips sched_submit_work() because none + * its work will be done. Asserts ensure that it is indeed the case. + */ +void rt_mutex_futex_pre_schedule(void) +{ + lockdep_assert(!(current->flags & (PF_WQ_WORKER | PF_IO_WORKER))); + lockdep_assert(!current->plug); + lockdep_assert(!fetch_and_set(current->sched_rt_mutex, 1)); +} + void rt_mutex_schedule(void) { lockdep_assert(current->sched_rt_mutex); @@ -7649,6 +7660,11 @@ void rt_mutex_post_schedule(void) lockdep_assert(fetch_and_set(current->sched_rt_mutex, 0)); } +void rt_mutex_futex_post_schedule(void) +{ + lockdep_assert(fetch_and_set(current->sched_rt_mutex, 0)); +} + /* * rt_mutex_setprio - set the current priority of a task * @p: task to boost -- cgit v1.2.3