summaryrefslogtreecommitdiff
path: root/kernel/time/hrtimer.c
AgeCommit message (Collapse)Author
26 hoursMerge branch into tip/master: 'timers/urgent'Ingo Molnar
# New commits in timers/urgent: f322d7f7ae56 ("clocksource: Prevent immediate watchdog rerun after a delay") cbd0f65b48f5 ("vdso/gettimeofday: Correct the read of tz_minuteswest and tz_dsttime") e389ae428847 ("tick/nohz: Reevaluate the tick for kicks flushed outside hardirq") 45aeb7b30393 ("selftests: timers: Add a test for disarmed SIGEV_NONE timers") 141a27750ec3 ("posix-timers: Report zero remaining time for disarmed SIGEV_NONE timers") d69cfd33b5a8 ("time/namespace: Fix overflow when converting absolute deadlines") ef60f61208a3 ("timers/itimer: Keep a nonzero value at CPU timer expiry") 5b819071e434 ("posix-timers: Accept NULL clock_getres() output in stubs") bd091346971c ("sched/idle: Use hrtimer cleanup class in play_idle_precise()") 064bcd117e8a ("hrtimer: Add cleanup class for on-stack timers") Signed-off-by: Ingo Molnar <mingo@kernel.org>
3 dayshrtimer: Add cleanup class for on-stack timersHui Su
hrtimer_setup_on_stack() requires callers to destroy the timer before its backing stack storage goes out of scope, but the setup interface does not encode the required teardown pairing. Add an hrtimer_on_stack cleanup class that pairs on-stack setup with destroy_hrtimer_on_stack(). Document the existing setup API's lifetime requirement. The cleanup class provides debug-object teardown, not timer cancellation. Callers must ensure that no hrtimer operation can still access the timer before the scope ends; in particular, the timer must no longer be queued and any running callback must have completed. Signed-off-by: Hui Su <sh_def@163.com> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/5f48c32490d19f51035e4209174f9cfe44472c71.1792083743.git.sh_def@163.com
2026-09-29hrtimer: Apply READ_ONCE() to lockless base->running loadsPaul E. McKenney
Updates to base->running are protected by the hrtimer base lock, but some loads are lockless. Therefore, prevent compiler mischief by applying READ_ONCE() to the lockless loads. KCSAN located this issue. Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260919001428.3133388-12-paulmck@kernel.org
2026-09-29hrtimer: Update hrtimer_resolution only if value changesPaul E. McKenney
Currently, the hrtimer_switch_to_hres() function unconditionally updates hrtimer_resolution to HIGH_RES_NSEC. This causes KCSAN false positives in kernels built with CONFIG_KCSAN_STRICT=y. It also causes unnecessary cache-line traffic on the many code paths that load hrtimer_resolution, which is especially regrettable because the value of hrtimer_resolution normally changes only once during the lifetime of a given kernel. Therefore, update the value of hrtimer_resolution to HIGH_RES_NSEC only if it does not already have that value. Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260919001428.3133388-10-paulmck@kernel.org
2026-09-29hrtimer: Mark data-racy accesses to hrtimer_sleeper ->task fieldPaul E. McKenney
The hrtimer_sleeper structure's ->task field is used as a flag to indicate that the associated hrtimer has expired. This means that the hrtimer handler can be storing to this field while other code is loading from it to check for expiry. Note that additional races appear for hrtimers that can be restarted, which could be argued to be a user error. However, that is no reason to let the compiler introduce additional confusion. Therefore, mark data-racy accesses to the hrtimer_sleeper ->task field using READ_ONCE() (using a new hrtimer_sleeper_task_get() access function) and WRITE_ONCE() (using a new hrtimer_sleeper_task_set() access function). KCSAN located this issue. Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com> Link: https://patch.msgid.link/20260919001428.3133388-1-paulmck@kernel.org
2026-09-11hrtimer: Use hard expiry when updating timers on the same baseAndrea Parri
Rearming a queued timer with nonzero slack can leave the timerqueue out of order. remove_and_enqueue_same_base() checks the new soft expiry against its neighbours' hard expiries, then stores the new hard expiry in the node without requeueing it. For example, with A at 10 and B at 20, rearming A at 11 with slack 30 passes the neighbour check but leaves A's hard expiry of 41 before B's 20. The same function also caches the soft expiry in base->expires_next when updating or inserting the first timer, giving next-event selection an earlier deadline than the queue head's hard expiry. Set the timer expiry before handling the queue. Use its stored hard expiry for the in-place ordering check and both updates to base->expires_next. The early update is safe because remove_and_enqueue_same_base() runs with base->cpu_base->lock held. The lock keeps the queue stable while hrtimer_can_update_in_place() checks the new expiry against both neighbours. If the check fails, timerqueue_linked_del() removes the node without comparing expiry values before it is reinserted. Fixes: eddffab8282e3 ("hrtimer: Keep track of first expiring timer per clock base") Fixes: 343f2f4dc5425 ("hrtimer: Try to modify timers in place") Signed-off-by: Andrea Parri <parri.andrea@gmail.com> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Assisted-by: LLM Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260910143442.2018-1-parri.andrea@gmail.com
2026-08-11hrtimer: Add a lockdep assertion to hrtimer_update_base()Thomas Weißschuh (Schneider Electric)
Document and verify that the hrtimer_cpu_base::lock is held at this point. Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-3-910cbd485390@linutronix.de
2026-08-11timekeeping: Use u32 for clock_was_set_seqThomas Weißschuh (Schneider Electric)
Use an explicitly sized type to make the code a bit more consistent with other fields of the datastructure and other sequence counters. Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260803-auxclock-nanosleep-prep-v2-2-910cbd485390@linutronix.de
2026-08-11hrtimer: Account nr_retries on recovered interrupt retriesLiang Hao
Re-arranging hrtimer_interrupt() switched the retry path to a local counter and dropped the update of cpu_base->nr_retries, leaving the field exported via /proc/timer_list stuck at zero. Increment nr_retries only when another pass through the expiry loop is started; the third attempt that falls through to hang handling is still accounted by nr_hangs alone. Fixes: 288924384856 ("hrtimer: Re-arrange hrtimer_interrupt()") Signed-off-by: Liang Hao <haohlliang@gmail.com> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260731150408.19554-1-haohlliang@gmail.com
2026-07-07hrtimer: Explicitly include linux/hrtimer_bases.hThomas Weißschuh (Schneider Electric)
This header uses some definitions from linux/hrtimer_bases.h. Currently this header is included transitively, which will change. Include the header explicitly. Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-5-c50b19bda473@linutronix.de
2026-07-07hrtimer: Move hrtimer_update_function() to hrtimer.cThomas Weißschuh (Schneider Electric)
The usage of the hrtimer base forces hrtimer.h to also expose the base structure definitions. Move the function to hrtimer.c to avoid this. Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260702-hrtimer-header-dependencies-v1-3-c50b19bda473@linutronix.de
2026-07-07hrtimer: Don't take cpu_base::lock in hrtimer_get_next_event() when hres_activeUsama Arif
hrtimer_get_next_event() runs on every tick-stop decision via get_next_timer_interrupt() -> cmp_next_hrtimer_event(). When high resolution timers are active it must return KTIME_MAX -- the caller documents and depends on this. The function takes cpu_base->lock, checks hres_active, and returns KTIME_MAX. Taking the lock is not required because cpu_base->hres_active is only written by the local CPU in hrtimer_switch_to_hres() from hard interrupt context and in hrtimers_cpu_starting() during bring-up. All callers of hrtimer_get_next_event() reach it from the tick-stop / cpuidle paths with interrupts disabled on that CPU. No writer can therefore race with the read, so an unlocked hres_active check is stable and the lock can be skipped in this case. On a 176-thread AMD EPYC 9D64 running a production workload, bucketing callers of native_queued_spin_lock_slowpath(), the slowpath had 199 samples in total, of which 54 are attributed to hrtimer_get_next_event(), i.e. this accounts for ~27% of slowpath hits on this specific workload. Move the hres_active check before the lock guard region to address this. [ tglx: Massaged change log and comments ] Signed-off-by: Usama Arif <usama.arif@linux.dev> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260707153800.542394-1-usama.arif@linux.dev
2026-07-06hrtimer: Remove unused next_timer argument from __hrtimer_reprogram()Zhan Xusheng
__hrtimer_reprogram() takes a @next_timer argument but never references it; it only stores @expires_next into cpu_base->expires_next and reprograms the clock event device. The argument has been unused since commit b14bca97c9f5 ("hrtimer: Consolidate reprogramming code"). Drop it and update the two callers. No functional change. Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260706084557.3845091-2-zhanxusheng@xiaomi.com
2026-06-30hrtimer: Remove unused clock_base_next_timer_safe()Zhan Xusheng
clock_base_next_timer_safe() was added by commit a64ad57e41c7 ("hrtimer: Simplify run_hrtimer_queues()") but has never had a caller; the queue iteration in __hrtimer_run_queues() uses clock_base_next_timer() instead. The two are functionally equivalent: struct hrtimer embeds the timerqueue node at offset 0, so container_of() of a NULL node yields NULL. Thus clock_base_next_timer() already returns NULL on an empty queue and the explicit NULL check in the _safe variant is redundant. Being a static __always_inline function it does not trigger -Wunused-function, so the dead code has gone unnoticed. Remove it. No functional change. Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260625140901.929554-1-zhanxusheng@xiaomi.com
2026-06-17hrtimer: Correct CONFIG_NO_HZ_COMMON macro name in commentEthan Nelson-Moore
A comment in kernel/time/hrtimer.c incorrectly refers to CONFIG_NOHZ_COMMON instead of CONFIG_NO_HZ_COMMON. Correct it. Discovered while searching for CONFIG_* symbols referenced in code but not defined in any Kconfig file. Signed-off-by: Ethan Nelson-Moore <enelsonmoore@gmail.com> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260609034314.25029-1-enelsonmoore@gmail.com
2026-05-13hrtimer: Fix the bogus return type of __hrtimer_start_range_ns()Thomas Gleixner
__hrtimer_start_range_ns() has a bool return type, but returns actually three different values, which are checked at the call site. Make the return type int. Fixes: bd5956166d20 ("hrtimer: Provide hrtimer_start_range_ns_user()") Reported-by: Dan Carpenter <error27@gmail.com Signed-off-by: Thomas Gleixner <tglx@kernel.org>
2026-05-06hrtimer: Return ktime_t from ↵Thomas Weißschuh
hrtimer_get_next_event()/hrtimer_next_event_without() These functions really work in terms of ktime_t and not u64. Change their return types and adapt the callers. Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260504-hrtimer-next_event-v2-1-7a5d0550b42f@linutronix.de
2026-05-01hrtimer: Use hrtimer_start_expires_user() for hrtimer sleepersThomas Gleixner
Most hrtimer sleepers are user controlled and user space can hand arbitrary expiry values in as long as they are valid timespecs. If the expiry value is in the past then this requires a full loop through reprogramming the clock event device, taking the hrtimer interrupt, waking the task and reprogram again. Use hrtimer_start_expires_user() which avoids the full round trip by checking the timer for expiry on enqueue. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Tested-by: Calvin Owens <calvin@wbinvd.org> Reviewed-by: Frederic Weisbecker <frederic@kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260408114952.062400833@kernel.org
2026-05-01hrtimer: Provide hrtimer_start_range_ns_user()Thomas Gleixner
Calvin reported an odd NMI watchdog lockup which claims that the CPU locked up in user space. He provided a reproducer, which set's up a timerfd based timer and then rearms it in a loop with an absolute expiry time of 1ns. As the expiry time is in the past, the timer ends up as the first expiring timer in the per CPU hrtimer base and the clockevent device is programmed with the minimum delta value. If the machine is fast enough, this ends up in a endless loop of programming the delta value to the minimum value defined by the clock event device, before the timer interrupt can fire, which starves the interrupt and consequently triggers the lockup detector because the hrtimer callback of the lockup mechanism is never invoked. The clockevents code already has a last resort mechanism to prevent that, but it's sensible to catch such issues before trying to reprogram the clock event device. Provide a variant of hrtimer_start_range_ns(), which sanity checks the timer after queueing it. It does not so before because the timer might be armed and therefore needs to be dequeued. also we optimize for the latest possible point to check, so that the clock event prevention is avoided as much as possible. If the timer is already expired _before_ the clock event is reprogrammed, remove the timer from the queue and signal to the caller that the operation failed by returning false. That allows the caller to take immediate action without going through the loops and hoops of the hrtimer interrupt. The queueing code can't invoke the timer callback as the caller might hold a lock which is taken in the callback. Add a tracepoint which allows to analyze the expired at start situation. Reported-by: Calvin Owens <calvin@wbinvd.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Tested-by: Calvin Owens <calvin@wbinvd.org> Reviewed-by: Frederic Weisbecker <frederic@kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260408114951.995031895@kernel.org
2026-04-11Merge branch 'timers/urgent' into timers/coreThomas Gleixner
to resolve the conflict with urgent fixes.
2026-04-10clockevents: Prevent timer interrupt starvationThomas Gleixner
Calvin reported an odd NMI watchdog lockup which claims that the CPU locked up in user space. He provided a reproducer, which sets up a timerfd based timer and then rearms it in a loop with an absolute expiry time of 1ns. As the expiry time is in the past, the timer ends up as the first expiring timer in the per CPU hrtimer base and the clockevent device is programmed with the minimum delta value. If the machine is fast enough, this ends up in a endless loop of programming the delta value to the minimum value defined by the clock event device, before the timer interrupt can fire, which starves the interrupt and consequently triggers the lockup detector because the hrtimer callback of the lockup mechanism is never invoked. As a first step to prevent this, avoid reprogramming the clock event device when: - a forced minimum delta event is pending - the new expiry delta is less then or equal to the minimum delta Thanks to Calvin for providing the reproducer and to Borislav for testing and providing data from his Zen5 machine. The problem is not limited to Zen5, but depending on the underlying clock event device (e.g. TSC deadline timer on Intel) and the CPU speed not necessarily observable. This change serves only as the last resort and further changes will be made to prevent this scenario earlier in the call chain as far as possible. [ tglx: Updated to restore the old behaviour vs. !force and delta <= 0 and fixed up the tick-broadcast handlers as pointed out by Borislav ] Fixes: d316c57ff6bf ("[PATCH] clockevents: add core functionality") Reported-by: Calvin Owens <calvin@wbinvd.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Tested-by: Calvin Owens <calvin@wbinvd.org> Tested-by: Borislav Petkov <bp@alien8.de> Link: https://lore.kernel.org/lkml/acMe-QZUel-bBYUh@mozart.vkv.me/ Link: https://patch.msgid.link/20260407083247.562657657@kernel.org
2026-04-01hrtimer: Fix incorrect #endif comment for BITS_PER_LONG checkZhan Xusheng
The #endif comment says "BITS_PER_LONG >= 64", but the corresponding #if guard is "BITS_PER_LONG < 64". The comment was originally correct when the block had a three-way #if/#else/#endif structure, where the #else branch provided a 64-bit inline version. Commit 79bf2bb335b8 ("[PATCH] tick-management: dyntick / highres functionality") removed the #else branch but did not update the #endif comment, leaving it inconsistent with the remaining #if condition. Fix the comment to match the preprocessor guard. Signed-off-by: Zhan Xusheng <zhanxusheng@xiaomi.com> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260331074811.26147-1-zhanxusheng@xiaomi.com
2026-03-12hrtimer: Add a helper to retrieve a hrtimer from its timerqueue nodeThomas Weißschuh (Schneider Electric)
The container_of() call is open-coded multiple times. Add a helper macro. Use container_of_const() to preserve constness. Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260311-hrtimer-cleanups-v1-12-095357392669@linutronix.de
2026-03-12hrtimer: Drop unnecessary pointer indirection in hrtimer_expire_entry eventThomas Weißschuh (Schneider Electric)
This pointer indirection is a remnant from when ktime_t was a struct, today it is pointless. Drop the pointer indirection. Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260311-hrtimer-cleanups-v1-9-095357392669@linutronix.de
2026-03-12hrtimer: Don't zero-initialize ret in hrtimer_nanosleep()Thomas Weißschuh (Schneider Electric)
The value will be assigned to before any usage. No other function in hrtimer.c does such a zero-initialization. Signed-off-by: Thomas Weißschuh (Schneider Electric) <thomas.weissschuh@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260311-hrtimer-cleanups-v1-7-095357392669@linutronix.de
2026-03-11hrtimer: Less agressive interrupt 'hang' handlingPeter Zijlstra
When the hrtimer_interrupt needs to restart more than 3 times and still has expired timers, the interrupt is considered hung. To give the system a little time to recover, the hardware timer is programmed a little into the future. Prior to commit 288924384856 ("hrtimer: Re-arrange hrtimer_interrupt()"), this was relative to the amount of time spend serving the interrupt with a max of 100 msec. However, in order to simplify, and because this condition 'should' not happen, the timeout was unconditionally set to 100 msec. 'Obviously' there is a benchmark that hits this hard, by programming a ton of very short timers :-/ Since reprogramming is decoupled from the interrupt handling, the actual execution time is lost, however the code does track max_hang_time. Using that, rather than the 100 ms max restores performance. stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --timermix 64 bogo ops/s 288924384856^1: 23715979.93 288924384856: 11550049.77 patched: 23361116.78 Additionally, Thomas noted that cpu_base->hang_detected should not be cleared until the next interrupt, such that __hrtimer_reprogram() won't undo the extra delay. Fixes: 288924384856 ("hrtimer: Re-arrange hrtimer_interrupt()") Reported-by: kernel test robot <oliver.sang@intel.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260311121500.GF652779@noisy.programming.kicks-ass.net Closes: https://lore.kernel.org/oe-lkp/202603102229.74b9dee4-lkp@intel.com
2026-02-27hrtimer: Try to modify timers in placeThomas Gleixner
When modifying the expiry of a armed timer it is first dequeued, then the expiry value is updated and then it is queued again. This can be avoided when the new expiry value is within the range of the previous and the next timer as that does not change the position in the RB tree. The linked timerqueue allows to peak ahead to the neighbours and check whether the new expiry time is within the range of the previous and next timer. If so just modify the timer in place and spare the enqueue and requeue effort, which might end up rotating the RB tree twice for nothing. This speeds up the handling of frequently rearmed hrtimers, like the hrtick scheduler timer significantly. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163431.873359816@kernel.org
2026-02-27hrtimer: Use linked timerqueueThomas Gleixner
To prepare for optimizing the rearming of enqueued timers, switch to the linked timerqueue. That allows to check whether the new expiry time changes the position of the timer in the RB tree or not, by checking the new expiry time against the previous and the next timers expiry. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163431.806643179@kernel.org
2026-02-27hrtimer: Optimize for_each_active_base()Thomas Gleixner
Give the compiler some help to emit way better code. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163431.599804894@kernel.org
2026-02-27hrtimer: Simplify run_hrtimer_queues()Thomas Gleixner
Replace the open coded container_of() orgy with a trivial clock_base_next_timer() helper. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163431.532927977@kernel.org
2026-02-27hrtimer: Rework next event evaluationThomas Gleixner
The per clock base cached expiry time allows to do a more efficient evaluation of the next expiry on a CPU. Separate the reprogramming evaluation from the NOHZ idle evaluation which needs to exclude the NOHZ timer to keep the reprogramming path lean and clean. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163431.468186893@kernel.org
2026-02-27hrtimer: Keep track of first expiring timer per clock baseThomas Gleixner
Evaluating the next expiry time of all clock bases is cache line expensive as the expiry time of the first expiring timer is not cached in the base and requires to access the timer itself, which is definitely in a different cache line. It's way more efficient to keep track of the expiry time on enqueue and dequeue operations as the relevant data is already in the cache at that point. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163431.404839710@kernel.org
2026-02-27hrtimer: Avoid re-evaluation when nothing changedThomas Gleixner
Most times there is no change between hrtimer_interrupt() deferring the rearm and the invocation of hrtimer_rearm_deferred(). In those cases it's a pointless exercise to re-evaluate the next expiring timer. Cache the required data and use it if nothing changed. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163431.338569372@kernel.org
2026-02-27hrtimer: Push reprogramming timers into the interrupt return pathPeter Zijlstra
Currently hrtimer_interrupt() runs expired timers, which can re-arm themselves, after which it computes the next expiration time and re-programs the hardware. However, things like HRTICK, a highres timer driving preemption, cannot re-arm itself at the point of running, since the next task has not been determined yet. The schedule() in the interrupt return path will switch to the next task, which then causes a new hrtimer to be programmed. This then results in reprogramming the hardware at least twice, once after running the timers, and once upon selecting the new task. Notably, *both* events happen in the interrupt. By pushing the hrtimer reprogram all the way into the interrupt return path, it runs after schedule() picks the new task and the double reprogram can be avoided. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163431.273488269@kernel.org
2026-02-27hrtimer: Rename hrtimer_cpu_base::in_hrtirq to deferred_rearmThomas Gleixner
The upcoming deferred rearming scheme has the same effect as the deferred rearming when the hrtimer interrupt is executing. So it can reuse the in_hrtirq flag, but when it gets deferred beyond the hrtimer interrupt path, then the name does not make sense anymore. Rename it to deferred_rearm upfront to keep the actual functional change separate from the mechanical rename churn. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.935623347@kernel.org
2026-02-27hrtimer: Re-arrange hrtimer_interrupt()Peter Zijlstra
Rework hrtimer_interrupt() such that reprogramming is split out into an independent function at the end of the interrupt. This prepares for reprogramming getting delayed beyond the end of hrtimer_interrupt(). Notably, this changes the hang handling to always wait 100ms instead of trying to keep it proportional to the actual delay. This simplifies the state, also this really shouldn't be happening. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.870639266@kernel.org
2026-02-27hrtimer: Separate remove/enqueue handling for local timersThomas Gleixner
As the base switch can be avoided completely when the base stays the same the remove/enqueue handling can be more streamlined. Split it out into a separate function which handles both in one go which is way more efficient and makes the code simpler to follow. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.737600486@kernel.org
2026-02-27hrtimer: Use NOHZ information for localityThomas Gleixner
The decision to keep a timer which is associated to the local CPU on that CPU does not take NOHZ information into account. As a result there are a lot of hrtimer base switch invocations which end up not switching the base and stay on the local CPU. That's just work for nothing and can be further improved. If the local CPU is part of the NOISE housekeeping mask, then check: 1) Whether the local CPU has the tick running, which means it is either not idle or already expecting a timer soon. 2) Whether the tick is stopped and need_resched() is set, which means the CPU is about to exit idle. This reduces the amount of hrtimer base switch attempts, which end up on the local CPU anyway, significantly and prepares for further optimizations. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.673473029@kernel.org
2026-02-27hrtimer: Optimize for local timersThomas Gleixner
The decision whether to keep timers on the local CPU or on the CPU they are associated to is suboptimal and causes the expensive switch_hrtimer_base() mechanism to be invoked more than necessary. This is especially true for pinned timers. Rewrite the decision logic so that the current base is kept if: 1) The callback is running on the base 2) The timer is associated to the local CPU and the first expiring timer as that allows to optimize for reprogramming avoidance 3) The timer is associated to the local CPU and pinned 4) The timer is associated to the local CPU and timer migration is disabled. Only #2 was covered by the original code, but especially #3 makes a difference for high frequency rearming timers like the scheduler hrtick timer. If timer migration is disabled, then #4 avoids most of the base switches. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.607935269@kernel.org
2026-02-27hrtimer: Convert state and properties to booleanThomas Gleixner
All 'u8' flags are true booleans, so make it entirely clear that these can only contain true or false. This is especially true for hrtimer::state, which has a historical leftover of using the state with bitwise operations. That was used in the early hrtimer implementation with several bits, but then converted to a boolean state. But that conversion missed to replace the bit OR and bit check operations all over the place, which creates suboptimal code. As of today 'state' is a misnomer because it's only purpose is to reflect whether the timer is enqueued into the RB-tree or not. Rename it to 'is_queued' and make all operations on it boolean. This reduces text size from 8926 to 8732 bytes. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.542427240@kernel.org
2026-02-27hrtimer: Replace the bitfield in hrtimer_cpu_baseThomas Gleixner
Use bool for the various flags as that creates better code in the hot path. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.475262618@kernel.org
2026-02-27hrtimer: Evaluate timer expiry only onceThomas Gleixner
No point in accessing the timer twice. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.409352042@kernel.org
2026-02-27hrtimer: Cleanup coding style and commentsThomas Gleixner
As this code has some major surgery ahead, clean up coding style and bring comments up to date. No functional change intended. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.342740952@kernel.org
2026-02-27hrtimer: Use guards where appropriateThomas Gleixner
Simplify and tidy up the code where possible. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.275551488@kernel.org
2026-02-27hrtimer: Reduce trace noise in hrtimer_start()Thomas Gleixner
hrtimer_start() when invoked with an already armed timer traces like: <comm>-.. [032] d.h2. 5.002263: hrtimer_cancel: hrtimer= .... <comm>-.. [032] d.h1. 5.002263: hrtimer_start: hrtimer= .... Which is incorrect as the timer doesn't get canceled. Just the expiry time changes. The internal dequeue operation which is required for that is not really interesting for trace analysis. But it makes it tedious to keep real cancellations and the above case apart. Remove the cancel tracing in hrtimer_start() and add a 'was_armed' indicator to the hrtimer start tracepoint, which clearly indicates what the state of the hrtimer is when hrtimer_start() is invoked: <comm>-.. [032] d.h1. 6.200103: hrtimer_start: hrtimer= .... was_armed=0 <comm>-.. [032] d.h1. 6.200558: hrtimer_start: hrtimer= .... was_armed=1 Fixes: c6a2a1770245 ("hrtimer: Add tracepoint for hrtimers") Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.208491877@kernel.org
2026-02-27hrtimer: Add debug object init assertionThomas Gleixner
The debug object coverage in hrtimer_start_range_ns() happens too late to do anything useful. Implement the init assert assertion part and invoke that early in hrtimer_start_range_ns(). Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163430.143098153@kernel.org
2026-02-27hrtimer: Provide LAZY_REARM modePeter Zijlstra
The hrtick timer is frequently rearmed before expiry and most of the time the new expiry is past the armed one. As this happens on every context switch it becomes expensive with scheduling heavy work loads especially in virtual machines as the "hardware" reprogamming implies a VM exit. Add a lazy rearm mode flag which skips the reprogamming if: 1) The timer was the first expiring timer before the rearm 2) The new expiry time is farther out than the armed time This avoids a massive amount of reprogramming operations of the hrtick timer for the price of eventually taking the alredy armed interrupt for nothing. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163429.408524456@kernel.org
2026-02-27hrtimer: Provide a static branch based hrtimer_hres_enabled()Thomas Gleixner
The scheduler evaluates this via hrtimer_is_hres_active() every time it has to update HRTICK. This needs to follow three pointers, which is expensive. Provide a static branch based mechanism to avoid that. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260224163429.136503358@kernel.org
2026-02-27hrtimer: Avoid pointless reprogramming in __hrtimer_start_range_ns()Peter Zijlstra
Much like hrtimer_reprogram(), skip programming if the cpu_base is running the hrtimer interrupt. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Juri Lelli <juri.lelli@redhat.com> Reviewed-by: Thomas Gleixner <tglx@kernel.org> Link: https://patch.msgid.link/20260224163429.069535561@kernel.org
2026-02-10Merge tag 'timers-core-2026-02-09' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip Pull timer core updates from Thomas Gleixner: - Inline timecounter_cyc2time() as that is now used in the networking hotpath. Inlining it significantly improves performance. - Optimize the tick dependency check in case that the tracepoint is disabled, which improves the hotpath performance in the tick management code, which is a hotpath on transitions in and out of idle. - The usual cleanups and improvements * tag 'timers-core-2026-02-09' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: time/kunit: Document handling of negative years of is_leap() tick/nohz: Optimize check_tick_dependency() with early return time/sched_clock: Use ACCESS_PRIVATE() to evaluate hrtimer::function hrtimer: Drop _tv64() helpers hrtimer: Remove public definition of HIGH_RES_NSEC hrtimer: Remove unused resolution constants time/timecounter: Inline timecounter_cyc2time()