summaryrefslogtreecommitdiff
path: root/kernel/workqueue.c
AgeCommit message (Collapse)Author
2026-08-13workqueue: annotate racy p->wake_cpu accesses in kick_pool_pick()Breno Leitao
kick_pool_pick() reads and writes p->wake_cpu while the scheduler can update it concurrently. KCSAN reports: BUG: KCSAN: data-race in kick_pool_pick+0xf8/0x2d8 race at unknown origin, with read to 0xffff000663229da4 of 4 bytes by task 1817002 on cpu 40: kick_pool_pick+0xf8/0x2d8 process_scheduled_works+0x2bc/0x888 worker_thread+0x394/0x548 kthread+0x1b8/0x1f0 ret_from_fork+0x10/0x20 value changed: 0x0000002b -> 0x0000002f The race is harmless. wake_cpu is a best-effort placement hint: every writer stores a valid CPU id and the wakeup path validates it through select_task_rq(), so a stale value only affects which CPU the worker wakes up on. Mark both accesses with READ_ONCE() and WRITE_ONCE() to document that they are intentionally racy and to stop the compiler from reloading or tearing them. Signed-off-by: Breno Leitao <leitao@debian.org> Reviewed-by: Bradley Morgan <include@grrlz.net> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-12workqueue: BUG_ON() instead of returning NULL in wq_node_nr_active()Breno Leitao
wq_node_nr_active() warns and returns NULL when @wq is not unbound, but every caller dereferences the result right away, so the WARN_ON_ONCE() only moves the oops one frame up, as raised by Tejun. Fix it by BUGing_ON() instead of this silly WARN_ON_ONCE(); Fixes: b72fdc651056 ("workqueue: account nr_active by the backing pool") Suggested-by: Tejun Heo <tj@kernel.org> Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-12workqueue: use RCU accessors when populating wq->cpu_pwqBreno Leitao
wq->cpu_pwq holds RCU-protected pwq pointers, but the percpu allocation path fills it in with plain loads and stores, which sparse flags: kernel/workqueue.c:5682:57: sparse: incorrect type in initializer (different address spaces) @@ expected struct pool_workqueue **pwq_p @@ got struct pool_workqueue [noderef] __rcu ** @@ Allocate the array as __rcu pointers and publish each pwq with rcu_assign_pointer() once it is initialized and linked, the order install_unbound_pwq() uses. The warnings are not new: commit 79f23600bc7b ("workqueue: factor out get_percpu_pool()") only turned the flagged assignment into an initializer. Reported-by: kernel test robot <lkp@intel.com> Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/ Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-12workqueue: use rcu_dereference_sched() in workqueue_congested()Breno Leitao
workqueue_congested() fetches the pwq out of wq->cpu_pwq with a plain load, so sparse complains about the dropped __rcu: kernel/workqueue.c:6304:13: sparse: incorrect type in assignment (different address spaces) @@ expected struct pool_workqueue *pwq @@ got struct pool_workqueue [noderef] __rcu * @@ A pwq is released with kfree_rcu() and the read is protected by the surrounding preempt_disable(), which is what commit fd5081f4ef33 ("workqueue: Remove redundant rcu_read_lock/unlock() in workqueue_congested()") relied on when it dropped the rcu_read_lock() here. Use the rcu_dereference_sched() helper to make that explicit. Reported-by: kernel test robot <lkp@intel.com> Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/ Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-10workqueue: skip the node_nr_active update for non-unbound workqueuesBreno Leitao
apply_wqattrs_commit() updates node_nr_active->max unconditionally. wq->node_nr_active[] is only allocated for unbound workqueues, so guard the call before per-cpu workqueues start using this path. No functional change: only unbound workqueues reach apply_wqattrs_*() today. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-10workqueue: rename alloc_unbound_pwq() to alloc_pwq()Breno Leitao
This allocates a pwq and binds it to the pool @attrs asks for. Which pool that is becomes a property of the attrs (once per-cpu becomes an affinity scope). Remove the 'unbound" from the function name, given it will be bigger than unbound. No functional change. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-10workqueue: allocate attrs for all workqueuesBreno Leitao
The attrs are where the affinity scope lives, and a per-cpu workqueue will need one once per-cpu becomes a scope rather than a separate backend. Allocate them unconditionally. wq_dump.py used a non-NULL wq->attrs as its test for an unbound workqueue, which no longer holds; test WQ_UNBOUND there instead. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-10workqueue: rename wq->unbound_attrs to wq->attrsBreno Leitao
The unbound prefix says which workqueues currently have the field rather than what it holds, and the next patch allocates it for every workqueue. Rename it first so that change stays a single line. tools/workqueue/wq_dump.py reads the field by name, so rename it there too. wq_sysfs_unbound_attrs[] keeps its name: it is the set of sysfs files that only unbound workqueues expose. No functional change. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-10workqueue: test WQ_UNBOUND explicitly in the hotplug loopsBreno Leitao
workqueue_online_cpu() and workqueue_offline_cpu() decide whether a workqueue needs a pod affinity update by testing wq->unbound_attrs for NULL, which is only meaningful because the attrs are allocated for unbound workqueues alone. Test the flag instead, so the attrs can later be allocated for every workqueue. No functional change. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-10workqueue: account nr_active by the backing poolBreno Leitao
pwq_tryinc_nr_active() and pwq_dec_nr_active() choose between the shared per-node nr_active and the plain per-pwq one by testing wq_node_nr_active() for NULL. Test the backing pool with is_percpu_pool() instead, so the accounting follows the pool that runs the work rather than the workqueue type. No functional change. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-10workqueue: release pwq pools by pool typeBreno Leitao
Add is_percpu_pool() and test the pool directly for per cpu. Convert the other open-coded pool->cpu checks -- in put_unbound_pool(), pool_allowed_cpus() and the workqueue watchdog -- to the same helper. No functional change. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-10workqueue: factor out alloc_and_link_percpu_pwqs()Breno Leitao
Move the per-cpu pwq allocation loop out of alloc_and_link_pwqs() into a helper. The inner allocation-failure path now returns -ENOMEM and the caller jumps to the existing enomem cleanup, equivalent to the previous goto. No functional change. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-08-10workqueue: factor out get_percpu_pool()Breno Leitao
Move the static per-cpu worker_pool lookup in alloc_and_link_pwqs() into a helper, get_percpu_pool(), so the lookup can be shared by other pool-selection paths. No functional change. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-07-06workqueue: annotate racy sum_exec_runtime reads for CPU-intensive detectionBreno Leitao
The automatic CPU-intensive work item detection reads the worker task's se.sum_exec_runtime without a lock in wq_worker_running(), wq_worker_tick() and process_one_work(). The scheduler updates that field under the rq lock (from the tick via update_curr(), or cross-CPU via task_sched_runtime()), raising: BUG: KCSAN: data-race in wq_worker_running+0xa8/0xe8 race at unknown origin, with read to 0xffff0009a11d1df8 of 8 bytes by task 238535 on cpu 68: wq_worker_running schedule schedule_preempt_disabled __mutex_lock mutex_lock_nested cgroup_bpf_release process_one_work worker_thread kthread ret_from_fork value changed: 0x0000000088482ba0 -> 0x00000000884893c0 The value only feeds a heuristic, so the race is benign-ish. Unlike commit ecf5aad9a441 ("workqueue: annotate racy PWQ_STAT_CPU_TIME update in wq_worker_tick()") that only needs data_race(), these are plain reads whose result drives a subtraction and comparison, so use READ_ONCE() for a single, non-torn load, which also silences KCSAN. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-07-02workqueue: annotate racy PWQ_STAT_CPU_TIME update in wq_worker_tick()Breno Leitao
wq_worker_tick() bumps pwq->stats[PWQ_STAT_CPU_TIME] on every scheduler tick before pool->lock is taken. For unbound workqueues the pool_workqueue is shared by all workers of the pool across CPUs, so concurrent ticks on different CPUs perform an unsynchronized 64-bit read-modify-write on the same counter. KCSAN reports this as a data-race: BUG: KCSAN: data-race in wq_worker_tick / wq_worker_tick read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 29: wq_worker_tick+0x70/0x418 sched_tick+0x248/0x3a0 update_process_times+0x200/0x260 tick_nohz_handler+0x230/0x2f8 __hrtimer_run_queues+0x1ec/0x6c8 hrtimer_interrupt+0x174/0x4b8 ... read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 24: wq_worker_tick+0x70/0x418 sched_tick+0x248/0x3a0 ... value changed: 0x000000000010a1d0 -> 0x000000000010a9a0 The counter is purely advisory, so an occasional lost update is harmless, and every other stats[] update already runs under pool->lock. Annotate the update with data_race(). Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-07-01workqueue: dump the last woken worker for stalled poolsBreno Leitao
To identify the task most likely responsible for a stall, add last_woken_worker (L: pool->lock) to worker_pool and record it in kick_pool() just before wake_up_process(). This captures the idle worker that was kicked to take over when the last running worker went to sleep; if the pool is now stuck with no running worker, that task is the prime suspect and its backtrace is dumped by show_pool_no_running_worker(). Using struct worker * rather than struct task_struct * avoids any lifetime concern: workers are only destroyed via set_worker_dying() which requires pool->lock, and set_worker_dying() clears last_woken_worker when the dying worker matches. show_cpu_pool_busy_workers() holds pool->lock while calling sched_show_task(), so last_woken_worker is either NULL or points to a live worker with a valid task. More precisely, set_worker_dying() clears last_woken_worker before setting WORKER_DIE, so a non-NULL last_woken_worker means the kthread has not yet exited and worker->task is still alive. Suggested-by: Petr Mladek <pmladek@suse.com> Reviewed-by: Petr Mladek <pmladek@suse.com> Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-06-30workqueue: trigger a single-CPU backtrace for stalled poolsBreno Leitao
When a CPU pool is stalled with no running worker, the task occupying the CPU may not be a workqueue worker at all. Trigger a single-CPU backtrace for the stalled CPU to capture what it is currently executing. The CPU is snapshotted under pool->lock and the backtrace is triggered after releasing the lock to avoid any potential issues with NMI delivery. Skip the backtrace when the CPU is offline. A pool disassociated by CPU hotplug keeps its pool->cpu, and an NMI to an offline CPU is never acked, so nmi_trigger_cpumask_backtrace() would busy-wait for its full timeout in the watchdog's timer context. Suggested-by: Petr Mladek <pmladek@suse.com> Reviewed-by: Petr Mladek <pmladek@suse.com> Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-06-30workqueue: only show running workers in stall diagnosticsBreno Leitao
show_cpu_pool_busy_workers() dumps every in-flight worker in the pool's busy_hash, including workers that are not currently running on the CPU. Restore the task_is_running() filter so only running workers are dumped. When no running worker is found the pool may be stuck, unable to wake an idle worker to process pending work, and the watchdog would otherwise give no feedback. Add show_pool_no_running_worker() to report the pool id, CPU, idle state, and worker counts in that case. The pool info message is printed inside pool->lock using printk_deferred_enter/exit, the same pattern used by the existing busy-worker loop, to avoid deadlocks with console drivers that queue work while holding locks also taken in their write paths. This has been running on the Meta fleet for a while and caught some real issues, for instance EFI stalls stalling the workqueue [1]. Link: https://lore.kernel.org/all/20260616-efi_timeout-v3-0-76dd1d26657b@debian.org/ [1] Suggested-by: Petr Mladek <pmladek@suse.com> Fixes: 8823eaef45da7 ("workqueue: Show all busy workers in stall diagnostics") Reviewed-by: Petr Mladek <pmladek@suse.com> Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-06-29workqueue: defer the worker wakeup outside pool->lock in process_one_work()Breno Leitao
Use kick_pool_pick() to select and claim the worker under pool->lock and issue the wakeup with wake_up_process() after the lock is dropped. Unlike __queue_work(), this path has no surrounding RCU section, so take rcu_read_lock() before dropping pool->lock to keep the picked worker's task_struct valid across the wakeup. Signed-off-by: Breno Leitao <leitao@debian.org> Tested-by: Krishna Magar <kmagar@redhat.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-06-29workqueue: defer the worker wakeup outside pool->lock in __queue_work()Breno Leitao
__queue_work() is the enqueue hot path: it inserts the work item and calls kick_pool() while holding pool->lock. kick_pool() ends in a wakeup, which takes the target task's rq->lock, so rq->lock nests under pool->lock on every enqueue that wakes a worker on a contended unbound pool. Use kick_pool_pick() to select and claim the worker under pool->lock and issue the wakeup with wake_up_process() right after dropping the lock. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-06-29workqueue: split kick_pool() into kick_pool_pick()Breno Leitao
Factor the worker selection out of kick_pool() into kick_pool_pick(), which picks and claims the worker under pool->lock but, instead of waking it, returns the worker's task via an out-param so the caller can issue the wakeup after dropping pool->lock. BH kicks and wake_cpu setup still happen under the lock. kick_pool() becomes a thin wrapper that wakes the returned task, so all existing callers keep waking under pool->lock. Pure refactor, no functional change. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-06-17Merge tag 'wq-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wqLinus Torvalds
Pull workqueue updates from Tejun Heo: - Continued progress toward making alloc_workqueue() unbound by default: more callers converted to WQ_PERCPU / system_percpu_wq / system_dfl_wq, and new warnings for queues that use neither WQ_PERCPU nor WQ_UNBOUND or the legacy system_wq / system_unbound_wq. - Misc: drop the now-trivial apply_wqattrs_lock()/unlock() wrappers, forbid the TEST_WORKQUEUE benchmark from being built-in, and fix a spurious pointer level in the worker debug-dump path. * tag 'wq-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq: drm/bridge: anx7625: Add WQ_PERCPU add to alloc_workqueue wifi: ath6kl: fix invalid workqueue flags in ath6kl_usb_create() btrfs: Drop WQ_PERCPU from ordered_flags in btrfs_init_workqueues() workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is present workqueue: Add warnings and fallback if system_{unbound}_wq is used workqueue: drop spurious '*' from print_worker_info() fn declaration workqueue: forbid TEST_WORKQUEUE from being built-in workqueue: drop apply_wqattrs_lock()/unlock() wrappers umh: replace use of system_unbound_wq with system_dfl_wq rapidio: rio: add WQ_PERCPU to alloc_workqueue users media: ddbridge: add WQ_PERCPU to alloc_workqueue users platform: cznic: turris-omnia-mcu: replace use of system_wq with system_percpu_wq media: synopsys: hdmirx: replace use of system_unbound_wq with system_dfl_wq virt: acrn: Add WQ_PERCPU to alloc_workqueue users
2026-06-15Merge tag 'sched-core-2026-06-14' of ↵Linus Torvalds
gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip Pull scheduler updates from Ingo Molnar: "SMP load-balancing updates: - A large series to introduce infrastructure for cache-aware load balancing, with the goal of co-locating tasks that share data within the same Last Level Cache (LLC) domain. By improving cache locality, the scheduler can reduce cache bouncing and cache misses, ultimately improving data access efficiency. Implemented by Chen Yu and Tim Chen, based on early prototype work by Peter Zijlstra, with fixes by Jianyong Wu, Peter Zijlstra and Shrikanth Hegde. - A series to simplify CONFIG_SCHED_SMT ifdef usage (Shrikanth Hegde) Fair scheduler updates: - A series to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness (Andrea Righi, K Prateek Nayak) - A series to optimize cfs_rq and sched_entity allocation for better data locality (Zecheng Li) - A preparatory series to change fair/cgroup scheduling to a single runqueue, without the final change (Peter Zijlstra) - Auto-manage ext/fair dl_server bandwidth (Andrea Righi) - Fix cpu_util runnable_avg arithmetic (Hongyan Xia) - Optimize update_tg_load_avg()'s rate-limiting code (Rik van Riel) - Allow account_cfs_rq_runtime() to throttle current hierarchy (K Prateek Nayak) - Update util_est after updating util_avg during dequeue, to fix the util signal update logic, which reduces signal noise (Vincent Guittot) Scheduler topology updates: - Allow multiple domains to claim sched_domain_shared (K Prateek Nayak) - Add parameter to split LLC (Peter Zijlstra) Core scheduler updates: - Use trace_call__<tp>() to save a static branch (Gabriele Monaco) Scheduler statistics updates: - Drop now-stale mul_u64_u64_div_u64() cputime over-approximation guard (Nicolas Pitre) Deadline scheduler updates: - Reject debugfs dl_server writes for offline CPUs (Andrea Righi) - Fix replenishment logic for non-deferred servers (Yuri Andriaccio) RT scheduling updates: - Turn RT_PUSH_IPI default off for non PREEMPT_RT (Steven Rostedt) - Update default bandwidth for real-time tasks to 1.0 (Yuri Andriaccio) Proxy scheduling updates: - A series to implement Optimized Donor Migration for Proxy Execution (John Stultz, Peter Zijlstra) - Various proxy scheduling cleanups and fixes (Peter Zijlstra, K Prateek Nayak) Misc fixes, improvements and cleanups by Aaron Lu, Andrea Righi, Zenghui Yu, Chen Yu, Guanyou.Chen, John Stultz, Shrikanth Hegde, Peter Zijlstra, Liang Luo and Yiyang Chen" * tag 'sched-core-2026-06-14' of gitolite.kernel.org:pub/scm/linux/kernel/git/tip/tip: (91 commits) sched/fair: Fix newidle vs core-sched sched/deadline: Use task_on_rq_migrating() helper sched/core: Combine separate 'else' and 'if' statements sched/fair: Fix cpu_util runnable_avg arithmetic sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() sched/fair: Move the throttled tasks to a local list in tg_unthrottle_up() sched/fair: Call update_curr() before unthrottling the hierarchy sched/fair: Use throttled_csd_list for local unthrottle sched/fair: Convert cfs bandwidth throttling to use guards sched/fair: Allocate cfs_tg_state with percpu allocator sched/fair: Remove task_group->se pointer array sched/fair: Co-locate cfs_rq and sched_entity in cfs_tg_state sched: restore timer_slack_ns when resetting RT policy on fork MAINTAINERS: Fix spelling mistake in Peter's name sched: Simplify ttwu_runnable() sched/proxy: Remove superfluous clear_task_blocked_in() sched/proxy: Remove PROXY_WAKING sched/proxy: Switch proxy to use p->is_blocked sched/proxy: Only return migrate when needed sched: Be more strict about p->is_blocked ...
2026-05-29workqueue: Add warnings and ensure one among WQ_PERCPU or WQ_UNBOUND is presentMarco Crivellari
Currently there are no checks in order to enforce the use of one between WQ_PERCPU or WQ_UNBOUND. So act as following: - if neither of them is present, set WQ_PERCPU - if both are present, remove WQ_PERCPU Along with this change, WARN_ONCE(), so that the code still uses both or neither of them, can be changed. Link: https://lore.kernel.org/all/20250221112003.1dSuoGyc@linutronix.de/ Suggested-by: Tejun Heo <tj@kernel.org> Signed-off-by: Marco Crivellari <marco.crivellari@suse.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-05-29workqueue: Add warnings and fallback if system_{unbound}_wq is usedMarco Crivellari
Currently many users transitioned already to the new introduced workqueue (system_percpu_wq, system_dfl_wq), but there are new users who still use the older system_wq and system_unbound_wq. This change try to push this transition forward, by warning whether the old workqueues are used. Link: https://lore.kernel.org/all/20250221112003.1dSuoGyc@linutronix.de/ Suggested-by: Tejun Heo <tj@kernel.org> Signed-off-by: Marco Crivellari <marco.crivellari@suse.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-05-27workqueue: drop spurious '*' from print_worker_info() fn declarationBreno Leitao
print_worker_info() declares its local 'fn' as work_func_t * but worker->current_func has type work_func_t (a function pointer). The extra level of indirection is wrong and only happens to be harmless today because every supported Linux architecture has sizeof(work_func_t) == sizeof(work_func_t *): copy_from_kernel_nofault() reads the correct number of bytes by accident, and %ps still resolves the printed address because the stored value is the function address regardless of declared type. On any future ABI where sizeof(void (*)()) differs from sizeof(void *), the nofault copy would transfer the wrong number of bytes and the subsequent %ps would print an incorrect address. Match the field type so the intent is explicit and the code does not silently rely on equal pointer sizes. Fixes: 3d1cb2059d93 ("workqueue: include workqueue info when printing debug dump of a worker task") Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-05-19sched: Simplify ifdeffery around cpu_smt_maskShrikanth Hegde
Now, that cpu_smt_mask is defined as cpumask_of(cpu) for CONFIG_SCHED_SMT=n, it is possible to get rid of the ifdeffery. Effectively, - This makes sched_smt_present is defined always - cpumask_weight(cpumask_of(cpu)) == 1. So sched_smt_present_inc/dec will never enable the sched_smt_present. Which is expected. - Paths that were compile-time eliminated become runtime guarded using static keys. - Defines set_idle_cores, test_idle_cores, etc which could likely benefit the CONFIG_SCHED_SMT=n systems to use the same optimizations within the LLC at wakeups. - This will expose sched_smt_present symbol for CONFIG_SCHED_SMT=n. Likely not a concern. - There is a bloat of code CONFIG_SCHED_SMT=n. (NR_CPUS=2048) add/remove: 24/18 grow/shrink: 26/28 up/down: 6396/-3188 (3208) Total: Before=30629880, After=30633088, chg +0.01% - No code bloat for CONFIG_SCHED_SMT=y, which is expected. - Add comments around stop_core_cpuslocked on why ifdefs are not removed. - This leaves the remaining uses of CONFIG_SCHED_SMT mainly for topology building bits which has a policy based decision. Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Phil Auld <pauld@redhat.com> Reviewed-by: Valentin Schneider <vschneid@redhat.com> Acked-by: Tejun Heo <tj@kernel.org> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com> Link: https://patch.msgid.link/20260515172456.542799-3-sshegde@linux.ibm.com
2026-05-11workqueue: drop apply_wqattrs_lock()/unlock() wrappersBreno Leitao
The apply_wqattrs_lock()/unlock() helpers were introduced by commit a0111cf6710b ("workqueue: separate out and refactor the locking of applying attrs") to encapsulate the get_online_cpus() (later cpus_read_lock()) + mutex_lock(&wq_pool_mutex) acquire pair that was duplicated across the apply-attrs paths. Since commit 19af45757383 ("workqueue: Remove cpus_read_lock() from apply_wqattrs_lock()") removed the cpus_read_lock() (pwq creation and installation now operate on wq_online_cpumask, so CPU hotplug no longer needs to be excluded), the wrappers have been one-line forwarders to mutex_lock(&wq_pool_mutex)/mutex_unlock(&wq_pool_mutex). They no longer encode any non-trivial locking rule and obscure the fact that callers just take the existing wq_pool_mutex. This align with the "unnecessary" helpers that got discussed in [1] Inline the eight call sites and remove the wrappers. No functional change. Link: https://lore.kernel.org/all/afs_44-6ToJJVZTn@gmail.com/ [1] Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-05-08workqueue: Fix wq->cpu_pwq leak in alloc_and_link_pwqs() WQ_UNBOUND pathBreno Leitao
For WQ_UNBOUND workqueues, alloc_and_link_pwqs() allocates wq->cpu_pwq via alloc_percpu() and then calls apply_workqueue_attrs_locked(). On failure it returns the error directly, bypassing the enomem: label which holds the only free_percpu(wq->cpu_pwq) in this function. The caller's error path kfree()s wq without touching wq->cpu_pwq, leaking one percpu pointer table (nr_cpu_ids * sizeof(void *) bytes) per failed call. If kmemleak is enabled, we can see: unreferenced object (percpu) 0xc0fffa5b121048 (size 8): comm "insmod", pid 776, jiffies 4294682844 backtrace (crc 0): pcpu_alloc_noprof+0x665/0xac0 __alloc_workqueue+0x33f/0xa20 alloc_workqueue_noprof+0x60/0x100 Route the error through the existing enomem: cleanup and any error before this one. Cc: stable@kernel.org Fixes: 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-05-08workqueue: Release PENDING in __queue_work() drain/destroy reject pathBreno Leitao
The caller of __queue_work() owns WORK_STRUCT_PENDING, won via test_and_set_bit() in queue_work_on()/__queue_delayed_work(). The state machine documented above __queue_work() requires that owner to either hand the token to a pwq (insert_work() -> set_work_pwq()), hand it to a timer, or release it via set_work_pool_and_clear_pending(). try_to_grab_pending() relies on this: when it observes "PENDING && off-queue" it busy-loops, trusting the current owner to make progress. The (__WQ_DESTROYING | __WQ_DRAINING) early-return path violates that contract. It WARN_ONCE()s and bare-returns, leaving work->data with PENDING set, WORK_STRUCT_PWQ clear, and work->entry empty. The path is reachable without explicit API abuse: queue_delayed_work() arms a timer with PENDING set; if drain_workqueue() runs while the timer is still pending, delayed_work_timer_fn() -> __queue_work() in softirq context hits the WARN, current is not a wq worker so is_chained_work() is false, and the work is silently dropped with PENDING leaked. Mirror what clear_pending_if_disabled() already does on its analogous reject path: unpack the off-queue data and call set_work_pool_and_clear_pending() to release the token before returning. I was able to reproduce this by queueing several slow works on a max_active=1 wq, arm a delayed_work whose timer fires while drain_workqueue() is blocked, then call cancel_delayed_work_sync(). Without this patch the cancel livelocks at 100% CPU; with it the cancel returns immediately. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-04-29workqueue: Annotate alloc_workqueue_va() with __printf(1, 0)Tejun Heo
alloc_workqueue_va() forwards its va_list to __alloc_workqueue() which ultimately feeds vsnprintf(). __alloc_workqueue() already carries __printf(1, 0); the new wrapper needs the same annotation so format string checking propagates through the forwarding. Fixes: 0de4cb473aed ("workqueue: fix devm_alloc_workqueue() va_list misuse") Reported-by: kernel test robot <lkp@intel.com> Closes: https://lore.kernel.org/oe-kbuild-all/202604300347.2LgXyteh-lkp@intel.com/ Signed-off-by: Tejun Heo <tj@kernel.org>
2026-04-28workqueue: fix devm_alloc_workqueue() va_list misuseBreno Leitao
devm_alloc_workqueue() built a va_list and passed it as a single positional argument to the variadic alloc_workqueue() macro: va_start(args, max_active); wq = alloc_workqueue(fmt, flags, max_active, args); va_end(args); C does not allow forwarding a va_list through a ... parameter. alloc_workqueue() expands to alloc_workqueue_noprof(), which runs its own va_start() over its ... params, so the inner vsnprintf(wq->name, sizeof(wq->name), fmt, args) in __alloc_workqueue() received the outer va_list object as the first variadic slot rather than the caller's actual format arguments. Add a new static helper alloc_workqueue_va() that wraps __alloc_workqueue() and runs wq_init_lockdep() on success, and fold both alloc_workqueue_noprof() and devm_alloc_workqueue_noprof() onto it as suggested by Tejun. The wq_init_lockdep() step is required on the devm path too, otherwise __flush_workqueue()'s on-stack COMPLETION_INITIALIZER_ONSTACK_MAP would NULL-deref wq->lockdep_map. No caller changes are required. devm_alloc_ordered_workqueue() is a macro forwarding to devm_alloc_workqueue() and inherits the fix. Two in-tree callers actively trigger the broken path on every probe: drivers/power/supply/mt6370-charger.c:889 drivers/power/supply/max77705_charger.c:649 both of which use devm_alloc_ordered_workqueue(dev, "%s", 0, dev_name(dev)). A standalone reproducer module is available at[1]. Link: https://github.com/leitao/debug/blob/main/workqueue/valist/wq_va_test.c [1] Fixes: 1dfc9d60a69e ("workqueue: devres: Add device-managed allocate workqueue") Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-04-15Merge tag 'wq-for-7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wqLinus Torvalds
Pull workqueue updates from Tejun Heo: - New default WQ_AFFN_CACHE_SHARD affinity scope subdivides LLCs into smaller shards to improve scalability on machines with many CPUs per LLC - Misc: - system_dfl_long_wq for long unbound works - devm_alloc_workqueue() for device-managed allocation - sysfs exposure for ordered workqueues and the EFI workqueue - removal of HK_TYPE_WQ from wq_unbound_cpumask - various small fixes * tag 'wq-for-7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq: (21 commits) workqueue: validate cpumask_first() result in llc_populate_cpu_shard_id() workqueue: use NR_STD_WORKER_POOLS instead of hardcoded value workqueue: avoid unguarded 64-bit division docs: workqueue: document WQ_AFFN_CACHE_SHARD affinity scope workqueue: add test_workqueue benchmark module tools/workqueue: add CACHE_SHARD support to wq_dump.py workqueue: set WQ_AFFN_CACHE_SHARD as the default affinity scope workqueue: add WQ_AFFN_CACHE_SHARD affinity scope workqueue: fix typo in WQ_AFFN_SMT comment workqueue: Remove HK_TYPE_WQ from affecting wq_unbound_cpumask workqueue: unlink pwqs from wq->pwqs list in alloc_and_link_pwqs() error path workqueue: Remove NULL wq WARN in __queue_delayed_work() workqueue: fix parse_affn_scope() prefix matching bug workqueue: devres: Add device-managed allocate workqueue workqueue: Add system_dfl_long_wq for long unbound works tools/workqueue/wq_dump.py: add NODE prefix to all node columns tools/workqueue/wq_dump.py: fix column alignment in node_nr/max_active section tools/workqueue/wq_dump.py: remove backslash separator from node_nr/max_active header efi: Allow to expose the workqueue via sysfs workqueue: Allow to expose ordered workqueues via sysfs ...
2026-04-13workqueue: validate cpumask_first() result in llc_populate_cpu_shard_id()Breno Leitao
On uniprocessor (UP) configs such as nios2, NR_CPUS is 1, so cpu_shard_id[] is a single-element array (int[1]). In llc_populate_cpu_shard_id(), cpumask_first(sibling_cpus) returns an unsigned int that the compiler cannot prove is always 0, triggering a -Warray-bounds warning when the result is used to index cpu_shard_id[]: kernel/workqueue.c:8321:55: warning: array subscript 1 is above array bounds of 'int[1]' [-Warray-bounds] 8321 | cpu_shard_id[c] = cpu_shard_id[cpumask_first(sibling_cpus)]; | ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is a false positive: sibling_cpus can never be empty here because 'c' itself is always set in it, so cpumask_first() will always return a valid CPU. However, the compiler cannot prove this statically, and the warning only manifests on UP configs where the array size is 1. Add a bounds check with WARN_ON_ONCE to silence the warning, and store the result in a local variable to make the code clearer and avoid calling cpumask_first() twice. Fixes: 5920d046f7ae ("workqueue: add WQ_AFFN_CACHE_SHARD affinity scope") Reported-by: kernel test robot <lkp@intel.com> Closes: https://lore.kernel.org/oe-kbuild-all/202604022343.GQtkF2vO-lkp@intel.com/ Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-04-07workqueue: use NR_STD_WORKER_POOLS instead of hardcoded valueManinder Singh
use NR_STD_WORKER_POOLS for irq_work_fns[] array definition. NR_STD_WORKER_POOLS is also 2, but better to use MACRO. Initialization loop for_each_bh_worker_pool() also uses same MACRO. Signed-off-by: Maninder Singh <maninder1.s@samsung.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-04-01workqueue: set WQ_AFFN_CACHE_SHARD as the default affinity scopeBreno Leitao
Set WQ_AFFN_CACHE_SHARD as the default affinity scope for unbound workqueues. On systems where many CPUs share one LLC, the previous default (WQ_AFFN_CACHE) collapses all CPUs to a single worker pool, causing heavy spinlock contention on pool->lock. WQ_AFFN_CACHE_SHARD subdivides each LLC into smaller groups, providing a better balance between locality and contention. Users can revert to the previous behavior with workqueue.default_affinity_scope=cache. On systems with 8 or fewer cores per LLC, CACHE_SHARD produces a single shard covering the entire LLC, making it functionally identical to the previous CACHE default. The sharding only activates when an LLC has more than 8 cores. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-04-01workqueue: add WQ_AFFN_CACHE_SHARD affinity scopeBreno Leitao
On systems where many CPUs share one LLC, unbound workqueues using WQ_AFFN_CACHE collapse to a single worker pool, causing heavy spinlock contention on pool->lock. For example, Chuck Lever measured 39% of cycles lost to native_queued_spin_lock_slowpath on a 12-core shared-L3 NFS-over-RDMA system. The existing affinity hierarchy (cpu, smt, cache, numa, system) offers no intermediate option between per-LLC and per-SMT-core granularity. Add WQ_AFFN_CACHE_SHARD, which subdivides each LLC into groups of at most wq_cache_shard_size cores (default 8, tunable via boot parameter). Shards are always split on core (SMT group) boundaries so that Hyper-Threading siblings are never placed in different pods. Cores are distributed across shards as evenly as possible -- for example, 36 cores in a single LLC with max shard size 8 produces 5 shards of 8+7+7+7+7 cores. The implementation follows the same comparator pattern as other affinity scopes: precompute_cache_shard_ids() pre-fills the cpu_shard_id[] array from the already-initialized WQ_AFFN_CACHE and WQ_AFFN_SMT topology, and cpus_share_cache_shard() is passed to init_pod_type(). Benchmark on NVIDIA Grace (72 CPUs, single LLC, 50k items/thread), show cache_shard delivers ~5x the throughput and ~6.5x lower p50 latency compared to cache scope on this 72-core single-LLC system. Suggested-by: Tejun Heo <tj@kernel.org> Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-04-01workqueue: Add pool_workqueue to pending_pwqs list when unplugging multiple ↵Matthew Brost
inactive works In unplug_oldest_pwq(), the first inactive work item on the pool_workqueue is activated correctly. However, if multiple inactive works exist on the same pool_workqueue, subsequent works fail to activate because wq_node_nr_active.pending_pwqs is empty — the list insertion is skipped when the pool_workqueue is plugged. Fix this by checking for additional inactive works in unplug_oldest_pwq() and updating wq_node_nr_active.pending_pwqs accordingly. Fixes: 4c065dbce1e8 ("workqueue: Enable unbound cpumask update on ordered workqueues") Cc: stable@vger.kernel.org Cc: Carlos Santa <carlos.santa@intel.com> Cc: Ryan Neph <ryanneph@google.com> Cc: Lai Jiangshan <jiangshanlai@gmail.com> Cc: Waiman Long <longman@redhat.com> Cc: linux-kernel@vger.kernel.org Signed-off-by: Matthew Brost <matthew.brost@intel.com> Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Waiman Long <longman@redhat.com>
2026-03-31workqueue: Remove HK_TYPE_WQ from affecting wq_unbound_cpumaskWaiman Long
For historical reason, wq_unbound_cpumask is initially set as intersection of HK_TYPE_DOMAIN, HK_TYPE_WQ and workqueue.unbound_cpus boot command line option. At run time, users can update the unbound cpumask via the /sys/devices/virtual/workqueue/cpumask sysfs file. Creation and modification of cpuset isolated partitions will also update wq_unbound_cpumask based on the latest HK_TYPE_DOMAIN cpumask. The HK_TYPE_WQ cpumask is out of the picture with these runtime updates. Complete the transition by taking HK_TYPE_WQ out from the workqueue code and make it depends on HK_TYPE_DOMAIN only from the housekeeping side. The final goal is to eliminate HK_TYPE_WQ as a housekeeping cpumask type. Signed-off-by: Waiman Long <longman@redhat.com> Acked-by: Frederic Weisbecker <frederic@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-25workqueue: unlink pwqs from wq->pwqs list in alloc_and_link_pwqs() error pathBreno Leitao
When alloc_and_link_pwqs() fails partway through the per-cpu allocation loop, some pool_workqueues may have already been linked into wq->pwqs via link_pwq(). The error path frees these pwqs with kmem_cache_free() but never removes them from the wq->pwqs list, leaving dangling pointers in the list. Currently this is not exploitable because the workqueue was never added to the global workqueues list and the caller frees the wq immediately after. However, this makes sure that alloc_and_link_pwqs() doesn't leave any half-baked structure, which may have side effects if not properly cleaned up. Fix this by unlinking each pwq from wq->pwqs before freeing it. No locking is needed as the workqueue has not been published yet, thus no concurrency is possible. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-25workqueue: Better describe stall checkPetr Mladek
Try to be more explicit why the workqueue watchdog does not take pool->lock by default. Spin locks are full memory barriers which delay anything. Obviously, they would primary delay operations on the related worker pools. Explain why it is enough to prevent the false positive by re-checking the timestamp under the pool->lock. Finally, make it clear what would be the alternative solution in __queue_work() which is a hotter path. Signed-off-by: Petr Mladek <pmladek@suse.com> Acked-by: Song Liu <song@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-21workqueue: Fix false positive stall reportsSong Liu
On weakly ordered architectures (e.g., arm64), the lockless check in wq_watchdog_timer_fn() can observe a reordering between the worklist insertion and the last_progress_ts update. Specifically, the watchdog can see a non-empty worklist (from a list_add) while reading a stale last_progress_ts value, causing a false positive stall report. This was confirmed by reading pool->last_progress_ts again after holding pool->lock in wq_watchdog_timer_fn(): workqueue watchdog: pool 7 false positive detected! lockless_ts=4784580465 locked_ts=4785033728 diff=453263ms worklist_empty=0 To avoid slowing down the hot path (queue_work, etc.), recheck last_progress_ts with pool->lock held. This will eliminate the false positive with minimal overhead. Remove two extra empty lines in wq_watchdog_timer_fn() as we are on it. Fixes: 82607adcf9cd ("workqueue: implement lockup detector") Cc: stable@vger.kernel.org # v4.5+ Assisted-by: claude-code:claude-opus-4-6 Signed-off-by: Song Liu <song@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-17workqueue: Remove NULL wq WARN in __queue_delayed_work()Tejun Heo
Remove the WARN_ON_ONCE(!wq) which doesn't serve any useful purpose. Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-13workqueue: fix parse_affn_scope() prefix matching bugBreno Leitao
parse_affn_scope() uses strncasecmp() with the length of the candidate name, which means it only checks if the input *starts with* a known scope name. Given that the upcoming diff will create "cache_shard" affinity scope, writing "cache_shard" to a workqueue's affinity_scope sysfs attribute always matches "cache" first, making it impossible to select "cache_shard" via sysfs, so, this fix enable it to distinguish "cache" and "cache_shard" Fix by replacing the hand-rolled prefix matching loop with sysfs_match_string(), which uses sysfs_streq() for exact matching (modulo trailing newlines). Also add the missing const qualifier to the wq_affn_names[] array declaration. Note that sysfs_streq() is case-sensitive, unlike the previous strncasecmp() approach. This is intentional and consistent with how other sysfs attributes handle string matching in the kernel. Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-10Merge branch 'for-7.1-devm-alloc-wq' into for-7.1Tejun Heo
2026-03-10workqueue: devres: Add device-managed allocate workqueueKrzysztof Kozlowski
Add a Resource-managed version of alloc_workqueue() to fix common problem of drivers mixing devm() calls with destroy_workqueue. Such naive and discouraged driver approach leads to difficult to debug bugs when the driver: 1. Allocates workqueue in standard way and destroys it in driver remove() callback, 2. Sets work struct with devm_work_autocancel(), 3. Registers interrupt handler with devm_request_threaded_irq(). Which leads to following unbind/removal path: 1. destroy_workqueue() via driver remove(), Any interrupt coming now would still execute the interrupt handler, which queues work on destroyed workqueue. 2. devm_irq_release(), 3. devm_work_drop() -> cancel_work_sync() on destroyed workqueue. devm_alloc_workqueue() has two benefits: 1. Solves above problem of mix-and-match devres and non-devres code in driver, 2. Simplify any sane drivers which were correctly using alloc_workqueue() + devm_add_action_or_reset(). Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com> Acked-by: Tejun Heo <tj@kernel.org> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-09workqueue: Add system_dfl_long_wq for long unbound worksMarco Crivellari
Currently there are users of queue_delayed_work() who specify system_long_wq, the per-cpu workqueue. This workqueue should be used for long per-cpu works, but queue_delayed_work() queue the work using: queue_delayed_work_on(WORK_CPU_UNBOUND, ...); This would end up calling __queue_delayed_work() that does: if (housekeeping_enabled(HK_TYPE_TIMER)) { // [....] } else { if (likely(cpu == WORK_CPU_UNBOUND)) add_timer_global(timer); else add_timer_on(timer, cpu); } So when cpu == WORK_CPU_UNBOUND the timer is global and is not using a specific CPU. Later, when __queue_work() is called: if (req_cpu == WORK_CPU_UNBOUND) { if (wq->flags & WQ_UNBOUND) cpu = wq_select_unbound_cpu(raw_smp_processor_id()); else cpu = raw_smp_processor_id(); } Because the wq is not unbound, it takes the CPU where the timer fired and enqueue the work on that CPU. The consequence of all of this is that the work can run anywhere, depending on where the timer fired. Introduce system_dfl_long_wq in order to change, in a future step, users that are still calling: queue_delayed_work(system_long_wq, ...); with the new system_dfl_long_wq instead, so that the work may benefit from scheduler task placement. Signed-off-by: Marco Crivellari <marco.crivellari@suse.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-06workqueue: Rename show_cpu_pool{s,}_hog{s,}() to reflect broadened scopeBreno Leitao
show_cpu_pool_hog() and show_cpu_pools_hogs() no longer only dump CPU hogs — since commit 8823eaef45da ("workqueue: Show all busy workers in stall diagnostics"), they dump every in-flight worker in the pool's busy_hash. Rename them to show_cpu_pool_busy_workers() and show_cpu_pools_busy_workers() to accurately describe what they do. Also fix the pr_info() message to say "stalled worker pools" instead of "stalled CPU-bound worker pools", since sleeping/blocked workers are now included. No functional change. Suggested-by: Tejun Heo <tj@kernel.org> Signed-off-by: Breno Leitao <leitao@debian.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-05workqueue: Show all busy workers in stall diagnosticsBreno Leitao
show_cpu_pool_hog() only prints workers whose task is currently running on the CPU (task_is_running()). This misses workers that are busy processing a work item but are sleeping or blocked — for example, a worker that clears PF_WQ_WORKER and enters wait_event_idle(). Such a worker still occupies a pool slot and prevents progress, yet produces an empty backtrace section in the watchdog output. This is happening on real arm64 systems, where toggle_allocation_gate() IPIs every single CPU in the machine (which lacks NMI), causing workqueue stalls that show empty backtraces because toggle_allocation_gate() is sleeping in wait_event_idle(). Remove the task_is_running() filter so every in-flight worker in the pool's busy_hash is dumped. The busy_hash is protected by pool->lock, which is already held. Signed-off-by: Breno Leitao <leitao@debian.org> Acked-by: Song Liu <song@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-05workqueue: Show in-flight work item duration in stall diagnosticsBreno Leitao
When diagnosing workqueue stalls, knowing how long each in-flight work item has been executing is valuable. Add a current_start timestamp (jiffies) to struct worker, set it when a work item begins execution in process_one_work(), and print the elapsed wall-clock time in show_pwq(). Unlike current_at (which tracks CPU runtime and resets on wakeup for CPU-intensive detection), current_start is never reset because the diagnostic cares about total wall-clock time including sleeps. Before: in-flight: 165:stall_work_fn [wq_stall] After: in-flight: 165:stall_work_fn [wq_stall] for 100s Signed-off-by: Breno Leitao <leitao@debian.org> Acked-by: Song Liu <song@kernel.org> Signed-off-by: Tejun Heo <tj@kernel.org>