workqueue: Make flush_workqueue() visit only pwqs active since the last flush

636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu
pool_workqueues") fixed unbound workqueue scalability on large machines but
made flush_workqueue() walk one pwq per possible CPU, cycling each pool
lock, up to three times per flush. 85f0d8e39aff ("workqueue: Reduce
expensive locks for unbound workqueue") coalesced same-pool locks for
unbound workqueues but the walk remains. Yao Kai reported the XFS CIL
workqueue, flushed on every log force, spending up to 64us per flush in the
walk on a 128-CPU machine.

Idle pwqs can't just be skipped: the lock-and-advance of every pwq's work
color is what keeps a concurrently queued work item from being stamped with
a retired color unseen by the flusher, which would let a later flush return
before it finishes.

Track active pwqs instead. Each workqueue gets a wq_flush_pnode per node
with a lock, a mirror of wq->work_color and a list of pwqs. Queueing to an
off-list pwq syncs pwq->work_color from the mirror and adds it under
fpn->lock, and flush_workqueue_prep_pwqs() advances the mirror and splices
the list in one fpn->lock section per node before visiting the pwqs under
pool->lock as before. That replaces the fence: a racing queueing either gets
its pwq on the list before the splice or stamps the advanced color. A pwq
stays on the list until a visit finds nothing in flight, so a cascade arming
an older color still finds it, barriers need no separate add as the work
item they follow keeps the pwq on the list, and pwq_release_workfn() removes
a released pwq under wq->mutex.

On a 192-CPU 2-node machine, flushing an idle per-cpu workqueue goes from
40k to 4.2M per second and an idle unbound one from 450k to 4.3M. 16 threads
each queueing a work item and flushing, go from 30k to 110k flushes per
second on a per-cpu workqueue and 60k to 180k on an unbound one. Dense
flushes with every pwq active are unchanged on per-cpu workqueues and 15-20%
slower on unbound ones. The queue path is unchanged.

Reported-by: Yao Kai <yaokai34@huawei.com>
Link: https://lore.kernel.org/all/0a030145-c108-4365-ba2d-ac1973a1e352@huawei.com/
Signed-off-by: Tejun Heo <tj@kernel.org>

0 files changed, 0 insertions, 0 deletions