Merge patch series "files: make closing files synchronous for close_range(), exec, exit"

Christian Brauner <brauner@kernel.org> says:

For tasks exiting with a large file descriptor table this means getting
rid of _a lot_ of cmpxchg()s and hammering on task->pi_lock. The cost of
this was mentioned in various threads over the years. I found at least
recent comments by Oleg and Neil.

The obvious problematic case are kernel threads. fput() punts their
final __fput() to a workqueue. kthreads never return to userspace to run
task work and some of them may need to finish a umount and so cannot
->release() inline. No path we change is taken by a kthread. They don't
own a descriptor table and kthreadd is created with CLONE_FILES and
every kthread inherits it. So all of them share init_files and init_task
pins that forever. Their exit_files() never drops the last reference.

kernel_execve() refuses PF_KTHREAD outright. Usermodehelpers are spawned
without the kthread flag. close_range() is a syscall. The one indirect
way is a failing copy_process() calling exit_files() on a
child from a kthread parent. A dup_fd() in copy_process() cannot hold
the last reference to any of its files while the source table is alive.

Worker threads such as io_uring workers are user threads without
PF_KTHREAD that share the fdtable. One of them can be the last to put
the fdtable at exit. Today fput() hands their final __fput()s to task
work. The worker runs that itself in exit_task_work() a few lines later.
Now the same thread does the same work in exit_files() instead. vhost
workers are created without a descriptor table at all, so they never
even put an fdtable. Neither kind ever execs or calls close_range().

Now the performance numbers. I measured this on a 64 vCPU KVM guest (AMD
EPYC 9754) on top of vfs-7.4.coredump with and without the series and
the same x86_64_defconfig for both. The benchmark opens N files in a
forked worker, moves them to random positions in the descriptor table so
that the table order has nothing to do with the allocation order, and
times the teardown:

- process exit
- execve() with N close-on-exec descriptors
- close_range() with and without CLOSE_RANGE_UNSHARE

Here are the medians over two boots which agree within 2%. With
/dev/null as the file and 4096, 65536 and 1M descriptors:

                         4096 fds        65536 fds        1M fds
  ----------------------------------------------------------------------
  exit                   1045 -> 895 us  18.9 -> 14.9 ms  429 -> 306 ms
                          -14.4%         -21.2%           -28.7%
  ----------------------------------------------------------------------
  execve, close-on-exec  1393 -> 1190 us 21.4 -> 16.9 ms  471 -> 325 ms
                          -14.6%         -21.0%           -31.1%
  ----------------------------------------------------------------------
  close_range()          1179 -> 988 us  21.9 -> 17.4 ms  484 -> 337 ms
                          -16.1%         -20.6%           -30.3%
  ----------------------------------------------------------------------
  close_range(UNSHARE)   1118 -> 958 us  22.1 -> 17.8 ms  487 -> 333 ms
                          -14.3%         -19.8%           -31.7%

So that's a 16-21% win or 40-150 ns per closed file. This sheds
init_task_work() and task_work_add() per file with its cmpxchg() and the
TIF_NOTIFY_RESUME test-and-set, the list walk with the indirect call
in the path out, and then a second pass over every struct file that
isn't cache-hot anymore by then.

The performance boost holds from 256 to 1M descriptors and is largest
at 1M. Note the thread count doesn't matter since one thread does the
close anyway.

So with eight processes tearing down 65536 descriptors each at the same
time the win is 16-21% (34.9 -> 29.5 ms for exit, 34.4 -> 27.0 ms
for execve). The old code walks the table once to queue the task work
and then walks all 16 MB of struct file a second time from
task_work_run().

So once eight CPUs compete for the cache that second pass gets
very expensive. With 32 or 64 processes at once both kernels are bound
by pushing millions of files through slab and RCU. So then the
difference shrinks to 4-8%. Still though...

Files whose own release dominates the cleanup gain a little less. 13%
for eventfds, 11% for pipes, 6% for AF_UNIX sockets and 7% for distinct
tmpfs inodes. For them a release costs about 2 us per descriptor anyway.

Using function profiling a 65536 descriptor exit spends 4.6 ms in
exit_files() and 11.1 ms in task_work_run() before and now 13.7 ms in
exit_files() and nothing in task_work_run() after.

close(2) is untouched and measures the same. will-it-scale (open1,
open3, dup1, eventfd1, unix1, pipe1, signal1, processes and threads)
is within 2% either way. dup1_threads at 16 tasks and getppid1 at 64
tasks a few percent better which I'd put down to layout.

* patches from https://patch.msgid.link/20260910-work-coredump-unlock-self-v4-0-a5c1800dc930@kernel.org:
  coredump: factor out coredump_wait_inactive()
  coredump: replace the startup completion with a thread count
  sched: add wait_var_event_state()
  coredump: drop core_state->dumper
  fs: make close_cloexec_files() synchronous
  fs: rename do_close_on_exec() to close_cloexec_files()
  fs: make close_range() synchronous
  fs: make close_files() synchronous
  fs: add filp_close_sync()
  fs: remove unshare_files()
  fs: move unshare_fd() to fs/file.c
  fs: add switch_files_struct()
  fs: don't open-code file_close_fd() in close_fd()

Link: https://patch.msgid.link/20260910-work-coredump-unlock-self-v4-0-a5c1800dc930@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>

0 files changed, 0 insertions, 0 deletions