Merge patch series "files: make closing files synchronous for close_range(), exec, exit"

Christian Brauner <brauner@kernel.org> says:

For tasks exiting with a large file descriptor table this means getting
rid of _a lot_ of cmpxchg()s and hammering on task->pi_lock. The cost of
this was mentioned in various threads over the years. I found at least
recent comments by Oleg and Neil.

The obvious problematic case are kernel threads. fput() punts their
final __fput() to a workqueue. kthreads never return to userspace to run
task work and some of them may need to finish a umount and so cannot
->release() inline. No path we change is taken by a kthread. They don't
own a descriptor table and kthreadd is created with CLONE_FILES and
every kthread inherits it. So all of them share init_files and init_task
pins that forever. Their exit_files() never drops the last reference.

kernel_execve() refuses PF_KTHREAD outright. Usermodehelpers are spawned
without the kthread flag. close_range() is a syscall. The one indirect
way is a failing copy_process() calling exit_files() on a
child from a kthread parent. A dup_fd() in copy_process() cannot hold
the last reference to any of its files while the source table is alive.

Worker threads such as io_uring workers are user threads without
PF_KTHREAD that share the fdtable. One of them can be the last to put
the fdtable at exit. Today fput() hands their final __fput()s to task
work. The worker runs that itself in exit_task_work() a few lines later.
Now the same thread does the same work in exit_files() instead. vhost
workers are created without a descriptor table at all, so they never
even put an fdtable. Neither kind ever execs or calls close_range().

Now the performance numbers. I measured this on a 64 vCPU KVM guest (AMD
EPYC 9754) on top of vfs-7.4.coredump with and without the series and
the same x86_64_defconfig for both. The benchmark opens N files in a
forked worker, moves them to random positions in the descriptor table so
that the table order has nothing to do with the allocation order, and
times the teardown:

- process exit
- execve() with N close-on-exec descriptors
- close_range() with and without CLOSE_RANGE_UNSHARE

Here are the medians over two boots which agree within 2%. With
/dev/null as the file and 4096, 65536 and 1M descriptors:

                         4096 fds        65536 fds        1M fds
  ----------------------------------------------------------------------
  exit                   1045 -> 895 us  18.9 -> 14.9 ms  429 -> 306 ms
                          -14.4%         -21.2%           -28.7%
  ----------------------------------------------------------------------
  execve, close-on-exec  1393 -> 1190 us 21.4 -> 16.9 ms  471 -> 325 ms
                          -14.6%         -21.0%           -31.1%
  ----------------------------------------------------------------------
  close_range()          1179 -> 988 us  21.9 -> 17.4 ms  484 -> 337 ms
                          -16.1%         -20.6%           -30.3%
  ----------------------------------------------------------------------
  close_range(UNSHARE)   1118 -> 958 us  22.1 -> 17.8 ms  487 -> 333 ms
                          -14.3%         -19.8%           -31.7%

So that's a 16-21% win or 40-150 ns per closed file. This sheds
init_task_work() and task_work_add() per file with its cmpxchg() and the
TIF_NOTIFY_RESUME test-and-set, the list walk with the indirect call
in the path out, and then a second pass over every struct file that
isn't cache-hot anymore by then.

The performance boost holds from 256 to 1M descriptors and is largest
at 1M. Note the thread count doesn't matter since one thread does the
close anyway.

So with eight processes tearing down 65536 descriptors each at the same
time the win is 16-21% (34.9 -> 29.5 ms for exit, 34.4 -> 27.0 ms
for execve). The old code walks the table once to queue the task work
and then walks all 16 MB of struct file a second time from
task_work_run().

So once eight CPUs compete for the cache that second pass gets
very expensive. With 32 or 64 processes at once both kernels are bound
by pushing millions of files through slab and RCU. So then the
difference shrinks to 4-8%. Still though...

Files whose own release dominates the cleanup gain a little less. 13%
for eventfds, 11% for pipes, 6% for AF_UNIX sockets and 7% for distinct
tmpfs inodes. For them a release costs about 2 us per descriptor anyway.

Using function profiling a 65536 descriptor exit spends 4.6 ms in
exit_files() and 11.1 ms in task_work_run() before and now 13.7 ms in
exit_files() and nothing in task_work_run() after.

close(2) is untouched and measures the same. will-it-scale (open1,
open3, dup1, eventfd1, unix1, pipe1, signal1, processes and threads)
is within 2% either way. dup1_threads at 16 tasks and getppid1 at 64
tasks a few percent better which I'd put down to layout.

* patches from https://patch.msgid.link/20260910-work-coredump-unlock-self-v4-0-a5c1800dc930@kernel.org:
  coredump: factor out coredump_wait_inactive()
  coredump: replace the startup completion with a thread count
  sched: add wait_var_event_state()
  coredump: drop core_state->dumper
  fs: make close_cloexec_files() synchronous
  fs: rename do_close_on_exec() to close_cloexec_files()
  fs: make close_range() synchronous
  fs: make close_files() synchronous
  fs: add filp_close_sync()
  fs: remove unshare_files()
  fs: move unshare_fd() to fs/file.c
  fs: add switch_files_struct()
  fs: don't open-code file_close_fd() in close_fd()

Link: https://patch.msgid.link/20260910-work-coredump-unlock-self-v4-0-a5c1800dc930@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>

3 files changed, 32 insertions, 5 deletions

diff --git a/include/linux/fdtable.h b/include/linux/fdtable.h
index afdfaad381df..8c28fdd15f31 100644
--- a/include/linux/fdtable.h
+++ b/include/linux/fdtable.h
@@ -100,7 +100,8 @@ static inline bool close_on_exec(unsigned int fd, const struct files_struct *fil
 struct task_struct;
 
 void put_files_struct(struct files_struct *fs);
-int unshare_files(void);
+void switch_files_struct(struct task_struct *tsk, struct files_struct *files);
+int unshare_fd(unsigned long unshare_flags, struct files_struct **new_fdp);
 enum fd_range_flags {
 	/* Leave behind all descriptors outside of the specified range. */
 	FD_RANGE_EXCEPT		= (1U << 0),
@@ -114,7 +115,7 @@ struct fd_range {
 	enum fd_range_flags flags;
 };
 struct files_struct *dup_fd(struct files_struct *, struct fd_range *) __latent_entropy;
-void do_close_on_exec(struct files_struct *);
+void close_cloexec_files(struct files_struct *);
 int iterate_fd(struct files_struct *, unsigned,
 		int (*)(const void *, struct file *, unsigned),
 		const void *);
diff --git a/include/linux/sched/signal.h b/include/linux/sched/signal.h
index d45a5476b97d..e039e29cd8c5 100644
--- a/include/linux/sched/signal.h
+++ b/include/linux/sched/signal.h
@@ -79,9 +79,9 @@ struct core_thread {
 };
 
 struct core_state {
-	atomic_t nr_threads;
-	struct core_thread dumper;
-	struct completion startup;
+	/* Threads the dumper still waits for. */
+	atomic_t threads_remaining;
+	struct core_thread *tasks;
 };
 
 /*
diff --git a/include/linux/wait_bit.h b/include/linux/wait_bit.h
index 553d7b23e3ad..af077ed4caf6 100644
--- a/include/linux/wait_bit.h
+++ b/include/linux/wait_bit.h
@@ -433,6 +433,32 @@ do {									\
 })
 
 /**
+ * wait_var_event_state - wait for a variable to be updated and notified
+ * @var: the address of variable being waited on
+ * @condition: the condition to wait for
+ * @state: the task state to sleep in, %TASK_UNINTERRUPTIBLE etc.
+ *
+ * Wait for a @condition to be true, only re-checking when a wake up is
+ * received for the given @var (an arbitrary kernel address which need
+ * not be directly related to the given condition, but usually is).
+ *
+ * Returns 0 if the condition became true, or %-ERESTARTSYS if a signal
+ * arrived which @state allows to interrupt.
+ *
+ * The condition should normally use smp_load_acquire() or a similarly
+ * ordered access to ensure that any changes to memory made before the
+ * condition became true will be visible after the wait completes.
+ */
+#define wait_var_event_state(var, condition, state)					\
+({											\
+	int __ret = 0;									\
+	might_sleep();									\
+	if (!(condition))								\
+		__ret = ___wait_var_event(var, condition, (state), 0, 0, schedule());	\
+	__ret;										\
+})
+
+/**
  * wait_var_event_any_lock - wait for a variable to be updated under a lock
  * @var: the address of the variable being waited on
  * @condition: condition to wait for