diff options
| author | Shakeel Butt <shakeel.butt@linux.dev> | 2026-09-09 21:54:06 -0700 |
|---|---|---|
| committer | Christian Brauner <brauner@kernel.org> | 2026-09-17 11:39:35 +0200 |
| commit | 23985e2b1f31623468244a2149b2388cda4925eb (patch) | |
| tree | f1945a2867c8d59d213cc661894b192b40116df7 /tools/lib/python | |
| parent | 5ad03af184342c760a83a18dba32043f7cec24c6 (diff) | |
| download | linux-next-23985e2b1f31623468244a2149b2388cda4925eb.tar.gz linux-next-23985e2b1f31623468244a2149b2388cda4925eb.zip | |
kernfs: don't hold kernfs_rwsem across notification delivery
Recently at Meta we noticed that on multi-tenant systems, a workload
under extreme memory pressure negatively impacts unrelated workloads and
system daemons like below [1] and fb-oomd [2]. Mainly we observed that
system daemons were stuck on kernfs_rwsem in cgroup-related interfaces
for long periods of time.
On further inspection, we found that the workload under pressure had
registered inotify watches on its memory.events and was continuously
receiving limit-hitting notifications. The cgroup notification is done in
kernfs_notify_workfn(), which holds kernfs_supers_rwsem and kernfs_rwsem
and calls fsnotify(), which allocates the event with
kmalloc(GFP_KERNEL_ACCOUNT|__GFP_RETRY_MAYFAIL) charged to the watching
cgroup, which is already under memory pressure. So the notify worker
ended up in memory reclaim of a cgroup already under pressure, and itself
kept triggering the limit notifications.
This notification path takes the locks in read mode, but a waiting writer
blocks all future readers. That is exactly what is happening in the Meta
fleet, and it leaves daemons that are critical to the reliability of the
system stuck for long periods of time.
Commit 400188ae361a ("kernfs: Acquire kernfs_rwsem in
kernfs_notify_workfn().") added kernfs_rwsem just to safely read
kernfs_node::name. Let's sample the name once before the loop and drop
the lock. kernfs_supers_rwsem still covers the list, and the removal
paths take that one for reading too, so a stalled worker can still hold
a removal up once a mount or unmount is queued behind it.
Two things change for a watcher. A name longer than NAME_MAX now gives
an event with no name, where before it gave one with the full name;
fsnotify() takes the name as optional, so the event still arrives. And
the name and the parent are no longer sampled under one lock, so a rename
between the two would name a file against the directory it moved to.
That cannot happen today: kernfs_notify() rejects anything that is not a
file, and no caller of kernfs_rename_ns() renames one.
[1] https://github.com/facebookincubator/below
[2] https://github.com/facebookincubator/oomd
Fixes: 400188ae361a ("kernfs: Acquire kernfs_rwsem in kernfs_notify_workfn().")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Link: https://patch.msgid.link/20260910045406.485295-1-shakeel.butt@linux.dev
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Diffstat (limited to 'tools/lib/python')
0 files changed, 0 insertions, 0 deletions
