diff options
| author | Jiawei Sun <abyssmystery@gmail.com> | 2026-07-07 23:38:00 +0800 |
|---|---|---|
| committer | Namhyung Kim <namhyung@kernel.org> | 2026-07-07 22:59:07 -0700 |
| commit | f94563fac26912ef5a51fd16ae1d83f17b24b19d (patch) | |
| tree | 64fa746df2ac8be201ca9eb788c7a13f2855db9e /tools/lib | |
| parent | b3665131e7a63b3ed2d91015a6db97c5f65a5328 (diff) | |
| download | linux-next-f94563fac26912ef5a51fd16ae1d83f17b24b19d.tar.gz linux-next-f94563fac26912ef5a51fd16ae1d83f17b24b19d.zip | |
perf record: fix poll storm when monitored threads exit
When `perf record` samples a multi-threaded process and one of the
target threads exits during the session, perf itself may start burning
100% CPU (up to 200% across two cores) until the session ends. A
single dead fd is sufficient to trigger this; it can be reproduced with
15 pthreads in a compute loop where one thread exits halfway through.
The root cause is two independent instances of the same defect: dead
perf_event ring-buffer fds are left in a pollfd array. When a monitored
thread exits, the kernel closes its ring-buffer fd, which then returns
POLLHUP. POSIX specifies that poll() always reports POLLHUP and POLLERR
regardless of the events mask, so any dead fd left in the array makes
poll() return immediately every time, spinning in a tight loop:
3 seconds: 256,600 poll() calls, 0 context switches, only 21 write()
Woken up count goes from ~0 to 1,300,000+
There are two affected poll paths, fixed together here:
1. Record main loop, via fdarray__filter() (tools/lib/api/fd/array.c).
Since commit 59b4412f27f1 ("libperf: Avoid internal moving of
fdarray fds") it only zeroes events/revents without setting fd to
-1, so poll() keeps reporting POLLHUP for the entry. Setting
fd = -1 makes poll() skip it, matching the pattern already used in
the control-fd path at tools/perf/builtin-record.c:1673.
2. BPF sideband thread, perf_evlist__poll_thread()
(tools/perf/util/sideband_evlist.c). This thread polls for
PERF_RECORD_BPF_EVENT but, unlike the main record loop, never calls
fdarray__filter() at all, so dead fds accumulate forever and it
spins at 100% CPU:
Before fix: dJiffies=101, wchan=0 (running)
After fix: dJiffies=0, wchan=do_sys_poll (blocking)
Fixed by calling the existing evlist__filter_pollfd() helper after
evlist__poll(), mirroring the main record loop. <poll.h> is
included for the POLLERR/POLLHUP macros (previously unused there).
The two fixes compose: fix 1 makes poll() ignore dead fds (fd=-1); fix
2 ensures the sideband thread actually performs the filtering. Both
paths are affected in all kernels from v5.1/v5.9 to the current master
(7.2-rc1); the source of both functions is byte-identical across them.
BPF event recording is preserved: after the fix, perf.data still
contains PERF_RECORD_BPF_EVENT records and bpf_prog_info entries.
Verified on perf 6.1.76, 6.6.143 and 7.2-rc1 with a minimal reproducer
(Woken up 1,300,000 -> 3, CPU 100% -> 0%) and an A/B orthogonal test:
keeping the unpatched binary but preventing the target thread from
exiting also makes the storm disappear, confirming the trigger.
Fixes: 59b4412f27f1 ("libperf: Avoid internal moving of fdarray fds")
Fixes: 657ee5531903 ("perf evlist: Introduce side band thread")
Signed-off-by: Jiawei Sun <abyssmystery@gmail.com>
Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Diffstat (limited to 'tools/lib')
| -rw-r--r-- | tools/lib/api/fd/array.c | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/tools/lib/api/fd/array.c b/tools/lib/api/fd/array.c index f0f195207fca..ffe8272af59b 100644 --- a/tools/lib/api/fd/array.c +++ b/tools/lib/api/fd/array.c @@ -122,6 +122,12 @@ int fdarray__filter(struct fdarray *fda, short revents, if (entry_destructor) entry_destructor(fda, fd, arg); + /* + * Set fd to -1 so poll() ignores this entry; otherwise + * POLLHUP/POLLERR are still reported for events=0 fds + * (POSIX: always checked), causing a poll storm. + */ + fda->entries[fd].fd = -1; fda->entries[fd].revents = fda->entries[fd].events = 0; continue; } |
