diff options
Diffstat (limited to 'Documentation/accounting')
| -rw-r--r-- | Documentation/accounting/delay-accounting.rst | 113 | ||||
| -rw-r--r-- | Documentation/accounting/taskstats.rst | 54 |
2 files changed, 125 insertions, 42 deletions
diff --git a/Documentation/accounting/delay-accounting.rst b/Documentation/accounting/delay-accounting.rst index 210c194d4a7b..e209c46241b0 100644 --- a/Documentation/accounting/delay-accounting.rst +++ b/Documentation/accounting/delay-accounting.rst @@ -107,22 +107,22 @@ Get sum and peak of delays, since system boot, for all pids with tgid 242:: TGID 242 - CPU count real total virtual total delay total delay average delay max delay min - 39 156000000 156576579 2111069 0.054ms 0.212296ms 0.031307ms - IO count delay total delay average delay max delay min - 0 0 0.000ms 0.000000ms 0.000000ms - SWAP count delay total delay average delay max delay min - 0 0 0.000ms 0.000000ms 0.000000ms - RECLAIM count delay total delay average delay max delay min - 0 0 0.000ms 0.000000ms 0.000000ms - THRASHING count delay total delay average delay max delay min - 0 0 0.000ms 0.000000ms 0.000000ms - COMPACT count delay total delay average delay max delay min - 0 0 0.000ms 0.000000ms 0.000000ms - WPCOPY count delay total delay average delay max delay min - 156 11215873 0.072ms 0.207403ms 0.033913ms - IRQ count delay total delay average delay max delay min - 0 0 0.000ms 0.000000ms 0.000000ms + CPU count real total virtual total delay total delay average delay max delay min delay max timestamp + 46 188000000 192348334 4098012 0.089ms 0.429260ms 0.051205ms 2026-01-15T15:06:58 + IO count delay total delay average delay max delay min delay max timestamp + 0 0 0.000ms 0.000000ms 0.000000ms N/A + SWAP count delay total delay average delay max delay min delay max timestamp + 0 0 0.000ms 0.000000ms 0.000000ms N/A + RECLAIM count delay total delay average delay max delay min delay max timestamp + 0 0 0.000ms 0.000000ms 0.000000ms N/A + THRASHING count delay total delay average delay max delay min delay max timestamp + 0 0 0.000ms 0.000000ms 0.000000ms N/A + COMPACT count delay total delay average delay max delay min delay max timestamp + 0 0 0.000ms 0.000000ms 0.000000ms N/A + WPCOPY count delay total delay average delay max delay min delay max timestamp + 182 19413338 0.107ms 0.547353ms 0.022462ms 2026-01-15T15:05:24 + IRQ count delay total delay average delay max delay min delay max timestamp + 0 0 0.000ms 0.000000ms 0.000000ms N/A Get IO accounting for pid 1, it works only with -p:: @@ -131,3 +131,84 @@ Get IO accounting for pid 1, it works only with -p:: linuxrc: read=65536, write=0, cancelled_write=0 The above command can be used with -v to get more debug information. + +After the system starts, use `delaytop` to get the system-wide delay information, +which includes system-wide PSI information and Top-N high-latency tasks. +Note: PSI support requires `CONFIG_PSI=y` and `psi=1` for full functionality. + +`delaytop` is an interactive tool for monitoring system pressure and task delays. +It supports multiple sorting options, display modes, and real-time keyboard controls. + +Basic usage with default settings (sorts by CPU delay, shows top 20 tasks, refreshes every 2 seconds):: + + bash# ./delaytop + System Pressure Information: (avg10/avg60vg300/total) + CPU some: 0.0%/ 0.0%/ 0.0%/ 106137(ms) + CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) + Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) + Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) + IO full: 0.0%/ 0.0%/ 0.0%/ 2240(ms) + IO some: 0.0%/ 0.0%/ 0.0%/ 2783(ms) + IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) + [o]sort [M]memverbose [q]quit + Top 20 processes (sorted by cpu delay): + PID TGID COMMAND CPU(ms) IO(ms) IRQ(ms) MEM(ms) + ------------------------------------------------------------------------ + 110 110 kworker/15:0H-s 27.91 0.00 0.00 0.00 + 57 57 cpuhp/7 3.18 0.00 0.00 0.00 + 99 99 cpuhp/14 2.97 0.00 0.00 0.00 + 51 51 cpuhp/6 0.90 0.00 0.00 0.00 + 44 44 kworker/4:0H-sy 0.80 0.00 0.00 0.00 + 60 60 ksoftirqd/7 0.74 0.00 0.00 0.00 + 76 76 idle_inject/10 0.31 0.00 0.00 0.00 + 100 100 idle_inject/14 0.30 0.00 0.00 0.00 + 1309 1309 systemsettings 0.29 0.00 0.00 0.00 + 45 45 cpuhp/5 0.22 0.00 0.00 0.00 + 63 63 cpuhp/8 0.20 0.00 0.00 0.00 + 87 87 cpuhp/12 0.18 0.00 0.00 0.00 + 93 93 cpuhp/13 0.17 0.00 0.00 0.00 + 1265 1265 acpid 0.17 0.00 0.00 0.00 + 1552 1552 sshd 0.17 0.00 0.00 0.00 + 2584 2584 sddm-helper 0.16 0.00 0.00 0.00 + 1284 1284 rtkit-daemon 0.15 0.00 0.00 0.00 + 1326 1326 nde-netfilter 0.14 0.00 0.00 0.00 + 27 27 cpuhp/2 0.13 0.00 0.00 0.00 + 631 631 kworker/11:2-rc 0.11 0.00 0.00 0.00 + +Interactive keyboard controls during runtime:: + + o - Select sort field (CPU, IO, IRQ, Memory, etc.) + M - Toggle display mode (Default/Memory Verbose) + q - Quit + +Available sort fields(use -s/--sort or interactive command):: + + cpu(c) - CPU delay + blkio(i) - I/O delay + irq(q) - IRQ delay + mem(m) - Total memory delay + swapin(s) - Swapin delay (memory verbose mode only) + freepages(r) - Freepages reclaim delay (memory verbose mode only) + thrashing(t) - Thrashing delay (memory verbose mode only) + compact(p) - Compaction delay (memory verbose mode only) + wpcopy(w) - Write page copy delay (memory verbose mode only) + +Advanced usage examples:: + + # ./delaytop -s blkio + Sorted by IO delay + + # ./delaytop -s mem -M + Sorted by memory delay in memory verbose mode + + # ./delaytop -p pid + Print delayacct stats + + # ./delaytop -P num + Display the top N tasks + + # ./delaytop -n num + Set delaytop refresh frequency (num times) + + # ./delaytop -d secs + Specify refresh interval as secs diff --git a/Documentation/accounting/taskstats.rst b/Documentation/accounting/taskstats.rst index 2a28b7f55c10..173c1e7bf5ef 100644 --- a/Documentation/accounting/taskstats.rst +++ b/Documentation/accounting/taskstats.rst @@ -76,41 +76,43 @@ The messages are in the format:: The taskstats payload is one of the following three kinds: 1. Commands: Sent from user to kernel. Commands to get data on -a pid/tgid consist of one attribute, of type TASKSTATS_CMD_ATTR_PID/TGID, -containing a u32 pid or tgid in the attribute payload. The pid/tgid denotes -the task/process for which userspace wants statistics. - -Commands to register/deregister interest in exit data from a set of cpus -consist of one attribute, of type -TASKSTATS_CMD_ATTR_REGISTER/DEREGISTER_CPUMASK and contain a cpumask in the -attribute payload. The cpumask is specified as an ascii string of -comma-separated cpu ranges e.g. to listen to exit data from cpus 1,2,3,5,7,8 -the cpumask would be "1-3,5,7-8". If userspace forgets to deregister interest -in cpus before closing the listening socket, the kernel cleans up its interest -set over time. However, for the sake of efficiency, an explicit deregistration -is advisable. + a pid/tgid consist of one attribute, of type TASKSTATS_CMD_ATTR_PID/TGID, + containing a u32 pid or tgid in the attribute payload. The pid/tgid denotes + the task/process for which userspace wants statistics. + + Commands to register/deregister interest in exit data from a set of cpus + consist of one attribute, of type + TASKSTATS_CMD_ATTR_REGISTER/DEREGISTER_CPUMASK and contain a cpumask in the + attribute payload. The cpumask is specified as an ascii string of + comma-separated cpu ranges e.g. to listen to exit data from cpus 1,2,3,5,7,8 + the cpumask would be "1-3,5,7-8". If userspace forgets to deregister + interest in cpus before closing the listening socket, the kernel cleans up + its interest set over time. However, for the sake of efficiency, an explicit + deregistration is advisable. 2. Response for a command: sent from the kernel in response to a userspace -command. The payload is a series of three attributes of type: + command. The payload is a series of three attributes of type: -a) TASKSTATS_TYPE_AGGR_PID/TGID : attribute containing no payload but indicates -a pid/tgid will be followed by some stats. + a) TASKSTATS_TYPE_AGGR_PID/TGID: attribute containing no payload but + indicates a pid/tgid will be followed by some stats. -b) TASKSTATS_TYPE_PID/TGID: attribute whose payload is the pid/tgid whose stats -are being returned. + b) TASKSTATS_TYPE_PID/TGID: attribute whose payload is the pid/tgid whose + stats are being returned. -c) TASKSTATS_TYPE_STATS: attribute with a struct taskstats as payload. The -same structure is used for both per-pid and per-tgid stats. + c) TASKSTATS_TYPE_STATS: attribute with a struct taskstats as payload. The + same structure is used for both per-pid and per-tgid stats. 3. New message sent by kernel whenever a task exits. The payload consists of a series of attributes of the following type: -a) TASKSTATS_TYPE_AGGR_PID: indicates next two attributes will be pid+stats -b) TASKSTATS_TYPE_PID: contains exiting task's pid -c) TASKSTATS_TYPE_STATS: contains the exiting task's per-pid stats -d) TASKSTATS_TYPE_AGGR_TGID: indicates next two attributes will be tgid+stats -e) TASKSTATS_TYPE_TGID: contains tgid of process to which task belongs -f) TASKSTATS_TYPE_STATS: contains the per-tgid stats for exiting task's process + a) TASKSTATS_TYPE_AGGR_PID: indicates next two attributes will be pid+stats + b) TASKSTATS_TYPE_PID: contains exiting task's pid + c) TASKSTATS_TYPE_STATS: contains the exiting task's per-pid stats + d) TASKSTATS_TYPE_AGGR_TGID: indicates next two attributes will be + tgid+stats + e) TASKSTATS_TYPE_TGID: contains tgid of process to which task belongs + f) TASKSTATS_TYPE_STATS: contains the per-tgid stats for exiting task's + process per-tgid stats |