| Age | Commit message (Collapse) | Author |
|---|
|
IPIs can now be monitored and accounted by osnoise top. When that is
the case, also record them when saving a trace file.
To match what is being recorded by the tool for its own analysis, event
filters are applied to the events recorded to the trace output.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Link: https://lore.kernel.org/r/20260902123942.695822-9-vschneid@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
In rare situations, sysfs might report a different number of CPUs than
the kernel sees, for example, when /sys is not real sysfs.
In tracefs mode, RTLA will discard any samples that are reported on CPUs
higher than the maximum in sysfs. However, in BPF mode, the incorrectly
sized BPF_MAP_LOOKUP_ELEM syscall on percpu maps will cause a buffer
overflow.
To cover for this corner case, write "all" to osnoise/cpus, and read
back the real kernel-side list of all possible cpus. If it doesn't match
RTLA's nr_cpus (read from sysfs), abort with an error.
Assisted-by: Claude:claude-opus-4-6
Link: https://lore.kernel.org/r/20260814135511.2207190-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
rtla uses get_nprocs_conf(), a standard libc routine, to retrieve the
total number of CPUs for the purpose of processing per-CPU data.
In some configurations, libc might return a different value than what is
seen by the kernel. To cover for this situation, replace the call to
get_nprocs_conf() with a newly added helper, get_possible_cpus().
get_possible_cpus() reads /sys/devices/system/cpu/possible and parses
the CPU list, identically to what libbpf_num_possible_cpus() does.
Additionally, it checks that the possible cpu list is zero-based and has
no holes by computing both the cpu count and the maximum cpu number.
Systems where /sys/.../cpu/possible is unreadable, is not zero-based,
or has holes, now report an error instead of crashing later due to unmet
assumptions.
Note that parse_cpu_set() cannot be used to parse
/sys/devices/system/cpu/possible, because it only supports CPU count of
1024 and lower. Higher CPU counts are not fully supported by rtla;
nr_cpus > 1024 is, though, so it has to be covered by the new
implementation in order to avoid a regression.
A new function, cpu_list_iterate(), is added to utils.c to contain the
cpu list parsing logic, which is now shared between the new function
get_possible_cpus() and the pre-existing parse_cpu_set().
Assisted-by: Claude:claude-opus-4-6
Link: https://lore.kernel.org/r/20260814135511.2207190-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Compiling RTLA 7.1.x with GCC 16 and uClibc as standard library fails
with these errors:
src/common.c: In function ‘set_signals’:
src/common.c:40:17: error: implicit declaration of function ‘alarm’ [-Wimplicit-function-declaration]
40 | alarm(params->duration);
| ^~~~~
src/common.c: In function ‘common_apply_config’:
src/common.c:187:44: error: implicit declaration of function ‘getpid’; did you mean ‘getpt’? [-Wimplicit-function-declaration]
187 | retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
| ^~~~~~
| getpt
In file included from src/common.c:9:
src/common.c: In function ‘run_tool’:
src/common.c:262:19: error: implicit declaration of function ‘sysconf’; did you mean ‘sscanf’? [-Wimplicit-function-declaration]
262 | nr_cpus = get_nprocs_conf();
| ^~~~~~~~~~~~~~~
src/common.c:262:19: error: ‘_SC_NPROCESSORS_CONF’ undeclared (first use in this function)
262 | nr_cpus = get_nprocs_conf();
| ^~~~~~~~~~~~~~~
src/common.c:262:19: note: each undeclared identifier is reported only once for each function it appears in
src/common.c:370:17: error: implicit declaration of function ‘sleep’ [-Wimplicit-function-declaration]
370 | sleep(1);
| ^~~~~
Restore the missing unistd.h include.
Fixes: 115b06a00875 ("tools/rtla: Consolidate nr_cpus usage across all tools")
Signed-off-by: Andreas Ziegler <br025@umbiko.net>
Link: https://lore.kernel.org/r/20260624033342.129970-1-br025@umbiko.net
[ capital letter in patch subject, remove EOL after Fixes ]
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Instead of using getopt_long() directly to parse the command line
arguments given to an RTLA tool, use libsubcmd's parse_options().
Utilizing libsubcmd for parsing command line arguments has several
benefits:
- A help message is automatically generated by libsubcmd from the
specification, removing the need of writing it by hand.
- Options are sorted into groups based on which part of tracing (CPU,
thread, auto-analysis, tuning, histogram) they relate to.
- Common parsing patterns for numerical and boolean values now share
code, with the target variable being stored in the option array.
To avoid duplication of the option parsing logic, RTLA-specific
macros defining struct option values are created:
- RTLA_OPT_* for options common to all tools
- OSNOISE_OPT_* and TIMERLAT_OPT_* for options specific to
osnoise/timerlat tools
- HIST_OPT_* macros for options specific to histogram-based tools.
Individual *_parse_args() functions then construct an array out of
these macros that is then passed to libsubcmd's parse_options().
All code specific to command line options parsing is moved out of the
individual tool files into a new file, cli.c, which also contains the
contents of the rtla.c file. A private header, cli_p.h, is added
alongside the public header cli.h, so that unit tests are able to test
statically declared option callbacks.
Minor changes:
- The return value of tool-level help option changes to 129, as this is
the value set by libsubcmd; this is reflected in affected test cases.
The implementation of help for command-level and tracer-level help
is set to 129 as well for consistency, and the change is reflected in
exit value documentation.
- Related to the above, {rtla,osnoise,timerlat}_usage() are marked
__noreturn and exit() is removed from after they are called for
cleaner code.
- The error messages for invalid argument for options --dma-latency and
-E/--entries were corrected, fixing off-by-one in the limits.
Note that unsetting options (using --no-<opt> syntax) is currently not
implemented for options that use custom callbacks. For --irq and
--thread, it will never be implemented, as they conflict with already
existing --no-irq and --no-thread with a different meaning.
Assisted-by: Composer:composer-1.5
Link: https://lore.kernel.org/r/20260528103254.2990068-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Before, when rtla got a signal, it stopped the main trace but not the
record trace. With "--on-end trace", this can lead to
save_trace_to_file() failing to keep up, especially on a debug kernel.
Plus, it adds post-stoppage noise to the trace file.
Signed-off-by: Crystal Wood <crwood@redhat.com>
Fixes: c73cab9dbed0 ("rtla/timerlat_hist: Stop timerlat tracer on signal")
Fixes: a4dfce7559d7 ("rtla/timerlat_top: Stop timerlat tracer on signal")
Fixes: 3aadb65db5d6 ("rtla/timerlat: Add action on end feature")
Link: https://lore.kernel.org/r/20260512173731.2151841-1-crwood@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Detach stop_trace() from SIGINT/SIGALRM on tool clean-up to prevent it
from crashing RTLA by accessing freed memory.
This prevents a crash when multiple SIGINTs are received.
Fixes: d6899e560366 ("rtla/timerlat_hist: Abort event processing on second signal")
Fixes: 80967b354a76 ("rtla/timerlat_top: Abort event processing on second signal")
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260310160725.144443-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Add proper error handling when pthread_create() fails to create the
timerlat user-space dispatcher thread. Previously, the code only logged
an error message but continued execution, which could lead to undefined
behavior when the tool later expects the thread to be running.
When pthread_create() returns an error, the function now jumps to the
out_trace error path to properly clean up resources and exit. This
ensures consistent error handling and prevents the tool from running
in an invalid state without the required user-space thread.
Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260309195040.1019085-10-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Several functions duplicate the logic for handling threshold actions.
When a threshold is reached, these functions stop the trace, perform
configured actions, and restart the trace if --on-threshold continue
is set.
Create common_threshold_handler() to centralize this shared logic and
avoid code duplication. The function executes the configured threshold
actions and restarts the necessary trace instances when appropriate.
Also add should_continue_tracing() helper to encapsulate the check
for whether tracing should continue after a threshold event, improving
code readability at call sites.
In timerlat_top_bpf_main_loop(), use common_params directly instead
of casting through timerlat_params when only common fields are needed.
Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260309195040.1019085-5-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
sysconf(_SC_NPROCESSORS_CONF) (via get_nprocs_conf) reflects
cpu_possible_mask, which is fixed at boot time, so querying it
repeatedly is unnecessary.
Replace multiple calls to sysconf(_SC_NPROCESSORS_CONF) with a single
global nr_cpus variable initialized once at startup.
`#pragma once` in timerlat_u.h is needed for pre-C23 compilers to avoid
redefinition errors.
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20260306194953.2511960-2-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
getopt_long() processes short and long options independently.
RTLA, like the majority of applications, uses both short and long
variants for each logical option.
Since the val member of struct option holds the letter of the short
variant, the string of short options can be reconstructed from the
array of long options.
Add getopt_auto() to generate optstring from an array of long options,
eliminating the need to maintain separate short option strings.
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260108095011.2115719-1-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The stop_tracing global variable is accessed from both the signal
handler context and the main program flow without synchronization.
This creates a potential race condition where compiler optimizations
could cache the variable value in registers, preventing the signal
handler's updates from being visible to other parts of the program.
Add the volatile qualifier to stop_tracing in both common.c and
common.h to ensure all accesses to this variable bypass compiler
optimizations and read directly from memory. This guarantees that
when the signal handler sets stop_tracing, the change is immediately
visible to the main program loop, preventing potential hangs or
delayed shutdown when termination signals are received.
Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260106133655.249887-16-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Each rtla tool duplicates parsing of -H/--house-keeping.
Migrate the option parsing from individual tools to the
common_parse_options().
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251209100047.2692515-8-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Each rtla tool duplicates parsing of -P/--priority.
Migrate the option parsing from individual tools to the
common_parse_options().
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251209100047.2692515-7-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Each rtla tool duplicates parsing of -e/--event.
Migrate the option parsing from individual tools to the
common_parse_options().
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251209100047.2692515-6-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Each rtla tool duplicates parsing of -d/--duration.
Migrate the option parsing from individual tools to the
common_parse_options().
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251209100047.2692515-5-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Each rtla tool duplicates parsing of -D/--debug.
Migrate the option parsing from individual tools to the
common_parse_options().
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251209100047.2692515-4-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Each rtla tool duplicates parsing of -C/--cgroup.
Migrate the option parsing from individual tools to the
common_parse_options().
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251209100047.2692515-3-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Each rtla tool duplicates parsing of -c/--cpus.
Migrate the option parsing from individual tools to the
common_parse_options().
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251209100047.2692515-2-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Each rtla tool duplicates parsing of many common options. This creates
maintenance overhead and risks inconsistencies when updating these
options.
Add common_parse_options() to centralize parsing of options used across
all tools.
Common options to be migrated in future patches.
Changes since v1:
- restore opterr
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251209100047.2692515-1-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The rtla tools have significant code quadruplication in their usage
functions. Each tool implements its own version of the same help text
formatting and option descriptions, leading to maintenance overhead and
inconsistencies. Documentation/tools/rtla/common_options.rst lists 14
common options.
Add common_usage() infrastructure to consolidate help formatting.
Subsequent patches will extend this to handle other common options.
The refactored output is almost identical to the original, with the
following changes:
- add square brackets to specify optionality: `usage: [rtla] ...`
- remove `-q` from timerlat hist because hist tools don't support it
- minor spacing
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251124063204.845425-1-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
This avoids startup races where one of the instances hit a threshold
before all instances were enabled, and thus tracing stops without
the relevant event. In particular, this is not uncommon with the
tests that set a very tight threshold and then complain if there's
no analysis.
This also ensures that we don't stop tracing during a warmup.
The downside is a small chance of having an event over the threshold
early in the output, without stopping on it, which could cause user
confusion. This should be less likely if the warmup feature is used, but
that doesn't eliminate the race window, just the odds of an unusual spike
right at that moment.
Signed-off-by: Crystal Wood <crwood@redhat.com>
Link: https://lore.kernel.org/r/20251112152529.956778-6-crwood@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Commit 8d933d5c89e8 ("rtla/timerlat: Add continue action") moved the
code performing on-threshold actions (enabled through --on-threshold
option) to inside the RTLA main loop.
The condition in the loop does not check whether the threshold was
actually exceeded or if stop tracing was requested by the user through
SIGINT or duration. This leads to a bug where on-threshold actions are
always performed, even when the threshold was not hit.
(BPF mode is not affected, since it uses a different condition in the
while loop.)
Add a condition that checks for !stop_tracing before executing the
actions. Also, fix incorrect brackets in hist_main_loop to match the
semantics of top_main_loop.
Fixes: 8d933d5c89e8 ("rtla/timerlat: Add continue action")
Fixes: 2f3172f9dd58 ("tools/rtla: Consolidate code between osnoise/timerlat and hist/top")
Reviewed-by: Crystal Wood <crwood@redhat.com>
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20251007095341.186923-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Currently a lot of code is duplicated between the different rtla tools,
making maintenance more difficult, and encouraging divergence such as
features that are only implemented for certain tools even though they
could be more broadly applicable.
Merge the various main() functions into a common run_tool() with an ops
struct for tool-specific details.
Implement enough support for actions on osnoise to not need to keep the
old params->trace_output path.
Cc: John Kacur <jkacur@redhat.com>
Cc: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/20250907022325.243930-5-crwood@redhat.com
Reviewed-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Crystal Wood <crwood@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
Merge the common bits of osnoise_apply_config() and
timerlat_apply_config(). Put the result in a new common.c, and move
enough things to common.h so that common.c does not need to include
osnoise.h.
Cc: John Kacur <jkacur@redhat.com>
Cc: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/20250907022325.243930-4-crwood@redhat.com
Reviewed-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Crystal Wood <crwood@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
