| Age | Commit message (Collapse) | Author |
|---|
|
Each osnoise tracer option (in /sys/kernel/tracing/osnoise) used by RTLA
requires four functions to be defined:
- static osnoise_get_<opt>() - to get the current value of the option
and save it into struct osnoise_context's orig_<opt> field,
- osnoise_set_<opt>() - to set the value of the option requested by the
user after reading and saving the original with osnoise_get_<opt>(),
and save it into <opt> field of struct osnoise_context,
- osnoise_restore_<opt>() - restore the value recorded in orig_<opt>,
- static osnoise_put_<opt>() - restore the value recorded in orig_<opt>
and update <opt> to reflect that.
The logic is duplicated for all the options, except for cpus (which is
the only string option) and period/runtime (which are handled together
and feature extra checks).
Deduplicate by moving the common code in get/set/restore/put into common
helper functions local to osnoise.c. The original set/restore/put
functions are reduced to wrappers generated using a set of X macros:
- OSNOISE_LL_OPTIONS, which invokes the OSNOISE_LL_OPTION macro for all
"long long" options,
- OSNOISE_FLAG_OPTIONS, which invokes the OSNOISE_FLAG_OPTION macro for
all flag options (boolean values in osnoise/options file).
Those are also used to auto-generate the corresponding fields in struct
osnoise_context, plus initialization and restoration code for them.
There are a few minor changes in the refactored code:
- osnoise_set_<opt>() now distinguishes between -2 (option cannot be
set) and -1 (option not present in kernel) for all options.
- OSNOISE_OPTION_INIT_VAL and OSNOISE_TIME_INIT_VAL are removed and
replaced with a plain -1; all unsigned long long fields are changed to
signed long long, consistent with pre-existing "%lld" format string in
osnoise_write_ll_config().
- osnoise_restore_<opt>() now correctly resets the <opt> field (not
orig_<opt>) for all options.
The change is intended to simplify adding and modifying tracer options.
It also makes the code shorter by a bit over 500 lines.
Assisted-by: Claude:claude-opus-4-6
Link: https://lore.kernel.org/r/20260929142944.1841462-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Give the user more control over the test execution when the test scripts
are run directly (not through prove), in order to do things like run a
particular test repeatedly to diagnose an intermittent failure.
If $SUBTESTS is provided, then only tests whose number is on the list will
run. For example:
# SUBTESTS="14 133 4 $(seq 98 103) 9" bash ./tests/timerlat.t
1..150
ok 4 - top verify -s/--stack
ok 9 - hist set the automatic trace mode
ok 14 - top disable auto-analysis
ok 98 - hist verify -H/--house-keeping
ok 99 - top verify -k/--kernel-threads
ok 100 - hist verify -k/--kernel-threads
ok 101 - top verify -u/--user-threads
ok 102 - hist verify -u/--user-threads
ok 103 - apply default timerlat_period_us
ok 133 - top trace output through --on-threshold trace
If $STOPONFAIL is set, then the script will exit with a non-zero exit code
on the first failure. If the output directory is non-empty, it will be
preserved, and the location shown.
Signed-off-by: Crystal Wood <crwood@redhat.com>
Link: https://lore.kernel.org/r/20260919062756.819701-4-crwood@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Make investigating test failures a little bit easier by not having
to recreate the command line by looking in the test script.
Unfortunately, this doesn't capture variables set outside of the
command line, but it's better than nothing.
Signed-off-by: Crystal Wood <crwood@redhat.com>
Link: https://lore.kernel.org/r/20260919062756.819701-3-crwood@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
When running the tests directly instead of from the makefile, RTLA
is likely to be unset. The default "./rtla" no longer works, due to
having changed the current diretory.
Fixes: ad5b50a0959fb ("rtla/tests: Run runtime tests in temporary directory")
Signed-off-by: Crystal Wood <crwood@redhat.com>
Link: https://lore.kernel.org/r/20260919062756.819701-2-crwood@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
IPIs can now be monitored and accounted by osnoise top. When that is
the case, also record them when saving a trace file.
To match what is being recorded by the tool for its own analysis, event
filters are applied to the events recorded to the trace output.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Link: https://lore.kernel.org/r/20260902123942.695822-9-vschneid@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
A later commit will apply a filter to events recorded to the trace
output. To prevent any user confusion, remove pre-existing filters when
enabling an event provided via the '-e' command line argument.
Suggested-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Link: https://lore.kernel.org/r/20260902123942.695822-8-vschneid@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Events are processed in LIFO order for ease of internal processing, but
this is can be confusing to users if the processing ordering is exposed to
them.
A following commit will do just that by making it so the last-defined event
options (filters/triggers) overrides any previous options for that same
event. Thus, process the events in the user-defined order.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Link: https://lore.kernel.org/r/20260902123942.695822-7-vschneid@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The events list is currently a single linked list. New events are prepended
to the list, making them the new head.
This makes it easy to fetch the last-defined event when handling trigger
and filter options on the cmdline, as they apply to the last-defined event,
but means the events are then processed in LIFO order with regards to the
command line order.
To prepare for processing the events in the user-defined order, make the
events list double linked.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Link: https://lore.kernel.org/r/20260902123942.695822-6-vschneid@redhat.com
[ capital letter in commit message ]
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Kernels pre v6.6 won't have:
39f7c41c908b ("tracing/filters: Enable filtering a cpumask field by another cpumask")
and thus won't be able to filter events using a user-provided cpumask, but
will still be capable of recording IPI events.
Make failing to set a filter for IPI events an acceptable error and fall
back to event handlers that do the filtering job themselves.
Suggested-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Link: https://lore.kernel.org/r/20260902123942.695822-5-vschneid@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Leverage the kernel event filtering infrastructure to only emit IPI events
if they target CPUs that are being traced, as specified by the -c cmdline
option.
Note that some post-processing is still required for the ipi_send_cpumask
event, as the event being emitted means *some* CPUs targeted by that event
are monitored, but not all of them - userspace has to recompute that
intersection.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Link: https://lore.kernel.org/r/20260902123942.695822-4-vschneid@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Leverage the ipi_send_cpu and ipi_send_cpumask trace events to record the
count of IPIs sent to monitored CPUs. These interferences are already
accounted by the IRQ count, but this split gives a better overall picture.
This uses the newly added --ipi cmdline option.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Link: https://lore.kernel.org/r/20260902123942.695822-3-vschneid@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Later commits will add IPI tracking to osnoise top. To avoid breaking
existing scripts, this new feature will be gated behind a new --ipi option.
Suggested-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Link: https://lore.kernel.org/r/20260902123942.695822-2-vschneid@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The get_max_cpu_from_list() function is used when parsing osnoise cpus
"all" value. Furthermore, the callback used by it is also used in
get_possible_cpus().
Add unit test to cover it.
Link: https://lore.kernel.org/r/20260814135511.2207190-6-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
cpu_list_iterate() was split out of parse_cpu_set() to hold shared code
between it and the newly added function get_possible_cpus().
As its semantics are more complex than parse_cpu_set() - it calls a
callback on each element of the list, with possible abort on failure,
while parse_cpu_set() cares only about the set defined by the list - it
deserves its own test.
Test the callback being called correctly as well as the return value and
early break on different combinations of comma-separated numbers and
ranges.
Link: https://lore.kernel.org/r/20260814135511.2207190-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
In rare situations, sysfs might report a different number of CPUs than
the kernel sees, for example, when /sys is not real sysfs.
In tracefs mode, RTLA will discard any samples that are reported on CPUs
higher than the maximum in sysfs. However, in BPF mode, the incorrectly
sized BPF_MAP_LOOKUP_ELEM syscall on percpu maps will cause a buffer
overflow.
To cover for this corner case, write "all" to osnoise/cpus, and read
back the real kernel-side list of all possible cpus. If it doesn't match
RTLA's nr_cpus (read from sysfs), abort with an error.
Assisted-by: Claude:claude-opus-4-6
Link: https://lore.kernel.org/r/20260814135511.2207190-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The "cpu" field of trace entries processed by rtla in tracefs mode is
used as an index into an array of size nr_cpus. In case the cpu numbers
mismatch because either the kernel or rtla reporting the number
incorrectly, out-of-bounds read/write may occur.
Guard against this by dropping trace entries with cpu >= nr_cpus in
collect_registered_events(). A new counter, "invalid_events", is added
to struct trace_instance, and printed (if non zero) next to the
pre-existing missed events counter.
Link: https://lore.kernel.org/r/20260814135511.2207190-3-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
rtla uses get_nprocs_conf(), a standard libc routine, to retrieve the
total number of CPUs for the purpose of processing per-CPU data.
In some configurations, libc might return a different value than what is
seen by the kernel. To cover for this situation, replace the call to
get_nprocs_conf() with a newly added helper, get_possible_cpus().
get_possible_cpus() reads /sys/devices/system/cpu/possible and parses
the CPU list, identically to what libbpf_num_possible_cpus() does.
Additionally, it checks that the possible cpu list is zero-based and has
no holes by computing both the cpu count and the maximum cpu number.
Systems where /sys/.../cpu/possible is unreadable, is not zero-based,
or has holes, now report an error instead of crashing later due to unmet
assumptions.
Note that parse_cpu_set() cannot be used to parse
/sys/devices/system/cpu/possible, because it only supports CPU count of
1024 and lower. Higher CPU counts are not fully supported by rtla;
nr_cpus > 1024 is, though, so it has to be covered by the new
implementation in order to avoid a regression.
A new function, cpu_list_iterate(), is added to utils.c to contain the
cpu list parsing logic, which is now shared between the new function
get_possible_cpus() and the pre-existing parse_cpu_set().
Assisted-by: Claude:claude-opus-4-6
Link: https://lore.kernel.org/r/20260814135511.2207190-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Commit fafb66e5903c ("rtla/timerlat_top: Fix on-threshold actions firing
on signal") fixed a bug where on-threshold actions were being triggered
even without a threshold being set.
Test all four tools with an --on-threshold action, but with no set
threshold, and verify the action does not trigger.
Link: https://lore.kernel.org/r/20260807094555.1244103-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The current handling of invalid command line option arguments is
inconsistent:
- opt_llong_callback() treats non-numerical input the same as "-1",
which might or might not be rejected later.
- opt_int_callback() returns -1 on non-numerical input without an error
message, which makes parsing fail silently (libsubcmd will
automatically print the usage of the option only, no error message).
- custom callbacks abort command line parsing using fatal(), which
displays an error message and exits, without libsubcmd printing the
usage.
Unify this such that all invalid options, regardless of the format,
print an error message similar to the out of range case:
Error: --opt: 'value' is not a valid XY
followed by the usage of the option, e.g.:
$ rtla timerlat hist --period=1us
Error: --period: '1us' is not a valid number
Usage: rtla timerlat hist [<options>] [-h|--help]
-p, --period <us> timerlat period in us
As this is a libsubcmd help path, all option parsing failures now return
the exit code of 129 (help).
The unified handling is implemented using a new error message helper,
opt_err(), which is called from two new CLI-specific parsing functions,
strtoll_safe() and strtoi_safe(), as well as from custom helpers.
Option callback tests are updated to cover the new behavior.
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260716144901.1187474-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Several RTLA options do range validation inside the CLI parser layer
(e.g. -p/--period). When RTLA migrated CLI parsing to libsubcmd, this
logic was moved unchanged inside opt_*() callbacks.
Unify range validation so that all options use two newly added
functions, check_llong_range() and check_int_range(), to validate the
range.
The new range validation returns -1 from opt_*() callbacks rather than
hard-exit with fatal(), allowing the help message for the specific
option to be automatically displayed by libsubcmd logic.
Many options no longer need a custom callback, as they use the unified
range validation of opt_llong_callback() and opt_int_callback().
Validation for several other options is improved:
- timerlat -p/--period: lower bound raised from 1 to 100 us to match
the kernel's timerlat_min_period in trace_osnoise.c.
- timerlat -A/--aligned: reject negative values.
- timerlat --deepest-idle-state: add range [-1, INT_MAX]; previously,
values <= -2 were read as "option not set".
- timerlat -p/--period, -A/--aligned, -b/--bucket-size: properly reject
negative values instead of passing them to the tracer.
Remove unit tests for removed callbacks and test the new range
validation functionality of opt_llong_callback() and opt_int_callback().
Update runtime tests for histogram options to account for the new error
messages and exit value.
Assisted-by: Claude:claude-opus-4-6
Link: https://lore.kernel.org/r/20260710131554.338335-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Currently, runtime tests only test the osnoise period option
(-p/--period of rtla-osnoise tools, backed by
/sys/kernel/tracing/osnoise/period_us), using the
check_with_osnoise_options function together with a hack relying on long
period (pre-set) timing out if RTLA fails to reset it to the default
value.
Extend tracer option testing to all options used by RTLA; test both RTLA
setting the default option by pre-setting the tracer to a different
value and user-requested value.
The tests are done using a script that reads the tracer values inside an
--on-threshold action, like existing tests for runtime behavior already
do. check_with_osnoise_option is modified to support grep filters, so
that it can be used together with the script pattern.
Assisted-by: Claude:claude-opus-4-6
Link: https://lore.kernel.org/r/20260709091755.58265-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Test parsing of command line that sets an option and then unsets it back
to the default value in all tools.
Only two CLI tests are added for each tool: short period option (-p ...
--no-period) and long period option (--period ... --no-period). The
logic specific for individual options is tested in opt callback tests
already.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260629083654.1548925-3-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Test for each opt callback that implements the unset option whether the
option sets the specified default value back correctly.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260629083654.1548925-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
libsubcmd implicitly allows the user to unset already set options using
a "no-" prefix for long options. For example, if I set the period like
this:
$ rtla timerlat -D
Loading BPF program
reading osnoise/timerlat_period_us returned 1000
setting osnoise/timerlat_period_us to 1000
reading osnoise/print_stack returned 0
setting osnoise/print_stack to 0
...
<timerlat top>
it can be unset by a subsequent --no-debug:
$ rtla timerlat -D --no-debug
...
<timerlat top>
Currently, this works only for boolean options. Extend the feature for
all options by implementing handling of the "unset" argument in opt_*()
callbacks defined in cli_p.h, except for list options, i.e. options that
can be passed multiple times (--event, --filter, --trigger,
--on-threshold, --on-end).
This allows, for example, unsetting of int/long long options, e.g. "-p":
$ rtla timerlat -D -p100 --no-period
...
setting osnoise/timerlat_period_us to 1000
...
By default, options in params struct are reset to zero. A constant is
added for every parameter with a different default value, which is then
used both in <tool>_hist_args() while setting the initial value and in
opt_*() when unsetting the option. This refactoring ensures there is no
duplicate "magic number".
The default value for opt_llong_callback() and opt_int_callback() is
passed in struct option's defval field; new macros
RTLA_OPT_{LLONG,INT}{,_DEFVAL} are added to define the field
conveniently. The default value for other callbacks is hardcoded inside
each callback's unset logic.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260629083654.1548925-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
A bug was reported when rtla-timerlat-top tool performs on-threshold
actions, even though no threshold was hit. This is reproduced even if no
threshold is set at all:
$ rtla timerlat top -q -c 0 --on-threshold shell,command='echo BAD'
BAD
Timer Latency
...
The bug is due to incorrect logic in timerlat_top_bpf_main_loop().
The loop uses timerlat_bpf_wait(), the return values of which are:
- > 0 (number of ringbuffer entries): at least 1 CPU hit threshold
- = 0: time out
- < 0: wait was interrupted by a signal
Commit 3138df6f0cd0 ("rtla/timerlat: Exit top main loop on any non-zero
wait_retval") changed the condition for "threshold hit" from
"wait_reval == 1" (exactly 1 CPU hit threshold) to "wait_retval != 0",
to fix a race where multiple CPUs hit the threshold at the same time.
That also made it incorrectly include a signal (< 0), coming from either
duration expired (SIGALRM) or user interrupt (SIGINT).
Check for wait_retval greater than zero in the if condition to cover all
return values correctly.
Fixes: 3138df6f0cd0 ("rtla/timerlat: Exit top main loop on any non-zero wait_retval")
Reported-by: Attila Fazekas <afazekas@redhat.com>
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260713141047.687877-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
rtla started to only link parts of the tools library. It now misses the
ctype information used by all the related string operations. Just add
another single file to make it build again.
Signed-off-by: Bastian Blank <waldi@debian.org>
Fixes: 48209d763c22 ("rtla: Add libsubcmd dependency")
Link: https://lore.kernel.org/r/ako2S4mzIqWwYuas@steamhammer.waldi.eu.org
[ remove duplicated spaces in commit message ]
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Compiling RTLA 7.1.x with GCC 16 and uClibc as standard library fails
with these errors:
src/common.c: In function ‘set_signals’:
src/common.c:40:17: error: implicit declaration of function ‘alarm’ [-Wimplicit-function-declaration]
40 | alarm(params->duration);
| ^~~~~
src/common.c: In function ‘common_apply_config’:
src/common.c:187:44: error: implicit declaration of function ‘getpid’; did you mean ‘getpt’? [-Wimplicit-function-declaration]
187 | retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
| ^~~~~~
| getpt
In file included from src/common.c:9:
src/common.c: In function ‘run_tool’:
src/common.c:262:19: error: implicit declaration of function ‘sysconf’; did you mean ‘sscanf’? [-Wimplicit-function-declaration]
262 | nr_cpus = get_nprocs_conf();
| ^~~~~~~~~~~~~~~
src/common.c:262:19: error: ‘_SC_NPROCESSORS_CONF’ undeclared (first use in this function)
262 | nr_cpus = get_nprocs_conf();
| ^~~~~~~~~~~~~~~
src/common.c:262:19: note: each undeclared identifier is reported only once for each function it appears in
src/common.c:370:17: error: implicit declaration of function ‘sleep’ [-Wimplicit-function-declaration]
370 | sleep(1);
| ^~~~~
Restore the missing unistd.h include.
Fixes: 115b06a00875 ("tools/rtla: Consolidate nr_cpus usage across all tools")
Signed-off-by: Andreas Ziegler <br025@umbiko.net>
Link: https://lore.kernel.org/r/20260624033342.129970-1-br025@umbiko.net
[ capital letter in patch subject, remove EOL after Fixes ]
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Multiple runtime tests in RTLA rely on the get_workload_pids() shell
helper function to get the PIDs of both kernel and user workloads.
On some systems (e.g. Fedora 43), pgrep matches kernel thread names
including square brackets: "[osnoise/0]"; on other systems (e.g.
RHEL 9.8), brackets are not included: "osnoise/0".
Accept both as valid workload PIDs rather that just the non-bracket form
to make the tests work on all systems.
Fixes: a98dad63cda3 ("rtla/tests: Add runtime test for -k and -u options")
Reported-by: Crystal Wood <crwood@redhat.com>
Link: https://lore.kernel.org/r/20260604140547.3616495-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
.gitignore includes several entries prone to unwanted matches in
subdirectories. One of them, the recently added "lib/", matches the
recently added directory "tests/scripts/lib/" in addition to the
intended top-level "lib/", which contains object files built from
sources in tools/lib.
Add "/" to all .gitignore entries that are intended to only match
top-level files or directories: rtla, rtla_static, unit_tests,
libsubcmd/.
Remove .gitignore entries that are not needed at all:
- lib/ (contains only object files, ignored by top-level .gitignore
already).
- .txt rtla output files added to .gitignore in commit 02689ae385c5
("rtla: Add generated output files to gitignore"). Since commit
ad5b50a0959f ("rtla/tests: Run runtime tests in temporary directory"),
those are created in a temporary directory, not in tools/tracing/rtla.
Keeping libsubcmd/ as that contains other generated files (headers,
archives, etc.).
Fixes: 48209d763c22 ("rtla: Add libsubcmd dependency")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202605291919.eszupseg-lkp@intel.com/
Closes: https://lore.kernel.org/oe-kbuild-all/202605300436.PqQ0Bc8q-lkp@intel.com/
Link: https://lore.kernel.org/r/20260601091835.3118094-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
RTLA tests are not documented anywhere. Mention both runtime and unit
tests in the README, with instructions on how to run them and a list of
dependencies and required system configuration.
Link: https://lore.kernel.org/r/20260514073038.204428-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Add both parse_args() and opt_* tests for the newly added -A/--aligned
option.
Assisted-by: Claude:claude-4.5-opus-high-thinking
Link: https://lore.kernel.org/r/20260527144928.2944472-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Add a new option, -A/--aligned, that enables timerlat thread alignment
implemented on the kernel-side in commit 4245bf4dc58f ("tracing/osnoise:
Add option to align tlat threads"). The option takes an argument,
representing alignment between timerlat threads in microseconds.
The feature is modeled after the option of the same name in the
cyclictest tool.
Link: https://lore.kernel.org/r/20260527144928.2944472-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
In addition to testing all tool_parse_args() functions, test also all
callbacks used for parsing custom option formats.
The callbacks represent a middle layer between the parsing functions
and utility functions dedicated to checking specific argument formats,
for example, scheduling class and duration. Callback tests are run
before parsing functions to make sure any issue in the former is
reported before it is encountered through the latter.
Tests verify both successful parsing and proper rejection of invalid
inputs (via exit tests). To enable testing static callbacks, a pragma
once guard is added to timerlat.h for safe inclusion by cli_p.h.
Add dependency of UNIT_TESTS_IN on LIBSUBCMD_INCLUDES, as the new test
file tests/unit/cli_opt_callback.c includes cli_p.h which includes
subcmd/parse-options.h.
Link: https://lore.kernel.org/r/20260528103254.2990068-7-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Add a test suite for the _parse_args() function of each tool that checks
the params structures (struct common_params, struct osnoise_params,
struct timerlat_params) returned by them for correctness.
One test case is added per option, as well as a few special cases for
tricky combinations of options. Test cases are ordered the same as the
option arrays and help message to allow easy checking of whether all
options are covered.
This should help clarify what the proper command line behavior of RTLA
is in case there are holes in the documentation and verify that the
intended behavior is implemented correctly.
A few necessary changes to the unit tests were done as part of this
commit:
- Unit tests now also link to libsubcmd and its dependencies.
- A new global variable in_unit_test is added to RTLA's CLI interface,
causing it to skip check for root if running in unit tests. This
allows the CLI unit tests to run as non-root, like existing unit
tests.
There is quite a lot of duplication, some of it is mitigated with macros,
but partially it is intentional so that future changes in behavior are
tracked across tools.
Link: https://lore.kernel.org/r/20260528103254.2990068-6-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Instead of using getopt_long() directly to parse the command line
arguments given to an RTLA tool, use libsubcmd's parse_options().
Utilizing libsubcmd for parsing command line arguments has several
benefits:
- A help message is automatically generated by libsubcmd from the
specification, removing the need of writing it by hand.
- Options are sorted into groups based on which part of tracing (CPU,
thread, auto-analysis, tuning, histogram) they relate to.
- Common parsing patterns for numerical and boolean values now share
code, with the target variable being stored in the option array.
To avoid duplication of the option parsing logic, RTLA-specific
macros defining struct option values are created:
- RTLA_OPT_* for options common to all tools
- OSNOISE_OPT_* and TIMERLAT_OPT_* for options specific to
osnoise/timerlat tools
- HIST_OPT_* macros for options specific to histogram-based tools.
Individual *_parse_args() functions then construct an array out of
these macros that is then passed to libsubcmd's parse_options().
All code specific to command line options parsing is moved out of the
individual tool files into a new file, cli.c, which also contains the
contents of the rtla.c file. A private header, cli_p.h, is added
alongside the public header cli.h, so that unit tests are able to test
statically declared option callbacks.
Minor changes:
- The return value of tool-level help option changes to 129, as this is
the value set by libsubcmd; this is reflected in affected test cases.
The implementation of help for command-level and tracer-level help
is set to 129 as well for consistency, and the change is reflected in
exit value documentation.
- Related to the above, {rtla,osnoise,timerlat}_usage() are marked
__noreturn and exit() is removed from after they are called for
cleaner code.
- The error messages for invalid argument for options --dma-latency and
-E/--entries were corrected, fixing off-by-one in the limits.
Note that unsetting options (using --no-<opt> syntax) is currently not
implemented for options that use custom callbacks. For --irq and
--thread, it will never be implemented, as they conflict with already
existing --no-irq and --no-thread with a different meaning.
Assisted-by: Composer:composer-1.5
Link: https://lore.kernel.org/r/20260528103254.2990068-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
In preparation for migrating RTLA to libsubcmd, build libsubcmd from the
appropriate directory next to the RTLA build proper, and link the
resulting object to RTLA.
libsubcmd uses str_error_r() and strlcpy() at several places. To support
these, also link the respective libraries from tools/lib.
For completeness, also add tools/include to include path. This will
allow other userspace functions and macros shipped with the kernel to be
used in RTLA; perf and bpftool, two other users of libsubcmd, already do
that.
To prevent a name conflict, rename RTLA's run_command() function to
run_tool_command(), and replace RTLA's own container_of implementation
with the one in tools/include/linux/container_of.h.
Assisted-by: Composer:composer-1
Link: https://lore.kernel.org/r/20260528103254.2990068-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
In case an action preceding the continue action fails, not only
the continue flag should not be set, it should be unset if it was set
from a previous run of actions_perform().
Add a runtime test to both osnoise and timerlat tools that checks that
this works properly by creating a temporary file.
Link: https://lore.kernel.org/r/20260526102523.2662391-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Create a temporary directory before each test case to serve as working
directory during the duration of the test.
This prevents littering of the original working directory as well as
allows tests to use it to avoid path conflicts.
In order not to break already existing tests, also add a new "testdir"
variable containing the directory where the test file is located. This
is then used to locate artifacts used during testing like BPF programs
and scripts for checking the tracer threads.
Link: https://lore.kernel.org/r/20260526102523.2662391-3-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
In case an action preceding the continue action fails, not only
the continue flag should not be set, it should be unset if it was set
from a previous run of actions_perform().
Add a unit test to check if this is implemented correctly.
Link: https://lore.kernel.org/r/20260526102523.2662391-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Currently, actions_perform() only ever sets the continue flag (when
performing the continue action), but never resets it. That leads to
RTLA continuing tracing even if the continue action was not performed in
the current iteration.
For example, the following command:
$ rtla timerlat hist -T 100 --on-threshold shell,command='
echo Spike!
if [ -f /tmp/a ]
then
exit 1
else
touch /tmp/a
fi' --on-threshold continue
should print Spike! at most once, because after hitting the threshold
for the first time, /tmp/a exists, the shell action will fail, and the
continue action is not performed. However, unless /tmp/a exists before
the measurement, it will print Spike! until stopped, as the continue
flag stays set.
Set the continue flag to false in the beginning of actions_perform() to
make RTLA continue only if the action was actually performed.
Fixes: 8d933d5c89e8 ("rtla/timerlat: Add continue action")
Link: https://lore.kernel.org/r/20260526102523.2662391-1-tglozar@redhat.com
[ correct Fixes tag to include 12 characters of hash ]
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Before, when rtla got a signal, it stopped the main trace but not the
record trace. With "--on-end trace", this can lead to
save_trace_to_file() failing to keep up, especially on a debug kernel.
Plus, it adds post-stoppage noise to the trace file.
Signed-off-by: Crystal Wood <crwood@redhat.com>
Fixes: c73cab9dbed0 ("rtla/timerlat_hist: Stop timerlat tracer on signal")
Fixes: a4dfce7559d7 ("rtla/timerlat_top: Stop timerlat tracer on signal")
Fixes: 3aadb65db5d6 ("rtla/timerlat: Add action on end feature")
Link: https://lore.kernel.org/r/20260512173731.2151841-1-crwood@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Add unit tests covering all functions in the actions module, including
both valid and invalid inputs and all action types, except for
actions_perform(), where only shell and continue actions are tested.
To support testing multiple modules, the unit test build was modified so
that it links the entire rtla-in.o file. For this to work, the main()
function in rtla.c was declared weak, so that the unit test main is able
to override it.
Other included minor changes to unit tests are:
- Make unit test output verbose to show which tests are being run, now
that we have more than 3 tests.
- Add unit_tests file to .gitignore.
- Split unit test sources to one file per test suite, and keep only
main() function in unit_tests.c.
- Fix Makefile dependencies so that "make unit-tests" will rebuild the
binary with the changes in the commit.
Also with the linking the entire rtla-in.o file, it now has rtla's
nr_cpus symbol, so the declaration in utils unit tests is made extern.
Assisted-by: Composer:composer-2-fast
Link: https://lore.kernel.org/r/20260424140244.958495-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Add a new script check-cgroup-match.sh that retrieves the cgroup of the
main rtla process and compares it to the cgroup of the rtla workload
threads.
Add a new test based on this script, for both osnoise and timerlat
tools, testing the variant of -C without argument (which sets the cgroup
of the workload to the cgroup of the rtla main process).
Note that this has to be tested in kernel mode to be significant for
timerlat tool, as user workloads inherit the parent rtla process cgroup
even without the option.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-10-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Add runtime test for rtla-timerlat's -k/--kernel-threads and
-u/--user-threads options using get_workload_pids.sh to check whether
the appropriate threads are being created.
The tests are implemented for both top and hist. Additionally, all tests
related to timerlat threads are moved to a separate section in the test
files. The latter is also done for rtla-osnoise tests.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-9-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Add a runtime test for -H/--house-keeping option for both osnoise and
timerlat tools, with affinity checking similar to what is done for
-c/--cpus.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-8-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Cover all options regarding histogram formatting for both
rtla-osnoise-hist and rtla-timerlat-hist tools. All options also have
output checking using positive or negative match, except for
-b/--bucket-size and -E/--entries, which cannot be tested in isolated
due to the output depending on the actual data collected.
Old -E/--entries test for rtla-osnoise was replaced with a new one
equivalent to the timerlat one.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-7-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
rtla-timerlat-top's --aa-only option is currently only tested for return
value.
Extend the tests to also check that only auto-analysis is being done via
a negative match for the "Timer Latency" text in the top header, and
further split the test case into two:
- one test case for --aa-only stopping on threshold
- one test case for --aa-only exiting without threshold being hit
For both cases, the expected output ("analyzing it" or "Max latency was"
respectively) is checked against in addition to the negative match.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-6-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
For testing the -a/--auto option in timerlat tool, the string "analyzing
it" is matched against to make sure auto-analysis was triggered.
Use the same string as a negative match for --aa-only option test.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
RTLA runtime tests verify the -c/--cpus options, but do not check
whether the correct affinity is actually applied.
Add a script named check-cpus.sh that retrieves the affinity of all
workload threads and use it to check the -c/--cpus option for both
osnoise and timerlat tools.
Also add missing -c/--cpus test for osnoise.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
RTLA runtime tests that check workload processes (currently the test
case "verify -P/--priority" of timerlat.t and "verify the --priority/-P
param" of osnoise.t) use "pgrep timerlatu/" or "pgrep osnoise/"
respectively to identify the workload.
Make them more robust by adding a get_workload_pids() helper that
finds the main rtla process and returns the PIDs of all siblings other
than the test script itself, plus all child processes of kthreadd that
have the osnoise/timerlat kthread pattern comm.
This filters out any spurious processes not related to the running test
that happen to have "timerlatu/" or "osnoise/" in their command, for
example, a user grepping the same names at the time of the running of
the test.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-3-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
