| Age | Commit message (Collapse) | Author |
|---|
|
In rare situations, sysfs might report a different number of CPUs than
the kernel sees, for example, when /sys is not real sysfs.
In tracefs mode, RTLA will discard any samples that are reported on CPUs
higher than the maximum in sysfs. However, in BPF mode, the incorrectly
sized BPF_MAP_LOOKUP_ELEM syscall on percpu maps will cause a buffer
overflow.
To cover for this corner case, write "all" to osnoise/cpus, and read
back the real kernel-side list of all possible cpus. If it doesn't match
RTLA's nr_cpus (read from sysfs), abort with an error.
Assisted-by: Claude:claude-opus-4-6
Link: https://lore.kernel.org/r/20260814135511.2207190-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The "cpu" field of trace entries processed by rtla in tracefs mode is
used as an index into an array of size nr_cpus. In case the cpu numbers
mismatch because either the kernel or rtla reporting the number
incorrectly, out-of-bounds read/write may occur.
Guard against this by dropping trace entries with cpu >= nr_cpus in
collect_registered_events(). A new counter, "invalid_events", is added
to struct trace_instance, and printed (if non zero) next to the
pre-existing missed events counter.
Link: https://lore.kernel.org/r/20260814135511.2207190-3-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
rtla uses get_nprocs_conf(), a standard libc routine, to retrieve the
total number of CPUs for the purpose of processing per-CPU data.
In some configurations, libc might return a different value than what is
seen by the kernel. To cover for this situation, replace the call to
get_nprocs_conf() with a newly added helper, get_possible_cpus().
get_possible_cpus() reads /sys/devices/system/cpu/possible and parses
the CPU list, identically to what libbpf_num_possible_cpus() does.
Additionally, it checks that the possible cpu list is zero-based and has
no holes by computing both the cpu count and the maximum cpu number.
Systems where /sys/.../cpu/possible is unreadable, is not zero-based,
or has holes, now report an error instead of crashing later due to unmet
assumptions.
Note that parse_cpu_set() cannot be used to parse
/sys/devices/system/cpu/possible, because it only supports CPU count of
1024 and lower. Higher CPU counts are not fully supported by rtla;
nr_cpus > 1024 is, though, so it has to be covered by the new
implementation in order to avoid a regression.
A new function, cpu_list_iterate(), is added to utils.c to contain the
cpu list parsing logic, which is now shared between the new function
get_possible_cpus() and the pre-existing parse_cpu_set().
Assisted-by: Claude:claude-opus-4-6
Link: https://lore.kernel.org/r/20260814135511.2207190-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Instead of using getopt_long() directly to parse the command line
arguments given to an RTLA tool, use libsubcmd's parse_options().
Utilizing libsubcmd for parsing command line arguments has several
benefits:
- A help message is automatically generated by libsubcmd from the
specification, removing the need of writing it by hand.
- Options are sorted into groups based on which part of tracing (CPU,
thread, auto-analysis, tuning, histogram) they relate to.
- Common parsing patterns for numerical and boolean values now share
code, with the target variable being stored in the option array.
To avoid duplication of the option parsing logic, RTLA-specific
macros defining struct option values are created:
- RTLA_OPT_* for options common to all tools
- OSNOISE_OPT_* and TIMERLAT_OPT_* for options specific to
osnoise/timerlat tools
- HIST_OPT_* macros for options specific to histogram-based tools.
Individual *_parse_args() functions then construct an array out of
these macros that is then passed to libsubcmd's parse_options().
All code specific to command line options parsing is moved out of the
individual tool files into a new file, cli.c, which also contains the
contents of the rtla.c file. A private header, cli_p.h, is added
alongside the public header cli.h, so that unit tests are able to test
statically declared option callbacks.
Minor changes:
- The return value of tool-level help option changes to 129, as this is
the value set by libsubcmd; this is reflected in affected test cases.
The implementation of help for command-level and tracer-level help
is set to 129 as well for consistency, and the change is reflected in
exit value documentation.
- Related to the above, {rtla,osnoise,timerlat}_usage() are marked
__noreturn and exit() is removed from after they are called for
cleaner code.
- The error messages for invalid argument for options --dma-latency and
-E/--entries were corrected, fixing off-by-one in the limits.
Note that unsetting options (using --no-<opt> syntax) is currently not
implemented for options that use custom callbacks. For --irq and
--thread, it will never be implemented, as they conflict with already
existing --no-irq and --no-thread with a different meaning.
Assisted-by: Composer:composer-1.5
Link: https://lore.kernel.org/r/20260528103254.2990068-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
In preparation for migrating RTLA to libsubcmd, build libsubcmd from the
appropriate directory next to the RTLA build proper, and link the
resulting object to RTLA.
libsubcmd uses str_error_r() and strlcpy() at several places. To support
these, also link the respective libraries from tools/lib.
For completeness, also add tools/include to include path. This will
allow other userspace functions and macros shipped with the kernel to be
used in RTLA; perf and bpftool, two other users of libsubcmd, already do
that.
To prevent a name conflict, rename RTLA's run_command() function to
run_tool_command(), and replace RTLA's own container_of implementation
with the one in tools/include/linux/container_of.h.
Assisted-by: Composer:composer-1
Link: https://lore.kernel.org/r/20260528103254.2990068-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Add a str_has_prefix() helper function that tests whether a string
starts with a given prefix. This function provides a cleaner interface
for prefix matching compared to using strncmp() with strlen() directly.
The function returns a boolean value indicating whether the string
starts with the specified prefix. This helper will be used in
subsequent changes to simplify prefix matching code throughout rtla.
Also add the missing string.h include which is needed for the strlen()
and strncmp() functions used by str_has_prefix() and the existing
strncmp_static() macro.
Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260309195040.1019085-11-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The actions_parse() function uses open-coded logic to extract arguments
from a string. This includes manual length checks and strncmp() calls,
which can be verbose and error-prone.
To simplify and improve the robustness of argument parsing, introduce a
new extract_arg() helper macro. This macro extracts the value from a
"key=value" pair, making the code more concise and readable.
Also, introduce STRING_LENGTH() and strncmp_static() macros to
perform compile-time calculations of string lengths and safer string
comparisons.
Refactor actions_parse() to use these new helpers, resulting in
cleaner and more maintainable code.
Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260309195040.1019085-4-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Most memory allocations in rtla happen during early initialization
before any resources are acquired that would require cleanup. In these
cases, propagating allocation errors just adds complexity without any
benefit. There's nothing to clean up, and the program must exit anyway.
This patch introduces fatal allocation wrappers (calloc_fatal,
reallocarray_fatal, strdup_fatal) that call fatal() on allocation
failure. These wrappers simplify the code by eliminating unnecessary
error propagation paths.
The patch converts early allocations to use these wrappers in
actions_init() and related action functions, osnoise_context_alloc()
and osnoise_init_tool(), trace_instance_init() and trace event
functions, and parameter structure allocations in main functions.
This simplifies the code while maintaining the same behavior: immediate
exit on allocation failure during initialization. Allocations that
require cleanup, such as those in histogram allocation functions with
goto cleanup paths, are left unchanged and continue to return errors.
Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260309195040.1019085-2-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
In the current implementation, the auto-analysis code for printing the
stack captured in the tracefs buffer of the aa instance stops at the
first encountered address that cannot be resolved into a function
symbol.
This is not always the desired behavior on all platforms; sometimes,
there might be resolvable entries after unresolvable ones, and
sometimes, the user might want to inspect the raw pointers for the
unresolvable entries.
Add a new option, --stack-format, with three values:
- truncate: stop at first unresolvable entry. This is the current
behavior, and is kept as the default.
- skip: skip unresolvable entries, but do not stop on them.
- full: print all entries, including unresolvable ones.
To make this work, the "size" field of the stack entry is now also read
and used as the maximum number of entries to print, capped at 64, since
that is the fixed length of the "caller" field.
Link: https://lore.kernel.org/r/20260119115222.744150-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The result enum defines custom values for PASSED, ERROR, and FAILED.
These values correspond to standard exit codes EXIT_SUCCESS and
EXIT_FAILURE.
Update the enum to use the standard macros EXIT_SUCCESS and
EXIT_FAILURE to improve readability and adherence to standard C
practices.
The FAILED value is implicitly assigned EXIT_FAILURE + 1, so there
is no need to assign an explicit value.
Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260106133655.249887-9-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The atoi() function does not perform error checking, which can lead to
undefined behavior when parsing invalid or out-of-range strings. This
can cause issues when parsing user-provided numerical inputs, such as
signal numbers, PIDs, or CPU lists.
To address this, introduce a new strtoi() helper function that safely
converts a string to an integer. This function validates the input and
checks for overflows, returning a negative value on failure.
Replace all calls to atoi() with the new strtoi() function and add
proper error handling to make the parsing more robust and prevent
potential issues.
Signed-off-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260106133655.249887-5-wander@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Historically four function declarations remain orphaned or duplicated.
Remove them to keep the source clean.
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Link: https://lore.kernel.org/r/20251012071133.290225-1-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
Currently, user can only specify cgroup to the tracer's thread the
following ways:
`-C[cgroup]`
`-C[=cgroup]`
`--cgroup[=cgroup]`
If user tries to specify cgroup as `-C [cgroup]` or `--cgroup [cgroup]`,
the parser silently fails and rtla's cgroup is used for the tracer
threads.
To make interface more user-friendly, allow user to specify cgroup in
the aforementioned way, i.e. `-C [cgroup]` and `--cgroup [cgroup]`.
Refactor identical logic between -t/--trace and -C/--cgroup into a
common function.
Change documentation to reflect this user interface change.
Fixes: a957cbc02531 ("rtla: Add -C cgroup support")
Signed-off-by: Ivan Pravdin <ipravdin.official@gmail.com>
Reviewed-by: Tomas Glozar <tglozar@redhat.com>
Link: https://lore.kernel.org/r/16132f1565cf5142b5fbd179975be370b529ced7.1762186418.git.ipravdin.official@gmail.com
[ use capital letter in subject, as required by tracing subsystem ]
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
The code contains some technical debt in error handling,
which complicates the consolidation of duplicated code.
Introduce an fatal() function to replace the common pattern of
err_msg() followed by exit(EXIT_FAILURE), reducing the length of an
already long function.
Further patches using fatal() follow.
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Reviewed-by: Tomas Glozar <tglozar@redhat.com>
Link: https://lore.kernel.org/r/20251011082738.173670-2-costa.shul@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
|
|
A test is considered failed when a sample trace exceeds the threshold.
Failed tests return the same exit code as passed tests, requiring test
frameworks to determine the result by searching for "hit stop tracing"
in the output.
Assign a distinct exit code for failed tests to enable the use of shell
expressions and seamless integration with testing frameworks without the
need to parse output.
Add enum type for return value.
Update `make check`.
Cc: Daniel Bristot de Oliveira <bristot@kernel.org>
Cc: John Kacur <jkacur@redhat.com>
Cc: "Luis Claudio R. Goncalves" <lgoncalv@redhat.com>
Cc: Eder Zulian <ezulian@redhat.com>
Cc: Dan Carpenter <dan.carpenter@linaro.org>
Cc: Jan Stancek <jstancek@redhat.com>
Link: https://lore.kernel.org/20250417185757.2194541-1-costa.shul@redhat.com
Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
Reviewed-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
Add functions to utils.c to disable idle states through functions of
libcpupower. This will serve as the basis for disabling idle states
per cpu when running timerlat.
Link: https://lore.kernel.org/20241017140914.3200454-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
glibc commit 21571ca0d703 ("Linux: Add the sched_setattr
and sched_getattr functions") now also provides 'struct sched_attr'
and sched_setattr() which collide with the ones from rtla.
In file included from src/trace.c:11:
src/utils.h:49:8: error: redefinition of ‘struct sched_attr’
49 | struct sched_attr {
| ^~~~~~~~~~
In file included from /usr/include/bits/sched.h:60,
from /usr/include/sched.h:43,
from /usr/include/tracefs/tracefs.h:10,
from src/trace.c:4:
/usr/include/linux/sched/types.h:98:8: note: originally defined here
98 | struct sched_attr {
| ^~~~~~~~~~
Define 'struct sched_attr' conditionally, similar to what strace did:
https://lore.kernel.org/all/20240930222913.3981407-1-raj.khem@gmail.com/
and rename rtla's version of sched_setattr() to avoid collision.
Link: https://lore.kernel.org/8088f66a7a57c1b209cd8ae0ae7c336a7f8c930d.1728572865.git.jstancek@redhat.com
Signed-off-by: Jan Stancek <jstancek@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
Since the sched_priority for SCHED_OTHER is always 0, it makes no
sence to set it.
Setting nice for SCHED_OTHER seems more meaningful.
Link: https://lkml.kernel.org/r/20240207065142.1753909-1-limingming3@lixiang.com
Cc: stable@vger.kernel.org
Fixes: b1696371d865 ("rtla: Helper functions for rtla")
Signed-off-by: limingming3 <limingming3@lixiang.com>
Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
|
|
Add the support for running timerlat threads in user-space. In this
mode, enabled with -u/--user-threads, timerlat dispatches user-space
processes that will loop in the timerlat_fd, measuring the overhead
for going to user-space and then returning to the kernel - in addition
to the existing measurements.
Here is one example of the tool's output with -u enabled:
$ sudo timerlat top -u -d 600 -q
Timer Latency
0 00:10:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) | Ret user Timer Latency (us)
CPU COUNT | cur min avg max | cur min avg max | cur min avg max
0 #600001 | 0 0 0 3 | 2 1 2 9 | 3 2 3 15
1 #600001 | 0 0 0 2 | 2 1 2 13 | 2 2 3 18
2 #600001 | 0 0 0 10 | 2 1 2 16 | 3 2 3 20
3 #600001 | 0 0 0 7 | 2 1 2 10 | 3 2 3 11
4 #600000 | 0 0 0 16 | 2 1 2 41 | 3 2 3 58
5 #600000 | 0 0 0 3 | 2 1 2 10 | 3 2 3 13
6 #600000 | 0 0 0 5 | 2 1 2 7 | 3 2 3 10
7 #600000 | 0 0 0 1 | 2 1 2 7 | 3 2 3 10
The tuning setup like -p or -C work for the user-space threads as well.
Link: https://lkml.kernel.org/r/758ad2292a0a1d884138d08219e1a0f572d257a2.1686066600.git.bristot@kernel.org
Cc: William White <chwhite@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Tested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
When the user sets -c <cpu-list> try to move rtla out of the <cpu-list>,
even without an -H option. This is useful to avoid having rtla
interfering with the workload.
This works by removing <cpu-list> from rtla's current affinity.
If rtla fails to move itself away it is not that of a problem as this
is an automatic measure.
Link: https://lkml.kernel.org/r/c54304d90c777310fb85a3e658d1449173759aab.1686066600.git.bristot@kernel.org
Cc: William White <chwhite@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Tested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
To avoid having rtla interfering with the measurement threads, add an
option for the user to set the CPUs in which rtla should run. For
instance:
# rtla timerlat top -H 0 -c 1-7
Will place rtla in the CPU 0, while running the measurement threads in
the CPU 1-7.
Link: https://lkml.kernel.org/r/6a6c78a579a96ba8b02ae67ee1e0ba2cb5e03c4a.1686066600.git.bristot@kernel.org
Cc: William White <chwhite@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Tested-by: Juri Lelli <juri.lelli@redhat.com>
Suggested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
The -C option sets a cgroup to the tracer's threads. If the -C option is
passed without arguments, the tracer's thread will inherit rtla's
cgroup. Otherwise, the threads will be placed on the cgroup passed
to the option.
Link: https://lkml.kernel.org/r/cb051477331d292f17c08bf1d66f0e0384bbe5a5.1686066600.git.bristot@kernel.org
Cc: William White <chwhite@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Tested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
Currently, timerlat displays a summary of the timerlat tracer results
saving the trace if the system hits a stop condition.
While this represented a huge step forward, the root cause was not
that is accessible to non-expert users.
The auto-analysis fulfill this gap by parsing the trace timerlat runs,
printing an intuitive auto-analysis.
Link: https://lkml.kernel.org/r/1ee073822f6a2cbb33da0c817331d0d4045e837f.1675179318.git.bristot@kernel.org
Cc: Daniel Bristot de Oliveira <bristot@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
Daniel Wagner reported to me that readproc.h got deprecated. Also,
while the procps-ng library was available on Fedora, it was not available
on RHEL, which is a piece of evidence that it was not that used.
rtla uses procps-ng only to find the PID of the tracers' workload.
I used the procps-ng library to avoid reinventing the wheel. But in this
case, reinventing the wheel took me less time than the time we already
took trying to work around problems.
Implement a function that reads /proc/ entries, checking if:
- the entry is a directory
- the directory name is composed only of digits (PID)
- the directory contains the comm file
- the comm file contains a comm that matches the tracers'
workload prefix.
- then return true; otherwise, return false.
And use it instead of procps-ng.
Link: https://lkml.kernel.org/r/e8276e122ee9eb2c5a0ba8e673fb6488b924b825.1652423574.git.bristot@kernel.org
Cc: John Kacur <jkacur@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Tao Zhou <tao.zhou@linux.dev>
Fixes: b1696371d865 ("rtla: Helper functions for rtla")
Reported-by: Daniel Wagner <dwagner@suse.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
Add the --dma-latency to set /dev/cpu_dma_latency to the
specified value, this aims to avoid having exit from idle
states latencies that could be influencing the analysis.
Link: https://lkml.kernel.org/r/72ddb0d913459f13217086dadafad88a7c46dd28.1646247211.git.bristot@kernel.org
Cc: Daniel Bristot de Oliveira <bristot@kernel.org>
Cc: Clark Williams <williams@redhat.com>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
|
|
This is a set of utils and tracer helper functions. They are used by
rtla mostly to parse config, display data and some trace operations that
are not part of libtracefs (because they are only useful it for this
case).
Link: https://lkml.kernel.org/r/a94c128aba9e6e66d502b7094f2e8c7ac95b12e5.1639158831.git.bristot@kernel.org
Cc: Tao Zhou <tao.zhou@linux.dev>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Tom Zanussi <zanussi@kernel.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Clark Williams <williams@redhat.com>
Cc: John Kacur <jkacur@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Daniel Bristot de Oliveira <bristot@kernel.org>
Cc: linux-rt-users@vger.kernel.org
Cc: linux-trace-devel@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
