| Age | Commit message (Collapse) | Author |
|---|
|
The klp test runner deletes each test's working directory when the test
finishes. On failure that removes diff.log, out.o and the fixture objects
before the developer can look at them, and the only hint was to re-run
(something) with --keep:
$ make -C tools/objtool tests
...
not ok - test-special-section: expected section '.kcfi_traps' in output
...
# pass:49 fail:1 static-skip:1 probe-skip:0 xfail:0 xpass:0
# re-run with --keep to hold on to what a failing test built
Change the default behavior to save failing test working directories. This
is friendlier to the drive-by tester. Provide the power-user with a KEEP
Makefile variable that can optionally retain 'all' or 'none' of those
directories. run-tests.sh --keep remains an alias for KEEP=all.
Now that a run can be holding objects worth looking at, say where they are
when it is cut short. A Ctrl-C left the run directory behind -- the tests
clean up after themselves as they exit, but the two levels the runner made
are its own to answer for, and nothing removed them. Take them away from
a trap if the tests left them empty, and print the path if they did not:
under the new default the test that was running keeps what it had built,
and that is the run most worth being told about.
The trap only ever rmdirs, so it cannot overrule what a test decided to
keep; the same helper now serves the end of a normal run.
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Assisted-by: Claude:claude-opus-5
Link: https://patch.msgid.link/20260916184351.2720310-11-song@kernel.org
|
|
Each test made its own mktemp directory, so a run scattered them through
/tmp among everything else using the same names, and there was no way to
inspect what a failing test built -- the objects are gone the moment it
exits, which is right every time but the one that matters.
Give the run one directory and each test a subdirectory of it, named for
the test and nested the way the source is:
/tmp/klp-tests.3zC6oyfd/
generic/test-basic/{orig.o,patched.o,out.o,Module.symvers,...}
x86/test-kcfi/...
--keep then reports one path instead of forty, and removing it is one
command. A test run by hand still falls back to a temp directory of its
own, and still says where that went.
Detecting a leak comes free. Cleanup is now "remove each test's directory,
then rmdir the run's", and the rmdir fails if anything is left -- so a test
which dies without running its own cleanup is reported rather than quietly
leaving something behind.
preflight names the directory the tests build under, since mktemp honours
TMPDIR and a run in a container is otherwise silent about where its work
went. The runner says what to do only when it is relevant: on a failure
without --keep, that the option exists; with it, how many directories are
waiting. Usage text is not read while something is broken.
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Assisted-by: Claude:claude-opus-5
Link: https://patch.msgid.link/20260916184351.2720310-8-song@kernel.org
|
|
A test could pass, fail, or skip, and the runner reported only the exit
status of the last one to fail. Skips are the interesting case: they are
how a suite quietly stops testing anything.
Three kinds, and the runner counts them separately:
declared the test said in advance it does not apply here, e.g. gcc_only
on a clang run. Expected indefinitely.
probe the construct did not turn up in the built object this time.
Weaker: one which becomes permanent is a fixture that stopped
testing anything.
undeclared counted as a failure. A test which gives up for a reason it
never declared is a hole, not an outcome.
xfail and xpass come with them, so a known failure is reported rather than
commented out and forgotten, and one which starts passing says so instead
of going quietly green.
The result line is what gets classified, not the whole of a test's output:
objtool warns on stderr and the runner captures it, and a stray line ahead
of the result would otherwise leave the exit status to decide -- counting
an expected failure, which exits 0, as a pass.
Every test owes the plan exactly one result line, so a substituted verdict
replaces the test's own rather than being printed beside it, and a test
which dies before reporting gets a result of its own instead of being
inferred from its exit status.
Also: a TAP version line, LC_ALL=C so the order tests run in and the
behaviour of grep's character ranges inside them do not depend on the
invoking shell's locale, and fail() on stdout with every other verdict.
Co-developed-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Assisted-by: Claude:claude-opus-5
Link: https://patch.msgid.link/20260916184351.2720310-5-song@kernel.org
|
|
Which architecture a test is for is expressed by where it lives: tests are
in generic/ or in a directory named for their architecture, each carrying
its own fixtures, and the runner executes generic/ plus the one that
matches. A test which cannot apply here is then not run at all, rather
than running in order to announce that it did not.
Layout does this better than a declaration would. There is no x86_only(),
and no lookup letting a fixtures/<arch>/ file shadow a generic one of the
same name -- an arch-specific test simply carries its own fixtures.
Compilers cannot work the same way: CI varies CC over the same tree, so a
compiler requirement stays a declaration in the test.
What a run leaves out is reported once:
# not run: 5 tests in x86/ (this run is arm64)
Silence would have been cheaper and wrong. A run covering less than the
tree holds must not look like a run that covered all of it.
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Assisted-by: Claude:claude-opus-5
Link: https://patch.msgid.link/20260916184351.2720310-4-song@kernel.org
|
|
Every test checked for itself that objtool exists, was built with klp
support, and that $CC runs. Three problems with that: it is the same work
done 40 times, a missing objtool reads as a per-test skip rather than as a
suite which cannot run, and a run in which everything skipped still exits
0.
Do it once, before any test, in klp_preflight() in lib.sh, and export the
answers where the tests can read them. If the suite cannot run the whole
run fails and says why; a test which gets as far as running can assume its
environment.
Nothing is written down, so nothing can go stale. Sourcing lib.sh runs the
checks, which means a test run by hand establishes its own answers rather
than inheriting those of some earlier run with a different CC, and a test
is never handed an empty value for where objtool is or which architecture
it is on: either the exports are set or the run has already failed.
preflight answers only whether the suite can run at all -- not what the
compiler is capable of. A test needing a particular compiler feature
probes for it and skips; that costs one compile and keeps the reason next
to the test that has to justify it.
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Assisted-by: Claude:claude-opus-5
Link: https://patch.msgid.link/20260916184351.2720310-3-song@kernel.org
|
|
"objtool klp checksum" and "objtool klp diff" work on object files alone,
with no kernel, vmlinux or configuration involved, so they can be tested
directly. That is worth doing: most klp generation bugs so far have been in
symbol correlation, special section extraction and relocation conversion,
and several of them failed silently, producing a livepatch which built
cleanly but was missing data.
A test compiles a fixture twice, as the original and (with -DPATCHED) the
patched object, runs both through klp checksum, diffs them and asserts on
the result. Fixtures are compiled at test time rather than committed as
binaries: codegen varies between compilers and architectures, and that
variation is where a good number of these bugs come from.
Assertions check properties rather than compare against recorded output.
Golden files would need re-recording for every compiler change and would
report churn instead of regressions.
klp diff resolves symbols against Module.symvers, so the harness writes
one. Whether a symbol is listed there decides between an ordinary
relocation and a klp relocation, which makes it the main knob tests use.
Run with:
make -C tools/objtool tests
Tests skip when objtool was built without klp support or when a fixture
does not build for the target architecture. A missing objtool binary fails
instead of skipping, since that means a broken invocation rather than an
environment which cannot run the test.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Co-developed-by: Joe Lawrence <joe.lawrence@redhat.com>
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Assisted-by: Claude:claude-opus-5
Link: https://patch.msgid.link/20260916184351.2720310-2-song@kernel.org
|
