.. SPDX-License-Identifier: GPL-2.0 ====================================== Design of the objtool klp test harness ====================================== tools/objtool/tests/ holds unit tests for the klp subcommands of objtool -- ``klp checksum``, ``klp diff``, ``klp post-link`` and ``--klp-symids`` -- which together turn two builds of the kernel into a livepatch module. This document explains how the harness is built and why. For the rules to follow when adding a test, see klp-write-tests.txt. TL;DR ===== One run covers one compiler and one architecture; CI runs the combinations. Build objtool first -- it needs libelf and libxxhash -- and the same ARCH is used for both steps. Natively, with gcc:: make -C tools/objtool make -C tools/objtool tests Natively, with clang -- LLVM=1 additionally selects the LLVM binutils:: CC=clang make -C tools/objtool tests LLVM=1 make -C tools/objtool tests Cross, with gcc -- an arm64 host running the x86 tests:: ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool tests Cross, with clang. It defaults to the host triple however it is invoked, so --target= is what makes it emit x86; OBJCOPY is needed because BFD's is usually built for the host's target alone:: ARCH=x86_64 make -C tools/objtool ARCH=x86_64 CC="clang --target=x86_64-linux-gnu" OBJCOPY=llvm-objcopy \ make -C tools/objtool tests A run ends with a totals line; anything other than fail:0 is a real result:: # pass:48 fail:0 static-skip:1 probe-skip:0 xfail:0 xpass:0 Useful extras:: tools/objtool/tests/run-tests.sh basic # one test, by name tools/objtool/tests/run-tests.sh --keep basic # and keep what it built Why unit tests are possible at all ================================== klp-build is a pipeline: build the kernel twice, checksum both, diff them, link the result. Testing that end to end means two kernel builds per case, which is too slow to run often and too heavy to keep in the tree. Three properties make a much cheaper test possible. **objtool has no configuration-dependent logic.** It never reads ``.config``. Every ``CONFIG_`` string in its source is a comment or one error message, and its only build-time conditionals are driven by host libraries and the target architecture. Configuration reaches objtool through exactly two channels: the ``objtool-args-$(CONFIG_*)`` lines in scripts/Makefile.lib, and the contents of the object handed to it. **The klp subcommands use none of the first channel.** Of objtool's options they consult three -- ``checksum``, ``debug_checksum``, ``dryrun`` -- all from their own command line. So klp behaviour varies with configuration *only* through the input object. **Therefore a test can reproduce any configuration's behaviour by reproducing its input.** Compile a small freestanding fixture with the flags that configuration would have used, and objtool cannot tell the difference. No kernel, no ``.config``, no object cache. The whole suite runs in a few seconds. Shape of a test =============== Each test compiles one fixture twice -- once plain, once with ``-DPATCHED`` -- runs ``klp checksum`` over both, diffs them, and asserts on properties of the output object:: . "$(dirname "$0")/../lib.sh" setup build_pair basic.c assert_input_symbol changed run_diff assert_patched changed assert_not_patched untouched pass "changed function cloned, unchanged function left alone" Assertions check properties, never recorded output. Codegen varies between compilers and versions, so a golden file would report churn rather than regressions. Layout ====== :: tools/objtool/tests/ lib.sh the harness: everything a test may call run-tests.sh selects, runs and classifies generic/ test-*.sh fixtures/*.c x86/ test-*.sh fixtures/*.c Which architecture a test is for is expressed by where it lives. The runner executes ``generic/`` plus the directory matching this architecture, so a test which cannot apply is not run rather than running in order to report that it did not. There is no ``x86_only`` helper, and no lookup letting an architecture fixture shadow a generic one: an architecture-specific test carries its own fixtures. Compilers cannot be expressed the same way, because CI varies ``CC`` over the same tree. A compiler requirement stays a declaration inside the test (``gcc_only``, ``clang_only``). The environment is established once =================================== Sourcing lib.sh runs ``klp_preflight``, which checks that objtool exists and has klp support, that ``$CC`` works, that the binutils are present, and which architecture this is. The answers are exported, so: * ``run-tests.sh`` sources lib.sh too, and therefore knows the architecture before it chooses which tests to run; * each test inherits the answers rather than repeating the work; * a test run on its own establishes them for itself. Preflight answers only whether the suite can run at all. A suite which cannot run must not exit 0 looking like one which passed, so a missing objtool fails the whole run with a TAP ``Bail out!`` rather than skipping each test in turn. What a *particular* compiler can do is a different question, left to the test which cares. Outcomes ======== Output is TAP. The distinction the harness cares most about is between kinds of skip, because a skip is how a suite quietly stops testing anything: ``declared`` The test said in advance it does not apply -- ``gcc_only`` on a clang run. Expected indefinitely. ``probe`` The construct did not turn up in the built object this time. Weaker: one which becomes permanent is a fixture that has stopped testing anything. ``undeclared`` Counted as a **failure**. A test which gives up for a reason it never declared is a hole, not an outcome. ``xfail``/``xpass`` come with them, so a known failure is reported rather than commented out, and one which starts passing says so instead of going quietly green. The runner classifies the TAP result line, not everything a test printed: objtool warns on stderr and that output is captured, so a stray line ahead of the result would otherwise leave the exit status to decide -- and an expected failure exits 0. A run ends with a totals line:: # pass:48 fail:0 static-skip:1 probe-skip:0 xfail:0 xpass:0 and reports what it left out:: # not run: 5 tests in x86/ (this run is arm64) Working directories =================== A run gets one directory; each test gets a subdirectory of it, mirroring the source layout:: /tmp/klp-tests.XXXXXXXX/ generic/test-basic/{orig.o,patched.o,out.o,Module.symvers,...} x86/test-kcfi/... By default (``KEEP=failed``) only failing tests keep their directories; the runner reports where they are. ``KEEP=all`` keeps every test's directory; ``KEEP=none`` removes them all. The runner ``rmdir``s the run directory when it is empty -- which fails if anything was left behind unexpectedly, so a test which dies without cleaning up is reported rather than silently leaking. Running ======= :: make -C tools/objtool # needs libelf and libxxhash make -C tools/objtool tests CC=clang make -C tools/objtool tests # the other toolchain LLVM=1 make -C tools/objtool tests # and its binutils too make -C tools/objtool tests KEEP=all # keep every test's workdir make -C tools/objtool tests KEEP=none # remove all workdirs tools/objtool/tests/run-tests.sh basic # one test; failures kept by default A run covers one compiler and one architecture; CI runs the combinations. Cross-compiled runs ------------------- objtool klp is built only where ARCH_HAS_KLP is set, which today means x86 -- so an arm64 machine cannot run any of this natively. It can run all of it cross, because objtool is a host tool that only reads and rewrites ELF, and the tests only compile fixtures and inspect the objects. Nothing has to execute target code. :: ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool tests objtool itself stays a native binary: it is built with HOSTCC, not CC, so setting a cross compiler cannot produce one the host is unable to run. ARCH selects both the objtool target and the directory of tests to run. clang needs telling, since it defaults to the host triple however it is invoked. LLVM=1 with CROSS_COMPILE does that for you -- the --target= it derives is forwarded to the tests -- and the fixtures include no kernel headers, so no sysroot is needed: :: ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- LLVM=1 \ make -C tools/objtool tests Naming the compiler by hand works too, and is what to reach for when the target triple is not the one CROSS_COMPILE implies: :: ARCH=x86_64 CC="clang --target=x86_64-linux-gnu" \ OBJCOPY=llvm-objcopy make -C tools/objtool tests CROSS_COMPILE picks the binutils, and each can be overridden on its own. readelf reads any target and rarely needs overriding; BFD's objcopy is usually built for the host's alone, hence OBJCOPY=llvm-objcopy above, or install binutils-multiarch. Either readelf will do. The assertions read readelf's output, and the two spell some of it differently -- GNU prints "OS [0xff20]" for SHN_LIVEPATCH where llvm-readelf prints "OS[0xff20]" -- so they accept both. Getting this wrong is easy and the harness refuses rather than producing a misleading result. "CC=clang ARCH=x86_64" alone selects the x86 tests and then builds arm64 objects; preflight compiles a probe object, hands it to objtool, and stops the run if they disagree about the architecture, or if ARCH does not match what the compiler emits. What this does not cover ======================== These are unit tests for objtool's klp subcommands. They do not build a kernel, do not run scripts/livepatch/klp-build, and do not load a livepatch. Behaviour which only appears when the kernel applies a patch -- the module loader refusing a relocation, late module patching ordering -- has to be tested by booting, and is out of scope here.