summaryrefslogtreecommitdiff
path: root/arch/x86/kernel/cfi.c
AgeCommit message (Collapse)Author
2026-10-12x86/kcfi: Decode the kCFI check from the trap addressKees Cook
Since commit b2250d63e6422 ("x86/kcfi: Optimize call sequence"), a CONFIG_FINEIBT=y kernel running in kCFI mode rewrites each kCFI call site at boot so that a failed check traps on the 0xd6 byte of "test $0xd6, %al", one byte past where the compiler's "ud2" was. handle_cfi_failure() steps "addr" back one byte to find the trap in .kcfi_traps, but decode_cfi_insn() still decodes from regs->ip, so it reads each instruction starting at its second byte. "movl $-<id>, %r10d" then decodes as "movl $-<id>, %edx", so the reported type is right by accident, but "addl -15(%r11), %r10d" decodes as "addl -15(%rbx), %edx", so the reported target is whatever %rbx holds: CFI failure at lkdtm_indirect_call+0x16/0x20 (target: 0x12; expected type: 0xc79a2d1b) Pass the adjusted address to decode_cfi_insn() and decode from it. Tested ARCH=x86_64 defconfig with CONFIG_CFI=y and CONFIG_LKDTM=y under QEMU, with LKDTM's CFI_FORWARD_PROTO and both Clang 24.0.0git and GCC 17.0.0 (experimental, with the kCFI series). With CONFIG_FINEIBT=y in kCFI mode, the report gives %rbx as the target without this change, and the expected lkdtm_increment_int() with it. With cfi=fineibt, with cfi=fineibt,paranoid, and with CONFIG_FINEIBT=n, it gives lkdtm_increment_int() both with and without this change. Fixes: b2250d63e6422 ("x86/kcfi: Optimize call sequence") Assisted-by: LLM Signed-off-by: Kees Cook <kees@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20261012090436.1676525-1-kees@kernel.org
2026-07-30x86/kcfi: Optimize call sequencePeter Zijlstra
As noted in commit 85a2d4a890dc ("x86,ibt: Use UDB instead of 0xEA") Jcc should be assumed not-taken, however the normal kCFI (ABI) emits the following sequence: movl $(-hash), %r10d addl -15(%r11), %r10d je 1f ud2 1: cs call __x86_indirect_thunk_r11 (when used in conjunction with -mretpoline-external-thunk). Notably, the Jcc here is always taken, resulting in lower throughput than would be ideal. Replace it with the following sequence on boot: movl $(-hash), %r10d addl -15(%r11), %r10d jne . + 3 test $0xd6, %al cs call __x86_indirect_thunk_r11 This jumps to the UDB instruction used as an immediate byte in the test instruction. The test instruction will clobber eflags, but that is immaterial, eflags is already changed by the preceding addl. Intel recommends the FineIBT sequence on platforms that support IBT; older platforms are still widely used and would benefit from this. An earlier PoC was benchmarked by Scott: Indirect branch miss rate (br_misp_retired.indirect:k / br_inst_retired.indirect:k) BHI_DIS_S=1 Benchmark Baseline IBT kCFI kCFI-opt ----------------------------------------------------------------------------- iperf3 UDP 0.103764 0.103180 0.104311 0.102945 hackbench 0.000885 0.000876 0.001996 0.000826 lmbench syscall 0.005089 0.004486 0.016990 0.005852 lmbench fork+exit 0.018454 0.019176 0.031085 0.015153 lmbench fork+exec 0.017147 0.021613 0.029129 0.016337 redis 0.032220 0.032655 0.045540 0.027946 nginx+wrk 0.109033 0.112765 0.132557 0.102417 fio randread 0.009704 0.009620 0.008548 0.000962 fio seqwrite 0.006927 0.006707 0.019372 0.004590 kbuild 0.056748 0.057324 0.064640 0.048136 BHI_DIS_S=0 Benchmark Baseline IBT kCFI kCFI-opt ----------------------------------------------------------------------------- iperf3 UDP 0.000077 0.000106 0.000186 0.000073 hackbench 0.000123 0.000132 0.000367 0.000097 lmbench syscall 0.023259 0.018319 0.040903 0.012772 lmbench fork+exit 0.011494 0.011887 0.029079 0.016415 lmbench fork+exec 0.037782 0.038994 0.055378 0.026381 redis 0.002481 0.003152 0.017073 0.000184 nginx+wrk 0.015478 0.016266 0.033637 0.000268 fio randread 0.009836 0.007949 0.007096 0.000143 fio seqwrite 0.014587 0.014165 0.041792 0.002157 kbuild 0.055774 0.055249 0.062590 0.046546 Suggested-by: Scott D Constable <scott.d.constable@intel.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Borislav Petkov (AMD) <bp@alien8.de> Tested-by: Nathan Chancellor <nathan@kernel.org> Link: https://patch.msgid.link/20260612071506.GQ187714%40noisy.programming.kicks-ass.net
2025-09-04x86/traps: Clarify KCFI instruction layoutKees Cook
Just a nit-picky change to the KCFI indirect call check instruction documentation. The addl offset isn't always -4 (it depends on patchable function entry configuration). Signed-off-by: Kees Cook <kees@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lore.kernel.org/r/20250904034656.3670313-2-kees@kernel.org
2025-02-26x86/traps: Allow custom fixups in handle_bug()Peter Zijlstra
The normal fixup in handle_bug() is simply continuing at the next instruction. However upcoming patches make this the wrong thing, so allow handlers (specifically handle_cfi_failure()) to over-ride regs->ip. The callchain is such that the fixup needs to be done before it is determined if the exception is fatal, as such, revert any changes in that case. Additionally, have handle_cfi_failure() remember the regs->ip value it starts with for reporting. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Reviewed-by: Kees Cook <kees@kernel.org> Link: https://lore.kernel.org/r/20250224124200.275223080@infradead.org
2025-02-14x86/ibt: Handle FineIBT in handle_cfi_failure()Peter Zijlstra
Sami reminded me that FineIBT failure does not hook into the regular CFI failure case, and as such CFI_PERMISSIVE does not work. Reported-by: Sami Tolvanen <samitolvanen@google.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Sami Tolvanen <samitolvanen@google.com> Link: https://lkml.kernel.org/r/20250214092619.GB21726@noisy.programming.kicks-ass.net
2023-12-15cfi: Flip headersPeter Zijlstra
Normal include order is that linux/foo.h should include asm/foo.h, CFI has it the wrong way around. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Sami Tolvanen <samitolvanen@google.com> Link: https://lore.kernel.org/r/20231215092707.231038174@infradead.org Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2022-09-26x86: Add support for CONFIG_CFI_CLANGSami Tolvanen
With CONFIG_CFI_CLANG, the compiler injects a type preamble immediately before each function and a check to validate the target function type before indirect calls: ; type preamble __cfi_function: mov <id>, %eax function: ... ; indirect call check mov -<id>,%r10d add -0x4(%r11),%r10d je .Ltmp1 ud2 .Ltmp1: call __x86_indirect_thunk_r11 Add error handling code for the ud2 traps emitted for the checks, and allow CONFIG_CFI_CLANG to be selected on x86_64. This produces the following oops on CFI failure (generated using lkdtm): [ 21.441706] CFI failure at lkdtm_indirect_call+0x16/0x20 [lkdtm] (target: lkdtm_increment_int+0x0/0x10 [lkdtm]; expected type: 0x7e0c52a) [ 21.444579] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 21.445296] CPU: 0 PID: 132 Comm: sh Not tainted 5.19.0-rc8-00020-g9f27360e674c #1 [ 21.445296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 21.445296] RIP: 0010:lkdtm_indirect_call+0x16/0x20 [lkdtm] [ 21.445296] Code: 52 1c c0 48 c7 c1 c5 50 1c c0 e9 25 48 2a cc 0f 1f 44 00 00 49 89 fb 48 c7 c7 50 b4 1c c0 41 ba 5b ad f3 81 45 03 53 f8 [ 21.445296] RSP: 0018:ffffa9f9c02ffdc0 EFLAGS: 00000292 [ 21.445296] RAX: 0000000000000027 RBX: ffffffffc01cb300 RCX: 385cbbd2e070a700 [ 21.445296] RDX: 0000000000000000 RSI: c0000000ffffdfff RDI: ffffffffc01cb450 [ 21.445296] RBP: 0000000000000006 R08: 0000000000000000 R09: ffffffff8d081610 [ 21.445296] R10: 00000000bcc90825 R11: ffffffffc01c2fc0 R12: 0000000000000000 [ 21.445296] R13: ffffa31b827a6000 R14: 0000000000000000 R15: 0000000000000002 [ 21.445296] FS: 00007f08b42216a0(0000) GS:ffffa31b9f400000(0000) knlGS:0000000000000000 [ 21.445296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 21.445296] CR2: 0000000000c76678 CR3: 0000000001940000 CR4: 00000000000006f0 [ 21.445296] Call Trace: [ 21.445296] <TASK> [ 21.445296] lkdtm_CFI_FORWARD_PROTO+0x30/0x50 [lkdtm] [ 21.445296] direct_entry+0x12d/0x140 [lkdtm] [ 21.445296] full_proxy_write+0x5d/0xb0 [ 21.445296] vfs_write+0x144/0x460 [ 21.445296] ? __x64_sys_wait4+0x5a/0xc0 [ 21.445296] ksys_write+0x69/0xd0 [ 21.445296] do_syscall_64+0x51/0xa0 [ 21.445296] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 21.445296] RIP: 0033:0x7f08b41a6fe1 [ 21.445296] Code: be 07 00 00 00 41 89 c0 e8 7e ff ff ff 44 89 c7 89 04 24 e8 91 c6 02 00 8b 04 24 48 83 c4 68 c3 48 63 ff b8 01 00 00 03 [ 21.445296] RSP: 002b:00007ffcdf65c2e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 21.445296] RAX: ffffffffffffffda RBX: 00007f08b4221690 RCX: 00007f08b41a6fe1 [ 21.445296] RDX: 0000000000000012 RSI: 0000000000c738f0 RDI: 0000000000000001 [ 21.445296] RBP: 0000000000000001 R08: fefefefefefefeff R09: fefefefeffc5ff4e [ 21.445296] R10: 00007f08b42222b0 R11: 0000000000000246 R12: 0000000000c738f0 [ 21.445296] R13: 0000000000000012 R14: 00007ffcdf65c401 R15: 0000000000c70450 [ 21.445296] </TASK> [ 21.445296] Modules linked in: lkdtm [ 21.445296] Dumping ftrace buffer: [ 21.445296] (ftrace buffer empty) [ 21.471442] ---[ end trace 0000000000000000 ]--- [ 21.471811] RIP: 0010:lkdtm_indirect_call+0x16/0x20 [lkdtm] [ 21.472467] Code: 52 1c c0 48 c7 c1 c5 50 1c c0 e9 25 48 2a cc 0f 1f 44 00 00 49 89 fb 48 c7 c7 50 b4 1c c0 41 ba 5b ad f3 81 45 03 53 f8 [ 21.474400] RSP: 0018:ffffa9f9c02ffdc0 EFLAGS: 00000292 [ 21.474735] RAX: 0000000000000027 RBX: ffffffffc01cb300 RCX: 385cbbd2e070a700 [ 21.475664] RDX: 0000000000000000 RSI: c0000000ffffdfff RDI: ffffffffc01cb450 [ 21.476471] RBP: 0000000000000006 R08: 0000000000000000 R09: ffffffff8d081610 [ 21.477127] R10: 00000000bcc90825 R11: ffffffffc01c2fc0 R12: 0000000000000000 [ 21.477959] R13: ffffa31b827a6000 R14: 0000000000000000 R15: 0000000000000002 [ 21.478657] FS: 00007f08b42216a0(0000) GS:ffffa31b9f400000(0000) knlGS:0000000000000000 [ 21.479577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 21.480307] CR2: 0000000000c76678 CR3: 0000000001940000 CR4: 00000000000006f0 [ 21.481460] Kernel panic - not syncing: Fatal exception Signed-off-by: Sami Tolvanen <samitolvanen@google.com> Reviewed-by: Kees Cook <keescook@chromium.org> Tested-by: Kees Cook <keescook@chromium.org> Tested-by: Nathan Chancellor <nathan@kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Tested-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Kees Cook <keescook@chromium.org> Link: https://lore.kernel.org/r/20220908215504.3686827-23-samitolvanen@google.com