| Age | Commit message (Collapse) | Author |
|---|
|
https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux-dt into soc/dt
Minor improvements in arm DTS for v7.4
A few more or less trivial style cleanups pointed out by dt-check-style
for Actions, Aspeed and ARM vexpress platforms.
* tag 'dt-cleanup-7.4' of https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux-dt:
ARM: dts: aspeed: Replace spaces indentation with tabs
ARM: dts: vexpress-v2p-ca5s: Replace spaces indentation with tabs
ARM: dts: actions: owl-s500: Replace spaces indentation with tabs
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux-dt into soc/dt
Minor improvements in arm64 DTS for v7.4
A few more or less trivial style cleanups pointed out by dt-check-style
for Actions and Cavium platorms.
* tag 'dt64-cleanup-7.4' of https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux-dt:
arm64: dts: cavium: thunder-88xx: Replace spaces indentation with tabs
arm64: dts: actions: Replace spaces indentation with tabs
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux into soc/dt
Samsung DTS ARM changes for v7.4
1. Cleanup of clock-frequency property in favor of assigned-clock-rates.
2. Fix bandwidth of Bluetooth on Exynos4210-based boards.
* tag 'samsung-dt-7.4' of https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux:
ARM: dts: samsung: exynos4210-*: Fix Bluetooth nodes
ARM: dts: samsung: exynos4412-midas: Replace clock-frequency in camera sensor node
ARM: dts: samsung: exynos4210-i9100: Replace clock-frequency in camera sensor node
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux into soc/dt
Samsung DTS ARM64 changes for v7.4
1. Several more or less trivial style cleanups pointed out by
dt-check-style.
2. Exynos850:
- Add Mali GPU, SRAM and mailbox nodes,
- Add new Samsung Galaxy A21s (SM-A217F) mobile phone.
3. Google GS101: improve description of the Type-C and fuel-gauge.
* tag 'samsung-dt64-7.4' of https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux: (23 commits)
arm64: dts: exynos: Add missing space in compatible
arm64: dts: exynos: gs101-pixel-common: add supply and regulator properties
arm64: dts: exynos850: Add acpm SRAM node
arm64: dts: exynos5433: Re-order properties to match coding style
arm64: dts: exynos5433: Re-order nodes to match coding style
arm64: dts: exynos: Add initial support for Samsung Galaxy A21s (SM-A217F)
dt-bindings: arm: samsung: Add compatible for Samsung Galaxy A21s (SM-A217F)
arm64: dts: exynos7870: Re-order properties to match coding style
arm64: dts: exynos2200: Drop redundant blank lines
arm64: dts: exynos2200-g0s: Re-order properties to match coding style
arm64: dts: exynos2200-g0s: Drop chosen address/size cells
arm64: dts: exynos: gs101: Re-order nodes to match coding style
arm64: dts: exynos: gs101: Re-order properties to match coding style
arm64: dts: exynos: axis: Re-order properties to match coding style
arm64: dts: exynos: axis: Re-order nodes to match coding style
arm64: dts: exynos850: Add ap2apm mailbox
arm64: dts: exynos: Add GPU/Mali Bifrost node to Exynos850
dt-bindings: gpu: arm,mali-bifrost: add exynos850-mali compatible
arm64: dts: fsd: Re-order properties to match coding style
arm64: dts: fsd: Re-order nodes to match coding style
...
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux into soc/dt
Google Tensor G5 SoC family - new platform
Introduce new ARCH_GOOGLE for the Google Tensor G5 (Laguna) family of
SoCs. The very early designs - Google GS101 - were derivatives of
Samsung SoC, thus they were grouped under Samsung SoC. The G5 however
went its own way, so no commonality with Samsung and a new ARCH_GOOGLE
symbol. Currently upstreamed SoC is used for example in Google Pixel 10
phones. Add basic support of three of their variants (Pixel 10, Pixel
10 Pro, Pixel 10 Pro XL).
* tag 'google-lga-7.4' of https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux:
arm64: dts: google: lga: fixup #size-cells DT warning
arm64: defconfig: enable Tensor G5 SoC family
arm64: dts: google: Add initial dts for frankel/blazer/mustang
arm64: dts: google: Add dts directory for Google-designed silicon
dt-bindings: arm: google: Add dt bindings for frankel/blazer/mustang
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
traceprobe_expand_meta_args expands probe arguments based on BTF info.
Return an error if we have arguments that need expanding but BTF is not
supported.
The params variable is now unused, it can be removed.
Link: https://lore.kernel.org/all/20261014083838.31020-1-martin@kaiser.cx/
Fixes: d157d7694460 ("tracing/probes: Support BTF field access from $retval")
Signed-off-by: Martin Kaiser <martin@kaiser.cx>
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
|
|
The two parts of '__builtin_choose_expr()' had different arithmetic
expressions. This patch corrects it so that both parts calculate:
COUNT * sizeof(*MEMBER) + __must_be_array(MEMBER)
In practice, '__must_be_array' is meant to assert if 'MEMBER' isn't an
array and return '0' otherwise. However, symmetry is less prone to
subtle errors (and far more satisfying).
Signed-off-by: Bill Wendling <morbo@google.com>
Link: https://patch.msgid.link/20261013203146.387344-1-morbo@google.com
Signed-off-by: Kees Cook <kees@kernel.org>
|
|
Daniel Borkmann says:
====================
bpf: Various x86-64 BPF JIT improvements
This series changes the x86-64 BPF JIT to emit shorter and faster native
code for the instruction patterns that LLVM-compiled programs hit in the
real world. Every change stays within the lowering of a single BPF
instruction, nothing is optimized across instructions. The patterns were
checked against what clang's BPF backend actually emits, and the native
forms follow what LLVM's own X86 backend picks for the same operations.
The last two patches carry the instruction tests of the test_bpf.ko module
over to test_progs as result-checked tests which run on every JIT, add
sweeps for the rewritten patterns with every register as operand, and
finally remove the module given it is legacy code and the BPF selftests
today are the authoritative source of truth for BPF-based testing.
To see what the series does to code that is actually deployed, the BPF
objects of Cilium (7c11a349512b), Tetragon (3f20b90ee580), Katran
(2cd783e43a07) and the sched_ext schedulers (the Rust ones of the scx
repository at fead94e15184 plus the C examples under tools/sched_ext) were
built with clang 20 and every program of them that loads (526, another 39
fail on both kernels) was loaded on two VMs run through vmtest.sh. Both
kernels (before/after this series) got the same objects through the same
loader; the size below is jited_prog_len from bpf_prog_get_info_by_fd().
There are no programs which got bigger in size.
File Program Size (A) Size (B) Size (DIFF)
------------------------------------ ------------------------ --------- --------- ------------------
tetragon/bpf_prog_iter.o iter 157 108 -49 (-31.21%)
scx/scx_simple.bpf.o simple_running 86 80 -6 (-6.98%)
scx/scx_mitosis.bpf.o mitosis_init_task 3362 3130 -232 (-6.90%)
scx/scx_simple.bpf.o simple_enqueue 256 239 -17 (-6.64%)
cilium/bpf_host.o tail_drop_notify 838 786 -52 (-6.21%)
katran/xdp_pktcntr.o pktcntr 179 168 -11 (-6.15%)
scx/scx_cpu0.bpf.o cpu0_enqueue 328 308 -20 (-6.10%)
scx/scx_tickless.bpf.o tickless_init 248 233 -15 (-6.05%)
scx/scx_cosmos.bpf.o cosmos_init 662 623 -39 (-5.89%)
scx/scx_userland.bpf.o userland_init 123 116 -7 (-5.69%)
scx/scx_lavd.bpf.o lavd_tick 1798 1698 -100 (-5.56%)
scx/scx_lavd.bpf.o set_power_profile 471 445 -26 (-5.52%)
cilium/bpf_wireguard.o tail_drop_notify 716 677 -39 (-5.45%)
scx/scx_layered.bpf.o tp_task_rename 382 362 -20 (-5.24%)
scx/scx_lavd.bpf.o lavd_cgroup_exit 59 56 -3 (-5.08%)
scx/scx_bpfland.bpf.o bpfland_init 1128 1071 -57 (-5.05%)
scx/scx_simple.bpf.o simple_select_cpu 240 228 -12 (-5.00%)
tetragon/bpf_execve_map_update_v61.o execve_map_update 322 306 -16 (-4.97%)
[...]
For a feel of the native code, the two loads through an untrusted pointer
at the start of Tetragon's bpf_prog_iter.o iter program.
Before:
24: 49 c7 c2 00 00 60 ff mov $0xffffffffff600000,%r10
2b: 49 89 f3 mov %rsi,%r11
2e: 49 81 c3 50 00 00 00 add $0x50,%r11
35: 4d 29 d3 sub %r10,%r11
38: 49 ba 00 00 a0 00 00 movabs $0x800000a00000,%r10
3f: 80 00 00
42: 4d 39 d3 cmp %r10,%r11
45: 77 04 ja 0x4b
47: 31 f6 xor %esi,%esi
49: eb 04 jmp 0x4f
4b: 48 8b 76 50 mov 0x50(%rsi),%rsi
4f: 49 c7 c2 00 00 60 ff mov $0xffffffffff600000,%r10
After:
21: 4c 8d 9e 50 00 a0 00 lea 0xa00050(%rsi),%r11
28: 49 c1 eb 30 shr $0x30,%r11
2c: 75 04 jne 0x32
2e: 31 f6 xor %esi,%esi
30: eb 04 jmp 0x36
32: 48 8b 76 50 mov 0x50(%rsi),%rsi
36: 4c 8d 9e 20 00 a0 00 lea 0xa00020(%rsi),%r11
The same JITed image dumps were run through llvm-mca 20 as straight-line
code: one pass over every instruction of the image on the scheduling model
of a given CPU, with the helper calls left out since llvm-mca cannot model
them and they are the same on both sides. This is a static model, the path
actually taken through the program, branch prediction and cache misses are
not part of it, but both sides get the same program so the delta is what
the series in here rewrote.
** llvm-mca -mcpu=skylake:
Project Progs Cycles (A) Cycles (B) Cycles (DIFF) uOps (A) uOps (B) uOps (DIFF)
---------- ----- ---------- ---------- ------------------ --------- --------- ------------------
cilium 156 87259 67879 -19380 (-22.21%) 256445 244684 -11761 (-4.59%)
katran 5 1213 1213 +0 (+0.00%) 3944 3944 +0 (+0.00%)
scx 275 72560 70781 -1779 (-2.45%) 133616 131221 -2395 (-1.79%)
tetragon 90 61907 61885 -22 (-0.04%) 188155 188015 -140 (-0.07%)
** llvm-mca -mcpu=icelake-server:
Project Progs Cycles (A) Cycles (B) Cycles (DIFF) uOps (A) uOps (B) uOps (DIFF)
---------- ----- ---------- ---------- ------------------ --------- --------- ------------------
cilium 156 78114 61902 -16212 (-20.75%) 256445 244684 -11761 (-4.59%)
katran 5 1149 1149 +0 (+0.00%) 3944 3944 +0 (+0.00%)
scx 275 66903 65717 -1186 (-1.77%) 133616 131221 -2395 (-1.79%)
tetragon 90 54849 54827 -22 (-0.04%) 188155 188015 -140 (-0.07%)
** llvm-mca -mcpu=alderlake:
Project Progs Cycles (A) Cycles (B) Cycles (DIFF) uOps (A) uOps (B) uOps (DIFF)
---------- ----- ---------- ---------- ------------------ --------- --------- ------------------
cilium 156 51850 50979 -871 (-1.68%) 230693 230022 -671 (-0.29%)
katran 5 1065 1063 -2 (-0.19%) 3816 3809 -7 (-0.18%)
scx 275 54339 54255 -84 (-0.15%) 102485 102317 -168 (-0.16%)
tetragon 90 43254 43225 -29 (-0.07%) 162817 162681 -136 (-0.08%)
** llvm-mca -mcpu=znver4:
Project Progs Cycles (A) Cycles (B) Cycles (DIFF) uOps (A) uOps (B) uOps (DIFF)
---------- ----- ---------- ---------- ------------------ --------- --------- ------------------
cilium 156 66985 66665 -320 (-0.48%) 274895 275648 +753 (+0.27%)
katran 5 1387 1390 +3 (+0.22%) 5188 5188 +0 (+0.00%)
scx 275 117153 116630 -523 (-0.45%) 304094 303980 -114 (-0.04%)
tetragon 90 81172 81156 -16 (-0.02%) 277077 277069 -8 (-0.00%)
On Skylake and Ice Lake class cores the modeled cycles drop by a tenth and
Cilium's by a fifth: those programs divide by constants, where llvm-mca
puts a 64-bit div at 32 uops and 76 cycles of latency (its Ice Lake model
carries over Skylake's divider, the hardware is at about 15 cycles), and
shift by a register, which is 3 uops through cl against 1 with shlx.
Cilium's BPF code holds 416 such divisions in 88 of its programs, all by
NSEC_PER_SEC from its monotonic clock helper, which conntrack, NAT and the
load balancer run for every entry they stamp or check; Tetragon and Katran
have none, the schedulers 75, mostly by 100 and 1000000. Both are cheap on
Zen 4 already, so there the model stays flat at the program level, and the
few programs it rates slower come from its Zen 4 model charging a 64-bit
immediate move a full issue cycle where Agner measures a quarter.
Golden Cove, here the Alder Lake P model which LLVM 20 also uses for the
later client cores and which gives the same numbers as its Sapphire Rapids
model, has a divider at 3 uops and 18 cycles. The division drops from 24 to
10 cycles there, and the totals move by -0.66%, Cilium's by -1.68%. That
model counts register moves and adds as zero uops and puts shlx at 3 cycles
of latency, so it rates the shift by a register a cycle slower. The 27
programs it rates slower are 1 to 6 cycles each, at most +1.16%.
The rewrites on their own, cycles and uops of llvm-mca for one instance of
each sequence in a single pass (the pipeline fill included, and for the
division without the rax/rdx save and restore both sides share):
pattern Skylake Ice Lake Alder Lake Zen 4
r1 /= 1000000000 81 > 10 81 > 10 24 > 10 21 > 8
uops 35 > 6 35 > 6 5 > 5 5 > 6
PROBE_MEM address check 8 > 6 8 > 6 8 > 6 7 > 6
uops 7 > 3 7 > 3 5 > 3 7 > 3
cast to a user address 9 > 6 9 > 6 10 > 7 8 > 6
uops 7 > 5 7 > 5 8 > 5 7 > 5
shift by a register 6 > 4 6 > 4 5 > 6 4 > 4
uops 3 > 1 3 > 1 2 > 1 1 > 1
multiply by 3 6 > 4 6 > 4 6 > 4 6 > 5
uops 1 > 1 1 > 1 1 > 1 1 > 2
16-bit swap, inc/dec, 32-bit and, test for cmp: same cycles and uops,
except one uop less for the 16-bit swap on Alder Lake
====================
Link: https://patch.msgid.link/20261013171230.523677-1-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The instruction tests of the test_bpf module live in the BPF selftests
now, as the jit_insns test of test_progs, which BPF CI runs on every
architecture it covers, and its skb_segment tests have a KUnit successor
in net/core/net_test.c.
Remove the module together with its Kconfig entry, the test_kmod.sh and
test_bpf.sh wrappers of the bpf and net selftests, the config fragments
and defconfigs that enabled it, as well as all other references to it and
symbol exports in core kernel.
For archs wanting to support BPF JITs, it's time they move over to the BPF
selftests for proper coverage.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-19-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Carry only the relevant instruction tests of the test_bpf kernel module
over into test_progs, so that BPF CI runs them on every JIT: ALU,
atomic, 64-bit immediate load and jump instructions for the power-of-two
magnitudes and signs of the operands, all shift counts, all operand
register combinations, the division and modulo by constants, jumps with
staggered offsets, here in both directions and also for the 32-bit
unconditional jump, and the maximal jump over instructions which expand
in the JITs. Add sweeps for the instruction patterns a JIT has a shorter
or faster native form for, with every register as operand. The families
with an immediate run once more with constant blinding.
# LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t jit_insns
[...]
#191/1 jit_insns/arena access with zero offset:OK
#191/2 jit_insns/cast to a user address:OK
#191/3 jit_insns/alu_imm:OK
#191/4 jit_insns/alu_reg:OK
#191/5 jit_insns/alu_imm_regs:OK
#191/6 jit_insns/div_const:OK
#191/7 jit_insns/alu_reg_pairs:OK
#191/8 jit_insns/shifts:OK
#191/9 jit_insns/atomics:OK
#191/10 jit_insns/atomic_reg_pairs:OK
#191/11 jit_insns/xadd_chain:OK
#191/12 jit_insns/ld_imm64:OK
#191/13 jit_insns/jmp_imm:OK
#191/14 jit_insns/jmp_reg:OK
#191/15 jit_insns/jmp_reg_pairs:OK
#191/16 jit_insns/staggered_jumps:OK
#191/17 jit_insns/max_jmp:OK
#191/18 jit_insns/ldst_zero_off:OK
#191/19 jit_insns/ldst_offsets:OK
#191/20 jit_insns/acq_rel:OK
#191/21 jit_insns/movsx:OK
#191/22 jit_insns/jset_imm:OK
#191/23 jit_insns/cmp_imm:OK
#191/24 jit_insns/alu_consts:OK
#191/25 jit_insns/bswap:OK
#191/26 jit_insns/probe_mem:OK
#191 jit_insns:OK
#192/1 jit_insns_blinded/alu_imm:OK
#192/2 jit_insns_blinded/alu_imm_regs:OK
#192/3 jit_insns_blinded/div_const:OK
#192/4 jit_insns_blinded/shifts:OK
#192/5 jit_insns_blinded/xadd_chain:OK
#192/6 jit_insns_blinded/ld_imm64:OK
#192/7 jit_insns_blinded/jmp_imm:OK
#192/8 jit_insns_blinded/ldst_zero_off:OK
#192/9 jit_insns_blinded/ldst_offsets:OK
#192/10 jit_insns_blinded/jset_imm:OK
#192/11 jit_insns_blinded/cmp_imm:OK
#192/12 jit_insns_blinded/alu_consts:OK
#192 jit_insns_blinded:OK
Summary: 2/38 PASSED, 0 SKIPPED, 0/0 FAILED
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-18-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The cast of an arena pointer to its user address zero extended the low
half, shifted it up, or'ed the user base in, rotated it back down and
replaced the result with zero from a cleared r11 through a cmove for a
NULL pointer: seven instructions with a dependency chain of six. Or the
user base into r11 instead and pick it with a cmovne when the low half
is not zero, five instructions with a chain of three. LLVM emits the
cast for every arena pointer handed back to user space.
Before, for r1 = addr_space_cast(r7, 1, 0) with a user base of
0x7f1200000000:
44 89 ef movl %r13d,%edi
48 c1 e7 20 shlq $0x20,%rdi
48 81 cf 12 7f 00 00 orq $0x7f12,%rdi
48 c1 c7 20 rolq $0x20,%rdi
4d 31 db xorq %r11,%r11
85 ff testl %edi,%edi
49 0f 44 fb cmoveq %r11,%rdi
After:
44 89 ef movl %r13d,%edi
49 bb 00 00 00 00 12 7f 00 00 movabsq $0x7f1200000000,%r11
49 09 fb orq %rdi,%r11
85 ff testl %edi,%edi
49 0f 45 fb cmovneq %r11,%rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-17-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The trampoline prologue open codes the choice between the imm8 and the
imm32 form of sub rsp which emit_sub_rsp() already makes. Use it.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-16-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
LLVM turns a multiplication by a power of two into a shift, but keeps
the other constants as a multiplication since BPF has no lea to lower
them into, so r *= 3 and friends reach the JIT as BPF_MUL with BPF_K
and were emitted as imul with an immediate, a 3 cycle instruction on
Intel and AMD. Emit lea dst, [dst + dst * 2/4/8] for 3, 5 and 9
instead, which is the same size and a single cycle on Intel and on
Zen 3 and later, two cycles on older AMD cores. r13 keeps the imul,
since it needs a displacement byte as base register, which makes the
lea a byte longer and a slow three operand form with 3 cycles of
latency on Intel.
Before/after for r1 *= 3:
48 6b ff 03 imulq $0x3,%rdi,%rdi
48 8d 3c 7f leaq (%rdi,%rdi,2),%rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-15-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
LLVM strength-reduces 32-bit divisions by a constant into a 64-bit
multiplication, but has no high-half multiply for the 64-bit ones, so
these reach the JIT as BPF_DIV/BPF_MOD with BPF_K and were emitted as
div or idiv, which take 20-100 cycles on Intel before Ice Lake, about
15 cycles on Ice Lake and later, and 9-19 cycles on AMD Zen 3-5,
whereas a mul is 3 to 4 cycles on all current cores.
Emit the multiplication with a magic number instead, with the constants
computed per Hacker's Delight, 2nd ed., figures 10-1 and 10-2, which is
also what compilers do for native code: an unsigned division becomes a
mul plus a shift, or plus a sub/shr/add/shr sequence when the magic
number needs 65 bits, which an even divisor avoids by shifting the
dividend by its trailing zeros first, a signed one an imul, an optional
add or sub of the dividend, a sar and the sign bit adjustment. A modulo
multiplies the quotient back and subtracts it from the dividend.
Divisors 0 and +-1 as well as unsigned divisors above 2^63 (negative
imm32) are left to div/idiv as before, and so are the 32-bit divisions.
The magic numbers were validated in user space against the bounds from
Hacker's Delight for every divisor an imm32 can hold, in both
signednesses, and the emitted sequences were executed with dividends
around every multiple boundary of the divisor, at the extremes and at
random.
Before, for r1 /= 1000:
50 pushq %rax
52 pushq %rdx
49 c7 c3 e8 03 00 00 movq $0x3e8,%r11
48 89 f8 movq %rdi,%rax
31 d2 xorl %edx,%edx
49 f7 f3 divq %r11
48 89 c7 movq %rax,%rdi
5a popq %rdx
58 popq %rax
After:
50 pushq %rax
52 pushq %rdx
48 89 f8 movq %rdi,%rax
48 c1 e8 03 shrq $0x3,%rax
49 bb cf f7 53 e3 a5 9b c4 20 movabsq $0x20c49ba5e353f7cf,%r11
49 f7 e3 mulq %r11
48 c1 ea 04 shrq $0x4,%rdx
48 89 d7 movq %rdx,%rdi
5a popq %rdx
58 popq %rax
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-14-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The PROBE_MEM check rejects src_reg + off - VSYSCALL_ADDR up to a limit
of TASK_SIZE_MAX + PAGE_SIZE - VSYSCALL_ADDR, which does not fit into a
32-bit immediate and therefore needed a movabs into r10 and a register
compare. Compare against the next power of two above the limit instead,
which is a shift by fls64(limit) followed by a jne on the result. The
addresses between the limit and that power of two are non-canonical on
every x86-64 layout, since user space ends at TASK_SIZE_MAX and the
kernel half starts far above 2^57, so they faulted and had the register
zeroed by the exception handler before. Now they are zeroed by the check
without taking the fault. The compare and branch stay two uops on Intel
and AMD, while the sequence shrinks from 15 to 6 bytes.
Before, for the compare of a 4-level paging kernel:
49 ba 00 00 a0 00 00 80 00 00 movabsq $0x800000a00000,%r10
4d 39 d3 cmpq %r10,%r11
77 05 ja 0x...
After:
49 c1 eb 30 shrq $0x30,%r11
75 05 jne 0x...
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-13-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The kernel address check in front of a BPF_PROBE_MEM load computed
src_reg + off - VSYSCALL_ADDR with a mov of VSYSCALL_ADDR, a mov of
the base register, an add of the offset in imm32 form and a sub, that
is 20 bytes and 4 uops. VSYSCALL_ADDR is -10 MiB, so together with
the 16-bit instruction offset it fits into a 32-bit displacement and
a single lea does the same in 7 bytes and 1 uop on Intel and AMD.
Before, for the check on r0 = *(u64 *)(r1 + 16):
49 c7 c2 00 00 60 ff movq $0xffffffffff600000,%r10
49 89 fb movq %rdi,%r11
49 81 c3 10 00 00 00 addq $0x10,%r11
4d 29 d3 subq %r10,%r11
After:
4c 8d 9f 10 00 a0 00 leaq 0xa00010(%rdi),%r11
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-12-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Commit 77d8f5d47bfb ("bpf,x64: use shrx/sarx/shlx when available") kept
the legacy shift by cl when the count already is in rcx, since that
encoding is two bytes shorter. However, shl/shr/sar r,cl are 3 uops on
Intel from Haswell through Skylake and still 2 uops on Ice Lake, as a
zero count must leave the flags untouched, whereas shlx/shrx/sarx are a
single uop with a latency of 1. On AMD both forms are a single uop, so
the two extra bytes are the only cost there. Use the BMI2 form whenever
it is available. LLVM keeps the count in whichever register it was
allocated to, r4 included.
Before/after for r1 <<= r4:
48 d3 e7 shlq %cl,%rdi
c4 e2 f1 f7 ff shlxq %rcx,%rdi,%rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-11-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The 16-bit byte swap was emitted as a 16-bit rotate followed by a zero
extending move. bswap of the 32-bit register followed by a right shift
by 16 has the same result, is two bytes shorter, and avoids the operand
size prefix and the 16-bit partial register write. Both sequences are
two single cycle uops on Intel and AMD. LLVM emits be16 for bpf_htons()
and bpf_ntohs() with cpu v3 and bswap16 with cpu v4, both of which
take this path.
Before, for r1 = be16 r1:
66 c1 cf 08 rorw $0x8,%di
0f b7 ff movzwl %di,%edi
After:
0f cf bswap %edi
c1 ef 10 shrl $0x10,%edi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-10-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
An add or sub of 1 or -1 is an inc or dec, which is a byte shorter and a
single uop on all current cores, and the flags they leave untouched are
not read across BPF instructions anyway. An add of 128 is the sub of
-128 and the other way round, which fits into an imm8. An and of a 64-bit
register with a non-negative immediate clears the upper half just like
the 32-bit and does, which drops the REX.W prefix on the lower eight
registers. A shift left by one is the add of the register to itself, the
same size and a single uop that runs on more ports. LLVM emits all of
them, r += 1 for every counter, r &= mask for the masks beyond the
byte and word ones and r <<= 1 for x * 2 and x + x, and does the same
replacements for native code.
Before, for r1 += 1, r1 += 128, r1 &= 0x3ff and r1 <<= 1:
48 83 c7 01 addq $0x1,%rdi
48 81 c7 80 00 00 00 addq $0x80,%rdi
48 81 e7 ff 03 00 00 andq $0x3ff,%rdi
48 d1 e7 shlq %rdi
After:
48 ff c7 incq %rdi
48 83 ef 80 subq $-0x80,%rdi
81 e7 ff 03 00 00 andl $0x3ff,%edi
48 01 ff addq %rdi,%rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-9-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The add, sub, and, or and xor with an immediate or a register, the shifts
by an immediate and neg are open coded in the ALU cases of do_jit(), the
add with an immediate once more in the tail call and KASAN check sequences.
Add emit_alu_imm(), emit_alu_reg(), emit_shift_imm() and emit_neg() for
them and use them there, together with emit_mov_reg(), emit_mov_imm32()
and emit_add_rsp() for the open coded moves and stack adjustment in the
tail call, the division and the trampoline. The emitted code is unchanged.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-8-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
LLVM lowers u8 and u16 truncations to r &= 0xff and r &= 0xffff, and
a bitwise not to r ^= -1, since BPF has neither a zero extending move
from a sub-register nor a not instruction. On x86 these masks need the
imm32 form of and as they do not fit into a sign-extended imm8. Emit
a movzx from the low byte or word instead, which is 3 to 4 bytes and
zero extends into the full register for both the 32 and 64-bit variant,
and a not for the xor with -1. Both are single cycle instructions on
Intel and AMD cores.
Before/after for r1 &= 0xff, r1 &= 0xffff and r1 ^= -1:
48 81 e7 ff 00 00 00 andq $0xff,%rdi
40 0f b6 ff movzbl %dil,%edi
48 81 e7 ff ff 00 00 andq $0xffff,%rdi
0f b7 ff movzwl %di,%edi
48 83 f7 ff xorq $-1,%rdi
48 f7 d7 notq %rdi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-7-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
A strict compare against 1 or 128 is the non-strict compare against 0
or 127, and a strict compare against -129 or, with a signed condition,
-1 the non-strict compare against -128 or 0. LLVM canonicalizes x <= 0
to x < 1, x >= 0 to x > -1 and x <= 127 to x < 128, so these reach the
JIT as compares against 1, -1 and 128. Adjust the condition and the
immediate, so that the compare against 0 becomes the test without an
immediate already emitted for it and 127 and -128 fit into an imm8.
Before, for if r1 s> -1 goto ... and if r1 < 128 goto ...:
48 83 ff ff cmpq $-0x1,%rdi
7f xx jg ...
48 81 ff 80 00 00 00 cmpq $0x80,%rdi
72 xx jb ...
After:
48 85 ff testq %rdi,%rdi
7d xx jge ...
48 83 ff 7f cmpq $0x7f,%rdi
76 xx jbe ...
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-6-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
test has no sign-extended imm8 form, so a JSET against a constant was
always emitted as a 6 to 7 byte test with imm32. When the mask fits
into the low byte, the result only depends on that byte, so test the
8-bit register instead, which takes 2 to 4 bytes. Reading the low byte
of a register that was written as a whole has no partial register
penalty on Intel or AMD cores. Also use the one byte shorter accumulator
forms of test and cmp with imm32 when the destination is r0, like the
ALU immediates already do.
Before/after for if r1 & 0x40 goto ...:
48 f7 c7 40 00 00 00 testq $0x40,%rdi
40 f6 c7 40 testb $0x40,%dil
Before/after for if r0 == 0x1000 goto ...:
48 81 f8 00 10 00 00 cmpq $0x1000,%rax
48 3d 00 10 00 00 cmpq $0x1000,%rax
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-5-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The prologue and the tail call sequences always used the imm32 forms
of sub/add rsp and a disp32 load of the tail call counter pointer, even
when the rounded stack depth fits into a byte. Route them through the
existing emit_sub_rsp()/emit_add_rsp() and emit_ldx() helpers which pick
the short forms, and derive the ip adjustment for the counter load in
front of a subprog call from the emitted length instead of hardcoding
the 7 bytes. Nothing depends on the size of these instructions: the tail
call entry offset covers only the code in front of the stack adjustment,
and the poke targets are computed from the emitted bytes.
Before/after for a program with 32 bytes of stack:
48 81 ec 20 00 00 00 subq $0x20,%rsp
48 83 ec 20 subq $0x20,%rsp
48 8b 85 d0 ff ff ff movq -0x30(%rbp),%rax
48 8b 45 d0 movq -0x30(%rbp),%rax
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-4-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
A memory operand with a zero offset was encoded with a mod 01 ModR/M and
an explicit zero displacement byte. Use the mod 00 form instead, which
saves a byte for every load, store and atomic with a zero offset, as
LLVM emits them for the first field of a struct and for every plain
pointer dereference. The exception are rbp and r13 as base register,
where mod 00 with r/m 101 selects a RIP-relative disp32 and with SIB
base 101 a disp32 without base, so these keep their displacement byte.
Also let emit_st() reuse emit_insn_suffix() instead of open-coding the
ModR/M encoding.
Before/after for r0 = *(u64 *)(r1 + 0):
48 8b 47 00 movq 0x0(%rdi),%rax
48 8b 07 movq (%rdi),%rax
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-3-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
movzx into a 32-bit register already clears the upper 32 bits, so the
REX.W prefix on the 8 and 16-bit BPF_LDX loads only costs a byte. This
also matches what the arena variants of these loads emit. Likewise,
the 32-bit movsx from an 8-bit register only needs a REX prefix when
the destination is an extended register or the source is sil, dil, bpl
or an extended register. LLVM emits these loads for u8 and u16 fields
and, with cpu v4, the movsx for a cast to s8 or s16.
Before/after for r0 = *(u8 *)(r1 + 0):
48 0f b6 47 00 movzbq 0x0(%rdi),%rax
0f b6 47 00 movzbl 0x0(%rdi),%eax
Before/after for w1 = (s8)w0:
40 0f be f8 movsbl %al,%edi
0f be f8 movsbl %al,%edi
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013171230.523677-2-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
iommufd_check_iova_range() subtracts one from bitmap->length before it
checks the length, so a zero-length request wraps to SIZE_MAX. With
iova = 0 the wrapped value passes the overflow and alignment checks, and
the request reaches iova_bitmap_alloc() with length 0.
iova_bitmap_alloc() then sizes the bitmap from a length - 1 of SIZE_MAX,
so mapped_total_index is effectively unbounded. The scan uses
last_iova = ULONG_MAX, which spans the whole IOAS when an area is mapped
at IOVA 0. Reject a zero length before the subtraction.
Fixes: b9a60d6f850e ("iommufd: Add IOMMU_HWPT_GET_DIRTY_BITMAP")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Link: https://patch.msgid.link/20260925161142.189813-4-parri.andrea@gmail.com
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
|
|
The iova_bitmap_set() function reports dirty IOVAs into a user-provided
bitmap, it does so by pinning the bitmap pages lazily one chunk at a time
as the reported IOVA advances. When advancing to the next chunk fails,
the function simply returns without recording the dirty bits.
if (iova_bitmap_advance_to(bitmap, iova))
return;
So on an IOMMU_HWPT_GET_DIRTY_BITMAP ioctl, if any bitmap chunk past the
first cannot be pinned, the dirty pages in that chunk are cleared in the
page table, not set in the bitmap, and the ioctl will still return
success. This means that the destination VM will observe stale memory.
This patch records the first such error in the iova_bitmap and propagates
it so the ioctl fails instead of silently succeeding.
Fixes: 4a00f9434891 ("iommupt: Add read_and_clear_dirty op")
Signed-off-by: Samarth Narang <samarth.colleges@gmail.com>
Link: https://patch.msgid.link/20261013171045.3104619-1-samarth.colleges@gmail.com
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
|
|
iommufd_eventq_fops_release() drops the file's users reference before
reading eventq->ictx. Once that reference is dropped, IOMMU_DESTROY in
another thread can remove and free the event queue, so the later load
accesses freed memory.
Read ictx while the users reference still pins the event queue, then
drop the reference and put the saved context. The context reference
taken by iommufd_eventq_init() keeps ictx valid until the put. Fault and
vEVENTQ file descriptors share this release path.
Fixes: 07838f7fd529 ("iommufd: Add iommufd fault object")
Cc: stable@vger.kernel.org
Assisted-by: LLM, Specula
Signed-off-by: Nirmoy Das <nirmoyd@nvidia.com>
Link: https://patch.msgid.link/20261014133632.3440297-1-nirmoyd@nvidia.com
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
|
|
A VFIO PCI dmabuf mapped into an IOAS is mapped with IOMMU_MMIO only in
the domains present when it is mapped. A domain added later, or a copy
of the area into another IOAS, maps the same BAR as cacheable CPU
memory: IOMMU_CACHE set and IOMMU_MMIO clear.
Once the area is in pages->domains_itree, pfn_reader_fill_span() reads
its PFNs back from the storage domain through batch_from_domain(), which
adds them with batch_add_pfn() as BATCH_CPU_MEMORY. Only a hole in the
span reaches pfn_reader_fill_dmabuf() and gets BATCH_MMIO, so
batch_to_domain() programs the later domain with the wrong prot.
The effect depends on the page table format. io-pgtable-arm maps the BAR
as Normal cacheable instead of Device memory, the AMDv1 and x86_64
generic_pt formats set the SME C-bit on it when the tables are encrypted,
and the RISC-V format with Svpbmt maps it as normal memory instead of IO.
Read dmabuf PFNs from the recorded phys for every span, before the
xarray and domain paths. The phys range is always available, and the
read-back from a domain only serves to avoid re-pinning user pages.
Sashiko pointed this out while reviewing an RFC that plumbs the dma-buf
memory type through pfn_reader_fill_dmabuf().
Fixes: 74014a4b55f5 ("iommufd: Have pfn_reader process DMABUF iopt_pages")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://lore.kernel.org/all/20260716154123.32DC01F000E9@smtp.kernel.org/
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Link: https://patch.msgid.link/20261008125206.11990-2-parri.andrea@gmail.com
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
|
|
iommufd_device_do_replace() installs the group's reserved IOVA ranges in
the new hwpt's IOAS only when it differs from the old hwpt's IOAS. If the
replace fails in iommufd_device_do_replace() or
iommufd_group_do_replace_reserved_iova(), the error paths remove the
ranges from the new hwpt's IOAS whether or not the IOAS is shared
between the old and new hwpt, which removes the ranges that the old hwpt
still needs.
For example, on x86 with intel-iommu, after a replace within a shared
IOAS fails with -ENODEV, IOMMU_IOAS_IOVA_RANGES goes from
0x0-0xfedfffff
0xfef00000-0xffffffffffff
to
0x0-0xffffffffffff
and IOMMU_IOAS_MAP at 0xfee00000 succeeds.
Remove the ranges on failure only when the replace installed them, and
look up the old hwpt_paging before the replace so that the error path
can compare the IOAS.
Fixes: e88d4ec154a8 ("iommufd: Add iommufd_device_replace()")
Assisted-by: LLM
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Link: https://patch.msgid.link/20260929031402.1998253-1-donggeunyoo.kernel@gmail.com
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/mmind/linux-rockchip into soc/dt
New boards: MangoPi M28K, NanoPi R28S, DFRobot UNIHIKER M10 V1.2
The gpio controllers on the RK3576 are always on, so they can get aliases
in the main soc dtsi.
Apart from that some cleanup of MMC properties and some board changes.
* tag 'v7.4-rockchip-dts64-2' of https://git.kernel.org/pub/scm/linux/kernel/git/mmind/linux-rockchip:
arm64: dts: rockchip: Add MangoPi M28K
dt-bindings: arm: rockchip: Add MangoPi M28K
arm64: dts: rockchip: Add devicetree for the FriendlyElec NanoPi R28S
arm64: dts: rockchip: Split out the common NanoPi RK3528 parts
dt-bindings: arm: rockchip: Add FriendlyElec NanoPi R28S
arm64: dts: rockchip: Add DFRobot UNIHIKER M10 V1.2
dt-bindings: arm: rockchip: Add DFRobot UNIHIKER M10 V1.2
arm64: dts: rockchip: Add supplies for es8316 on RockPro64
arm64: dts: rockchip: Add PWM fan to Radxa ROCK 3B
arm64: dts: rockchip: Remove cap-mmc-highspeed from SD-only slots
arm64: dts: rockchip: Remove broken-cd on Rock 5 ITX SDIO
arm64: dts: rockchip: Add missing audio codec supplies to Pinebook Pro
arm64: dts: rockchip: Set gpioN aliases on RK3576
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/gclement/mvebu into soc/dt
mvebu dt64 for 7.4 (part 1)
Add Sophos XGS 107w NPU board (CN9130 based)
* tag 'mvebu-dt64-7.4-1' of git://git.kernel.org/pub/scm/linux/kernel/git/gclement/mvebu:
arm64: dts: marvell: Add Sophos XGS 107w NPU board
dt-bindings: arm: marvell: Add Sophos XGS 107w NPU
dt-bindings: vendor-prefixes: Add Sophos
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
A recent LLVM commit [1] renamed the internal LLVM plugin option
'-split-machine-functions' to '-function-splitting=all', breaking the
AutoFDO + ThinLTO build with:
ld.lld: error: -plugin-opt=-: ld.lld: Unknown command line argument '-split-machine-functions'. Try: 'ld.lld --help'
The plugin option is needed for in-process ThinLTO, so handle the rename
accordingly using ld-option.
Cc: stable@vger.kernel.org
Closes: https://github.com/ClangBuiltLinux/linux/issues/2186
Fixes: 2fd65f7afd5a ("AutoFDO: Enable machine function split optimization for AutoFDO")
Link: https://github.com/llvm/llvm-project/commit/3e6acc87c58843e551d6e8e4bf06c26712b04622 [1]
Link: https://patch.msgid.link/20261014-update-autofdo-split-machine-function-option-v1-1-5c71b3005e89@kernel.org
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq
Pull workqueue fix from Tejun Heo:
- Fix a NULL dereference in the check for whether the current task is
running a memory reclaim work item when called from a kworker outside
work item execution
* tag 'wq-for-7.3-rc7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq:
workqueue: Fix NULL current_pwq deref in mem-reclaim helper
|
|
Persistent and remote buffer memory is pre-allocated by the time
__rb_allocate_pages() is called. Skip si_mem_available() to avoid
wrongly failing the allocation under memory pressure.
For the same reason, unset user_thread.
Link: https://patch.msgid.link/20261013102047.1488347-2-vdonnefort@google.com
Fixes: be68d63a139b ("ring-buffer: Add ring_buffer_alloc_range()")
Fixes: 2e67fabd8b77 ("ring-buffer: Introduce ring-buffer remotes")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
tracefs_d_revalidate() reads ->is_freed of the eventfs_inode that
dentry->d_fsdata points to, and ->d_revalidate() is called in rcu
pathwalk mode with nothing but rcu_read_lock() held. The dentry itself
stays around for such a walker, but tracefs_d_release() drops its
reference to the eventfs_inode when the dentry is killed, and
release_ei() then frees it after a grace period of eventfs_srcu. An
SRCU grace period doesn't wait for rcu_read_lock() sections, so the
walker can read an eventfs_inode that is already gone.
With a pathwalk held up in tracefs_d_revalidate() while the dentry goes
away:
BUG: KASAN: slab-use-after-free in tracefs_d_revalidate+0x76/0x90
Read of size 1 at addr ff1100010152d8d0 by task walker/97
Call Trace:
<TASK>
tracefs_d_revalidate+0x76/0x90
lookup_fast+0x320/0x640
path_lookupat+0x160/0xb00
filename_lookup+0x1f1/0x5b0
vfs_statx+0xd7/0x3b0
vfs_fstatat+0x4d/0x90
__do_sys_newfstatat+0xa0/0x100
do_syscall_64+0x10e/0x5e0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
[...]
Allocated by task 1:
__kmalloc_cache_noprof+0x249/0x560
eventfs_create_dir+0x7a/0x590
event_create_dir+0x2fb/0xcf0
trace_add_event_call+0xbf/0x150
[...]
Freed by task 11:
kfree+0x1db/0x5b0
srcu_invoke_callbacks+0x3a3/0x620
process_one_work+0x937/0x1810
worker_thread+0x4f1/0xd60
Before commit 8e556432477e ("eventfs: Use SRCU for freeing
eventfs_inodes") release_ei() used call_rcu(), which covered the walker,
while eventfs_iterate() needs the SRCU grace period. Wait for both and
chain call_rcu() from the SRCU callback.
Cc: stable@vger.kernel.org
Fixes: 8e556432477e ("eventfs: Use SRCU for freeing eventfs_inodes")
Link: https://patch.msgid.link/20261009-work-fixes-rcu-v1-4-7446d2d335e6@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
If trace buffer allocation fails during early boot (e.g. in
early_trace_init() -> allocate_trace_buffers()), tracing_disabled is
left set to 1 and the global trace array events list remains
uninitialized.
Later during boot, tracer_init_tracefs() (invoked via fs_initcall)
queues tracerfs_init_work asynchronously without checking if tracing was
disabled. When tracer_init_tracefs_work_func() runs, it calls
event_trace_init() and init_tracer_tracefs(&global_trace, NULL)
unconditionally. This leads to __find_event_file() dereferencing an
uninitialized tr->events list, causing an early boot NULL pointer
dereference Oops:
BUG: Kernel NULL pointer dereference on read at 0x00000010
Faulting instruction address: 0xc000000000485be0
Oops: Kernel access of bad area, sig: 7 [#1]
...
NIP [c000000000485be0] __find_event_file+0x70/0x3c0
LR [c000000000448874] init_tracer_tracefs+0x274/0xc80
Call Trace:
init_tracer_tracefs+0x274/0xc80
tracer_init_tracefs_work_func+0x50/0x320
process_one_work+0x1e8/0x5c0
worker_thread+0x1dc/0x3d0
kthread+0x194/0x1b0
start_kernel_thread+0x14/0x18
Check tracing_disabled at the beginning of tracer_init_tracefs_work_func()
and return immediately if tracing is disabled or failed to initialize,
preventing the crash.
Cc: stable@vger.kernel.org
Fixes: 6621a7004684 ("tracing: make tracer_init_tracefs initcall asynchronous")
Link: https://patch.msgid.link/20261009163557.88467-2-amachhiw@linux.ibm.com
Reported-by: Michal Suchánek <msuchanek@suse.de>
Closes: https://lore.kernel.org/all/arYskzbiaNzBR9MD@kunlun.suse.cz/
Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com>
Tested-by: Michal Suchánek <msuchanek@suse.de>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Syzkaller triggered a WARN_ON_ONCE(!nest) warning:
WARNING: kernel/trace/ring_buffer.c:821 at ring_buffer_event_time_stamp
Call trace:
ring_buffer_event_time_stamp
hist_field_timestamp
hist_fn_call
event_hist_trigger
event_triggers_call
__event_trigger_test_discard
trace_event_buffer_commit
do_trace_event_raw_event_sched_switch
trace_event_raw_event_sched_switch
The warning is probabilistically reproducible and is related to syzkaller
executing the following commands:
echo 'prev_pid == 999999' > events/sched/sched_switch/filter
echo 'hist:keys=common_timestamp' > events/sched/sched_switch/trigger
The specific triggering process is as follows:
CPU 0 CPU 1
(context switching) (echo 'hist:keys=common_timestamp' > \
events/sched/sched_switch/trigger)
do_trace_event_raw_event_##call
trace_event_buffer_lock_reserve
if (!tr->no_filter_buffering_ref &&
trace_file->flags & EVENT_FILE_FL_FILTERED)
return entry; // return directly
event_hist_trigger_parse
hist_register_trigger
tracing_set_filter_buffering(file->tr, true);
tr->no_filter_buffering_ref++;
__trace_buffer_lock_reserve
rb_start_commit
local_inc(&cpu_buffer->committing); // skipped and not executed
trace_event_buffer_commit
__event_trigger_test_discard
event_triggers_call
event_hist_trigger
hist_fn_call
hist_field_timestamp
ring_buffer_event_time_stamp
nest = local_read(&cpu_buffer->committing);
WARN_ON_ONCE(!nest) // trigger warning
The root cause is that when an event file has a filter attached and the
hist trigger has not yet been attached, trace_event_buffer_lock_reserve()
writes the event into the per-CPU trace_buffered_event temp buffer instead
of reserving it in the ring buffer, so cpu_buffer->committing is not
incremented. When the event is later processed at commit time by a hist
trigger, hist_field_timestamp() -> ring_buffer_event_time_stamp() sees a
zero committing count, triggering WARN_ON_ONCE(!nest).
tracing_event_time_stamp() was added by commit d8279bfc5e959 ("tracing:
Add tracing_event_time_stamp() API") for exactly this case: if the event
is the per-CPU trace_buffered_event, it returns the current ring buffer
timestamp. But it never gained a caller. Use it in hist_field_timestamp(),
so that a hist trigger that races onto a buffered event records the
current time, which is only marginally later than when the event was
recorded, instead of warning and recording a bogus timestamp.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260929104015.1689921-1-wutengda@huaweicloud.com
Fixes: b94bc80df648 ("tracing: Use a no_filter_buffering_ref to stop using the filter buffer")
Signed-off-by: Tengda Wu <wutengda@huaweicloud.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
When tr->allocated_snapshot is true, resize_buffer_duplicate_size() is
called to update the snapshot buffer size. However, its return value was
not checked. In case of a failed resize this could lead next snapshot
being incomplete.
Add resize_buffer_duplicate_size() result check and return error if
resize fails.
Compile tested only.
Found by Linux Verification Center (linuxtesting.org) with SVACE.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260928152634.4354-1-d.kandybka@gmail.com
Fixes: 180e4e390978 ("tracing: Add snapshot refcount")
Signed-off-by: Dmitry Kandybka <d.kandybka@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The mmap reader can warn when it catches up with a writer that is still
committing events. rb_get_reader_page() returns NULL in this case, but
ring_buffer_map_get_reader() treats that result as an error.
The initial rb_per_cpu_empty() check filters out an empty buffer, but it
does not guarantee that the next reader page has committed data. Writers
do not take reader_lock and can advance commit_page before publishing the
committed length on the new page. rb_get_reader_page() can swap to that
page, catch up with the writer and return NULL.
Handle NULL through the existing no-data exit without warning. Remove the
caller's reader_page == commit_page check as well, since the reader-page
helper already handles that case. The existing metadata update and return
path remain in place.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260924084946.20104-1-krystianmkaniewski@gmail.com
Fixes: 117c39200d9d ("ring-buffer: Introducing ring-buffer mapping functions")
Assisted-by: Gemini:gemini-3.8-flash syzbot
Reported-by: syzbot+de3d7f9bcc9212f3fae1@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=de3d7f9bcc9212f3fae1
Link: https://syzkaller.appspot.com/ai_job?id=488adc18-a10e-42d2-a205-25327c38837f
Signed-off-by: Krystian Kaniewski <krystianmkaniewski@gmail.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
simple_ring_buffer relies on headers that end up including
<asm/thread_info.h>. On sparc64, this emits the undefined symbol
current_thread_info_reg. Allow this symbol.
Link: https://patch.msgid.link/20260923074213.3077182-4-vdonnefort@google.com
Fixes: 1211907ac0b5 ("tracing: Generate undef symbols allowlist for simple_ring_buffer")
Reported-by: Guenter Roeck <linux@roeck-us.net>
Closes: https://lore.kernel.org/all/5574c9e6-9ed2-4ee6-a2b2-fb6ee81e83a1@roeck-us.net/
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
simple_ring_buffer uses local_set() and cmpxchg() on unsigned long. Add
them to undefsyms_base.c to extend the allowlist for the parisc
architecture.
Link: https://patch.msgid.link/20260923074213.3077182-3-vdonnefort@google.com
Fixes: 1211907ac0b5 ("tracing: Generate undef symbols allowlist for simple_ring_buffer")
Reported-by: Guenter Roeck <linux@roeck-us.net>
Closes: https://lore.kernel.org/all/5574c9e6-9ed2-4ee6-a2b2-fb6ee81e83a1@roeck-us.net/
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Sparc64 emits empty symmbol names. Skips them when establishing the
allowlist and when extracing undefined symbols.
Link: https://patch.msgid.link/20260923074213.3077182-2-vdonnefort@google.com
Fixes: 1211907ac0b5 ("tracing: Generate undef symbols allowlist for simple_ring_buffer")
Reported-by: Guenter Roeck <linux@roeck-us.net>
Closes: https://lore.kernel.org/all/5574c9e6-9ed2-4ee6-a2b2-fb6ee81e83a1@roeck-us.net/
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
Pull sched_ext fixes from Tejun Heo:
- A task that skipped the BPF scheduler on its way back to its CPU's
local queue could stall there until the watchdog fired, as the CPU
had already decided to go idle.
- Use-after-free of a destroyed user dispatch queue by a deferred
re-enqueue request.
- A task could be moved off a dispatch queue while the dispatching CPU
was still running ops.dequeue() on it.
- Bypass mode mishandled a sub-scheduler bypass overlapping an
ancestor's, as across a system suspend. Descendants could leave
bypass mid-teardown and tasks could be stranded on bypass queues.
- The idle CPU selection helper could return a CPU outside the allowed
CPUs of a pinned or migration-disabled task.
* tag 'sched_ext-for-7.3-rc7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext:
sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs
sched_ext: Rename scx_reenq_wait_dispatching() to scx_wait_dispatching()
sched_ext: idle: Respect task affinity in restricted CPU selection
sched_ext: Fix the bypass dispatch claim for schedulers bypassed by an ancestor
sched_ext: Propagate every bypass depth change to the descendants
sched_ext: Fix use-after-free of a destroyed user DSQ by a deferred reenqueue
sched_ext: Fix stall when a task enqueued with SCX_ENQ_LAST lands back on its CPU's local DSQ
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci
Pull PCI fixes from Bjorn Helgaas:
- Leave ANFE masked on Apple T2 Secure Enclave, which fixes spontaneous
poweroff regressions on MacBookPro16,1 and likely other models with
the T2 caused by new Advisory Non-Fatal Error support (Jason Perlow)
- Move dw-rockchip INTx IRQ setup from rockchip_pcie_host_init(), where
it runs after every reset, to the rockchip_pcie_probe() path to avoid
installing new IRQ handler (which causes INTx IRQs to be lost) and
leaking an IRQ domain on every reset; fixes a regression added this
cycle (Shawn Lin)
* tag 'pci-v7.3-fixes-5' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci:
PCI: dw-rockchip: Move INTx IRQ setup to probe
PCI/AER: Keep ANFE masked on Apple T2 Secure Enclave
|
|
Eduard Zingerman says:
====================
bpf: follow-up for scalar evolution series
Assorted follow-ups for the series [1]:
- Handle ARG_SCALAR and ARG_PTR_TO_DYNPTR in
bpf_global_subprog_stack_access_bytes().
- Fixes for other nits reported by bots for the series.
Changelog v1 -> v2:
- dropped patch #1,
"bpf: static subprogram arguments liveness should use BTF type info",
see the v1 lore link for explanation.
v1: https://lore.kernel.org/bpf/20261010-scev-minimal-v3-followups-v1-0-6000e95ef993@gmail.com/
---
====================
Link: https://patch.msgid.link/20261013-scev-minimal-v3-followups-v2-0-6d9d61965486@gmail.com
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
|
|
The test case indexes inside `char bytes[32]` array with unaligned
offset. Use BPF_F_ANY_ALIGNMENT flag to avoid the test rejection for
unrelated reason.
Fixes: e187fac8ac6b ("selftests/bpf: Tests for varying offset access to PTR_TO_BTF_ID")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013-scev-minimal-v3-followups-v2-7-6d9d61965486@gmail.com
|
|
The `{{.*}}` in the pattern for R5 below might hide the `step=...`
logging from the __no_step annotation. Replace it with full expected
log instead. Also drop __no_step for R3 above, where it is
unnecessary.
Fixes: 2db5837f9ada ("selftests/bpf: Tests for register base/step arithmetic")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013-scev-minimal-v3-followups-v2-6-6d9d61965486@gmail.com
|
|
The original pattern:
__msg("3: (85) call pc+{{[0-9]+}}\n")
Is ambigous, as this line is printed on the main verification pass as
well. Harden the test case by ensuring a __msg_next sequence matching
the stack use/def results specifically.
Fixes: 063c85f5d8c7 ("selftests/bpf: Tests for BTF-based global subprog stack liveness")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20261013-scev-minimal-v3-followups-v2-5-6d9d61965486@gmail.com
|
