| Age | Commit message (Collapse) | Author |
|---|
|
Pull bpf fixes from Alexei Starovoitov:
- Remove BPF_F_ARENA_SCALAR flag and .arena.data section in libbpf that
were added for rust-bpf in this merge window. We found a cleaner
approach that does it on compiler side instead of kernel/libbpf. No
released kernel has them (Alexei Starovoitov)
- Fix bpf_send_signal_task() returning -EPERM when it is called from a
kernel thread (Aditya Sharma)
- Various fixes (Alexei Starovoitov):
- Use emit_rsb_call() to call bpf_unwind_target() in x86 JIT to fix
objtool warning during the build
- Fix scalar evolution: clear delta of a widened register, don't
widen a pointer to a buffer, don't clamp a register whose type
changed in the loop, and reset the range of a clamped packet
pointer
- Finish the insn in the state that maybe_fork_scalars() splits
off, so that pruning doesn't drop it at the head of a counted
loop
- Preserve the registers that carry stack arguments across
arch_bpf_timed_may_goto() on x86-64, arm64 and riscv64
- Don't re-sort subprog_info[] for every new subprogram
- Fix flaky map_kptr and rcu_reentry selftests
- Use memmove() in bpf_skb_load_bytes(), bpf_skb_load_bytes_relative()
and bpf_xdp_load_bytes(), since the destination can be a slice of the
same packet (Ayushman Rout)
- Fixes and follow-ups for scalar evolution: reject pointer operands in
32-bit recurrences and latches, don't intersect with the current
register range on clamp to avoid a false 'verifier bug', avoid false
infinite loop reports, and recognize a 32-bit copy of the induction
variable (Eduard Zingerman)
- Fix out-of-bounds write to btf->types[] for split BTF and propagate
btf_add_type() errors in btf_check_all_metas() (Feng Yang)
- Fix use-after-free in bpf_timer callback that fires after its inner
map was freed (Mani Tofigh)
- Fix x86 JIT stopping before jumps converge: size backward jumps by
the insns they jump over (Mykyta Yatsenko)
- Fix use-after-free of struct_ops trampoline: dissociate struct_ops
progs before the grace period of the map (Qiliang Yuan)
- Validate stack argument reads against the caller state to fix
out-of-bounds read in the verifier, and reject callbacks with stack
arguments (Weiming Shi)
- Tie borrowed iterator results and pointers loaded from trusted fields
to the lifetime of their source, so that iterator next, destroy and
reference release invalidate them (Xu Yunxiang)
* tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf: (48 commits)
bpf: Don't re-sort subprog_info[] for every new subprogram
bpf: Fix out-of-bounds btf->types write for split BTF
selftests/bpf: Test stack arguments across may_goto
bpf: Keep stack arguments in arch_bpf_timed_may_goto()
selftests/bpf: Test maybe_fork_scalars() at the head of a counted loop
bpf: Finish the insn in the state that maybe_fork_scalars() splits off
selftests/bpf: Test the clamp of pointers in scalar evolution
bpf: Reset the range of a packet pointer that scalar evolution clamps
bpf: Don't clamp a register whose type changed in the loop
selftests/bpf: Test a loop that moves a pointer to a buffer
bpf: Don't widen a pointer to a buffer in scalar evolution
selftests/bpf: Avoid a page fault in rcu_reentry while the handler is armed
selftests/bpf: Wait for the last maps of map_kptr to be freed
selftests/bpf: Test that a widened register is not linked
bpf: Clear delta of a register that scalar evolution widens
bpf, x86: Remove r0_on_stack from fetching atomics in arena
bpf: Remove BPF_F_ARENA_SCALAR
selftests/bpf: Remove tests for arena access through numbers
libbpf: Keep format strings of bpf_printk() in .rodata
libbpf: Remove .arena.data
...
|
|
add_subprog() appends new subprog to env->subprog_info[] and calls sort()
on the whole array, though only the last entry can be out of order.
Move subprogs that start after the new one up by one entry instead.
'start' is unique, so the order of the array doesn't change. The new entry
can land in a used slot, so initialize all of it.
add_subprog() returns the index of the new subprog instead of the last
one. The callers check for error only.
It's a trivial prep for rust-bpf [1] that bumps BPF_MAX_SUBPROGS to 1024.
Rust crates have more than 256 functions that are not inlined.
Time in add_subprogs() when veristat loads sched_ext schedulers from scx
repo, 8 cpu VM, usec:
subprogs before after
arena_selftest 101 386 27
lavd_enqueue 52 96 16
lavd_dispatch 48 78 11
[1] https://github.com/4ast/rust-bpf
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Mahe Tardy <mahe.tardy@gmail.com>
Link: https://patch.msgid.link/20261023150302.4097865-1-alexei.starovoitov@gmail.com
|
|
btf_add_type() stores the types of a BTF in btf->types[], which can hold at
most BTF_MAX_TYPE entries. The types of a split BTF get IDs start_id ..
start_id + nr_types - 1, so only BTF_MAX_TYPE - start_id of them fit, but
neither the limit check nor the size used to grow the array accounts for
start_id: the check is an equality test against BTF_MAX_TYPE
('start_id + btf->types_size == BTF_MAX_TYPE') and the new size is clamped
to BTF_MAX_TYPE.
The check therefore only fires if the array happens to grow to exactly
BTF_MAX_TYPE - start_id entries. Otherwise, if the module has more than
BTF_MAX_TYPE types, the array is filled up to BTF_MAX_TYPE entries. The
next type that needs to be added reallocates the array to the same size,
after which btf->types_size no longer equals btf->nr_types and the growth
path is not taken anymore, so every following type is written past the end
of the array:
BUG: unable to handle page fault for address: ffffc90006200000
RIP: 0010:btf_add_type+0x74/0x270
Call Trace:
btf_check_all_metas
btf_parse_module
Bound both the array size and the limit check by BTF_MAX_TYPE - start_id.
Base BTF (start_id == 0) is unaffected.
Fixes: 951bb64621b8 ("bpf: Add in-kernel split BTF support")
Signed-off-by: Feng Yang <yangfeng@kylinos.cn>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Mahe Tardy <mahe.tardy@gmail.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://patch.msgid.link/20261028031933.277550-1-yangfeng59949@163.com
|
|
When a register is -1 or 0, maybe_fork_scalars() splits the state in two
before an 'and' or an 'or' with a constant. The result is more accurate for
each value alone. The verifier continues with -1. It pushes a copy of the
state with 0 and verifies the copy later, from the same insn.
today this patch
N: r1 &= 8 r1 is -1 or 0 N: r1 &= 8 r1 is -1 or 0
/ \ / \
r1 = -1 r1 = 0 r1 = -1 r1 = 0
r1 &= 8 push to N r1 &= 8 r1 &= 8
| | push to N+1
N+1: ... N+1: ...
At insn N the copy is a part of the state that it came from: 0 is in
[-1, 0]. If pruning compares the two states, it drops the copy, and no one
verifies the case r1 = 0.
The first state has branches that are not explored. Usually pruning does
not use such states. In a loop that scalar evolution counted it does, and
the head of such a loop always keeps its state. Thus a fork at the head of
a counted loop loses the case r1 = 0:
r9 = a map value of 8 bytes
r6 = -1
1: r6 &= 8 r6 is -1 or 0, only -1 is verified: 8
r6 ^= 8 0. It is 8 when r6 was 0.
r1 = r9
r1 += r6
r0 = *(u8 *)(r1 + 0)
r6 = -1 or 0
r7 += 1
if r7 < 100 goto 1b
from 11 to 11: safe
The verifier accepts this prog. The prog reads the byte that follows the
map value. With a constant of 64 Tbyte in a register it reads an address
that is not mapped, and the kernel crashes.
Do the insn in the copy too, and push the copy to the next insn. All other
code that pushes a state does the same.
No verdict changes in selftests, Cilium and scx. Each fork processes one
insn less, and the sequence in which the verifier explores states changes
slightly. 62 of 161 progs of Cilium process a different number of insns,
from -0.7% to +3.3%. The total is 986970 -> 987013. 2 of 563 progs of scx
process one insn less.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261027190314.1440199-17-alexei.starovoitov@gmail.com
|
|
A compare with pkt_end sets 'range' in all packet pointers that have the
same id. Then 'range - offset' bytes from the pointer are in the packet.
When a state returns to the head of a counted loop, bpf_clamp_scev_regs()
gives a new id to a packet pointer, and the offsets of all iterations. It
does not change the range. But the range is valid only together with the
old id and the old offsets.
1: ; R6=pkt(id=1,r=0,off=0..100)
r5 = r6
r5 += 100
if r5 > pkt_end goto 2f ; R6=pkt(id=1,r=200,off=0..100)
r6 += 100 ; R6=pkt(id=1,r=200,off=100..200)
r9 += 1
if r9 < 2 goto 1b ; clamp: R6=pkt(id=2,r=200,off=0..100)
after bytes from r6 that the verifier accepts
---------------------- ---------------------------------------
the compare 100, correct
r6 += 100 0, correct
the clamp 100, incorrect. Nothing compared this r6
Usually the verifier prunes the state at the head, and the range has no
effect. The verifier continues with the state when a different part of it
is not within the state at the head. A flag in a 32-bit spill is
sufficient. Then the prog reads and writes after pkt_end. A write to
skb_shared_info crashes the kernel.
Take the range that the pointer has at the entry of the loop. No compare in
the loop found this range, so it is valid in all iterations. The state at
the head has the same range, so the verifier prunes as before.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261027190314.1440199-15-alexei.starovoitov@gmail.com
|
|
When a state returns to the head of a counted loop, bpf_clamp_scev_regs()
replaces the range of each register that changes by a constant step. The
new range is the range over all iterations. The function calculates it from
the register in the state that entered the loop. For a pointer this range
is the range of its offsets.
A call in the loop can invalidate the pointer. bpf_skb_pull_data() does it
to packet pointers, bpf_ringbuf_discard() does it to ringbuf memory. Then
the register is an unknown scalar. Scalar evolution does not see that, and
the clamp gives the offsets of the pointer to the scalar.
r6 = bpf_ringbuf_reserve() ; R6=ringbuf_mem
r7 = r6
1: if first iteration goto 2f
r1 = map_value
r1 += r6 ; accepted: R6 is 0 or 1
r0 = *(u8 *)(r1 + 0)
goto 3f
2: bpf_ringbuf_discard(r7) ; R6=scalar()
3: r6 += 1
r9 += 1
if r9 < 2 goto 1b ; clamp: R6=scalar(umax=1)
At run time r6 is the address of the ringbuf memory plus 1. The prog adds
it to a pointer to a map value, and the kernel crashes in the access.
Do not clamp a register that has a different type than at the entry of the
loop. Such a state is not within the state at the head. Thus the verifier
continues with it, and r6 stays an unknown scalar.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261027190314.1440199-14-alexei.starovoitov@gmail.com
|
|
Scalar evolution widens counted loops. Since that change the verifier
rejects a loop over the per-cpu values of a map element:
SEC("iter/bpf_map_elem")
...
for (i = 0; i < n; i++) {
out[i] = *(__u32 *)pptr;
pptr += 8;
}
loop header at 12, widening r2 to 0..56 step 8
R2 invalid variable buffer offset: off=0, var_off=(0x0; 0x38)
pptr is a pointer to a buffer (PTR_TO_BUF). Scalar evolution replaces its
offset with the range over all iterations. But __check_buffer_access()
accepts only a constant offset. Thus a prog cannot use a widened
PTR_TO_BUF.
Do not widen it. Then the verifier verifies the loop one iteration at a
time, as it did before.
loop before SCEV with SCEV this patch
-------------------------------- ----------- ---------- ----------
counts in i, reads through pptr loads rejected loads
counts in p ('p < end'), no read 1M insns loads 1M insns
counts in p, reads through p 1M insns rejected 1M insns
The verifier cannot decide a comparison of two pointers to a buffer. Thus
it follows 'p < end' until the limit of 1M insns.
bpf_iter_bpf_percpu_array_map.c and bpf_iter_bpf_percpu_hash_map.c have the
first loop. They load today because num_cpus is volatile. The prog reads
the bound again in each iteration. Scalar evolution cannot count such a
loop, and it widens nothing.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261027190314.1440199-12-alexei.starovoitov@gmail.com
|
|
After 'r3 = r8; r3 += 4' the verifier knows that r3 == r8 + 4. The two
registers have the same id, and r3 keeps the difference in reg->delta. When
the prog compares one of them with a constant, sync_linked_regs()
calculates the range of the other register from the difference of the two
deltas.
At a loop head scalar evolution widens a register: it replaces the register
with its range over all iterations. Then the register is not linked to
other registers. scratch_widened_reg_id() clears reg->id, but it does not
clear reg->delta. A subsequent copy of the register gets the stale delta.
Then the range of the copy is incorrect by this delta.
r3 = r8
r3 += 4 ; R3=scalar(id=1+4)
1: ; r3 is widened: no id, delta is still 4
r4 = r3 ; id=2 and delta 4 in both
r4 += 1 ; R4=scalar(id=2+1)
if r3 > 10 goto 3f ; R3 <= 10, R4 = R3 + 1 - 4 <= 7
if r4 <= 7 goto 2f
... ; not verified, r4 is up to 11 here
The verifier does not verify code that can run. Use clear_scalar_id(). It
clears id and delta together. Move it to bpf_verifier.h, so that scev.c can
call it.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261027190314.1440199-8-alexei.starovoitov@gmail.com
|
|
Rust has no address spaces. Thus a prog in Rust accesses arena through
scalars. With BPF_F_ARENA_SCALAR the verifier accepts each load and each
store through a scalar as an access to arena.
The verifier rewrites one insn at a time. Thus it must add one insn to each
access. The compiler knows the source of each pointer. It can emit one
addr_space_cast for each pointer, as it does for the __arena pointers of C.
flag: one move for each access cast: one cast for each pointer
ax = (u32)r1 r1 = addr_space_cast(r1, 0, 1)
r2 = *(u64 *)(ax + 0) r2 = *(u64 *)(r1 + 0)
ax = (u32)r1 r3 = *(u64 *)(r1 + 8)
r3 = *(u64 *)(ax + 8)
20 progs of rust-bpf, with the flag and with the casts that a prototype of
an LLVM pass emits:
flag casts
moves or casts 1568 571
insns 14864 13938
jited bytes 72351 69111
No released kernel has the flag. LLVM does not emit the casts for Rust at
this time. Until it does, Rust progs that access arena do not load.
scx_mavd and scx_nitosis set the flag too. They are experimental schedulers
in C in sched-ext/scx, and they use the flag to omit __arena. They must use
__arena again.
This reverts commit 4c651a91bdfc
("bpf: Treat load and store through a number as arena access"). One part
stays: BPF_REG_AX in reg2pt_regs[] of the x86 JIT. The fetching AND/OR/XOR
atomics in arena use it now.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261027190314.1440199-6-alexei.starovoitov@gmail.com
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull more MM updates from Andrew Morton:
- "mm: make VMA flag semantics explicit, eliminate VM_SPECIAL" (Lorenzo
Stoakes):
Replace VM_SPECIAL with helpers describing mapping ownership and
behavior. Enforce the mmap flag rules and convert affected drivers,
removing ambiguous tests shared by driver mappings and ordinary
memory.
- "support kselftest on nommu platform" (Hajime Tazaki):
Add opt-in mmap and mremap tests for NOMMU systems. Teach the runner
to avoid its fork-dependent timeout command and document the build
and runtime requirements.
- "mm/collapse: separate a collapse from its callers" (Kiryl
Shutsemau):
Separate huge-page collapse from caller policy. Replace khugepaged
identity checks with explicit permissions, and separate scanning from
collapse with the mmap_lock drop in one place.
- "mm: implement and use vma_has_anon_rmap(), silence KCSAN" (Lorenzo
Stoakes):
Provide a common helper for testing whether a VMA has anonymous
reverse mappings. Document the locking rules, annotate intentional
concurrent reads for KCSAN, and convert the core MM callers.
- "mm/sparse: remove SECTION_MARKED_PRESENT and further cleanups"
(David Hildenbrand):
Remove SECTION_MARKED_PRESENT in favor of the early-section flag.
Simplify sparse memory initialization, section metadata handling and
related PFN validity checks.
- "sh: remove NUMA and SPARSEMEM support" (Mike Rapoport):
Remove SuperH NUMA and SPARSEMEM support. The extra memory banks have
been unusable since the NO_BOOTMEM conversion, leaving only the code
and data overhead.
- "mm: Unify device DAX and HugeTLB vmemmap population paths" (Muchun
Song):
Unify the generic device DAX and HugeTLB vmemmap population paths and
shared tail-page handling. Remove the duplicate DAX machinery while
retaining the powerpc radix implementation.
- "kselftest: mm: fix some failure of split_huge_page_test" (Yeoreum
Yun):
Fix huge-page split tests racing with khugepaged recollapse. Use
pagemap and page flags instead of VMA-boundary assumptions which fail
when glibc has already enabled THP.
- "memcg: move memcgid refcount to objcg to unpin dying memcgs"
(Bingfang Guo):
Let dying memcgs be freed while swapped-out shmem still holds their
IDs. Move ID references to reparentable objcgs, with the necessary
swap accounting and RCU lifetime changes.
- "mm/truncate: fix data loss when truncating straddling large folios"
(Zhang Yi):
Preserve out-of-range data when truncating straddling large folios.
Fix a lookup race and split-failure handling, and handle minimum
folio orders for the forthcoming ext4 iomap buffered I/O path.
- "mm/damon: introduce damos quota goal target metric complement flag"
(SJ Park):
Let DAMOS quota goals use a metric's complement, so utilization can
also express free-memory goals without adding separate metrics. Add
the sysfs interface, documentation and tests.
- "arm64/mm: Standardize printing for pgtable entries" (Anshuman
Khandual):
Use the generic formatter for arm64 page-table entries, ready for
future wider entries. Move the safe-PTE-update check out of the
header to avoid an include cycle.
- "mm: kmsan: fix iounmap metadata teardown" (Dima Koziuk):
Fix KMSAN metadata leaks on iounmap and failed ioremap allocations.
Free backing pages before removing their mappings, and share the
teardown with partial-allocation cleanup.
- "mm/damon/core: preserve quota state when constructing schemes" (Karl
Mehltretter):
Preserve running DAMOS quota state while validating parameter
updates. Even rejected updates could reset accounting and exceed the
quota window; add tests for both accepted and rejected updates.
- "mm/vmalloc: fix vmalloc_dump_obj VA lookup" (Ye Liu):
Fix vmalloc_dump_obj() lookups for last-page addresses and
allocations spanning vmap zones. These allocations were being missed
in diagnostic output.
- "mm: various (urgent?) fixes" (David Hildenbrand):
Fix mmap charge leaks and hugetlb use-after-free, wrong-address
unmapping, stuck locks and endless faults. Also preserve reservations
on failed mremap and handle stale userfaultfd bits.
- "maple_tree: fix maple_range_64 crashes in RCU mode" (Dimitris
Charisis):
Fix pointer and metadata confusion during RCU destruction of
maple_range_64 nodes, avoiding double frees and invalid accesses. Add
a regression test; no in-tree user currently uses this combination.
- "mm: fix hugetlb NR_HUGETLB accounting on folio migration" (Hongfu
Li):
Fix hugetlb migration and preallocation accounting so /proc/vmstat
retains the correct count and memory.numa_stat follows the folio to
its new node.
- "mm/damon: fix a minor bug, clean code, and add a test" (SJ Park):
Fix 32-bit DAMON age overflow which made old regions look young.
Share hugetlb folio acquisition code and add a regression test for
the previously fixed last-region skipping bug.
- "mm/damon/core: cleanup code, reduce stack usage, and add kunit" (SJ
Park):
Reduce kdamond stack usage and remove unused enums. Add deterministic
PSI quota-goal tests which do not depend on the machine's current
memory pressure.
- "mm/damon: fix the temporal goal tuner's size quota conversion"
(Donggeun Yoo):
Bound the temporal tuner's quota conversion to prevent overflow and
zero effective quotas, which could stop schemes making progress. Add
boundary tests.
- "Don't use GFP_DMA when calling dma_alloc_coherent" (Baoquan He):
Remove redundant GFP_DMA flags from coherent DMA allocation calls.
The DMA API already selects the zone from device limits and ignores
the caller's flag.
- "kselftest: mm: fix intermittent failure khugepaged test" (Yeoreum
Yun):
Propagate child-process failures in the khugepaged tests and prevent
background collapse from racing their checks. This removes false
passes and intermittent failures.
- "MAINTAINERS: cover all hugetlb headers under HUGETLB SUBSYSTEM"
(Enze Li):
Cover all hugetlb headers in MAINTAINERS so get_maintainer.pl finds
the right maintainers. Also label the closing hugetlb_cgroup.h
include guard.
Summary of the 65 singletons in this merge:
The singletons include a few worthwhile performance fixes. Avoiding
unnecessary reverse-map tree updates gave a reported 15.9% improvement
in UnixBench's execl test on a 192-core x86 system. A reclaim retry
fix stops defrag_mode from spinning under memory.low protection; a
reported file-creation stress test ran 5.7 times faster, with far less
memory pressure. Narrower hugetlb invalidations also spare KVM guests
needless refaults: the reported Redis-in-VM P99 QPS penalty while
punching holes fell from 7.09% to 1.45%.
Several fixes prevent lost data or corrupted memory. Lazyfree
huge-page races with mremap()/munmap() could lose data or access freed
memory, and MADV_COLD could lose a concurrent write's dirty bit.
Another fix prevents stale migration entries from outliving large
folios. On filesystems with blocks smaller than a page, extending a
file by 4 GiB or more could expose bytes written past the old EOF
through mmap instead of zeroes. That is fixed, along with
use-after-free races in PAGEMAP_SCAN during process exit and in
memory-cgroup accounting on PREEMPT_RT kernels.
Other fixes stop UFFDIO_MOVE spinning forever on pinned large folios,
which could wedge other memory operations, and stop mremap() leaking
commit charges until allocations fail under strict overcommit. A
further fix keeps userfaultfd tracking intact on neighboring mappings.
NUMA binding no longer prevents long-term pins when the chosen node
has only movable memory, fixing QEMU/VFIO startup and io_uring buffer
registration failures. MGLRU also gets a fix for early-boot crashes
with memory cgroups disabled at boot, notably in small crash kernels.
* tag 'mm-stable-2026-10-25-19-27' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (213 commits)
mm: multi-gen LRU: fix NULL lruvec->pgdat deref with memcg disabled on cmdline
mm/page_counter: use WRITE_ONCE() for emin/elow when parent is the root
mm/memcontrol: publish null pointer before dropping css reference
test_maple_tree: test maple_range_64 node in RCU mode
maple_tree: fix maple_range_64 crashes in RCU mode
mm/vma: avoid broken uffd merge on mremap
tools/testing/shared: catch a failed posix_memalign()
mm/damon/core: remove unreachable damos charge_{target,addr}_from
mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache
riscv: mm: exclude invalid THP PMDs from page table check
mm/damon/core: filter out damos address filter-split failed region
fs/proc/task_mmu: pin mm_users in do_pagemap_scan()
mm/mempolicy: apply bind policies against the scoped allocation zone
mm/vmscan: reclaim every zone in per-node proactive reclaim
mm/mremap: correctly uncharge accountable VMAs in error path
mm/mremap: populate calculated fields in vrm state correctly
MAINTAINERS: add an entry for MEMORY MANAGEMENT - MEMORY MAP
mm: bdi: reset min_ratio after the sysfs attributes are gone
mm/page_vma_mapped: continue past nonmatching huge PMDs
selftests/mm: skip hugetlb cgroup v2 tests without the needed controllers
...
|
|
A bpf_timer in an inner map can fire after the map has been freed, and
bpf_timer_cb() then reads the freed map:
[ 50.709520] BUG: KASAN: slab-use-after-free in bpf_timer_cb+0x249/0x290
[ 50.709540] Read of size 4 at addr ff11000003bc1030 by task bpftimer/72
...
[ 50.710783] Freed by task 12:
...
[ 50.710826] bpf_map_free+0x1a2/0x380
When user space closes the map, its timers are cancelled and their
callbacks are cleared. Since commit 1bfbc267ec91 ("bpf: Enable
bpf_timer and bpf_wq in any context"), bpf_timer_set_callback() and
bpf_timer_start() run without a lock, so a program that found the timer
just before the close can still use it afterwards:
Task A Task B Task C
------ ------ ------
bpf_timer_start()
finds the timer
bpf_timer_set_callback()
finds the timer
close(map_fd)
clears the callback
cancels the timer
sets the callback
arms the timer
deletes the map from
the outer map
map is freed
the timer fires
bpf_timer_cb() reads the freed map
The timer is cancelled once more when it is finally freed, but that
happens well after the inner map is freed, so a timer started this way
can still fire on the freed map.
Clear the timer's map pointer in bpf_async_cancel_and_free() before the
map can be freed, and make the bpf_timer and bpf_wq callbacks return
early when it is NULL. The map is not freed until callbacks that are
already running have finished, so a callback either sees the map while
it is still valid or sees NULL.
The reproducer below calls the two helpers in a loop on two CPUs while a
third CPU keeps creating an inner map with a timer, closing it and
removing it from the outer map. This bug was identified and confirmed
with the help of a deterministic hypervisor currently under development,
but the non-deterministic C reproducer can typically trigger it in the
1-50s time range. With the fix, nothing is triggered after 600 seconds.
Fixes: 1bfbc267ec91 ("bpf: Enable bpf_timer and bpf_wq in any context")
Signed-off-by: Mani Tofigh <mt4022@columbia.edu>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Tested-by: Mahe Tardy <mahe.tardy@gmail.com>
Link: https://patch.msgid.link/20261027-bpf-timer-freed-map-v1-1-30402b0ee99b@columbia.edu
|
|
- Print large 32-bit bounds without sign extension.
- Omit 32-bit bounds when they coincide with 64-bit bounds.
Before:
loop header at ..., widening r0 to 0x80000000..0xffffffff (s32 0xffffffff80000000..-1) step 1
loop header at ..., widening r0 to 0..3 (s32 0..3) step 1
After:
loop header at ..., widening r0 to 0x80000000..0xffffffff (s32 0x80000000..-1) step 1
loop header at ..., widening r0 to 0..3 step 1
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261026-scev-simpler-clamp-v2-6-f373aec9825f@gmail.com
|
|
The compiler can preserve an induction variable across a call by
copying it to a callee-saved register. For example:
w1 = 0;
loop:
w7 = w1;
call foo;
w1 = w7;
w1 += 2;
if w7 < 10 goto loop;
When building expressions for the loop body, each 32-bit move adds a
zero extension:
w7 = w1; r7 = (zext32 r1)
w1 = w7; r1 = (zext32 (zext32 r1))
w1 += 2; r1 = (zext32 (+ (zext32 (zext32 r1)) 2))
The existing ALU32 simplification rules allow recognizing the header
SCEV for r1 as:
(zext32 (linear r1 2))
Substituting this SCEV into r7's expression at the latch produces:
(zext32 (zext32 (linear r1 2)))
Add the simplification rule:
(zext32 (zext32 X)) -> (zext32 X)
This allows recognizing the saved induction variable in r7 as a
32-bit linear SCEV and computing the loop iteration bound.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261026-scev-simpler-clamp-v2-5-f373aec9825f@gmail.com
|
|
The comment before `total_change` computation:
/*
* compute_backedge_taken_count() guarantees that 'total_change'
* does not overflow in 64-bit domain.
*/
total_change = initial + step * backedge_taken_count;
Is not true before the check:
if (backedge_taken_count >= U32_MAX - 1)
return false;
Move the check before `total_change` computation in order to reconcile
the code with comments. Old position is not a bug the check was
performed either way.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261026-scev-simpler-clamp-v2-4-f373aec9825f@gmail.com
|
|
Clamping now reproduces the widened range at every back edge,
so induction variable ranges no longer vary between iterations.
This can produce identical checkpoint states in terminating programs,
for example:
int main(...) {
for (int i = 0; i < 10; i++) {
foo(...);
for (int j = 0; j < 3; j++)
...;
}
}
void foo(...) {
...;
}
Handle two cases in states.c:bpf_is_state_visited():
- For checkpoints inside foo(), look for active loops in upper frames
as well. Otherwise current_loop() returns NULL and an identical
callee state can trigger an infinite loop report.
- For checkpoints inside the inner loop, skip the infinite loop check
when the current state is inside a terminating loop in any frame.
That loop bounds how many times its body is re-executed, so an
identical state on a later iteration does not imply nontermination.
This covers the inner loop above, re-entered with a fresh entry
state on each outer iteration, and inner loops SCEV cannot analyze.
The nested_terminating_prune test still rejects such an outer loop,
but now reaches the jump complexity limit instead of reporting an
infinite loop. Update its expected message accordingly.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261026-scev-simpler-clamp-v2-3-f373aec9825f@gmail.com
|
|
scev.c:bpf_clamp_scev_regs() is called each time verifier returns to a
loop header via a backedge. It's purpose is to re-establish computed
variable ranges for registers with linear SCEVs.
It can be defined as:
- For each register R with linear SCEV compute:
- loop_range(R) := entry_range(R) * iterations_count
- range(R) := range(R) ∩ loop_range(R)
Such formulation is necessary to avoid ranges over-approximation in
examples like below:
r1 = 0; SCEVs: r1 = (linear r1 1)
r3 = 0; r3 = (linear r3 1)
loop: r3 += 1;
r1 += 1;
if r1 < 5 goto loop;
r0 = 0;
exit;
- At loop entry r3 is widened to be in a range r3 ∈ {0,1,2,3,4}.
- After increment it becomes r3 ∈ {1,2,3,4,5}.
- On a second iteration the bpf_clamp_scev_regs() forces r3 to become
r3 ∈ {1,2,3,4}. W/o clamping logic r3 range wouldn't shrink
preventing loop from converging in states.c:states_equal().
However, intersection logic might lead to assigning an empty range to
a register, as in example below:
r1 = 0; verification
r3 = 0; path
loop: r3 += 1; | <---------.
if r3 != 5 goto +0; | r3 ∈ {5} |
r1 += 1; | |
if r1 != 5 goto loop; '-----------'
r0 = 0;
exit;
- At header r3 ∈ {0,1,2,3,4}.
- While exploring the branch `if r3 != 5 goto ...` with r3 ∈ {5}.
- On a second iteration:
r3 := {0,1,2,3,4} ∩ {5} = ∅
Which stops verification with the following error:
verifier bug: scev clamp produced empty range for r3
Technically this means that the current verification path is
impossible at runtime and should not be traced further.
This commit mitigates the above by removing the intersection logic,
the function is now defined as:
- For each register R with linear SCEV compute:
- range(R) := entry_range(R) * iterations_count
This is still sound, provided that SCEV expressions loop execution
counts are correct.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261026-scev-simpler-clamp-v2-2-f373aec9825f@gmail.com
|
|
Avoid inferring loop bounds for the following program shapes:
32-bit latch (A) | 32-bit recurrence (B)
-------------------------+--------------------------
r1 = ... pointer ... | r1 = ... pointer ...
r2 = ... pointer ... | r2 = ... pointer ...
l: r1 += 8 | l: w1 += 8
... | ...
if w1 < w2 goto l | if r1 < r2 goto l
(A) Scalar bounds for pointers represent only offsets from their
common origin values. The 32-bit operations also depend on the
runtime base address: the addresses can wrap at 32 bits without
the tracked offsets wrapping. Thus, offsets alone do not determine
the iteration count.
(B) 32-bit operations on pointers either prohibited (wX += 1)
or convert them to scalars (wX = wX). No need to deal with
this at SCEV level.
Fixes: 32e962577d2d ("bpf: Support 32-bit linear scalar evolution")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261026-scev-simpler-clamp-v2-1-f373aec9825f@gmail.com
|
|
check_stack_arg_read() bounds a stack argument index with the callee's
BTF-derived argument count, then uses the index in the caller's
stack_arg_regs[] array. The array's actual logical length is recorded in
caller->out_stack_arg_cnt, so the bound and indexed object can diverge.
A crafted BPF_PROG_LOAD can use an argument mismatch to mark a static
subprogram's BTF information unreliable. Later calls return from
btf_check_subprog_call() before check_outgoing_stack_args(), while
check_func_call() continues the static call on -EINVAL. The callee can
then read a slot that the caller never allocated or initialized. The
earlier commit 4286f5deee14 ("bpf: Validate outgoing stack args when
btf_prepare_func_args fails") closed this hole for the first call, but
not for later calls after the subprogram has been marked unreliable.
Callbacks have a related problem. The verifier can make incoming stack
arguments readable from outgoing slots at the helper call site. However,
helpers and kfuncs invoke callbacks through an ABI that passes at most five
arguments, so the runtime callback receives no stack arguments. Async
callback verification can also reach check_stack_arg_read() without a
caller frame.
On x86-64, KASAN reports the direct-call out-of-bounds copy as:
BUG: KASAN: slab-out-of-bounds in check_load_mem+0xb51/0xc60
Read of size 80 at addr ffff8881009af550 by task poc/131
Call Trace:
check_stack_arg_read at kernel/bpf/verifier.c:4234
(inlined by) check_load_mem at kernel/bpf/verifier.c:6645
do_check_common at kernel/bpf/verifier.c:19568
bpf_check at kernel/bpf/verifier.c:21318
bpf_prog_load at kernel/bpf/syscall.c:3134
Reject callback subprograms with incoming stack arguments before their
state is pushed. For direct calls, retrieve the slot with
bpf_get_spilled_stack_arg(), which validates the index against the caller's
out_stack_arg_cnt and rejects uninitialized slots before copying the
register state.
Fixes: 0f6bd5e7a804 ("bpf: Support stack arguments for bpf functions")
Reported-by: <co+eada64ce90ecebce@bugs.sh>
Signed-off-by: Weiming Shi <bestswngs@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261026015250.1674676-1-bestswngs@gmail.com
|
|
An open-coded iterator can return a struct pointer borrowed from its
current element. The verifier has not tied such results to iterator
state since the task_vma iterator was introduced, so a saved result
can still be used after destroy. Commit bee9ef4a40a27 ("bpf: switch
task_vma iterator from mmap_lock to per-VMA locks") also made next
release the protection of the prior element. A later snapshot
conversion made vm_file explicitly borrowed from that element.
Associate non-RCU, non-acquired struct results with the iterator
reference by default. Invalidate dependents before next forks its
NULL and non-NULL paths, while retaining the iterator itself. Let
destroy propagate child-reference errors. Add KF_ITER_RET_INDEPENDENT
for results that outlive the current element, preserving existing
RCU and acquired-reference rules.
Trusted pointer field loads also need to preserve the source lifetime.
Loading a trusted field does not acquire a reference to the pointed-to
object. Set the field pointer's parent_id to the source's reference ID
if the source is referenced, or inherit its parent_id if it is borrowed.
This is a generic rule, including accesses unrelated to iterators. For
example, the following access is now rejected:
file = bpf_get_task_exe_file(task);
if (!file)
return 0;
inode = file->f_inode;
bpf_put_file(file);
ino = inode->i_ino;
The inode pointer borrows the file's lifetime. Releasing the acquired
file reference removes that lifetime guarantee, so the verifier must
invalidate the derived inode pointer as well. This does not imply that
bpf_put_file() immediately frees the file or inode. Reading the inode
while the file reference is held remains valid. Fields with independent
RCU protection retain their existing lifetime rules.
For an iterator result, the same propagation keeps trusted fields such
as vm_file and f_inode tied to the current element. Propagate the backing
object's lifetime through dynptr construction too, so its descendants
remain in the invalidation graph. Share the invalidation traversal
between next and release, so advancing the iterator preserves its own
reference.
Fixes: 4ac454682158 ("bpf: Introduce task_vma open-coded iterator kfuncs")
Suggested-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Xu Yunxiang <xyx2021@mail.ustc.edu.cn>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://patch.msgid.link/20261024100647.2579244-2-xyx2021@mail.ustc.edu.cn
|
|
The bpf_map->ops->map_mmap callback invoked by bpf_map_mmap() can be set to
one of ringbuf_map_mmap_kern(), ringbuf_map_mmap_user(), array_map_mmap()
or arena_map_mmap().
It is convention in mm to mark mappings whose pages the kernel manages
itself with VM_MIXEDMAP, so core mm knows not to treat them as ordinary
page cache or anonymous memory.
The map_mmap callbacks ringbuf_map_mmap_kern() and ringbuf_map_mmap_user()
use remap_vmalloc_range(), which ultimately invokes vm_insert_page() and so
marks the ranges VM_MIXEDMAP, and array_map_mmap() sets VM_MIXEDMAP
explicitly.
However, the exception to this is arena_map_mmap(), which doesn't set the
flag.
This patch corrects this and updates the comment to reflect it.
The pages are refcounted and vm_normal_page() finds them regardless of the
flag, and VM_DONTEXPAND remains set (marking the memory as VM_SPECIAL and
thus unmergeable). The one effect is that NUMA balancing now skips these
VMAs, as it already does for the other bpf map mappings, which is the
reason array_map_mmap() gives for setting the flag.
The intent of this patch is to be able to establish the invariant that only
PFN-mapped or mixed map ranges may clear the VM_MAYWRITE flag, as is done
in bpf_map_mmap().
Link: https://lore.kernel.org/20261013-b4-mmap-prepare-vma-flag-sanify-v6-14-1aa3c9b139f9@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Paul Moore <paul@paul-moore.com>
Cc: Stephen Smalley <stephen.smalley.work@gmail.com>
Cc: Jaroslav Kysela <perex@perex.cz>
Cc: Takashi Iwai <tiwai@suse.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Eduard Zingerman <eddyz87@gmail.com>
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Doug Gilbert <dgilbert@interlog.com>
Cc: James Bottomley <james.bottomley@hansenpartnership.com>
Cc: Martin K. Petersen <mkp@kernel.org>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: Helge Deller <deller@gmx.de>
Cc: Sebastian Reichel <sre@kernel.org>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Harry Yoo <harry@kernel.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Will Deacon <will@kernel.org>
Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Oliver Upton <oupton@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Anup Patel <anup@brainfault.org>
Cc: Paul Walmsley <pjw@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Andreas Larsson <andreas@gaisler.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Youngjun Park <youngjun.park@lge.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Xu Xin <xu.xin@linux.dev>
Cc: Ihor Solodrai <ihor.solodrai@linux.dev>
Cc: Bert Karwatzki <spasswolf@web.de>
|
|
btf_check_all_metas() ignores the return value of btf_add_type(), so a
type that could not be registered is silently dropped while the loop
keeps walking over the type section.
Both failure modes are reachable:
- -E2BIG. Once the types array reaches its capacity (BTF_MAX_TYPE),
every following btf_add_type() fails the same way, so
btf_check_all_metas() still returns 0 and the BTF object is created
with btf->nr_types capped while hdr->type_len claims more records.
- -ENOMEM. btf->nr_types is not incremented and the array is not
grown, so the next type is recorded at the slot of the dropped one
and btf->types[] stays off by one with respect to the type ids for
all remaining types. btf_type_by_id() then returns the wrong type
for those ids for the whole lifetime of the object, and the last
type is silently left unresolved.
Propagate the error so that the object is rejected instead.
Fixes: 69b693f0aefa ("bpf: btf: Introduce BPF Type Format (BTF)")
Signed-off-by: Feng Yang <yangfeng@kylinos.cn>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Mahe Tardy <mahe.tardy@gmail.com>
Acked-by: Alan Maguire <almagui@kernel.org>
Link: https://patch.msgid.link/20261022064727.747304-1-yangfeng59949@163.com
|
|
A struct_ops prog is associated with its map without a reference on the
map, and finds the struct_ops through bpf_prog_get_assoc_struct_ops()
under RCU. For a struct_ops map, bpf_map_put() waits for an RCU grace
period before it calls bpf_struct_ops_map_free(), which clears the
association and then frees the trampoline and the kdata right away.
Nothing waits for a prog that reads the association after that grace
period started and before the association is cleared. A timer callback
of a struct_ops prog that calls test_1 of bpf_testmod's multi_st_ops
through the association while the map is freed returns into the freed
trampoline, whose image is filled with int3. RAX holds MAP_MAGIC (1234),
which test_1 has just returned:
Oops: int3: 0000 [#1] SMP KASAN NOPTI
CPU: 2 UID: 0 PID: 31 Comm: ksoftirqd/2 ...
RIP: 0010:0xffffffffc043114d
Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc <cc> cc cc cc
RAX: 00000000000004d2 ...
Call Trace:
bpf_prog_ec591f4f9d13f364_timer_cb+0x96/0x1e2
bpf_timer_cb+0x15d/0x240
__hrtimer_run_queues+0x264/0x5a0
hrtimer_run_softirq+0x1a7/0x3b0
handle_softirqs+0x18c/0x5b0
Dissociate the progs in map_free_pre_rcu(), so that the grace period of
bpf_map_put() follows it, and make that an RCU tasks trace grace period
for every struct_ops map. It then also waits for the progs that read
the association under RCU tasks trace, e.g. from a bpf_wq callback or a
sleepable prog, as a tasks trace grace period implies an RCU one.
Stop setting free_after_rcu_gp, which bpf_map_put() no longer reads for
a struct_ops map, and update the comment of free_after_mult_rcu_gp in
struct bpf_struct_ops, which now only picks the grace period of a cgroup
prog array.
The st_ops_assoc_in_timer_free subtest of test_progs reproduces it. Its
timer callback keeps calling test_1 through the association while the
map is freed, and test_1 spins in bpf_loop() when called from there:
$ cd tools/testing/selftests/bpf
$ make test_progs
$ ./test_progs -t struct_ops_assoc/st_ops_assoc_in_timer_free
On kernels of the same bpf-next commit with KASAN enabled, x86_64 VM
with 32 vCPUs, before and after this patch:
before after
outcome Oops (int3) on run 1 100 of 100 runs passed, no Oops
Fixes: 5db69b0fbdc8 ("bpf: Make struct_ops tasks_rcu grace period optional")
Signed-off-by: Qiliang Yuan <odys.yuan@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261022-selftests-bpf-struct-ops-assoc-timer-v6-2-2c0f18e82582@gmail.com
|
|
bpf_prog_assoc_struct_ops() and bpf_prog_disassoc_struct_ops() update
prog->aux->st_ops_assoc under st_ops_assoc_mutex, and the latter puts
the map of a non-struct_ops prog while holding it.
A struct_ops map is about to dissociate its progs from
map_free_pre_rcu(), which bpf_map_put() calls in whatever context drops
the last reference, including softirq, e.g. bpf_struct_ops_put() from
tcp_cleanup_congestion_control(). A mutex can't be taken there, and the
put of the last reference under the lock of one prog would take the
lock of other progs.
Replace the mutex with a spinlock taken with interrupts disabled, and
put the map after dropping it.
Signed-off-by: Qiliang Yuan <odys.yuan@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261022-selftests-bpf-struct-ops-assoc-timer-v6-1-2c0f18e82582@gmail.com
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
Pull bpf updates from Alexei Starovoitov:
"Major changes:
- Introduce scalar evolution (SCEV) analysis in the verifier.
It infers how registers and stack slots change in a loop and lets
the verifier check bounded loops with induction variables as ranges
instead of walking every iteration. The number of processed insns
drops ten fold in programs with loops (Eduard Zingerman)
- Add 'callx' instruction: indirect call of a static BPF subprogram.
The address is taken with ld_imm64 or loaded from read-only data.
Supported by x86-64, arm64 and riscv64 JITs (Alexei Starovoitov, Pu
Lehui)
- Add unwind_goto and unwind_exit instructions, so that Rust panic or
C++ exception can run destructors of every frame it passes through.
The normal path is free of unwind code. Supported by x86-64 and
arm64 JITs (Alexei Starovoitov)
- Support programs compiled by rust-bpf: treat load and store through
a number as arena access with BPF_F_ARENA_SCALAR, accept rust names
in BTF (Alexei Starovoitov)
- Raise BPF program stack size from 512 bytes to 2 KiB on x86-64 and
arm64 (Kumar Kartikeya Dwivedi)
- Support struct, union and __int128 up to 16 bytes passed and
returned by value in BPF subprograms and kfuncs, and fix x86-64 and
arm64 trampolines for such arguments (Yonghong Song)
- Introduce a common way to attach struct_ops to a cgroup via
bpf_link. bpf_tcp_ops is the first user and mirrors TCP sock_ops
callbacks (Martin KaFai Lau, Amery Hung)
- Describe inlined functions in BTF. New kinds tell where the inline
sites are and how to fetch the arguments there.
CONFIG_DEBUG_INFO_BTF_INLINE keeps it apart from the main BTF in
.BTF.inline section or in btf_vmlinux_inline.ko (Alan Maguire)
- Fix many memory safety, lifetime and locking bugs in BPF core,
verifier, iterators, local storage, dynptr, devmap, cpumap and
seg6local (Josef Bacik, Alexei Starovoitov)
- Diff stat is dominated by selftests and by removal of
lib/test_bpf.c. Its 16k lines of instruction tests were converted
to BPF selftests that BPF CI runs on every architecture it covers
(Daniel Borkmann)
Other features and fixes:
- Add KASAN checks to memory accesses of JITed programs on x86-64
(Alexis Lothoré)
- Complete unification of helper, kfunc and subprog argument checks
in the verifier (Amery Hung)
- Add bpf_copy_from_user_mm[_str]() kfuncs (Anastasios Papagiannis)
- Various features and fixes (Daniel Borkmann):
- Add .bpf keyring for program signature verification, modelled
after dm-verity keyring, and support ML-DSA in signing tools
- Emit shorter and faster code in x86-64 JIT for patterns that
LLVM generates
- Fix various issues in gotox handling
- Let BPF LSM set xattrs at inode creation with bpf_inode_init_xattr()
(David Windsor, Daniel Borkmann)
- Use sleepable page allocation in sleepable arena paths. Fix races
between arena free, allocation and user page fault (Emil Tsalapatis)
- Switch BPF crypto from lskcipher to AES-CBC and AES-ECB libraries
(Eric Biggers)
- Add bpf_proactive_reclaim() kfunc (Hui Zhu)
- Render decl_tags in libbpf btf_dump and add tests for bpftool C
dump (Ihor Solodrai)
- Speed up tracing_multi attach on x86 (Jiri Olsa)
- Various features and fixes (Kumar Kartikeya Dwivedi):
- Add BPF_PROG_STREAM_OPEN command to read BPF streams through a
file descriptor with poll support
- Let kernel structs publish arena pointers with __arena type tag
- Fix __uninit kfunc output buffers and fastcall rewrite with
indirect stack accesses
- Refuse CO-RE relocations for signed programs
- Add enqueue_rcvq and dequeue_rcvq hooks to bpf_tcp_ops to adjust
sk_rcvlowat and avoid unnecessary wakeups (Kuniyuki Iwashima)
- Add bpf_ksock_sendto() kfunc and allow ksock connect and send from
bpf_wq (Mahe Tardy)
- Fix racy plt target update in arm64 bpf_arch_text_poke()
(Matthew Wood)
- Restructure BPF selftests Makefile as a layered build
(Mykola Lysenko)
- Speed up hash map lookups for u32 and u64 keys (Mykyta Yatsenko)
- Add signed div/mod, movsx, bswap and gotol to MIPS JITs, signed
div/mod to parisc JITs (Nicholas Dudar, Johan Almbladh)
- Fix stack overflow in build ID parser on 32-bit (Nickolai
Zeldovich)
- Add stack arguments, signed arena loads and gotox to riscv64 JIT.
Pu Lehui is now co-maintainer of riscv BPF JITs (Pu Lehui, Chen
Pei, Feng Jiang)
- Add bpf_call_rcu() and bpf_call_rcu_tasks_trace() kfuncs. Support
fetching and/or/xor atomics in arena on x86-64 (Puranjay Mohan)
- Fix flaky selftests, selftests with clang 23 and bpf-gcc
(Qiliang Yuan, KaFai Wan, Vineet Gupta)
- Fix builds broken by fatal resolve_btfids warnings, BTF generation
on ARC, and define arch_cmpxchg_local() on arc, hexagon and
openrisc (Sasha Levin)
- Reduce memory use of hash maps by up to 50% per element
(T.J. Mercier)
- Mark pointer fields of sched_ext ops arguments as trusted (Tejun
Heo)
- Remove legacy bpf_asm, bpf_dbg and bpf_jit_disasm tools (Xia Yan)
- Fix index overflow in queue and stack maps, arena range tree after
allocation failure, tail call loop after freplace detach, lost
signal in bpf_send_signal() (Yuan Chen)
- Many more fixes and cleanups"
* tag 'bpf-next-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (660 commits)
selftests/bpf: Time tc_edt from its first packet with data
selftests/bpf: Test unwind_exit of an exception callback
selftests/bpf: Tests for 32-bit scalar evolution bounds computation
bpf: Reject unwind_goto and unwind_exit in offloaded progs
bpf: Support 32-bit linear scalar evolution
bpf: Make unwind_exit of an exception callback exit in JITs
selftests/bpf: Test forbidden bpf_ksock_sendto() LSM attach
selftests/bpf: Test ksock sendto()
bpf: Add bpf_ksock_sendto() kfunc
selftests/bpf: Test unwinding from R0: R2 subprog
selftests/bpf: Add a C++ prog with exceptions
selftests/bpf: Add tests for unwind_goto and unwind_exit
bpf, arm64: Support unwind_goto and unwind_exit
bpf, x86: Support unwind_goto and unwind_exit
bpf: Add unwind_goto and unwind_exit insns
tools/bpf: Remove legacy BPF tools
bpf: Annotate data races on bpf_lru_node type
selftests/bpf: Add arena allocation race tests
bpf: Fix arena race between page free and alloc leading to incoherency
bpf: Revalidate faulted arena pages before installing user mappings
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull MM updates from Andrew Morton:
- "selftests/mm: improve khugepaged coverage" (Kiryl Shutsemau):
Give khugepaged a proper workout. Extend the tests to all supported
mTHP orders, check exactly which windows collapsed, and make scan
completion deterministic. Race collapse against faults, pinning, fork
and remap; verify that the contents survive.
Add coverage for whole-page-table zaps and memory pressure too.
- "selftests/mm: separate GUP microbenchmarking from functional testing"
(Sarthak Sharma):
Untangle GUP benchmarking from correctness testing. Move the
benchmark into tools/mm, add a standalone harness-based GUP selftest,
and move shared helpers into tools/lib/mm. Give each tool one job
instead of mixing timing measurements with correctness checks.
Cover the mapping, pinning and batch-size combinations explicitly.
- "mm: Switch device DAX to section-based vmemmap optimization" (Muchun
Song):
Apply section-based vmemmap optimization to device DAX. Rework the
backing metadata layout to save memory while preserving the handling
of device memory hotplug and its different alignment and lifetime
requirements.
- "mm: Introduce section-based vmemmap optimization for HugeTLB"
(Muchun Song):
Cut the memory consumed by HugeTLB's struct page array with
section-based vmemmap optimization. Arrange the metadata so large
huge pages need fewer backing pages, while keeping initialization,
teardown and the various hugepage configurations straight.
- "mm/huge_memory: clean up and decouple the anon and file split
helpers" (Kairui Song):
Separate anonymous and file-backed folio splitting, which have rather
different locking and bookkeeping requirements. Move the work into
separate helpers, drop dead checks and tidy swap-cache handling. Fix
error paths, including one that could leave a folio frozen.
Keep the swap cache cases separate from file-cache splitting.
- "mm: Unconditional per-VMA locks and cleanups" (Dave Hansen):
Make per-VMA locking unconditional and clean up the old conditional
code around it. This removes configuration-dependent plumbing and
lets callers depend on one locking model rather than carrying both
cases along.
- "memcg: remove the v1 soft limit" (Shakeel Butt):
Remove the old cgroup v1 memory soft-limit machinery. This has long
since been superseded by the newer memory-control interfaces; hauling
around a separate reclaim scheme for it no longer buys us much.
- "read proc/pid/smaps_rollup under per-vma lock" (Suren Baghdasaryan):
Read /proc/PID/smaps_rollup using per-VMA locks where possible,
rather than holding mmap_lock across the entire walk. Retain the
contention fallback and add tearing tests. This considerably reduces
latency when another thread is busy changing its mappings.
- "mm/damon: introduce data access-as-a-data attribute" (SJ Park):
Teach DAMON to expose memory access observations as data attributes,
making them available to the probe and filtering machinery. Extend
the interfaces, documentation and tests so the new information can be
used rather than merely collected.
- "Remove PG_private by using page/folio->private checks instead" (Zi
Yan):
Get rid of PG_private and use the folio's private pointer to say
whether private data is attached. Convert the filesystem users,
including erofs, f2fs and md, and clean up the old accessors and
flags. There are a few special cases, notably swapcache and hugetlb,
which need explicit care.
- "mm: make userland page table freeing RCU-safe" (Lorenzo Stoakes):
Make freeing userspace page tables safe against RCU readers. Sort out
the page-table lifetime and locking rules so walkers cannot stumble
over freed tables while mappings are being torn down.
- "make unused huge shrinker memcg aware" (Qi Zheng):
Make shmem's unused-huge-folio shrinker aware of memory cgroups, so
reclaim can find those folios under the right memcg. Wire up the
per-memcg tracking and its lifetime rules, and give the shrinklist
its own lockdep class to avoid spurious dependency reports.
- "mm: Drop pxd_ERROR()" (Anshuman Khandual):
Remove the old pxd_ERROR() page-table checks and their
architecture-specific baggage. Simplify the common page-table
interfaces and callers now that these checks no longer serve a
useful purpose.
- "memcg: group struct fields by access pattern" (Shakeel Butt):
Rearrange memcg's hot and cold fields according to how they're
actually accessed. Keep frequently updated counters and their
neighboring state from fighting over cachelines and reduce needless
cacheline traffic.
- "mm/damon/core: fix the size charged for a filter-trimmed region"
(Donggeun Yoo):
Fix DAMOS quota accounting when a filter trims a monitoring region.
Charge only the part left after filtering, not the original region,
and add KUnit coverage. Previously the excluded memory could eat into
the quota and make schemes run more slowly than requested.
- "mm/damon: add kunit and selftests for probes and probe weights" (SJ
Park):
Add more tests for DAMON probes and their weights, including
validation and the results reported to userspace. This is mostly
about catching regressions in the newer probe infrastructure before
they become harder to diagnose.
- "mm: memcontrol: constify the read side of the read side of the memcg
API" (Tal Zussman):
Mark the read-only parts of the memcg interface const, from folio and
lruvec accessors to counters and swap helpers. Most of this is
mechanical, but it makes the contracts clearer and lets the compiler
catch accidental writes.
- "mm/damon/vaddr: support {prep,apply}_probes" (SJ Park):
Support the new prepare/apply probe operations in DAMON's
virtual-address backend. This brings vaddr monitoring into line with
the probe infrastructure and makes the added probe types useful there
too.
- "mm/mglru: clean up folio counters and flag usage" (Kairui Song):
Simplify MGLRU's folio counters and flag handling. Remove redundant
accounting paths and make the state transitions easier to follow
without changing the intended reclaim policy.
- "arch, mm/execmem: resolve confusion about set_direct_map_valid_noflush()"
(Mike Rapoport (Microsoft)):
Clarify and straighten out the set_direct_map_valid_noflush()
interface across architectures and execmem. Make the requested
mapping changes and the callers' expectations agree, rather than
relying on subtly different architecture behavior.
- "mm: zswap: free cold writeback folios promptly" (Alexandre Ghiti):
Free zswap writeback folios as soon as their disk write completes.
They are cold temporary buffers, so putting them on the LRU only
makes reclaim rediscover them later. Extend swap-cache dropbehind and
let zswap keep these folios off the LRU altogether.
- "mm: gup: cleanup gup_fast call chain" (Aristeu Rozanski):
Clean up the GUP-fast call chain and simplify how the fast-path
helpers pass their arguments and results around. Keep the page-pin
behavior the same while making the call sequence easier to maintain.
- "mm: optimize zone-device memmap initialization" (Li Zhe):
Speed up initialization of zone-device struct pages. Avoid redundant
work when setting up large device-memory ranges, where touching every
descriptor can get expensive.
- "mm/zswap: shrink zswap_entry via a pool id" (Jianyue Wu):
Reduce the size of zswap_entry by storing a compact pool identifier
instead of a full pointer. Keep the pool lookup and lifetime handling
correct while saving per-entry metadata.
- "mm/damon: support access monitoring of hugetlb-backed memory"
(Krishna Iyer):
Let DAMON monitor hugetlb-backed memory too. Extend the relevant
access checks and add coverage so those mappings no longer fall
through a hole in the monitoring machinery.
- "mm/execmem: fixes and cleanups for the ROX cache" (Mike Rapoport):
Tidy execmem's read-only-executable cache and fix its handling of the
awkward allocation and release cases. The aim is to make the cache's
ownership and permission transitions more predictable.
- "zsmallc: remove old object read API" (Sergey Senozhatsky):
Drop zsmalloc's old object-read API and move its remaining users to
the supported access mechanism. This removes a duplicate interface
which we no longer need.
- "mm/mempolicy: stop copying state in the interleave paths" (Gregory
Price):
Stop copying mempolicy state in the interleave paths when a reference
to the existing policy will do. This avoids extra work and makes the
ownership rules less obscure.
- "mm/mglru: speed up inc_min_seq() and fix cold/hot inversions" (Barry
Song):
Make MGLRU's generation advancement cheaper and fix cases where cold
and hot folios end up in the wrong order. Adjust the counter and list
handling so reclaim doesn't keep passing over the better eviction
candidates.
- "mm: Fix MF_DELAYED handling on memory failure" (Lisa Wang):
Treat MF_DELAYED as a partially completed memory-failure operation,
not an outright failure. Propagate that result correctly through
shmem and the generic handler, fix the extra-reference accounting,
and add tests for the resulting SIGBUS behavior.
- "tools/mm/page_owner_sort: fix --sort, add module filter, improve
usage" (Ye Liu):
Fix page_owner_sort's --sort handling, add module-based filtering and
make the usage output more helpful. This makes the tool less
frustrating when tracking down who is holding on to pages.
- "mm/damon: move damos filter range arguments validation to core" (SJ
Park):
Move DAMOS filter range validation into the core, where all callers
can share it. Also close the sysfs read-twice races which could pass
inconsistent ranges, and add tests for invalid input.
- "mm/damon: misc cleanups" (Zenghui Yu):
Clean up a collection of small DAMON rough edges in the core,
interfaces and tests. No large behavior change here, just simpler
code and fewer opportunities for confusion.
- "mm/damon: cleanup code, add test cases, and update guidances in
docs" (SJ Park):
Clean up DAMON's core and interfaces, add missing test cases and
bring the documentation into line with the code. These changes should
make the newer facilities easier to understand and maintain.
- "mm/damon: introduce hugepage_size probe filter" (SJ Park):
Add a DAMON probe filter for hugepage size. Let users select a range
of folio sizes through sysfs, validate the range and document the new
interface, so monitoring can concentrate on memory backed by the
desired hugepage sizes.
- "selftests/mm: pagemap_ioctl test fixes and cleanups" (Zenghui Yu):
Fix some pagemap ioctl test assumptions and clean up their setup and
reporting. Make failures point to the operation that went wrong
rather than to test harness bookkeeping.
- "mm: avoid large folio splits when swap is unavailable" (Johannes
Weiner):
Avoid splitting large folios merely to attempt swap when no swap
space is available. There is little point paying for an expensive
split when reclaim cannot actually swap the resulting pages out.
- "mm: workingset: fix the shadow node budget under MGLRU" (Hui Zhu):
Fix the workingset shadow-node budget with MGLRU enabled. Keep the
shadow entries within the intended limits so refault tracking does
not quietly consume more metadata than it should.
- "selftests/mm: TAP output and global-state fixes" (Song Hu):
Fix selftests which produce misleading TAP output or fail to restore
global settings after a run. A test ought to leave the system as it
found it, particularly when a whole suite follows behind.
- "mm: remove page_swap_entry()" (Tal Zussman):
Remove page_swap_entry() and convert its remaining users to the folio
and swap-entry helpers. Lose an old page-level interface and make the
swap paths a little more consistent.
- "mm: refactor zonelist constructors and iterators" (Gregory Price):
Refactor the zonelist construction and walk helpers to reduce
duplicated logic. Keep the allocation ordering rules intact while
making it clearer how zones are selected.
- "mm: kmemleak: batch the struct page scan" (Breno Leitao):
Scan adjacent struct pages in batches instead of taking kmemleak_lock
once for every single page. This preserves scan coverage and the
lock-hold bound, while cutting overhead substantially on debug
kernels.
- "mm/damon: misc improvements in tests and documents" (Zenghui Yu):
Fix some DAMON selftests which leave kdamond running or fail under
Python safe-path mode, and improve the KUnit coverage and
documentation. Nothing especially glamorous, but these are worth
doing.
- "mm/damon: introduce probe_hits_wsum DAMOS core filter" (SJ Park):
Add a DAMOS core filter using weighted probe-hit counts. This gives
schemes another way to select regions based on what the monitoring
probes observed.
- "mm/mglru: fix ineffective memory protection for non-kswapd reclaim"
(Ridong Chen):
Fix MGLRU's memory protection handling for reclaim outside kswapd.
The protection settings weren't having the intended effect in these
paths, leaving protected memory more exposed than requested.
- "mm/damon: improve readability, clarity and test coverage" (SJ Park):
Simplify a handful of DAMON core helpers, clarify their behavior and
expand test coverage. There are no intended userspace-visible changes
in this batch.
- "mm: replace PF_KCOMPACTD/PF_KSWAPD with kthread_func()" (Kefeng
Wang):
Stop identifying reclaim and compaction workers through the PF_KSWAPD
and PF_KCOMPACTD task flags. Use their kthread functions instead and
remove the redundant flag plumbing.
- "mm/hugetlb: fix surplus accounting and availability checks during
demotion" (Longlong Xia):
Fix HugeTLB surplus accounting and availability checks when demoting
huge pages. Keep the counters and reservation decisions consistent as
one huge page turns into several smaller ones.
- "mm/damon: add kunit tests for probe_hits handling and probe params
validation" (Jason Angelov):
Add KUnit tests for DAMON probe-hit handling and invalid probe
parameters. These cover edge cases that were too easy to miss in
ordinary monitoring tests.
- "mm/damon: fix misc bugs in kunit, quota goals and sysfs refresh_ms"
(SJ Park):
Fix a few DAMON bugs in KUnit tests, quota-goal handling and the
sysfs refresh interval. Small fixes, but the incorrect results and
control behavior were real.
- "mm: zswap: optimize zswap invalidate and store" (Kefeng Wang):
Simplify zswap's invalidate and store paths to avoid avoidable
lookups and work. Keep the entry lifetime rules intact while making
those frequent operations cheaper.
- "mm: replace custom ratelimiting logic" (Pedro Falcato):
Replace hand-rolled MM rate limiting with the common kernel helpers.
There is no need to maintain another local version of the same
mechanism.
- "mm/vmalloc: minor cleanups" (Ye Liu):
Tidy some vmalloc initialization and reporting code, and pull
duplicated vmap-area insertion work into a helper. No functional
changes are intended.
- "mm/mglru: clean up isolate_folios for readability and clarity"
(Ridong Chen):
Clean up MGLRU's isolate_folios() so the decisions are easier to read
and verify. Keep reclaim behavior unchanged while removing
unnecessary twists in the control flow.
- "Catch automatic storage in IDA and Maple Tree definitions" (Yury
Norov):
Catch IDA and maple-tree definitions placed in automatic storage
where the lifetime rules are wrong or easy to get wrong. Add checks
so such mistakes are harder to slip through unnoticed.
- "mm, swap: some random fixes and cleanups" (Kemeng Shi):
Fix and simplify a few unrelated swap paths, removing redundant work
and tidying up error handling. A small collection of things which
were better fixed than left lying around.
- "mm/page_io: folio conversion cleanups" (Tal Zussman):
Finish some page-to-folio conversions in swap I/O, removing obsolete
wrappers and using folio operations consistently. The I/O behavior is
intended to remain the same.
- "mm/swap: skip empty clusters in the swapoff scan" (Youngjun Park):
Skip empty swap clusters during swapoff instead of walking slots
which have nothing to release. This avoids pointless work on sparsely
used swap devices.
- "mm: trace: decode architecture-specific VMA flags" (Meijing Zhao):
Teach the VMA trace output to decode architecture-specific flags,
rather than leaving them as unexplained bits. That should make
tracing a good deal more useful when debugging unusual mappings.
- "mm, swap: don't spin on a bad swap entry" (Breno Leitao):
Stop repeatedly retrying a swap entry which is already known to be
bad. Make the error path terminate cleanly instead of wasting CPU
chasing an entry that cannot be used.
- "mm/page_isolation: fix UBSAN shift-out-of-bounds in
isolate_single_pageblock" (Qi Xi):
Fix an out-of-range shift in isolate_single_pageblock() and tighten
its associated boundary handling. UBSAN found the issue; the result
should be more robust pageblock isolation.
- "mm: khugepaged: fix tracepoint UAF" (Vernon Yang):
Fix a use-after-free in khugepaged tracepoint handling. Make sure the
information recorded by tracing stays valid for as long as the trace
event needs it.
- "mm/damon: fix DAMOS bugs in core, paddr and vaddr" (SJ Park):
Fix several DAMOS problems across the core and physical- and
virtual-address backends. The changes close error cases and bring the
implementations back into agreement.
- "mm/hugetlb_cgroup: move the per-node usage along with the folio"
(Hongfu Li):
Move HugeTLB per-node usage accounting when folios migrate between
nodes or are reparented to another cgroup. Otherwise numa_stat can
retain stale charges and even underflow when the folio is freed.
- "docs/core-api: memory-allocation: add k[mz]alloc_obj() and clarify
kmalloc" (Mike Rapoport):
Document the typed k[mz]alloc_obj() allocation helpers and clear up
some kmalloc guidance. Make it easier for callers to choose the right
interface.
- "selftests/mm: Validate selections and scope memfd_secret setup"
(Tianyi Chen):
Check selftest selections before running and confine memfd_secret
setup to the cases that actually need it. Avoid confusing failures
caused by the test harness itself.
- "mm/damon: move zero damos quota target_value handling to the core
layer" (SJ Park):
Handle a zero DAMOS quota goal target in the core rather than making
each user of the interface invent its own rule. Keep quota decisions
consistent across callers.
- "mm/damon: Introduce a huge page collapsing mechanism using auto
tuning" (Asier Gutierrez):
Add an auto-tuned DAMON mechanism for collapsing huge pages. Use
monitoring feedback to decide when collapse is worthwhile rather than
forcing one fixed policy onto every workload.
- "kmemleak: fix stale documentation and raise the verbose default"
(Breno Leitao):
Bring the kmemleak documentation up to date and require a third
confirming scan before verbose kernels print a suspected leak. The
extra scan helps suppress transient false positives in noisy debug
builds.
- "mm/memory: improve insert_pages() error handling" (Avi Weiss):
Tighten insert_pages() error handling so partial failures are
reported and cleaned up consistently. The caller should not have to
guess how far the operation got.
- "mm/damon: allow NULL or unstarted damon_ctx parameter for
damon_call()" (SJ Park):
Make damon_call() handle NULL and not-yet-started contexts safely.
Simplify the rules for callers and avoid an indefinite wait on an
invalid state.
- "mm/damon: introduce pgidle_set probe filter type" (SJ Park):
Add a pgidle_set DAMON probe filter, giving monitoring users another
way to select pages by their idle-tracking state. Wire it through the
filter interfaces and tests.
- "mm/damon/core: remove unused helper functions" (Cheng-Han Wu):
Remove DAMON core helpers which no longer have callers. Keep the API
surface smaller rather than retaining dead code just in case.
- "mm: stop calling pmd_folio() on special PMDs" (Gregory Price):
Avoid calling pmd_folio() on special PMD mappings that have no struct
page. The old assumption could fault in procfs walkers; use the
normal-folio checks and handle those mappings properly.
- "mm: remove three unused helpers from mm.h" (Tal Zussman):
Remove three unused helpers from mm.h. Fewer obsolete interfaces for
the next person to stumble over.
- "mm/damon: improvements in efficiency, error handling, documents"
(Liew Rui Yan):
Skip needless DAMOS quota work once the quota is full, propagate a
missing sysfs error and tidy a couple of comments. The ignored error
could make a monitoring-start request appear successful when it
wasn't.
- "docs/mm/damon/design: add explanation of nr_snapshots" (Liew Rui
Yan):
Explain DAMON's nr_snapshots accounting in the design documentation
and straighten out the associated terminology. Make it clearer what
users should expect from that counter.
- "selftests/cgroup: fixes for test_zswap on single core VM" (Wilson
Felipe Pereira):
Fix two test_zswap failures on single-CPU VMs. Wait for the exiting
child's cgroup to become unpopulated before reconfiguring it, and
avoid another test assumption which breaks under constrained
scheduling.
- "docs/ksm: fix advisor documentation and comment" (Jinjiang Tu):
Correct the KSM advisor documentation and a misleading code comment.
Keep the description aligned with how the advisor actually works.
- "Fix bugs in HugeTLB allocation when mem_cgroup_charge_hugetlb()
fails" (Ackerley Tng):
Fix HugeTLB allocation cleanup when the memcg charge fails. Make sure
failure exits undo what was allocated rather than leaving stale
accounting or resources behind.
- "mm, memcg: fix memory.peak reset clobbering other fds' watermark"
(Ridong Chen):
Fix memory.peak reset handling so resetting one file descriptor
doesn't overwrite the watermark kept for another. Keep those per-open
measurements independent as intended.
- "mm: reject zone device folios in more folio walkers" (Gregory
Price):
Reject zone-device folios in page walkers that cannot safely handle
them. Extend the existing checks rather than assuming every folio has
ordinary system-memory backing.
- "percpu: remove code with no effect" (Sang-Heon Jeon):
Remove a few pieces of percpu code which no longer do anything. There
is no reason to keep maintaining dead paths.
* tag 'mm-stable-2026-10-19-21-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (539 commits)
mm: shmem: assign a lock class to the unused huge shrinklist
mm: vmscan: put rotation-missed folios at the LRU tail
mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches
mm: kmemleak: move the struct page scan into a helper
mm/damon: document that a zero sample_interval is accepted
mm/damon: fix typos in comments
mm/damon/sysfs: propagate damon_call() error in turn_damon_on
mm/damon/core: skip quota score setup when the quota is full
mm/damon/tests/core-kunit: test the size charged for a filter-trimmed region
mm/damon/core: charge only the part of a region the filter left
mm: disallow raw PFN mappings of huge/shared zeropage
selftests/mm: zap whole PTE tables in the khugepaged race harness
selftests/mm: add memory-pressure threads to the khugepaged race harness
selftests/mm: race the collapse of windows with holes
selftests/mm: add khugepaged race harness
selftests/mm: check that one khugepaged pass collapses one window
selftests/mm: run every supported collapse order by default
selftests/mm: cover a shared-source collapse write race
selftests/mm: parameterize the mixed-source collapse case by source order
selftests/mm: add order-parameterized khugepaged collapse cases
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux
Pull hardening updates from Kees Cook:
- Provide kmalloc_obj*-style API for devm_kmalloc()
- Improve seq_buf API for saner C String handling with several bug
fixes along the way (Kees Cook, Bill Wendling)
- Overflow helper readability improvements (Bill Wendling)
- Unexport native_write_cr4() (Dennis Chen)
- um: fix CONFIG_GCOV for built-in code (Johannes Berg)
- Improve stackinit test coverage (Justin Stitt)
- KMSAN: fix memset() when using fortify-source, again (Arnd Bergmann)
- lkdtm: use kmalloc() instead of __get_free_page (Mahad Ibrahim)
- signalfd: don't dequeue the forced fatal signals (Oleg Nesterov)
- Fix randstruct to recognize new container_of() behavior
- coccinelle: kmalloc_obj: Convert multi-byte integral allocations too
- Fix a couple remaining allocation return type mismatches
* tag 'hardening-v7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux: (29 commits)
overflow: Match arithmetic in both parts of __builtin_choose_expr
lkdtm: Make UNSET_SMEP built-in-only and unexport native_write_cr4()
extcon: realtek: Remove const from type_c_cfg allocation type
coresight: cti: Add const to con_groups allocation type
docs: core-api: Document the seq_buf API
nvdimm: ndtest: Return the string length from flags_show()
powerpc/papr_scm: Return the string length from the sysfs show functions
seq_buf: Add seq_buf_init_append()
bpf: Remove dead newline stripping from format_disasm_line()
seq_buf: Add seq_buf_terminate()
seq_buf: Add seq_buf_strlen()
seq_buf: Clear what a writer did not claim when a seq_buf overflows
seq_buf: Copy what fits when seq_buf_puts() and seq_buf_putmem() overflow
scsi: lpfc: Print rx monitor records straight into the seq_buf
seq_buf: Do not pop from an overflowed seq_buf
seq_buf: Do not print an empty line from an overflowed seq_buf_do_printk()
.get_maintainer.ignore: Add Kees Cook
devres: Provide kmalloc_obj*-style API for devm_kmalloc
mm: Move default_gfp() into gfp_types.h
kunit/stackinit: Clean up XFAIL under instrumentation
...
|
|
A prog with unwind_goto or unwind_exit has to be JITed by a JIT that
supports them. The driver JITs an offloaded prog, and nobody looks at
jit_required there.
nfp rejects unwind_goto in nfp_verify_insn(). netdevsim takes everything.
Nothing runs such a prog, but it's not jited for bpf_insn_prepare_dump(),
which takes unwind_goto for BPF_CALL_ARGS and shows it as a call.
Fixes: a2d274771321 ("bpf: Add unwind_goto and unwind_exit insns")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261020231942.2568163-3-alexei.starovoitov@gmail.com
Closes: https://lore.kernel.org/bpf/sashiko-outbox-179499@kernel.org/
|
|
Representation
--------------
Represent ALU32 recurrences as:
w0 += step -> (zext32 (linear r0 step))
Extend simplify() rules to maintain the above form for latch
expressions. E.g. for a post-condition loop such as:
w0 = 0;
l: w0 += 1;
if w0 != 3 goto l;
exit;
SCEV at header: (zext32 (linear r0 1))
Initial SCEV at latch: (zext32 (+ (zext32 (linear r0 1)) 1))
Simplified SCEV at latch: (zext32 (+ (zext32 (linear r0 1)) 1))
-> (zext32 (+ (linear r0 1) 1))
-> (zext32 (linear (+ r0 1) 1))
Iterations count computation
----------------------------
- Extract the main iteration-count logic from compute_max_iters() into
compute_backedge_taken_count(). Reuse its 64-bit arithmetic after
sign/zero extending the latch operands.
- Reject the computed count if the predicted exit value falls outside
signed/unsigned 32-bit range, e.g.:
w0 = 0x7ffffffe;
l: w0 += 1;
if w0 s<= 0x7fffffff goto l;
The 64-bit computation predicts an exit at S32_MAX + 1, but the
32-bit value wraps to S32_MIN and the comparison remains true.
- Handle mixed 64-bit/32-bit comparisons/recurrences only when initial
bounds and step fit in 32-bit ranges, e.g.:
w0 = 0;
r1 = 0x100000003 ll;
l: w0 += 1;
if r0 != r1 goto l;
Truncating the bound to 32 bits would predict three header visits.
However, `w0 += 1` keeps r0 within [0, U32_MAX], so it can never
be equal to the 64-bit bound.
Widened bounds computation
--------------------------
- Update linear_bounds() to truncate the initially computed 64-bit
range to 32 bits, then zero-extend it to obtain the 64-bit range
of iteration values. Step computation needs to handle:
- unsigned 32-bit wraparound;
- entry and zero-extended iteration values disagreeing on base/step.
- Union both ranges with the corresponding register ranges at entry,
since the entry value need not be zero-extended.
- Extend `struct bounds` to keep both {32,64}-bit ranges and pass both
to bpf_set_reg_range(). A 64-bit range alone loses useful 32-bit
information, e.g.:
entry = 0, step = -1, two header visits
r64: [0, U32_MAX]
r32: [-1, 0]
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261020-scev-32bit-v1-1-838f34f9f0cd@gmail.com
|
|
Rust Drop, C++ destructors and __attribute__((cleanup)) in C have to run
when a panic or an exception passes through a frame.
v1 [1] did it with a flag in r3 and 'if r3 != 0 goto pad' after the call.
Three bugs that the bots found there had the same cause: r3 is an ordinary
register, and JITs guessed the meaning from the shape of the code.
Kumar suggested new insns instead [2]:
unwind_goto pad BPF_JMP | BPF_JCOND, src_reg = BPF_UNWIND_GOTO
unwind_exit BPF_JMP | BPF_EXIT, src_reg = BPF_UNWIND_EXIT
'call foo; unwind_goto pad' goes to the pad when foo left with unwind_exit
and falls through when it left with exit. No register says which one it
was. In everything else unwind_exit is exit.
llvm lowers unwinding to them [3]:
ret exit
invoke foo unwind pad call foo; unwind_goto pad
resume r0 = 0; unwind_exit
call _Unwind_RaiseException r0 = 0; unwind_exit
invoke _Unwind_RaiseException goto pad
In a function that returns 16 bytes it's 'r0 = 0; r2 = 0; unwind_exit'.
Landing pads are ordinary blocks, so the verifier checks them like any
other code. A pad that forgets to drop a lock or a reference is a leak on
that path. There are no tables, no stack walk, no kfuncs, no BTF and
nothing to do for libbpf and other loaders.
In the verifier:
- unwind_goto has to follow a call or callx of a subprog and cannot be
a jump target. For cfg, liveness and the rest it's one more conditional
jump, like may_goto.
- How the callee left is one bit of the verifier state. It lives from
the exit of the callee to the next insn. states_equal() looks at it,
since the registers can be the same either way.
- unwind_exit is rejected when unwind_goto doesn't follow the call.
- A global subprog that has unwind_exit can leave either way. Both are
explored after the call, like it's done for might_throw.
- A sync callback cannot unwind. The helper that calls it doesn't expect
that: bpf_loop() would go on.
- unwind_exit of the main prog is exit, unless it's an extension. So it is
in async and exception callbacks. The kernel calls them, and there is no
frame to unwind to.
- An extension leaves either way, like the subprog that it replaced.
freplace is a jump, so the extension returns to the caller of
the subprog. An extension that has unwind_exit can replace only a subprog
that has it too. Otherwise there is no unwind_goto after the calls.
- Of tracing progs only fentry can attach to a subprog or an extension
that has unwind_exit. It would return to the trampoline that called it.
- unwind_exit changes its return address, so the cpu mispredicts
the return and runs what follows unwind_goto speculatively. Without
CAP_PERFMON the verifier looks at that path too.
- unwind_goto goes away together with a pad that turned out to be dead
code. It stays when what follows it is dead code. The cpu predicts that
the callee returns there, so 'goto pc-1' takes the place of the dead
code. Otherwise its removal would move other code there.
JITs don't emit a branch:
unwind_goto:
extable[n++] = { ret_addr - 1 -> pad }
unwind_exit:
my_ret_addr = bpf_unwind_target(my_ret_addr)
epilogue
So exit and the normal path after the call cost nothing. Every frame fixes
only its own return address. There is no stack walk. The lookup is what
the page fault handler does for bpf progs.
The interpreter cannot do that, so the prog has to be JITed, and JITs opt
in with bpf_jit_supports_unwind(). x86-64 and arm64 are in the next
patches.
BPF_JCOND is the same opcode as BPF_CALL_ARGS of the interpreter, and
BPF_UNWIND_GOTO is the same number as BPF_PSEUDO_CALL. may_goto is gone by
the time a prog is dumped, unwind_goto is not. bpf_insn_prepare_dump()
tells them apart: a prog with unwind_goto is always JITed.
Compared to v1 there is no BTF decl_tag and no 'r3 = 0' before every exit.
Tail call and ld_abs are fine in a subprog that can unwind, since they
leave it like exit does.
unwind_exit doesn't set r0. An implicit definition of r0 would need
special cases in backtracking, in liveness and in the check of the return
value of the main prog. A plain 'r0 = 0' in front of it needs none.
Rust
rust-bpf [4] builds with that llvm as it is. The programs are built with
-C panic=unwind. The panic handler prints the message and calls
_Unwind_RaiseException().
exc_cleanup.rs panics three frames deep, with preemption disabled in one
frame and rcu_read_lock held in another. All Drop impls run and the prog
returns 0.
scx_simple.rs is 2195 insns with 48 unwind_goto, 20 of them after callx,
and 16 unwind_exit. With r3 it was 2216 insns. scx_cosmos.rs is 3694 insns
with 68 and 18. It was 3716.
C++
Exceptions work too. This compiles, loads and runs:
struct rcu_guard {
rcu_guard() { bpf_rcu_read_lock(); }
~rcu_guard() { bpf_rcu_read_unlock(); }
};
static __noinline int inner(int len)
{
rcu_guard guard;
if (len > 100)
throw 1;
return len;
}
SEC("tc") int prog(struct __sk_buff *skb)
{
try {
return inner(skb->len);
} catch (...) {
return 42;
}
}
It returns 64 for a packet of 64 bytes and 42 for a packet of 200 bytes.
The destructor runs in both cases, otherwise the verifier would reject
the prog. 'throw' is a jump to the pad with the destructor. 'catch' is
unwind_goto:
inner:
w6 = w1
call bpf_rcu_read_lock
if w6 s< 101 goto ret
r1 = exc_obj ll
*(u32 *)(r1 + 0) = 1
goto pad
ret:
call bpf_rcu_read_unlock
w0 = w6
exit
pad:
call bpf_rcu_read_unlock
r0 = 0
unwind_exit
prog:
w6 = *(u32 *)(r1 + 0)
w1 = w6
call inner
unwind_goto catch
out:
w0 = w6
exit
catch:
w6 = 42
goto out
It needs -fexceptions -funwind-tables and four one line __cxa_*()
functions, where __cxa_throw() calls _Unwind_RaiseException().
Every pad is entered whatever was thrown, so llvm takes catch (...) only.
It's a selftest in patch 5.
[1] https://lore.kernel.org/bpf/20261018010246.1643877-1-alexei.starovoitov@gmail.com/
[2] https://lore.kernel.org/bpf/DM7KR7J8VY2T.1CZIRUF6QRCWG@gmail.com/
[3] https://github.com/llvm/llvm-project/pull/233649
[4] https://github.com/4ast/rust-bpf
Suggested-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261020181320.1284798-2-alexei.starovoitov@gmail.com
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs idmap constification from Christian Brauner:
"This makes struct mnt_idmap const throughout the tree.
A mount's idmapping is immutable. The only thing modified after
alloc_mnt_idmap() is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only reads from
it. That is the same model struct cred uses and the idmapping is
sensitive enough to deserve it.
The conversion goes bottom up through the vfs{g,u}id and fs{g,u}id
helpers, the privilege checks, acl, xattr and quota and then every
inode operation that takes an idmap, so ->permission(), ->getattr(),
->setattr(), ->create(), ->mkdir(), ->mknod(), ->symlink(),
->rename(), ->tmpfile(), ->get_acl(), ->set_acl() and ->fileattr_set()
with their implementations in every filesystem, plus the LSM hooks.
struct vfsmount, struct mount_kattr and struct kstatmount store a
const pointer and nop_mnt_idmap and invalid_mnt_idmap live in rodata,
so anything that tries to modify them faults instead of silently
corrupting the initial idmapping"
* tag 'vfs-7.4-rc1.idmap' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (27 commits)
fs: make nop_mnt_idmap and invalid_mnt_idmap const
fs: port mnt_idmap() and file_mnt_idmap() to const mnt_idmap
fs: port vfs_*() helpers to const mnt_idmap
fs: port ->setattr() to pass const mnt_idmap
fs: port ->getattr() to pass const mnt_idmap
fs: port ->create() to pass const mnt_idmap
fs: port ->symlink() to pass const mnt_idmap
fs: port ->mkdir() to pass const mnt_idmap
fs: port ->rename() to pass const mnt_idmap
fs: port ->mknod() to pass const mnt_idmap
fs: port ->tmpfile() to pass const mnt_idmap
fs: port ->get_acl() to pass const mnt_idmap
fs: port ->set_acl() to pass const mnt_idmap
fs: port ->fileattr_set() to pass const mnt_idmap
fs: port xattr to const mnt_idmap
fs: port ->permission() to pass const mnt_idmap
fs: port acl to const mnt_idmap
fs: port inode_init_owner() to const mnt_idmap
fs: port inode_owner_or_capable() to const mnt_idmap
fs: port privilege checking helpers to const mnt_idmap
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull file table updates from Christian Brauner:
"This annotates struct fdtable's fd array with __counted_by_ptr so
KASAN and the fortified functions can bounds check it against max_fds
and adds a KUnit suite for fdtable. A follow-up uses __struct_size()
in the test since sparse doesn't know __builtin_dynamic_object_size().
FD_PREPARE() is simplified. It was built as an ACQUIRE-style guard on
infrastructure that was never needed. The ACQUIRE machinery and
CLASS_INIT() go away, the variable is declared with __cleanup()
directly, struct fd_prepare loses its err field since the fd either
holds the descriptor or the error, and callers get a const pointer to
the guard behind a UNIQUE_ID() name so they can't shadow it. Code
generation is identical"
* tag 'vfs-7.4-rc1.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
file: make struct fd_prepare const and kill its err field
cleanup: remove CLASS_INIT()
file: declare the FD_PREPARE() variable with __cleanup() directly
file: simplify FD_PREPARE()
vfs: use __struct_size() in the fdtable KUnit test
vfs: Add KUnit tests for fdtable
vfs: Annotate struct fdtable's fd field with __counted_by_ptr
|
|
KCSAN reports a data race on bpf_lru_node::type between the LRU list
rotation and the free path of the common LRU:
BUG: KCSAN: data-race in __bpf_lru_list_rotate / bpf_lru_push_free
write to 0xffff88811b758022 of 1 bytes by task 4756 on cpu 1:
__bpf_lru_node_move
__bpf_lru_list_rotate_active
__bpf_lru_list_rotate
bpf_lru_list_pop_free_to_local
bpf_common_lru_pop_free
bpf_lru_pop_free
htab_lru_map_update_elem
read to 0xffff88811b758022 of 1 bytes by task 4752 on cpu 0:
bpf_lru_list_push_free
bpf_common_lru_push_free
bpf_lru_push_free
htab_lru_push_free
htab_lru_map_update_elem
value changed: 0x00 -> 0x01
bpf_lru_list_push_free() reads node->type for a WARN_ON_ONCE() sanity
check before taking l->lock. __bpf_lru_node_move(), called right after
under l->lock, does the same check, so drop the unlocked one.
bpf_common_lru_push_free() re-checks node->type under loc_l->lock after
the lockless READ_ONCE() at function entry. By then the node has been
removed from the htab, so only flush and rotate can change its type,
through __bpf_lru_node_move_in() and __bpf_lru_node_move() under
l->lock. Mark that re-check with READ_ONCE() and those two writers with
WRITE_ONCE(), following commit 6df8fb83301d ("bpf_lru_list: Read
double-checked variable once without lock"). The remaining writers
operate on a node with a single owner that nobody else reads and are
left alone.
Tested on bpf-next with CONFIG_KCSAN=y in a 4-vCPU virtme-ng guest,
using a reproducer that runs 8 threads issuing BPF_MAP_UPDATE_BATCH
on a 64-entry BPF_MAP_TYPE_LRU_HASH with overlapping keys.
Reported-by: syzbot+1ea4f564bf6f1f3d1dc4@syzkaller.appspotmail.com
Signed-off-by: Nicholas Velten <nicholasfvelten@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019-fix-lru-node-type-kcsan-v2-1-c10a410b3565@gmail.com
Closes: https://syzkaller.appspot.com/bug?extid=1ea4f564bf6f1f3d1dc4
|
|
Existing arena kfunc code has an underlying race condition
that can lead to writes being lost from the BPF program's
point of view:
a) A memory range gets freed by operation (1), and its range
is added back to the arena range tree.
b) A concurrent allocation (2) reallocates the range, and does
writes to it. Writes from that CPU may follow the stale TLB
entries into the pages that are about to be freed.
c) (1) invalidates the TLB. The old pages, and any writes done
to them, are now inaccessible. zap_pages() similarly removes the
mappings for userspace threads.
This can be triggered by particularly demanding BPF arena data
structures that constantly allocate and deallocate memory, like
hash table allocations.
Solve this ABA problem by keeping the range allocated until TLB
invalidation and unmapping are complete. Under the arena spinlock,
claim any free pages in the range and clear its kernel PTEs. Then drop
the spinlock, lock and unlock each cleared page to wait for faults that
are still installing a user PTE, flush the kernel TLB and zap the user
page tables. Finally, retake the spinlock to release the range. Frees
serialize on zap_mutex, so an overlapping free cannot release the range
early. Claiming the range only allocates a range tree node when the
whole range is free, and then there is nothing to free.
User faults on a range that is allocated without a page now raise
SIGBUS instead of populating it. That covers a range reserved with
bpf_arena_reserve_pages() and a range whose kernel PTEs a free has
already cleared. Touching a page while a BPF program frees it is a user
space bug, and in a BPF_F_SEGV_ON_FAULT arena it already raises SIGSEGV.
Sleepable frees run synchronously and are deferred only when the arena
spinlock cannot be taken. The deferred free worker retries such a free
in place instead of requeueing itself, so the flush_work() in
arena_map_free() waits for all of its work.
Fixes: b8467290edab ("bpf: arena: make arena kfuncs any context safe")
Reported-by: Mykola Lysenko <nickolay.lysenko@gmail.com>
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019201108.4552-3-emil@etsalapatis.com
|
|
A user fault on an arena pins the page that the kernel arena maps,
drops the arena spinlock and returns the page to the core fault code,
which then locks the folio and installs the user PTE. A concurrent
free can clear the kernel mapping in that window. The page reference
prevents a use-after-free, but the fault can still install a user PTE
for a page the arena has already released. User space and BPF programs
then see different memory at the same arena offset.
Close the window from the fault side. Lock the page, recheck that the
kernel mapping still points to it, and return it with VM_FAULT_LOCKED so
that it stays locked until the core fault code has installed the user
PTE. If a free removed the page, drop it and return VM_FAULT_NOPAGE so
that the fault starts over. Return VM_FAULT_HWPOISON for a poisoned
page.
This change alone does not close the race, because the free path does
not yet wait for the page lock. The following change makes frees wait
for in-flight faults before zapping user mappings.
Fixes: 317460317a02 ("bpf: Introduce bpf_arena.")
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019201108.4552-2-emil@etsalapatis.com
|
|
The signature of a program covers its insns as submitted, along with the
frozen metadata maps of a signed loader, and bpf_prog_verify_signature()
checks it before the verifier does anything else with the program. CO-RE
relocations are the one input besides the insns that changes them: the
loader hands them in with the program, and bpf_check_core_relo() applies
them after the signature check, rewriting the immediate or the offset of
every insn a record names with a value the loader chooses, a field
offset, a type size or id or an enum value of the kernel BTF, or a type
id of the BTF the loader supplies itself. Whoever holds a signed program
can so change its constants once the signature has passed, the offsets a
signed loader reads its metadata map with for instance, and run a
program the signer never saw. The exclusive metadata map of a signed
loader does not make up for it: the hash that binds the map to a program
is chosen by whoever creates the map, so it can name the relocated
program as well.
Refuse a signed load that carries relocations, before the signature is
examined, like the other checks on the shape of a signed load. A signed
loader has none to lose: the programs it embeds are relocated by the
loader itself through bpf_core_apply(), and nothing relocates the loader.
The BTF of the program itself can stay: without relocations it steers
only the verification, through func_info, line_info and the annotations
of global function arguments, and rewrites nothing.
Fixes: 349271568303 ("bpf: Implement signature verification for BPF programs")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://patch.msgid.link/20261019144851.4052056-2-memxor@gmail.com
|
|
kernel/bpf/Makefile builds arena.o only for CONFIG_MMU && CONFIG_64BIT,
but special_kfunc_list references bpf_arena_alloc_pages() and
bpf_arena_free_pages() unconditionally, so on 32-bit and !MMU kernels
resolve_btfids can't resolve them:
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
WARN: resolve_btfids: unresolved symbol bpf_arena_alloc_pages
Since commit 201a454df948 ("resolve_btfids: Fix the --fatal_warnings
exit status") these warnings fail the build with CONFIG_WERROR=y, which
breaks i386 builds with CONFIG_DEBUG_INFO_BTF=y:
Failed to generate BTF for vmlinux
Try to disable CONFIG_DEBUG_INFO_BTF
Only reference them under the condition arena.o is built with, and use
BTF_ID_UNUSED otherwise to keep the list in sync with enum
special_kfunc_type. An unresolved ID is zero, like BTF_ID_UNUSED, so the
contents of the list don't change.
Fixes: b8467290edab ("bpf: arena: make arena kfuncs any context safe")
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019211006.2574350-3-sashal@kernel.org
|
|
Commit 6f033615ef8f ("bpf: Trim special_kfunc_list in verifier") dropped
bpf_arena_reserve_pages, bpf_stream_vprintk and bpf_stream_print_stack
from special_kfunc_list once nothing used them anymore, but commit
ab7fff217fd1 ("bpf: Check helper and kfunc arguments in one path")
brought all three back without adding any users.
Besides being dead, the bpf_arena_reserve_pages entry can't be resolved
on 32-bit and !MMU kernels, which don't build arena.o, and since commit
201a454df948 ("resolve_btfids: Fix the --fatal_warnings exit status")
the resulting resolve_btfids warning fails the build with
CONFIG_WERROR=y:
WARN: resolve_btfids: unresolved symbol bpf_arena_reserve_pages
Drop the three entries again.
Fixes: ab7fff217fd1 ("bpf: Check helper and kfunc arguments in one path")
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019211006.2574350-2-sashal@kernel.org
|
|
A JIT sets up the arena base in the prologue of every function of a
program that has an arena, a callee-saved register loaded with a 64-bit
immediate and spilled around the function on x86. That is paid for by
every program with an arena, whether it touches the arena or not, and
now that libbpf binds the arena of an object to every program in it, by
every program of an object that happens to contain an arena.
Decide per function, on its final insns, right before it is JITed.
bpf_prog_has_arena_insns() looks for an insn that works with the base:
an access through an arena pointer, or through a number in a program
loaded with BPF_F_ARENA_SCALAR, has been rewritten to a BPF_PROBE_MEM32
form by then, a cast to the user address space keeps
BPF_ADDR_SPACE_CAST, and a kfunc call with an __arena argument, which
the JIT rebases, has a function model that says so. A load of an
__arena struct member is rewritten with an immediate and needs none;
dereferencing the result does.
Looking at the final insns rather than recording the uses during
verification keeps the answer in step with what the JIT sees. A
constant NULL passed to a nullable __arena kfunc argument takes the
early exit for NULL arguments in the verifier, and the JIT rebases the
call all the same; an insn that is dead code has disappeared by then.
bpf_prog_jit_compile() caches the answer in bpf_prog_aux for a single
function program, jit_subprogs() for each subprogram once it has its
kfunc table and arena. It implies that the program has an arena, as the
scan says no without one. The JITs with arena support pick up the arena
base and user address only when bpf_prog_uses_arena_insns() says so,
and so generate the prologue and epilogue of a program bound to an
arena it never touches as if it had none.
The other users of the arena address of a program stay as they are on
purpose. The struct_ops trampoline converts __arena arguments on entry
and arena_member_fixup() converts loaded __arena members after the load,
both with the base as an immediate. Both have to happen whether or not
the program dereferences the result, it may compare, store or pass it
on, so neither depends on the base register nor on the flag. The arena
fault report reads the user address of the arena the faulting insn
belongs to, and such an insn sets the flag anyway.
check_map_prog_compatibility() checks the user address when the arena
is bound, before any use is known. The rest, in arena.c and sched_ext,
works with the map and not with a program.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019132203.3034698-14-memxor@gmail.com
|
|
A kernel struct shared with BPF programs may want to point at arena
memory, e.g. a sched_ext argument struct that the kernel builds in the
scheduler's arena and that refers to another arena object. A pointer
member reaches the program typed as a kernel pointer, so sched_ext
publishes a plain u64 holding a pre-translated arena address that the
program casts before use.
Translate in the verifier instead. When a load through a kernel struct
pointer lands on a member tagged __arena, btf_struct_walk() reports it
and the destination register becomes PTR_TO_ARENA. The member holds the
kernel address of the arena memory; the fixup pass appends the
conversion to the arena pointer form that the struct_ops trampoline
applies to __arena arguments, after a plain load or a load-acquire of
the member:
r1 = *(u64 *)(r2 + off)
if r1 == 0 goto +1 // __nullable members and probed loads
w1 -= lo32(kern_vm_start)
A probed load, through a plain or an untrusted pointer, keeps the NULL
test as well: it reads as 0 when it faults, and the subtraction must
not turn that into an arena pointer. An acquire load only comes through
a pointer that cannot fault.
Stores to kernel structs are rejected as before, and with them the
atomics that write such a member. A member
inside a multi-member union is rejected too: an offset into a union
resolves to its first member, so a program reading another member there
would receive an arena pointer it did not ask for. Loads through
program-allocated objects and program BTF are unaffected, there the tag
is what the compiler emits for the arena address space.
The rewrite is per insn while verification is per path, so the aux data
records what an insn loaded on any path: an insn that loaded an arena
pointer member on one path and a different member on another is
rejected by the fixup pass, while paths disagreeing only on __nullable
keep the NULL test, which a never-NULL member can afford. As for
__arena arguments, the program needs an associated arena, which also
supplies the base to subtract.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019132203.3034698-8-memxor@gmail.com
|
|
A pointer member of a kernel struct that the verifier derives as trusted
or RCU-protected is assumed to be non-NULL, so a load through it is a
plain dereference. The BTF_TYPE_SAFE_*_OR_NULL lists exist for members
known to be NULL at times, but they live in the verifier and have to be
extended by hand.
Teach btf_struct_walk() to report the "nullable" BTF type tag on a
pointer member as PTR_MAYBE_NULL. The tag combines with any other tag
on the member; it only says that the pointer may be NULL. The verifier
keeps the flag where NULL would otherwise be assumed away: on trusted
and RCU results, which a program dereferences as they are, and on
per-CPU results, which the per-CPU helpers shift without a check. Such
a pointer has to be tested before it is dereferenced or passed on.
Plain and untrusted results are probed anyway and lose the flag, as the
BTF_TYPE_SAFE_*_OR_NULL lists never applied to them either, so a tagged
member of an existing struct asks no more of the programs that read it
than a list entry does, and the tag can take the place of the lists.
The next patch uses the tag on __arena members to decide whether the
injected address translation has to preserve NULL.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019132203.3034698-7-memxor@gmail.com
|
|
An exclusive map is bound to the digest of its program, which the
verifier compared against prog->digest in check_map_prog_compatibility()
as the map was added to the program. For a map an insn refers to that
happens in check_and_resolve_insns(), after the digest is computed at
the top of that function. For a map bound through a continuous fd_array
it happens in process_fd_array(), which runs earlier, so for an unsigned
program the comparison saw a zeroed digest and the load failed with
-EACCES, "program's hash doesn't match map's excl_prog_hash", for the
very program the map is exclusive to.
Computing the digest earlier would not do: it has to cover the insns as
the kernel verifies and runs them, after the CO-RE relocations it
applies, or a loader could submit the insns of the exclusive program
together with relocations that change them and pass the check with a
different program. Leave the digest where it is and move the comparison
instead: check every used map against the digest at the end of
check_and_resolve_insns(), whether an insn referred to it or the array
bound it.
The claims a program makes on its maps move behind the check as well:
bpf_map_claim(), which records the program as the user of the map for
resolve_func_ptrs(), and the takeover of an insn array by
bpf_insn_array_init(). Both ran as the map was added, so a program that
is not the owner of an exclusive map could have flipped the owner's
single-user state to shared, which a failed load does not undo, or held
the owner's insn array while it was being verified. Now they run once
every map has passed the check, and a rejected program leaves the maps
as it found them. Nothing consults a claim before resolve_func_ptrs()
and the fixup passes, which run later.
The early digest of a signed program goes with the early comparison.
Commit b707068e0ed9 ("bpf: Verify signed loader metadata at load time")
computed it only so that the maps of a signed loader passed the check in
process_fd_array(); the signature itself is verified over the submitted
insns and the frozen maps by bpf_prog_verify_signature(), which does not
read the digest. The maps of a signed program are now checked against
the digest of the final insns like everyone else's, which covers the
relocations the kernel applies and the signature does not; a loader
program carries none.
Fixes: baefdbdf6812 ("bpf: Implement exclusive map creation")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261019132203.3034698-2-memxor@gmail.com
|
|
An extension that writes to the packet of a tc, lwt_xmit or sk_skb program
starts with bpf_unclone_prologue(). If the skb is cloned, that gives it
a new head.
It does not count as changing packet data. changes_pkt_data goes by the
helpers and kfuncs that a program calls, and whether it gets a prologue
is known only after it has been compared with its target. So the extension
can replace a global function that leaves the packet alone, and the callers
of that function keep their packet pointers across the call. These point
into the old head, which is left to the other clone. Once that is freed,
by the reader of a packet socket for example:
BUG: KASAN: slab-use-after-free in bpf_prog_53bc21b9810cadbc_t_read+0x2e8/0x36e
Read of size 1 at addr ffff888107bfe73c by task tt11/102
Freed by task 112:
kfree+0x161/0x500
skb_release_data+0x2dc/0x340
consume_skb+0xa6/0x1c0
packet_recvmsg+0x4b3/0xa20
Nothing is wrong if the target program writes to the packet too: its own
prologue has uncloned the skb, and that of the extension does nothing.
Nor if the callers hold no packet pointers, like a dispatcher, or if the
main function is replaced.
Note for every global function that leaves the packet alone if it is
called while packet pointers, or slices of the dynptr of a packet, are
held. What the callers of a global function hold is not known while it
is verified, so the functions that it calls are taken for such as well.
Refuse an extension with a prologue for such a function, unless the target
program has a prologue, both when the extension is loaded and when it
is attached to another target.
Functions can share a type id in func_info. The first of them is the one
that is replaced, so look it up in one place.
Fixes: 7e40781cc8b7 ("bpf: verifier: Use target program's type for access verifications")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261019011328.1843764-1-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
A global subprogram parameter tagged __arg_trusted is verified with
PTR_TRUSTED, but the common call-site type check accepts bare PTR_TO_BTF_ID
and MEM_RCU registers. The callee can consequently pass a pointer with
weaker provenance to a kfunc that does not accept RCU-protected pointers.
RCU protection only guarantees that the object remains allocated through a
grace period. It does not guarantee that an object's refcount is nonzero,
so an RCU-protected pointer cannot satisfy the trusted-pointer contract.
For example, an RCU-protected bpf_cpumask loaded from a map may have a
zero refcount. Passing it through __arg_trusted lets the callee pass it
to bpf_cpumask_acquire(), whose plain refcount_inc() can retain an object
that will be freed after the grace period.
Make the existing provenance check contract-driven. Mark generated BTF ID
kfunc argument types with PTR_TRUSTED, and add MEM_RCU only for KF_RCU
kfuncs. Normalize PTR_MAYBE_NULL when the argument contract permits it.
Then use the argument flags to require trusted or referenced provenance
and allow MEM_RCU only when the contract says so.
Split is_trusted_reg() so the normalized type is used for type and modifier
checks while reference lookup still uses the original register. Conversion
of scalar-struct BTF ID arguments to fixed-size memory continues to
construct a new ARG_PTR_TO_MEM type without PTR_TRUSTED.
Fixes: e2b3c4ff5d18 ("bpf: add __arg_trusted global func arg tag")
Suggested-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20261007-a3-arg-trusted-v4-v4-1-253ef3a1a69a@mails.tsinghua.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
For BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH maps,
htab_map_check_update_flags() and check_flags() are not considering
the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags
with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the
(BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.
For example, when using (BPF_F_CPU | BPF_EXIST) or
(BPF_F_CPU | BPF_NOEXIST) flag combinations with bpf_map_update_elem()
on a percpu hash map, the BPF_EXIST flag does not prevent new
insertions as expected and BPF_NOEXIST flag does not prevent
modification of existing entries as expected.
This patch fixes the bug by adding proper flag validations and checks.
Before this patch, htab_map_check_update_flags() rejected
(BPF_F_ALL_CPUS | BPF_EXIST) and (BPF_F_ALL_CPUS | BPF_NOEXIST) with
-EINVAL. After the patch those combinations are accepted.
This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST),
with or without BPF_F_CPU, which was previously accepted.
Fixes: c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps")
Signed-off-by: Masoud Aghasi <maghasi@disroot.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://patch.msgid.link/20261008120734.1565972-3-maghasi@disroot.org
|
|
For BPF_MAP_TYPE_PERCPU_ARRAY map, bpf_percpu_array_update()
is not considering the possibility of a combination of
(BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags.
This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect
in some cases.
For example, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination
with bpf_map_update_elem() results in an incorrect EINVAL error
response, even though the flag combination is valid.
This patch fixes the bug by adding proper flag validations and checks.
Before this patch, bpf_percpu_array_update() rejected
BPF_F_ALL_CPUS | BPF_EXIST and BPF_F_ALL_CPUS | BPF_NOEXIST with
-EINVAL. Also the BPF_F_CPU | BPF_NOEXIST were accepted.
After the patch BPF_F_ALL_CPUS | BPF_EXIST is accepted and
BPF_F_ALL_CPUS | BPF_NOEXIST and BPF_F_CPU | BPF_NOEXIST
result in -EEXIST.
This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST),
with or without BPF_F_CPU, which was previously accepted.
Fixes: 8eb76cb03f0f ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps")
Signed-off-by: Masoud Aghasi <maghasi@disroot.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://patch.msgid.link/20261008120734.1565972-2-maghasi@disroot.org
|
|
bpf_check_attach_target() rejects a freplace whose target is a static
function, using the subprog "unreliable" flag as a proxy for "static".
That flag is raised whenever a subprogram's BTF prototype disagrees with
the registers at its call site.
`147400e9c27c ("bpf: Compute parameters liveness in
bpf_compute_live_registers()")` made that work for static callees: a
static function with an unused argument no longer matches its BTF
prototype at the call site and is marked unreliable.
Since LLVM commit `29b36a16f111 ("[BPF] Emit ABI-accurate BTF
prototypes for DW_CC_nocall")` in 23.1 [1] that premise is gone:
BTFDebug may now drop arguments removed by the optimizer, so for
__noinline static int subprog(struct __sk_buff *ctx)
{
return bpf_ktime_get_ns();
}
compiled with clang 23 the BTF describes subprog() as
[75] FUNC_PROTO '(anon)' ret_type_id=2 vlen=0
[76] FUNC 'subprog' type_id=75 linkage=static
There is then no argument left to compare, the subprog stays reliable,
and the check in bpf_check_attach_target() is skipped. The freplace is
then rejected by the later linkage check instead, reporting "subprog()
is not a global function", and the selftest "exceptions" fails.
Determine staticness from the BTF linkage instead, and keep the
unreliable check for global functions whose BTF does not match their
call site.
There is no functional change: freplacing a static function was rejected
before and is still rejected with the same -EINVAL. Selftest "exceptions"
is updated to match.
[1] https://github.com/llvm/llvm-project/pull/198426
Fixes: 147400e9c27c ("bpf: Compute parameters liveness in bpf_compute_live_registers()")
Signed-off-by: KaFai Wan <kafai.wan@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261018005348.2195130-2-kafai.wan@linux.dev
|
|
bpf_get_func_arg() takes a u32 index, but the inlined expansion compared
only the low 32 bits of R2 with the argument count and then shifted the
full 64-bit register to compute the context address. R2 is ARG_ANYTHING,
so nothing guarantees that its upper bits are zero: an index of
1ULL << 32 passes the bounds check as index zero while the load is
performed at ctx + (1ULL << 35).
Shift the index as a 32-bit value. When the 32-bit bounds check passes,
the low 32 bits of R2 are below R0, and R0 is at most 255: the trampoline
arm masks the argument count with 0xFF and the raw_tp arm uses the BTF
argument count minus one, which the verifier caps at MAX_BPF_FUNC_ARGS.
The 3-bit shift therefore cannot overflow and it clears the upper bits of
the register. Widening the comparison instead would reject indices whose
low 32 bits are valid. The check itself is unchanged, so this cannot
change which programs are accepted, only the address that is computed.
Fixes: f92c1e183604 ("bpf: Add get_func_[arg|ret|arg_cnt] helpers")
Signed-off-by: Woojin Ji <random6.xyz@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261018-get-func-arg-v2-1-048848b3eddf@gmail.com
|
|
The helpers which may change packet data do not take a packet pointer as a
memory argument: they have no pkt_access. They make the skb writable, or
make room in it, before they read what the argument points to, and that
can reallocate the head which it points into.
A slice from bpf_dynptr_slice() of the dynptr of the skb points into the
same head, but it is a PTR_TO_MEM, and the verifier lets it pass.
bpf_lwt_push_encap(), bpf_lwt_seg6_store_bytes() and bpf_lwt_seg6_action()
then read the old head, which has been freed. bpf_skb_store_bytes() follows
the head by now, but a slice is no better than a packet pointer there.
Refuse a slice of a packet where a packet pointer is refused for that
reason. All other helpers take slices as before. bpf_dynptr_write() and
bpf_dynptr_copy() can be used to copy inside of a packet.
Fixes: 2012c867c800 ("bpf: verifier: Accept dynptr mem as mem in helpers")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-17-alexei.starovoitov@gmail.com
|
|
bpf_dynptr_copy(), bpf_dynptr_memset() and the kfuncs that read user or
kernel memory into a dynptr ask bpf_dynptr_slice_rdwr() for a slice and
write to it. For the linear part of an skb that is a pointer into its
head. bpf_dynptr_slice_rdwr() counts on the verifier to have uncloned the
skb, see bpf_unclone_prologue().
The verifier does so if the program itself calls bpf_dynptr_slice_rdwr()
or writes to the packet. A program that only uses these kfuncs gets no
prologue, and changes the data of all the clones of the skb. A packet
socket, whose clone is taken before tc ingress runs, sees what
bpf_dynptr_memset() has written there.
Treat a call that may write to an skb like a direct write.
Fixes: daec295a7094 ("bpf/helpers: Introduce bpf_dynptr_copy kfunc")
Fixes: a498ee7576de ("bpf: Implement dynptr copy kfuncs")
Fixes: 5fc5d8fded57 ("bpf: Add bpf_dynptr_memset() kfunc")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-13-alexei.starovoitov@gmail.com
|
|
A tc, lwt_xmit or sk_skb program that writes to packet data starts with
bpf_unclone_prologue(). It takes the skb from R1, and if the skb is cloned
it calls bpf_skb_pull_data() and restores R1 only.
That suits the main function of a program. An extension gets the prologue
too, but it starts with the arguments of the function that it replaces.
If the first one is not the context, something else is taken for an skb:
BUG: KASAN: stack-out-of-bounds in skb_ensure_writable+0x84/0x1a0
Read of size 8 at addr ffffc90000270ad0 by task uncl/105
skb_ensure_writable+0x84/0x1a0
bpf_skb_pull_data+0x32/0xf0
bpf_prog_853eb1991d35e093_ext_slice+0x4f/0x321
bpf_prog_9eb18752b5d0959a_t_glob+0x2ac/0x2de
BUG: unable to handle page fault for address: 0000000082298821
And whatever R2-R5 held is lost, while the verifier goes on trusting
pointers in them.
Refuse an extension that needs a prologue unless it gets the context as
its first argument and scalars beside it. A scalar is still clobbered if
the skb is cloned, which the verifier can cope with. Extensions of XDP,
sk_msg and sockopt programs get no prologue and load as before.
Fixes: be8704ff07d2 ("bpf: Introduce dynamic program extensions")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261017230950.802052-12-alexei.starovoitov@gmail.com
|
