| Age | Commit message (Collapse) | Author |
|---|
|
SCEV needs to find an exit condition that dominates a loop's backedge;
such conditions bound every loop iteration. Compute the immediate
dominator tree as a prerequisite for identifying such conditions.
Use the iterative reverse-postorder algorithm from:
"A Simple, Fast Dominance Algorithm" by Cooper et al.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-23-bcba9bc53ac9@gmail.com
|
|
Tests for multiple analyses performed by the verifier need the
verifier log to contain analysis results alongside the program
disassembly. This patch moves the log-level-2 program dump from
bpf_compute_live_registers() to a standalone function called from
bpf_check(), in order to provide a common logging function for such
analyses (and thus avoid printing program disassembly multiple times).
Rename the dump header from "Live regs before insn:" to
"Program dump (scc? insn#: live_regs_before):" and print available
source-line information before the corresponding instructions.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-22-bcba9bc53ac9@gmail.com
|
|
Dominator intersections and SCEV worklist scheduling need to compare
instructions by their DFS postorder rank. Save per-instruction
postorder numbers alongside the existing postorder sequence.
This commit modifies bpf_compute_postorder() to actually do the DFS
traversal, instead of scheduling traversal of all siblings at once.
Consider a graph:
A -> B
A -> C
C -> B
Old result: C, B, A
DFS result: B, C, A
Fixes: efcda22aa541 ("bpf: compute instructions postorder per subprogram")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-21-bcba9bc53ac9@gmail.com
|
|
Relax the requirements for reads like `p->arr[i].field`, where `p` is a
PTR_TO_BTF_ID, by allowing `i` to be a non-constant value.
This is useful for loops over arrays accessed from a PTR_TO_BTF_ID
when the verifier widens the loop induction variable.
Check that the access stays within array bounds based on the access
register's min and max bounds. Check that the access is aligned with
array elements using the register's `base` and `step`.
Technically, modify btf_struct_access() such that:
- it identifies `field` by walking BTF from the minimum offset
(the register's reg_smin + static instruction offset);
- it remembers which arrays contain `field`;
- it checks that offsets `k * step` stay within the array
and point to the same `field`;
- it does not check upper bounds for flexible arrays
(as is already the case for constant-offset access).
Support for flexible arrays nested inside trailing struct members
(e.g. p->nested.data[i], with data declared as int data[]) is left
for future work.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-20-bcba9bc53ac9@gmail.com
|
|
Extend scalar register tracking with a linear "base + step * k" description:
each scalar carries two u16 fields, base and step (invariant: base < step),
recording that its value is some point on the line base + step * k.
A fresh scalar starts as base=0, step=1, i.e. any integer.
This lets the verifier reason about strided values, e.g. an index multiplied
by an element size, which neither the min/max bounds nor the tnum can capture
for a non-power-of-2 stride. Such reasoning is necessary to allow
array access with a variable offset through PTR_TO_BTF_ID, e.g.
for expressions like p->arr[i].field.
reg_bounds_sync() uses the description in two ways:
- __reg_bound_offset() clears the low bits of var_off implied by the number
of trailing zeros shared by base and step.
- __update_bounds_from_step() intersects the 64-bit range with the line via
cnum64_intersect_linear(), snapping smin/smax to actual points on it.
This is the last step of the sync, as refining the range from var_off
can move the bounds off the line.
range_within() is extended to check whether the cur register's equation
describes a subset of the points described by the old register's equation.
Scalar ALU operations update base/step as follows:
- ADD of a constant (including pointer + scalar): keep step, shift base;
- MUL by a constant: scale step and base by the constant;
- LSH by a constant: scale step and base by the corresponding power of two;
- sign extension: keep the register unchanged if all values fit the target
type, reset to base=0, step=1 otherwise;
- All other operations reset to base=0, step=1.
Branch (eq/neq) checks do not take into account base/step yet.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-19-bcba9bc53ac9@gmail.com
|
|
regsafe() currently requires an exact match for PTR_TO_STACK registers.
This prevents an explored state from pruning a current state whose
variable-offset range is a subset of the explored range.
This change allows a checkpoint containing a widened stack pointer to
cover narrower offsets reached on subsequent loop iterations,
for example [-56, -8] within [-64, -8].
Merge the PTR_TO_STACK case with PTR_TO_MEM and others, as the set of
necessary checks is almost identical. The only difference is a map_uid
check, which is not needed for PTR_TO_STACK. All locations that
produce PTR_TO_STACK reset map_uid to zero.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-18-bcba9bc53ac9@gmail.com
|
|
For use in SCEV logging to make large widened bounds more readable.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-17-bcba9bc53ac9@gmail.com
|
|
When windening SCEV expressions like *fp-8 -> (any *fp-8 *fp-16),
fp-8 might be not yet allocated. Expose a utility function such
that widening logic can handle this.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-16-bcba9bc53ac9@gmail.com
|
|
Expose the existing helpers verifier.c:rev_opcode() and
verifier.c:flip_opcode() as bpf_rev_opcode() and bpf_flip_opcode()
and declare them in bpf_verifier.h. The helpers are used by SCEV
logic while analyzing loop conditions.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-15-bcba9bc53ac9@gmail.com
|
|
Add cnum64_intersect_linear(), which intersects a cnum64 interval with the
set of integers congruent to 'base' modulo 'step', i.e. the points
base + step * k. It tightens the interval by rounding its signed minimum up
and its signed maximum down to the nearest such point, and returns an empty
cnum when no point falls within the interval.
The arithmetic is done on the signed bounds [smin, smax]. The line is defined
over the integers, while a cnum64 is an arc on the u64 circle, so a u64-modular
computation would misplace the line for negative values whenever 'step' does
not divide 2^64. Working from the signed bounds keeps the residues consistent,
e.g. it can reason about a range like [-9, 3] with step 3.
Also add imod() to bpf_verifier.h: a mathematical modulo returning an
always-non-negative residue (unlike C's '%', which follows the sign of the
dividend). It is used by cnum64_intersect_linear() and by the base/step
tracking added in subsequent patches.
This is a building block for tracking scalar registers whose value is known
to lie on a line base + step * k.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-14-bcba9bc53ac9@gmail.com
|
|
Add cnum32_union() and cnum64_union() to compute a smallest circular
range containing both inputs.
Do so by enumerating the following configurations in a rotated
frame (a.base at the origin):
0 UT_MAX
|---------------------------------------------------|
[= a ==============================] |
[= b tail =] [= b main ===================>
[= union ===========================================]
0 UT_MAX
|---------------------------------------------------|
[= a =====================] |
[= b tail ======] [= b main =====>
[= union tail ============] [= union main =>
0 UT_MAX
|---------------------------------------------------|
[= a =======] |
[= b tail =============] [= b main =====>
[= union tail =========] [= union main =>
0 UT_MAX
|---------------------------------------------------|
[= a =========================] |
| [= b ====================] |
[= union ==================================] |
0 UT_MAX
|---------------------------------------------------|
[= a =====================================] |
| [= b ========] |
[= union =================================] |
0 UT_MAX
|---------------------------------------------------|
[= a ============] |
| [= b =======] |
Two possible covering arcs:
[= ab ======================================] |
[= ba tail ======] [= ba main =========>
Pick the smaller one.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-13-bcba9bc53ac9@gmail.com
|
|
Used by the SCEV widening logic later in the series.
The verifier checks alignment for some memory accesses.
When SCEV widens a pointer or an index variable, it must retain
alignment information in order to allow array access.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-12-bcba9bc53ac9@gmail.com
|
|
This computes exact liveness for the arguments of a call to a
subprogram instead of assuming that r1..r5 are read:
- For static subprograms compute use registers at the
bpf_compute_live_registers() fixpoint phase.
- For global subprograms use BTF information.
Add a separate pass doing try_btf_prepare_func_args() in order for
bpf_subprog_info->arg_slot_cnt to be available during
bpf_compute_live_registers().
The conservative liveness marks hindered SCEV widening logic on a set
of production programs.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-11-bcba9bc53ac9@gmail.com
|
|
Stack liveness currently analyzes global callees in the context of their
callers. The resulting read/write marks describe only the current body,
but freplace can replace it after the caller has been verified.
A no-op callee taking a stack pointer can be replaced with one writing
through that pointer, leaving the call's may_write marks incomplete.
SCEV could then preserve a stack counter recurrence across a call that
resets it, invalidating a loop termination proof.
Instead, derive global call effects from BTF argument types instead,
same is it done on the main verification pass.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-10-bcba9bc53ac9@gmail.com
|
|
Track spilled stack pointers through 64-bit atomic accesses:
- BPF_STORE_REL spills the source register.
- BPF_LOAD_ACQ restores the spilled value into dst_reg.
- BPF_XCHG and BPF_{ADD,AND,OR,XOR} | BPF_FETCH restore it into src_reg.
- BPF_CMPXCHG restores it into R0.
This prevents missing stack accesses through pointers recovered by these
instructions, matching the main verifier's spill/fill behavior.
Fixes: bf0c571f7feb ("bpf: introduce forward arg-tracking dataflow analysis")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-9-bcba9bc53ac9@gmail.com
|
|
Atomic read-modify-write instructions are recorded as reads only,
omitting their stack write effects from the may_write masks needed
by SCEV.
Record both read and write accesses for atomic RMW operations. Keep
LOAD_ACQ read-only and STORE_REL write-only. Share the precise/imprecise
stack-access dispatch so both accesses use the same address handling.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-8-bcba9bc53ac9@gmail.com
|
|
SCEV needs to know which registers might be stack pointers at a
particular instruction. This information is used to avoid widening
registers if that would hinder further verification.
See the patch [1] for additional information.
liveness.c:compute_subprog_args() already tracks this information.
This commit modifies it to save the information in insn_aux_data for
further usage.
[1] "bpf: avoid widening registers that hinder exact stack-slot tracking"
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-7-bcba9bc53ac9@gmail.com
|
|
SCEV needs live stack information in order to skip tracking dead
slots. Liveness tracks this per function instance, but SCEV is not
callchain-sensitive and needs a summary across calling contexts.
For each instruction, union the current-frame liveness masks from all
analyzed function instances. Represent 8-byte slots as a summary of
constituent halves. Store the summary in insn_aux_data.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-6-bcba9bc53ac9@gmail.com
|
|
SCEV needs may-write information to invalidate stack expressions after
indirect writes. Liveness tracks this per function instance,
but SCEV is not callchain-sensitive and needs a summary across
calling contexts.
For each instruction, union the current-frame may_write masks from all
analyzed instances. Represent 8-byte slots as a summary of constituent
halves. Store the summary in insn_aux_data and expose it via a helper.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-5-bcba9bc53ac9@gmail.com
|
|
Track stack slots that instructions may modify, in order to account
for such writes when constructing SCEV expressions.
Consume the may_write flag in arg_access_info separately from reads and
definite writes. Include partially covered slots and all candidate
offsets; imprecise writes mark whole candidate frames. Possible writes
do not kill liveness.
Merge analyzed instances as:
may_write(dst) |= may_write(src)
must_write(dst) &= must_write(src)
For each ancestor frame f, summarize callee writes at the callsite:
may_write(callsite, f) |= OR_{i in callee} may_write(i, f)
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-4-bcba9bc53ac9@gmail.com
|
|
Return kfunc stack effects through arg_access_info while preserving the
existing BTF size lookup and read/def classification. Retain __sz/__szk
handling and conservatively mark pointer arguments as possible writes.
Remove the temporary signed-size adapter from liveness. Add liveness
assertions for privileged and unprivileged __uninit outputs.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-3-bcba9bc53ac9@gmail.com
|
|
Adjust bpf_helper_stack_access_bytes() to return results as a struct
arg_access_info. The result is based on bpf_func_proto->argX_type
value as in the following table:
| Argument flags | allow_uninit_stack | Meaning |
|----------------------+--------------------+----------------------|
| MEM_WRITE | true | may_read, must_write |
| MEM_WRITE+MEM_UNINIT | true | must_write |
| MEM_WRITE | false | may_read, must_write |
| MEM_WRITE+MEM_UNINIT | false | may_read, may_write |
| 0 | true/false | may_read |
MEM_WRITE now produces must_write when allow_uninit_stack is true:
the verifier discards previous stack knowledge over the affected
range. Previously, liveness only treated MEM_UNINIT this way.
Also use resolve_map_arg_type() to account for bpf_map_peek_elem()
treating its buffer as an input for bloom filters.
Annotate ringbuf dynptr submit and discard arguments with MEM_WRITE.
Their existing OBJ_RELEASE handling already invalidates the dynptr, but
the prototypes need to describe the write as well as the release.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-2-bcba9bc53ac9@gmail.com
|
|
The following patches in the series add support for may_write property
inference. To properly track this property there is a need to
distinguish three ways the helpers/kfuncs affect stack:
- read;
- write and destroy all previous stack tracking information for
affected slots;
- both read and write the stack slots, potentially preserving some of
the tracking information.
The later property can't be encoded by signed access_byte convention.
Hence, introduce arg_access_info with a size and separate may_read,
may_write and must_write bits. Convert the liveness access recording
paths to use it. Proper may_write calculation and support will be
added in the following patches.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-1-bcba9bc53ac9@gmail.com
|
|
Patch series "mm: Unconditional per-VMA locks and cleanups", v7.
tl;dr: Make per-VMA locks available in all configs. Simplify some of the
per-VMA lock users now that they can rely on them being always available.
Binder and networking folks: Your code is the target of the cleanups. I'm
cc'ing you now on v2 because there's emerging consensus on the mm side
that the approach here is sane. I'm not quite sure how this pile would
get merged, but ack/review tags would be appreciated if this looks good to
you.
Longer version:
When working on some x86 shadow stack code, it was a real pain to avoid
causing recursive locking problems with mmap_lock. One way to avoid those
was to avoid mmap_lock and use per-VMA locks instead. They are great, but
they are not available in all configs which makes them unusable in generic
code, or if you want to completely avoid mmap_lock.
Make per-VMA locks available in all configs. Right now, they are only
available on select architectures when SMP and MMU are enabled. But all
of the primitives that per-VMA locks are built on (RCU, maple trees,
refcounts) work just fine without SMP or MMU.
The only real downside is that making VMAs a wee bit bigger on !MMU and
!SMP builds.
The upside is much cleaner code, lower complexity and less #ifdeffery.
Clean up a binder VMA locking site now that it can rely on per-VMA locks.
Building on top of universally-available per-VMA locks, introduce a new
helper. Since the new API does not require callers to have a fallback to
mmap_lock, it's much easier to use. Callers can potentially replace this
very common kernel idiom:
mmap_read_lock(mm);
vma = vma_lookup()
// fiddle with vma
mmap_read_unlock(mm);
with:
vma = vma_start_read_unlocked(mm, address);
// fiddle with vma
vma_end_read(vma);
Which avoids mmap_lock entirely in the fast path.
Use that new API for another binder site and one in the TCP code.
This patch (of 7):
The per-VMA locks have been around for several years. They've had some
bugs worked out of them and have seen quite wide use. However, they are
still only available when architectures explicitly enable them. Remove
the conditional compilation around the per-VMA locks, making them
available on all architectures and configs.
The approach up to now seemed to be to add ARCH_SUPPORTS_PER_VMA_LOCK when
the architecture started using per-VMA locks in the fault handler. But,
contrary to the naming, the Kconfig option does not really indicate
whether the architecture supports per-VMA locks or not. It is more of a
marker for whether the architecture is likely to benefit from per-VMA
locks.
To me, the most important thing side-effect of universal availability is
letting per-VMA locks be used in SMP=n configs. This lets us use
per-VMA locking in all x86 code without fallbacks.
Overall, this just generally makes the kernel simpler. Just look at the
diffstat. It also opens the door to users that want to use the per-VMA
locks in common code. Doing *that* brings additional simplifications.
The downside of this is adding some fields to vm_area_struct and
mm_struct. There are likely ways to optimize this, especially for things
like SMP=n configs. For now, do the simplest thing: use the same
implementation everywhere.
== Considerations for NOMMU config ==
NOMMU systems do not write-lock VMAs, therefore read-locking a VMA would
always succeed unless VMA is detached. Therefore for NOMMU config we make
vma_mark_attached() a NOOP, which keeps VMAs always in detached state.
This causes VMA read-locking to always fail and the caller falls back to
locking mmap_lock.
The following functions will have a different implementation in NOMMU
config:
- vma_mark_attached(), vma_mark_detached() are made NOOPs, keeping VMAs
always in a detached state and preventing assertions and refcount
underflows;
- vma_start_write(), vma_start_write_killable() are made NOOPs to avoid
warnings in __vma_start_write() due to VMAs being detached. These
functions are not used in NOMMU code but __vma_start_write() is an
exported function, therefore might be used by drivers.
- vma_assert_attached() is made NOOP because it's reachable from NOMMU
code via split_vma()->vma_iter_store_new()->vma_iter_store_overwrite();
- vma_assert_write_locked() is asserting vma->vm_mm is write-locked, as
was done before this change;
- vma_assert_locked() is asserting vma->vm_mm is locked, as was done
before this change;
The following functions work for both MMU and NOMMU configs:
- vma_lock_init() performs the same initialization as for MMU config;
- mm_lock_seqcount_init(), mm_lock_seqcount_begin(),
mm_lock_seqcount_end() are called from mmap_write_{lock|unlock} and
update mm_lock_seq correctly.
- mmap_lock_speculate_try_begin(), mmap_lock_speculate_retry() work as
is because mm_lock_seq is updated correctly;
- vma_start_read(), vma_start_read_locked() will always fail because
VMAs are always detached;
- vma_end_read() will never be called because vma_start_read() never
succeeds;
- vma_is_attached() always return false because VMAs are always
detached;
- vma_assert_detached() will never trigger because VMAs are never
attached;
- vma_start_read_locked() always return false because VMAs are always
detached;
- lock_vma_under_rcu() will be safe as the attempted read lock will bail;
Changes in the following files are not affecting NOMMU config:
task_mmu.c - not compiled when CONFIG_MMU=n;
pagewalk.c - not compiled when CONFIG_MMU=n;
userfaultfd.c - not compiled when CONFIG_MMU=n (CONFIG_USERFAULTFD depends
on CONFIG_MMU);
The following changes in the BPF code are made to keep NOMMU config
working like before:
stack_map_lock_vma() - keeps mmap_lock in NOMMU config;
bpf_iter_task_vma_new() - bails out in NOMMU config;
Link: https://lore.kernel.org/20260831203056.838265-1-surenb@google.com
Link: https://lore.kernel.org/20260831203056.838265-2-surenb@google.com
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Todd Kjos <tkjos@android.com>
Cc: Christian Brauner <christian@brauner.io>
Cc: Carlos Llamas <cmllamas@google.com>
Cc: Alice Ryhl <aliceryhl@google.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: David Ahern <dsahern@kernel.org>
Cc: Arve Hjønnevåg <arve@android.com>
|
|
A data slice is valid only while the specific dynptr that produced it
remains valid. prepare_func_exit() copies a subprogram return value into
the caller and then frees the callee state without applying the normal
destruction semantics to dynptrs in the callee stack.
A callee can therefore derive a slice from a local dynptr, return or spill
the slice into its caller, and then let the dynptr disappear with the
callee frame. The slice retains the id of a dynptr that is no longer in the
verifier state, so the verifier continues to accept accesses through it.
Before the object relationship refactor, bpf_dynptr_data() slices from
referenced dynptrs also carried the shared reference id, allowing a later
release to catch some of these escaped slices. The refactor made those
slices precise children of their source dynptr and exposed the missing
teardown as an accepted stale access even after the shared resource is
released.
Add destroy_dynptrs_in_stack_slots() to apply the existing dynptr teardown
to an inclusive range of stack slots. Reuse it for dynptr initialization,
variable-offset stack writes, and the entire callee stack before freeing
the frame. The teardown rejects losing the last dynptr for a referenced
resource and invalidates the dynptr and all its descendants, including
slices held in the caller. Propagate errors through prepare_func_exit().
Reuse the existing teardown API and diagnostic. Update the affected
callback test expectation in this change so the commit remains test-clean.
Fixes: 308c7a0ae885 ("bpf: Refactor object relationship tracking and fix dynptr UAF bug")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Suggested-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Xu Yunxiang <xyx2021@mail.ustc.edu.cn>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/r/CAMB2axOmdHwSrHihSRskDywkCmEGQOX+6dZPN_A9u-HhxY3UDA@mail.gmail.com
Link: https://lore.kernel.org/r/CAMB2axMmrTb+UZ84UD48MwMtXbk1s9bWtreU3AOfjzU0fPT0BA@mail.gmail.com
Link: https://lore.kernel.org/bpf/20260927120422.1042107-2-xyx2021@mail.ustc.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
LLVM splits a static of a Rust program into pieces. Every piece is a VAR
with the type of the whole static:
[223] STRUCT 'BpfCell<core::option::Option<...>>' size=32 vlen=1
[236] VAR '..scx_cosmos9TASK_CTXS.0' type_id=223, linkage=static
[248] DATASEC '.bss' size=0 vlen=9
type_id=236 offset=24648 size=1 (VAR '..TASK_CTXS.0')
and the kernel rejects such BTF with "Invalid size".
Allow it. The size of a variable in DATASEC is used in two places:
- btf_find_datasec_var() looks for timers, spin locks, kptrs and other
special fields. btf_find_field_one() skips a variable when its size is
not the size of its type, so there are no special fields in a piece.
- btf_datasec_show() prints variables by their types. It would read
past the piece and past the end of the map value when the piece is the
last one. Don't print the pieces.
"global data test #8" and "#10" in prog_tests/btf.c expect "Invalid
size". BTF of both is loaded now. The map of #10 is not created, since
its value is larger than the section.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261002124714.180012-10-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Some static functions in BTF of a Rust program have arguments without
names:
[71] FUNC_PROTO '(anon)' ret_type_id=0 vlen=1
'(anon)' type_id=72
[81] FUNC 'unwrap_failed' type_id=71 linkage=static
and the kernel rejects such BTF with "Invalid arg#1". It's 5 of 25
static functions in scx_cosmos and 6 of 24 in scx_simple.
The verifier looks at types of the arguments. The names are printed only,
as "(anon)" when there is none. Allow such static FUNC. Global functions
are checked as before.
The test "func (Some arg has no name)" in prog_tests/btf.c has a static
function. Make it global to keep the check.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Alan Maguire <alan.maguire@oracle.com>
Link: https://lore.kernel.org/bpf/20261002124714.180012-8-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Names of types and functions in BTF that LLVM makes for a Rust program
are not C identifiers:
[69] STRUCT 'NonNull<str>' size=16 vlen=1
[147] FUNC 'write_fmt<scx_cosmos::BpfStream>' type_id=146
and the kernel rejects such BTF with "Invalid name". scx_simple and
scx_cosmos schedulers written in Rust have them in STRUCT, FWD and FUNC,
made of letters, digits and " #&()*,:;<>[]{}", 430 characters at most.
Allow any printable character in btf_name_valid_identifier(), like it's
done for DATASEC. It checks names of types, functions, members,
enumerators, variables and arguments, so all of them can have such
characters now. The limit of KSYM_NAME_LEN stays.
The name of FUNC is a part of the name of the program in kallsyms, where
a space would break the parsers. Replace what is not a character of
an identifier with '_' there.
Tests in prog_tests/btf.c expect "Invalid name" for names with '!' and
'*', which are valid now. Put a character that is not printable there.
The type name '?foo' is expected to load.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261002124714.180012-6-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
There are no address spaces in Rust. LLVM emits plain loads and stores
for arena memory, without cast_kern:
r3 = 0x20 ll // R_BPF_64_64 .bss, libbpf puts it into arena
r2 = *(u64 *)(r3 + 0) // R3 invalid mem access 'scalar'
lock *(u64 *)(r2 + 8) += r1
Add BPF_F_ARENA_SCALAR flag of BPF_PROG_LOAD. When the program is loaded
with it and has an arena treat ldx, stx, st and atomics through a number
as arena access. It's as safe as access through PTR_TO_ARENA: JIT adds
the base of arena to the low 32 bits of the address. The program has
an arena only with CAP_BPF and CAP_PERFMON.
Registers of the program are not changed, since Rust compares and
stores the address after the access. bpf_do_misc_fixups() copies
the low 32 bits into BPF_REG_AX and the access goes through it:
r2 = *(u64 *)(r3 + 0) -> w12 = w3
r2 = *(u64 *)(r12 + 0)
Constant blinding needs BPF_REG_AX for immediates and leaves insns that
use BPF_REG_AX alone. So the immediate of st through a number is not
blinded, the rest of the program is:
*(u64 *)(r3 + 0) = 1 -> w12 = w3
*(u64 *)(r12 + 0) = 1
The same insn may see PTR_TO_ARENA on another path. It works for both.
It's a flag, since a number is also what the verifier makes of a pointer
that went away when the program has CAP_PERFMON: a ringbuf record after
bpf_ringbuf_submit(), a pointer to the packet after bpf_skb_pull_data().
Programs in C that have an arena keep "invalid mem access 'scalar'" for
such bugs.
JIT tells with bpf_jit_supports_arena_scalar() that it takes BPF_REG_AX
as the address of arena access. With other JITs the flag is rejected
with -EOPNOTSUPP. x86 and arm64 do:
- x86: the fault handler finds the register that holds the address
through reg2pt_regs[]. Add BPF_REG_AX, r10 of x86, there.
- arm64: nothing else is needed. The JIT takes any register as the
address and the fault handler reads it by its number. Compile tested
only.
Any number is an address of arena, NULL and small numbers included, like
it is for PTR_TO_ARENA: arena code in C dereferences NULL and relies on
the fault being handled.
Still rejected:
- insn that sees a number on one path and a pointer that is not
PTR_TO_ARENA on another.
- numbers passed to helpers and kfuncs, except __arena arguments.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261002124714.180012-4-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Rust's core::fmt keeps a flag in the low bit of a pointer:
r2 = *(u64 *)(r1 + 0)
r3 = r2
r3 &= 1
r2 >>= 1
The verifier rejects it:
r0 &= 8
R0 bitwise operator &= on pointer prohibited
Negation and byte swap of a pointer already produce a number when
allow_ptr_leaks is set. Do the same for bitwise ops, shifts, *=, /= and
%=, 64-bit and 32-bit. The result is an unknown number. With CAP_PERFMON
the program can store the pointer and load it back as a number already.
+= and -= are not changed: they keep the pointer, 32-bit += is rejected.
Pointers that allow no arithmetic and pointers that may be NULL are
still rejected.
Two tests in verifier_value_illegal_alu store through the result of
&= and /=. They are rejected at the store now. Update expected messages.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261002124714.180012-2-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
A freplace attach claims the target prog by bumping
tgt_prog->aux->freplace_link_cnt and setting tr->extension_prog.
bpf_arch_text_poke() then makes the extension take effect. If the
poke fails, the claims are never released: the attach unwinds
through bpf_link_cleanup(), which clears link->prog, so
bpf_trampoline_unlink_prog() never runs.
Drop the link count under ext_mutex on the error path, and set
tr->extension_prog only after the poke succeeded. The count is
still bumped before the poke: it blocks prog_array updates while
the entry is patched.
Fixes: d6083f040d5d ("bpf: Prevent tailcall infinite loop caused by freplace")
Fixes: be8704ff07d2 ("bpf: Introduce dynamic program extensions")
Suggested-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Yuan Chen <chenyuan@kylinos.cn>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://patch.msgid.link/20260928091121.219333-1-chenyuan_fl@163.com
|
|
Cross-merge BPF and other fixes after downstream PR.
Conflicts:
arch/arm64/net/bpf_jit_comp.c
arch/x86/net/bpf_jit_comp.c
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
specialize_kfunc() currently updates the canonical kfunc descriptor in
place. Different call sites therefore cannot select different
specializations of the same kfunc, and the selected target depends on
verification order.
Keep canonical descriptors unchanged for verifier lookups. Specialize a
copy for each call site, then reuse or append an immutable target
descriptor and store its index in the finalized call instruction.
Allow space for one canonical and one specialized target per kfunc. This
is conservative because most kfuncs do not specialize. Adding specialized
kfunc versions does not require resorting the array because lookups are
only used for deduplication and func_model lookup. Deduplication for
specialized kfuncs is handled by the specialization process itself,
while all specializations of a function share the same proto and
func_model.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-7-emil@etsalapatis.com
|
|
Currently, the verifier sort the kfunc desc table twice: Once during
kfunc collection by function ID, useful during verification, and
once by immediate address, useful during JIT bytecode lowering
time. The latter sort can be removed by storing in the instruction
the kfunc desc array index the desc is in and using that instead.
Modify the JITs to follow the same convention.
This change simplifies the subsequent patch that adds per-call site
function specialization.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-6-emil@etsalapatis.com
|
|
The bpf_arena_alloc_pages() function currently only allocates pages
inside a spinlock critical section with IRQs off. This forces the use of
alloc_pages_nolock() in the BPF allocator, even when the caller is a
sleepable BPF function. This in turn causes allocation failures even in
cases where falling into the allocator slow path and possibly sleeping
would eventually succeed. This can be triggered consistently by heavy
BPF arena users like scx.
Allocate the arena pages before taking the critical section and pass
whether the caller can sleep to bpf_alloc_pages(). This lets sleepable
callers use the blocking allocator while non-sleepable callers retain
the no-lock allocation behavior.
Fixes: b8467290edab ("bpf: arena: make arena kfuncs any context safe")
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-4-emil@etsalapatis.com
|
|
bpf_alloc_pages() currently decides whether it may use the
blocking page allocator from the current execution context
alone. Let callers further restrict that choice by passing
whether their context is sleepable.
Use the blocking allocator only when both the caller and
runtime context allow sleeping. Add __GFP_RETRY_MAYFAIL so
this path can reclaim without invoking the OOM killer when
the allocation is charged to another memcg. Existing
non-sleepable callers retain the no-lock allocation behavior.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-3-emil@etsalapatis.com
|
|
bpf_map_alloc_pages() does not use its map argument. Storing allocated
pages in an array also forces arena callers to allocate a separate
pointer array.
Expose the single-page allocator as bpf_alloc_page(), rename the bulk
helper to bpf_alloc_pages(), and return bulk allocations through an
llist using page->pcp_llist. Add bpf_free_pages() to safely release all
pages remaining on such a list.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261002105218.6171-2-emil@etsalapatis.com
|
|
bpf_mem_cache_free_rcu() uses this_cpu_ptr() which requires migration
to be disabled. All callers of rhtab_delete_elem() disable migration
except __rhtab_map_lookup_and_delete_batch(), which calls it under
rcu_read_lock() only.
On CONFIG_PREEMPT_RCU, rcu_read_lock() does not disable preemption or
migration, so the task can migrate between CPUs during the delete loop,
causing this_cpu_ptr() to trigger:
BUG: using smp_processor_id() in preemptible [00000000] code
Fix by wrapping the delete loop in migrate_disable()/migrate_enable()
in __rhtab_map_lookup_and_delete_batch(), matching the migration
protection that the other callers already provide.
Fixes: 818e00848227 ("bpf: Implement iteration ops for resizable hashtab")
Reported-by: syzbot+fd7e415d891073b83e1f@syzkaller.appspotmail.com
Signed-off-by: Ömer Mete Kaya <omermetekaya0@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://patch.msgid.link/20260929081609.557899-1-omermetekaya0@gmail.com
Closes: https://syzkaller.appspot.com/bug?extid=fd7e415d891073b83e1f
|
|
do_call_rcu_ttrace() returns early when call_rcu_ttrace_in_progress is set
and leaves the objects in free_by_rcu_ttrace. __free_rcu() frees
waiting_for_gp_ttrace only and clears the flag. Hence the objects that
free_bulk() or __free_by_rcu() added while RCU tasks trace GP was in flight
stay in free_by_rcu_ttrace until free_bulk() or alloc_bulk() is called for
the same bpf_mem_cache again, which may never happen. The number of such
objects is not bounded.
Turn call_rcu_ttrace_in_progress into three states:
0 - idle
1 - __free_rcu() is queued
2 - __free_rcu() is queued and free_by_rcu_ttrace got more objects since
do_call_rcu_ttrace() sets 2. __free_rcu() does cmpxchg(1 -> 0) and starts
the next GP when it fails. It cannot clear the flag first and check
free_by_rcu_ttrace later, since bpf_mem_alloc_destroy() frees bpf_mem_cache
without waiting for RCU callbacks when the flag is zero.
Now __free_rcu() queues itself, so the one that didn't see 'draining' may
do call_rcu_tasks_trace() after rcu_barrier_tasks_trace() in
free_mem_alloc(). Queue it under rcu_read_lock() and do synchronize_rcu()
before the barriers. Calling rcu_barrier_tasks_trace() twice works too, but
creating and destroying hash maps in a loop on many cpus slows down to one
free_mem_alloc() per GP and kworkers pile up.
Fixes: 8d5a8011b35d ("bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU.")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20260930095920.601738-3-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Move the part of do_call_rcu_ttrace() that runs after
call_rcu_ttrace_in_progress is set into __do_call_rcu_ttrace().
The next patch will call it from __free_rcu().
No functional change.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20260930095920.601738-2-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Since commit 022ac0750883 ("bpf: use reg->var_off instead of reg->off
for pointers"), find_good_pkt_pointers() sets the range of all packet
pointers sharing an id from the umax of the compared pointer, and
check_packet_access() requires umax + off + size <= range. That assumes
the umax of two such pointers differ by exactly their constant distance.
reg_bounds_sync() breaks it when var_off tightens one umax and not the
other:
r4 &= 0x38
if r4 > 50 goto exit ; umax 50, var_off (0x0; 0x38)
r5 = pkt + r4 ; umax 50
r6 = r5
r6 += 8 ; umax 56, not 58
Comparing r6 with pkt_end sets the range to 56, and the valid 8-byte
load at r5 is rejected (50 + 8 > 56). Comparing r5 sets it to 50, and
the out-of-bounds 1-byte load at r6 - 7, i.e. r5 + 1, is accepted
(56 - 7 + 1 <= 50).
Don't call reg_bounds_sync() on a packet pointer that keeps its id (a
constant was added or subtracted) or its range (an unknown non-negative
value was subtracted), so that var_off cannot tighten its umax. Only
update the 32-bit bounds from var_off: reg_bounds_sanity_check() wants
them constant when the lower half of var_off is, e.g. for pkt + 8.
This relies on nothing else changing the 64-bit bounds of a packet
pointer, which holds today.
var_off of such a pointer is no longer narrowed by its bounds. Adjust
three verifier_align expectations; the low bits, which the alignment
checks use, don't change. veristat on the selftests shows no verdict
changes and +0.8% insns in test_cls_redirect_subprogs.
Fixes: 022ac0750883 ("bpf: use reg->var_off instead of reg->off for pointers")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20261001145255.855630-1-alexei.starovoitov@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Two things in __cgroup_bpf_query() work only because CGROUP_TCP_SOCK_OPS is
the only struct_ops attach type.
It calls cgroup_bpf_enabled(atype) with an atype that
find_atype_by_struct_ops_id() works out at runtime. That macro is an asm
goto and needs a constant. Today the compiler can see there is only one
value; add a second type and the build breaks with "impossible constraint
in 'asm'". The check is not needed anyway. When the key is off nothing is
attached, so progs[atype] and effective[atype] are empty and the count loop
leaves total_cnt at 0. The other attach types already use that loop with
no such check. Drop the check and the skip_count label.
And find_atype_by_struct_ops_id() matches on type_id alone. An attach type
whose subsystem is not built keeps type_id 0, so a query for type 0 finds
it and returns success with nothing instead of -ENOENT. Skip such slots.
Fixes: 369d9dcd8fb8 ("bpf: Add infrastructure to support attaching struct_ops to cgroups")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Yafang Shao <laoar.shao@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20261001141901.3225830-1-shakeel.butt@linux.dev
|
|
Program Structure reports have two attribution gaps. A missing jump
table is reported at the beginning of its subprogram rather than at
the gotox that needs the table, and the subprogram-layout checks run
before func_info and line_info are validated and installed, so their
reports cannot name the source function or line.
Report the missing jump table at the gotox instruction that looks it up,
rather than at the start of its subprogram.
func_info and line_info validation only needs the subprogram boundaries
found by add_subprogs() and the LD_ABS and tail-call properties that
check_subprogs() collects while scanning instructions; it does not
depend on the layout checks themselves. Split the property collection
into its own nested subprogram and instruction scan, together with the
program's callx marker, and run bpf_check_btf_info() before
check_subprogs(). The jump-boundary and fallthrough reports then carry
validated source information without changing either check.
CO-RE relocations are unaffected: commit c26e97721b17 ("bpf: Apply
CO-RE relocations before subprogram validation") applies them before
add_subprogs(), and they stay there. Only the func_info and line_info
validation moves.
Fixes: a8f427835394 ("bpf: Report Program Structure CFG errors")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/cf2f420c2b21de440a7dc51b1565c0f06d4b539640ee5c03384e5d77bcfb5686@mail.kernel.org/
Link: https://lore.kernel.org/bpf/20260926133048.2962553-3-memxor@gmail.com
|
|
Ensure that vlen and flags combinations are consistent for
BTF_KIND_LOC_PARAMs.
Fixes: 33c5a3278bdb ("btf: Extend UAPI to support BTF location (inline site) info")
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260926175113.2368566-2-alan.maguire@oracle.com
|
|
All supported BPF-subprogram argument types now pass through
check_func_arg() from one guarded branch in the legacy validation loop.
Replace that loop and its outgoing-stack validation with
check_func_args().
The scratch prototype is indexed by BTF parameter and the call metadata
carries the BTF function prototype. The existing BTF argument iteration
therefore maps parameters to ABI slots for both kfunc and BPF subprogram
calls, including additional slots occupied by by-value aggregates.
The common checker can return non-EFAULT errors other than -EINVAL, as
the existing BTF-ID path already did. This is compatible with subprog
call handling: only -EFAULT is fatal. Any other error marks BTF
unreliable. Static subprogs can then fall back to inline verification,
while global subprogs reject the call.
Remove the caller-register plumbing that the dedicated loop required.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-12-ameryhung@gmail.com
|
|
Route fixed-size global ARG_PTR_TO_MEM arguments through
check_func_arg(). Preserve their read-write access check, nullable
contract, and support for BTF-defined allocated memory.
Recognize subprog calls explicitly in the common checker. Global
subprog stack liveness can prove that bytes in an argument are unused
by the callee. Retain the existing allowance for those poisoned stack
bytes when call metadata is present, and update the nullability log
expectation.
The verifier also rejects packet pointers when the callee may change
packet data. The concrete packet-backed register state is only available
while checking the call site; the independently verified callee sees
generic PTR_TO_MEM. Record this property as subprog_may_change_pkt in
subprog-only call metadata and retain the packet-pointer rejection in the
common fixed-memory path.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-11-ameryhung@gmail.com
|
|
Route global ARG_PTR_TO_BTF_ID arguments through check_func_arg(). Use
vmlinux BTF for their type match and retain the global-subprogram rules
instead of applying kfunc-only trusted-pointer validation or runtime type
resolution.
The common nullability check now rejects non-nullable global BTF-ID
arguments before register-type matching. Update the corresponding
verifier log expectations.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-10-ameryhung@gmail.com
|
|
Subprog and helper/kfunc dynptr arguments ultimately use the same
process_dynptr_func() validation. Route the subprog dynptr branch
through check_func_arg() so register type, offset, and dynptr state
are checked in the common order.
check_func_arg() validates the register against the PTR_TO_STACK and
CONST_PTR_TO_DYNPTR compatibility set before dispatching to
process_dynptr_func(). Once the subprog path uses the common checker,
process_dynptr_func() has no caller that bypasses this validation. Remove
its now-redundant register-type check.
The existing wrong-register-type test passed a NULL local to a callee
that did not use the argument. Clang left the context pointer in R1,
while GCC materialized zero. The common checker rejected the values at
different stages and emitted different diagnostics.
Obtain a PTR_TO_BTF_ID from bpf_get_current_task_btf() and keep its
callee argument live. This makes both compilers exercise the intended
register-type mismatch.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-9-ameryhung@gmail.com
|
|
ARG_PTR_TO_ARENA accepts both arena pointers and scalars. The existing
BPF-subprogram contract includes a constant-zero scalar; it is an arena
address value rather than a conventional pointer rejected by
nullability checking.
Kfunc prototype generation already records this zero acceptance with
PTR_MAYBE_NULL for both arena suffixes; separate JIT metadata determines
whether the kernel callee receives the arena base or NULL. Apply the same
call-site representation to the temporary BPF-subprogram prototype, then
extend the guarded check_func_arg() path to cover arena arguments.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-8-ameryhung@gmail.com
|
|
Subprog and helper/kfunc context arguments require PTR_TO_CTX with an
acceptable offset. Route subprog context arguments through
check_func_arg() and update the verifier-log expectation.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260928185334.1004200-7-ameryhung@gmail.com
|
