| Age | Commit message (Collapse) | Author |
|---|
|
tmp_ops.func_hash->filter_hash is annotated __rcu, but
update_ftrace_direct_mod() assigns hash to it directly. Sparse reports an
address-space mismatch.
Use rcu_assign_pointer() for the assignment.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260911142512.19344-1-leon.hwang@linux.dev
Fixes: 50b35c9e50a8 ("ftrace: Use hash argument for tmp_ops in update_ftrace_direct_mod")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202609110704.Q3M5vCDV-lkp@intel.com/
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
check_subprogs() treats gotox as a direct jump and validates its reserved
zero offset. When gotox is the final instruction, this produces a
synthetic successor one instruction past the end of the subprogram and
rejects an otherwise valid program.
Skip direct-offset validation for gotox and accept it as a
non-fallthrough terminal instruction. Its actual targets remain validated
from the instruction-array jump table during CFG construction.
Fixes: 493d9e0d6083 ("bpf, x86: add support for indirect jumps")
Signed-off-by: Siddharth Chintamaneni <sidchintamaneni@gmail.com>
Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>
Link: https://lore.kernel.org/r/20260902171414.96165-1-sidchintamaneni@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Generated kfunc prototypes now carry the argument kinds and metadata
needed by the helper argument checker, but kfunc calls still duplicate
the per-argument loop and switch.
Add check_func_args() to determine the argument count, validate outgoing
kfunc stack arguments, and drive check_func_arg() for both call kinds.
Move the remaining kfunc-only argument handlers, including the dynptr
clone runtime refinement, into check_func_arg(), derive each kfunc BTF
referent there once, and skip check_reg_arg() for stack-passed
arguments that check_outgoing_stack_args() already validated. Keep
special-kfunc ID checks behind small helpers so their table can remain
in place.
Keep the intentional differences selected by call metadata: helpers
retain their privileged ARG_ANYTHING pointer-leak behavior, while kfunc
scalars use strict ARG_SCALAR register admission. The shared
ARG_PTR_TO_BTF_ID case retains helper-only compatible-ID and
bpf_kptr_xchg() handling, performs kfunc trusted/RCU provenance
checks, and selects the expected BTF based on call kind. Helper and kfunc
memory arguments likewise
retain their access modes and share fixed-memory and memory/size
diagnostics.
Treat an accepted nullable NULL as a completed argument check, matching
the former kfunc continue path. The skipped helper-specific processing
is either a no-op for NULL or deferred to the paired size argument.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-24-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Helper and kfunc argument verification both require a live owning
reference for non-dynptr release arguments. Whether that reference may
be NULL is expressed by PTR_MAYBE_NULL in helper prototypes and the
__nullable BTF suffix for kfuncs. Factor the shared ownership policy
into check_func_arg_release().
Drop the helper-only rejection of maybe-null owning references. This
allows bpf_kptr_xchg() to accept its explicitly nullable source
directly, matching nullable kfunc release arguments. The verifier
consumes the reference regardless of its runtime nullness. Update the
tests so an owned nullable source succeeds while a nullable non-owning
source remains rejected for lacking ownership.
Preserve the identity of a nullable per-CPU allocation when
bpf_kptr_xchg() transfers its ownership inside an RCU critical section.
The converted MEM_RCU aliases no longer own a reference, but still need
a non-zero ID so a NULL check on one allocation cannot refine another
unrelated allocation.
Use common legacy and structured diagnostic text for both call kinds.
Tailor the structured suggestion to describe a pointer for helpers and
include the expected BTF type for kfuncs. Resolve that type lazily from
the kfunc BTF parameter only when reporting an ownership failure. The
kfunc path now keys the check directly on the cached OBJ_RELEASE
argument flag instead of comparing the argument register against
meta->release_regno, and uses the cached argument kind for the dynptr
exemption.
This is another prerequisite for routing both call types through
check_func_arg().
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-22-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_func_arg() checks whether a helper permits packet pointers before
dispatching argument-specific memory validation. check_kfunc_args() has
no equivalent check, even though kfunc memory arguments may be backed by
packet data.
Move packet-access validation to check_helper_mem_access(), where the
access direction is known and helper, kfunc, and global-subprogram memory
arguments converge. Pass call metadata there so helpers continue to
require bpf_func_proto::pkt_access, while writes through kfunc and global
subprogram arguments use the program-type policy and set
env->seen_direct_write.
Keep call metadata when variable-size memory disables raw mode by
clearing arg_raw_mem.regno instead, and pass it through the map-key path
as well.
This also makes kfunc and global-subprogram packet writes request the
required writable-packet prologue and rejects them for program types
that only support direct packet reads.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-20-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_kfunc_args() open-codes exact register-type tests in most of
its per-argument cases, duplicating what compatible_reg_types[]
already expresses for helpers. This leaves two admission paths and
prevents the helper and kfunc loops from converging.
Runtime argument resolution now converts a scalar-struct BTF
argument to fixed-size memory before register admission. Give the
remaining kfunc-only argument kinds compatibility entries and run
check_reg_type() once before the per-kind switch.
Kfunc memory arguments already accept BPF-allocated objects.
Normalize only the local comparison type to PTR_TO_MEM; subsequent
memory checks still inspect the original register type. Keep
allocated-object forms out of mem_types so helper calls continue
through the existing type-mismatch path and retain its diagnostic.
For ARG_PTR_TO_BTF_ID, let check_reg_type() admit BTF-backed
register types and reject incompatible register classes with its
standard diagnostic. Remove the now-unused lookup_reg2btf_ids().
Exact BTF identity and trust requirements remain checked later by
process_arg_ptr_to_btf_id(). ARG_IGNORE and ARG_PTR_TO_PROG_AUX remain
skipped because the verifier does not read those arguments from the
program.
Iterator arguments use the stack-pointer table. Graph nodes and
ARG_PTR_TO_REFCOUNTED_KPTR share an allocated-object table. It
admits owning and borrowed objects, including RCU-protected forms.
Their switch cases retain API-specific ownership and BTF-record
validation.
ARG_PTR_TO_ALLOC_BTF_ID uses a separate table for object-drop arguments.
Rename timer_types to map_value_types now that ARG_PTR_TO_WORKQUEUE
and ARG_PTR_TO_TASK_WORK share it. Similarly, rename spin_lock_types
to map_value_or_alloc_obj_types because graph roots and resource spin
locks share its map-value-or-allocated-object admission.
Moving admission checks into check_reg_type() must not discard the
structured call-argument diagnostics emitted by the individual cases.
Add bpf_diag_arg_type_plain() alongside bpf_diag_reg_type_plain() and
use it to preserve the existing per-kind Pass suggestions where
available. Other argument kinds retain the generic suggestion. The
reason continues to report the actual register type and all accepted
register types.
Two behavior changes fall out of running admission first:
- ARG_CONST_MEM_SIZE reaches process_const_arg(), and through it
mark_chain_precision(), only after the register is known to be a
scalar. Passing a pointer as a __szk argument used to reach
backtrack_insn() with a non-scalar and trip the backtracking-misuse
verifier bug.
- ARG_CONST_MAP_PTR no longer needs its own type_may_be_null()
test, because check_reg_type() compares whole register types.
Every kfunc argument that is not explicitly ignored now passes
through check_reg_type(), followed by the common register-offset check
in the same order as a helper argument.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-19-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Keep check_reg_type() focused on register admission. Helpers currently
match BTF-ID arguments there, while kfuncs use
process_kf_arg_ptr_to_btf_id(). Both paths ultimately call
btf_struct_ids_match().
Introduce process_arg_ptr_to_btf_id() for helpers, kfuncs, and global
subprograms. Callers provide the expected BTF and ID and retain their
call-specific metadata handling. Group the helper compatible-ID, poison,
and bpf_kptr_xchg() handling in a helper-only block in the
ARG_PTR_TO_BTF_ID case, leaving the common matcher outside it.
Derive strict matching from the generated argument type. This limits
KF_RELEASE strictness to the argument marked OBJ_RELEASE while
preserving the bpf_sk_release() exception and kfunc no-cast-alias rule.
Remove the post-admission BTF and nullability switch from
check_reg_type(), leaving it responsible for register admission. The
compatibility tables already limit helper MEM_ALLOC inputs to
ARG_PTR_TO_SPIN_LOCK and ARG_KPTR_XCHG_DEST, while the kptr source is
admitted only for bpf_kptr_xchg(). Drop the redundant helper-ID
whitelist, pointer-offset check, and constant-offset assertion.
The bpf_kptr_xchg() source match now follows offset validation, so a
source within a referenced object reports the release zero-offset error
before the kptr type error. Update the affected selftests and use
call-neutral wording for BTF mismatch diagnostics.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-18-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_reg_type() rejects a trusted PTR_TO_BTF_ID admitted by
mem_types unless the argument carries MEM_RDONLY. This duplicates
argument-specific memory access validation and is too conservative for
map keys and input map values, whose read direction is not represented
by MEM_RDONLY.
Since commit fa75b7c85b0d ("bpf: Enforce write checks for BTF pointer
helper access"), check_helper_mem_access() forwards the requested access
type to check_ptr_to_btf_access(). Map keys always request read access.
Map values request write access only when tagged with MEM_WRITE.
ARG_PTR_TO_MEM prototypes must carry either MEM_RDONLY or MEM_WRITE,
and size arguments use the direction of their associated memory
argument. check_ptr_to_btf_access() therefore checks the effective access
direction in every case.
Drop the early rejection and keep check_reg_type() limited to register
admission.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-17-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_kfunc_args() rejects a NULL or possibly-NULL register passed to
a non-nullable pointer argument up front. Helpers only do so inside
check_reg_type() for PTR_TO_BTF_ID arguments, leaving other pointer
kinds to fail indirectly through compatible_reg_types[].
Give both paths the same policy and factor it into
check_func_arg_nullability(). Gate it on arg_type_is_scalar() rather
than on the BTF shape of the kfunc parameter. This keeps a zero passed
to a size argument from being interpreted as a NULL pointer.
Taking nullability from the argument classification also avoids an
is_kfunc_arg_nullable() lookup for every kfunc argument on every
verification.
Keep release-argument nullability separate from ownership checking.
A possibly-NULL release argument must be rejected even if it is
otherwise nullable, while the ownership check must still permit a
literal NULL for bpf_kptr_xchg().
Use call-neutral structured diagnostics and retain the expected BTF
type when it is available for a kfunc argument. Resolve that type lazily
from the call metadata only when reporting an error. Helpers now report
NULL-ness rather than a register-type mismatch for non-BTF-ID pointer
arguments and literal NULL passed to a non-nullable pointer argument.
Update the affected selftest expectations.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-16-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
A function argument type can require call-site refinement before
register admission. Helpers refine map-value arguments according to the
recorded map type, while kfuncs reinterpret a pointer to a scalar-only
BTF struct as fixed-size memory when the register does not carry a
BTF ID.
Add resolve_func_arg_type() as the common entry point. It delegates
helper map-value refinement to the existing resolve_map_arg_type(),
then handles the kfunc BTF-to-memory fallback. It updates the effective
argument type and size consumed by the subsequent checks.
Preserve PTR_MAYBE_NULL when changing the base type so later
nullability validation observes the BTF contract. Keep OBJ_RELEASE
arguments in their original pointer class. The latter is not a bug fix:
existing kfunc checking already validates ownership before this
resolver. It preserves release metadata for the later shared checking
order, where ownership validation follows register admission.
Moving the kfunc fallback before admission lets the resulting
fixed-size memory argument enter the compatibility check directly.
This is another step toward routing both call types through
check_func_arg().
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-15-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
A generated kfunc prototype classifies a pointer to scalar memory
without an adjacent size argument as ARG_PTR_TO_MEM | MEM_FIXED_SIZE.
The kfunc path currently walks BTF and resolves the pointee size each
time it verifies a call. The common check_func_arg() path instead
expects that size in bpf_func_proto::arg_size[].
Resolve the size when generating the prototype and teach the existing
kfunc path to consume the cached value. This prepares fixed-size memory
arguments to move to the common checker.
arg_size[] shares storage with arg_btf_id[]. An ARG_PTR_TO_BTF_ID
argument that falls back to scalar-struct memory therefore keeps its BTF
ID pointer and continues resolving the size when that fallback is taken.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-14-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_kfunc_args() walks the kfunc's BTF on every verification of a call
to work out which BTF ID an ARG_PTR_TO_BTF_ID argument expects. Helpers
name theirs in bpf_func_proto::arg_btf_id[], as a pointer to a BTF ID
that resolve_btfids fills in at build time.
The ID of a kfunc argument's referent is already stored in the
immutable BTF records that describe its pointer and modifier chain. Make
arg_btf_id[] point to the BTF field containing the resolved ID. A __map
argument instead uses the existing vmlinux BTF ID pointer. Produce this
metadata alongside the argument classification in get_kfunc_arg_type().
Declare the argument BTF ID pointers const, since the verifier only
reads through them. The BTF object owns their storage and remains
alive while the generated prototype is used, so the pointers remain
valid when the kfunc descriptor array is reallocated or sorted.
Both helper and kfunc callers can now read the expected BTF ID through
the same bpf_func_proto field, which lets check_func_arg() take over
the ARG_PTR_TO_BTF_ID case.
arg_btf_id[] shares a union with arg_size[], so an argument cannot
store both. The scalar-struct memory fallback keeps resolving its byte
size at verification time, as it does today.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-13-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
The BTF ID argument of a kfunc has to be referenced or trusted unless
the kfunc is KF_RCU, in which case an RCU-protected pointer is accepted
too. check_kfunc_args() reads that off meta->kfunc_flags on every
verification of the call, which is the last thing keeping the argument
policy of ARG_PTR_TO_BTF_ID out of the generated prototype.
Set MEM_RCU in get_kfunc_arg_type() for ARG_PTR_TO_BTF_ID arguments of
KF_RCU kfuncs, the way it already sets OBJ_RELEASE for release
arguments, and test the generated flag instead. MEM_RCU represents the
KF_RCU relaxation applied by the ARG_PTR_TO_BTF_ID trust check; other
argument kinds have their own provenance requirements.
No functional change intended.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-12-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
The dynptr flavour a kfunc argument expects is rebuilt on every
verification of the call: check_kfunc_args() starts from
ARG_PTR_TO_DYNPTR, adds MEM_UNINIT from the __uninit suffix, and then
walks a chain of func_id comparisons to add DYNPTR_TYPE_SKB, _XDP,
_SKB_META or _FILE. All of that is fixed by the kfunc identity and BTF,
so move it into get_kfunc_arg_type() and let the generated prototype
carry it.
Inheriting the classification also means PTR_MAYBE_NULL reaches
process_dynptr_func() for a __nullable dynptr argument, where it did not
before, and is_dynptr_type_expected() tested for any type of dynptr with
an exact arg_type == ARG_PTR_TO_DYNPTR comparison. Test the
DYNPTR_TYPE_FLAG_MASK bits instead, which is what the comment there
already claims to mean and does not care about unrelated flags.
What cannot move is the KF_bpf_dynptr_clone arm, which takes its type
from meta->dynptr.type, recorded while verifying the parent dynptr
argument earlier in the same call. That stays at the call site, applied
on top of the cached classification.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-11-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_kfunc_args() ORs OBJ_RELEASE into arg_type on every verification
of every argument of every call, from a comparison against
meta->release_regno. Neither side of that comparison depends on the
call site. bpf_fetch_kfunc_arg_meta() sets release_regno to BPF_REG_1
when the kfunc is KF_RELEASE and leaves it zero otherwise, and regno
is derived from the argument index, so the test is true exactly for
argument 0 of a KF_RELEASE kfunc.
Set the flag in get_kfunc_arg_type() instead, where the rest
of the classification is built, and drop the comparison. The release
handling in check_func_proto() is helper-only, so an OBJ_RELEASE in a
generated kfunc prototype does not feed back into meta->release_regno.
meta->release_regno itself stays: it is still what tells the referenced
PTR_TO_BTF_ID check and release_reg() which register to act on.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-10-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
The verifier supplies a bpf_prog_aux argument to both
bpf_timer_set_callback() and kfuncs rather than reading it from the BPF
program. The helper prototype leaves its third argument unused, while
the kfunc path identifies the argument from BTF at every call.
Add ARG_PTR_TO_PROG_AUX, record it in the helper prototype, and classify
the kfunc argument when its prototype is generated. Because other
ignored and implicit arguments are already represented by ARG_IGNORE,
the generated prototype has no internal ARG_UNUSED gaps. Validate in
one place that it contains at most one prog-aux argument and that the
argument is register-passed, as required by the BPF_LD_IMM64 fixup.
Record the argument register in the per-instruction metadata for both
call kinds. Drive the helper fixup from that metadata instead of the
helper ID and its hard-coded R3, matching the existing kfunc fixup.
bpf_call_arg_meta::arg_prog is then no longer needed.
No functional change beyond reporting invalid kfunc signatures when
the call is added rather than when it is verified.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-9-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
The verifier does not inspect __ign arguments or implicit arguments,
except that bpf_prog_aux is handled specially. get_kfunc_arg_type()
leaves the ignored arguments unclassified, so gen_kfunc_arg_proto()
skips them and check_kfunc_args() repeats the same BTF predicates for
every call.
Add ARG_IGNORE and classify ignored arguments in get_kfunc_arg_type().
The generated prototype can then record them, and the call site can
consume the recorded type without deriving the classification again.
Keep bpf_prog_aux on its existing BTF-based path for now.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-8-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_kfunc_args() classifies arguments with enum kfunc_ptr_arg_type
while check_func_arg() uses enum bpf_arg_type, yet both classifications
are stored in bpf_func_proto::arg_type. The overlapping namespaces force
the kfunc path to translate argument kinds before calling shared checks.
Fold the kfunc kinds into enum bpf_arg_type. Reuse ARG_SCALAR and
the existing pointer and memory kinds where their semantics match,
map kfunc callbacks to ARG_PTR_TO_FUNC, and add enumerators for the
remaining kfunc-only kinds. check_kfunc_args() can then carry one
classification throughout verification.
Preserving the original argument kind also lets
check_func_arg_reg_off() derive the zero-offset requirement for
ARG_PTR_TO_REFCOUNTED_KPTR directly. Remove its separate
btf_id_fixed_off_ok parameter and wrapper, along with the now-empty
translation switch in check_kfunc_args().
Keep the PTR_TO_CTX offset decision based on the base argument kind.
The generated kind may carry flags such as PTR_MAYBE_NULL, while the
removed translation discarded them. This retains the zero-offset
requirement for context arguments regardless of such flags.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-7-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
ARG_DONTCARE is the zero-valued terminator for the fixed-size
helper argument array rather than an argument whose value is ignored.
Rename it to ARG_UNUSED and stop helper argument iteration explicitly
when it is encountered.
Commit c3fd8e5fd100 ("bpf: Reject non-scalar bpf_loop iteration
counts") introduced ARG_SCALAR for arguments that must be scalars.
Reuse it for integer and enum global subprogram arguments, which already
require SCALAR_VALUE despite being classified as ARG_ANYTHING, and for
exception callback arguments. This leaves ARG_ANYTHING with its legacy
helper behavior of accepting pointers when pointer leaks are allowed.
Also pass ARG_PTR_TO_MEM when checking a global subprogram memory
argument instead of using the prototype terminator as a placeholder.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-6-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Helpers are identified by enum bpf_func_id, while kfuncs are identified
by a BTF ID. Both values are stored in bpf_call_arg_meta::func_id, and
a helper ID can have the same numeric value as a kfunc BTF ID.
Introduce is_kfunc_call(), which first confirms that the metadata is for
a kfunc through meta->btf, and use it for comparisons against the special
kfunc BTF ID list. This complements is_helper_call() before later patches
move these checks into paths shared by helpers and kfuncs.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-5-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Helpers are identified by enum bpf_func_id, while kfuncs are identified
by a BTF ID. Both values are stored in bpf_call_arg_meta::func_id, and
a kfunc BTF ID can have the same numeric value as a BPF_FUNC_* constant.
Later patches extend check_reg_type() and check_func_arg() to kfuncs.
A bare func_id comparison in those common paths could then mistake a
kfunc for a helper.
Introduce is_helper_call(), which first excludes kfunc metadata through
meta->btf, and use it for every BPF_FUNC_* comparison. This keeps
helper-specific behavior out of the shared path from the start.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-4-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_func_arg() derives every argument from BPF_REG_1 + arg. That works
for helpers, which have at most five arguments, but kfunc arguments
beyond the fifth are passed on the stack.
Switch to the argno_t addressing used by check_kfunc_args(): obtain
argument state with get_func_arg_reg(), use argno for diagnostics, and
call check_reg_arg() only when the argument is held in a register. This
prepares check_func_arg() to be shared with kfunc verification.
There is no functional change for helpers. reg_arg_name() renders their
register arguments as R%d, so the updated diagnostics remain unchanged.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-3-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Helper, kfunc, and global subprogram argument paths supply callee
identity and state differently. check_reg_type() receives the call name
separately from bpf_call_arg_meta, while process_dynptr_func() receives
the call name, referenced-object state, and dynptr state as separate
arguments.
Populate meta->func_name for helper and global subprogram calls, keep
one metadata object for the entire global subprogram argument check, and
have both shared checking functions obtain their name and state from the
metadata. This gives dynptr validation one interface for every call kind
in preparation for unifying their argument-checking paths.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/r/20260911220415.1396439-2-ameryhung@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
rb_wake_up_waiters() is a irq_work callback which is initialized with
init_irq_work(). As such it will be invoked in thread context on
PREEMPT_RT. Invoking the callback in IRQ context on PREEMPT_RT is not an
option due its usage of wake_up_all(). Since this callback may run in
thread context, it needs to acquire ring_buffer_per_cpu::reader_lock with
disabling interrupts and may not assume that they are disabled.
Use raw_spinlock_irqsave() to acquire ring_buffer_per_cpu::reader_lock.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260911102152.YEtwkBj9@linutronix.de
Fixes: 68282dd930ea3 ("ring-buffer: Fix resetting of shortest_full")
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
When a tracer option file is opened, it is passed a descriptor that points
to an element on the trace_array's topts array. This element has
information to find the trace array and other information. It uses this
element to take a reference of the trace_array so that the trace_array
does not get removed while this file is opened.
Unfortunately, there's a race condition where the element itself could be
freed by the removal of the instance the trace_array represents causing a
use-after-free as this element that is used to find the trace_array to
increment its reference counter is also freed when the instance is
removed.
To solve this, add a trace_array_tracer_options_get() helper function that
will take the address of the element that is passed to the open function
by the inode->i_private pointer and search all the trace_arrays under a
lock to find the one that the element's address is in the range of the
trace_arrays topts array elements. When a match happens, that trace_array's
reference would be increased.
Note, there's a race where if an admin was deleting and creating trace
instances at the same time and the memory of the old trace_array's array
matched the memory of the new trace_array that it could in theory open the
option from the wrong trace array. But we do not care because it would be
stupid to perform that kind of action. As long as the only thing that can
happen is that the option from the wrong trace array is used and doesn't
crash the kernel it will only make the user confused. But if they are
doing something stupid like this, they are already confused, so no harm
done.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260910221209.62dad8d3@robin
Fixes: 7e2cfbd2d3c86 ("tracing: Have option files inc the trace array ref count")
Reported-by: sashiko-bot@kernel.org
Closes: https://lore.kernel.org/linux-trace-kernel/20260902121918.5a9e9d1b@gandalf.local.home/
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
ring_buffer_read_page_size() takes a parameter named rpage, but its
kernel-doc describes page. As a result, kernel-doc reports rpage as
undescribed and page as an excess parameter description.
Rename the documentation entry to match the function.
Link: https://patch.msgid.link/20260909062917.89482-1-kmehltretter@gmail.com
Fixes: dae8dda341d2 ("tracing: Fix subbuf resize races with trace_pipe_raw readers")
Assisted-by: LLM
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
While ftrace_set_clr_event() modifies its input buffer during parsing,
before returning to the caller the buffer is supposed to be restored
to its original state.
This works correctly for the colon between the subsystem and event
but not the colon at the beginning of :mod:.
Restore the colon, so the :mod: trailer is not stripped after
ftrace_set_clr_event().
Cc: stable@vger.kernel.org
Fixes: 4c86bc531e60 ("tracing: Add :mod: command to enabled module events")
Link: https://patch.msgid.link/20260908-tracing-cli-event-filter-v2-1-05396a3fb663@linutronix.de
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
"cpu", "CPU", "stacktrace" and "STACKTRACE" are generic fields, defined
with an offset and a size of zero so that the filter code can match them
by name. parse_field() maps them onto their common_* equivalents for
backward compatibility, but unlike the common_* names it hands the
placeholder back to the caller instead of NULL.
create_hist_field() takes a non-NULL field as a promise that the record
carries a stacktrace and picks HIST_FIELD_FN_STACK, so the __data_loc
word is read from offset 0, that is from common_type, and its low 16
bits are followed as an offset into the record. What is found there
becomes the length of an unbounded memcpy. Pick an event whose id is
small enough that the offset stays inside its own record and the length
is a kernel text address:
# cd /sys/kernel/tracing
# echo 'hist:keys=STACKTRACE' > events/ftrace/print/trigger
# echo hello > trace_marker
Oops: general protection fault, probably for non-canonical address
RIP: 0010:rb_next+0x23/0x60
</IRQ>
RIP: 0010:memcpy+0xc/0x30
event_hist_trigger+0x2e7/0x12c0
Kernel panic - not syncing: Fatal exception in interrupt
Leave the field NULL, which is what the comment above the branch says
the code does and what common_stacktrace already does. FILTER_CPU and
FILTER_COMM are left alone, their create_hist_field() branches never
look at the field.
Cc: stable@vger.kernel.org
Fixes: 4b512860bdbd ("tracing: Rename stacktrace field to common_stacktrace")
Link: https://patch.msgid.link/20260907155045.692664-3-donggeunyoo.kernel@gmail.com
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
parse_field() sets HIST_FIELD_FL_STACKTRACE from the ".stacktrace"
modifier before it looks the field name up, and nothing afterwards
checks that the name resolved to a field which holds a stacktrace.
create_hist_field() picks HIST_FIELD_FN_STACK on the strength of the
field pointer alone, which reads a __data_loc word from the record and
follows its low 16 bits as an offset into the same record.
event_hist_trigger() takes the first word there as an entry count and
copies that many longs into a 31 entry array:
n_entries = *stack;
memcpy(entries, ++stack, n_entries * sizeof(unsigned long));
Neither end of that copy is bounded, and the count is whatever the event
holds at the offset, so any field will do:
# cd /sys/kernel/tracing/events/sched/sched_process_fork
# echo 'hist:keys=parent_pid.stacktrace' > trigger
# (true)
BUG: kernel NULL pointer dereference, address: 0000000000000008
RIP: 0010:rb_insert_color+0x18/0x130
timerqueue_linked_add+0x7e/0xd0
enqueue_hrtimer+0x39/0xb0
__hrtimer_run_queues+0x10f/0x1f0
</IRQ>
RIP: 0010:memcpy+0xc/0x30
event_hist_trigger+0x165/0x690
The timer interrupt landed on the rbtree the copy had already run over.
No debug options are needed for this; KASAN reports the same write as an
out-of-bounds read of 13835058055416381440 bytes.
Documentation/trace/histogram.rst already states the rule, "must be a
long[] type", so enforce it once the name has been resolved. Names which
resolve to no field at all, "hitcount.stacktrace" and the common_*
pseudo-fields, are refused for the same reason: they hold no stacktrace
to read.
Cc: stable@vger.kernel.org
Fixes: cc5fc8bfc961 ("tracing/histogram: Add stacktrace type")
Link: https://patch.msgid.link/20260907155045.692664-2-donggeunyoo.kernel@gmail.com
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Commit 6f86bdeab633 ("tracing: Fix bad hist from corrupting named_triggers
list") described how a trigger that is registered but not on file->triggers
ends up freed while still on the global named_triggers list, and moved the
registration down so that hist_trigger_enable() follows it immediately. One
path still gets there. hist_trigger_enable() adds the trigger and takes it
straight back out when the event cannot be enabled:
list_add_tail_rcu(&data->list, &file->triggers);
update_cond_flag(file);
if (trace_event_trigger_enable_disable(file, 1) < 0) {
list_del_rcu(&data->list);
update_cond_flag(file);
ret--;
}
so the list walk in hist_unregister_trigger() matches nothing, test stays
NULL, and the ->free() that would call del_named_trigger() is skipped.
out_unreg falls through to out_free, which frees the trigger anyway:
BUG: KASAN: slab-use-after-free in find_named_trigger+0xac/0xc0
Read of size 8 at addr ffff8880091d3160 by task init/1
find_named_trigger+0xac/0xc0
hist_register_trigger+0xc1/0xa00
event_hist_trigger_parse+0x3146/0x6af0
event_trigger_write+0xce/0x160
Freed by task 69:
kfree+0x154/0x420
trigger_kthread_fn+0xfd/0x160
Leave the trigger where hist_unregister_trigger() can find it and let that
undo the registration, which is the only code that knows all of what
cmd_ops->init() took: the named list entry, the hist_pad reference, the
reference on the trigger a named histogram is shared with, and the copied
cmd_ops. It also pairs the failed trace_event_trigger_enable_disable(),
whose sm_ref and buffered event reference are otherwise left behind.
Since ->free() releases trigger_data and, for a trigger that does not share
its histogram, hist_data with it, out_unreg can no longer fall through to
out_free. For a trigger that does share, hist_register_trigger() has
already destroyed the caller's hist_data, so the fall-through was reading
freed memory there as well.
Move the enable_timestamps check in hist_unregister_trigger() above the
->free() call for the same reason: hist_data does not outlive it once the
trigger being removed is the one that owns it.
Cc: stable@vger.kernel.org
Fixes: 067fe038e70f ("tracing: Add variable reference handling to hist triggers")
Reported-by: Sashiko AI <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/linux-trace-kernel/20260907092944.3950E1F00A3D@smtp.kernel.org/
Link: https://patch.msgid.link/20260907124420.607097-3-donggeunyoo.kernel@gmail.com
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
event_hist_trigger_named_init() puts the trigger on the global
named_triggers list and only then takes the reference on the trigger it
shares its histogram with:
data->ref++;
save_named_trigger(data->named_data->name, data);
ret = event_hist_trigger_init(data->named_data);
if (ret < 0) {
kfree(data->cmd_ops);
data->cmd_ops = &trigger_hist_cmd;
}
return ret;
event_hist_trigger_init() fails when alloc_hist_pad() cannot allocate, and
nothing takes the trigger back off the list on the way out.
event_hist_trigger_parse() frees it, and the next lookup by name reads the
freed object:
BUG: KASAN: slab-use-after-free in find_named_trigger+0xac/0xc0
Read of size 8 at addr ffff888009346860 by task init/1
find_named_trigger+0xac/0xc0
hist_register_trigger+0xc1/0xa00
event_hist_trigger_parse+0x3146/0x6af0
event_trigger_write+0xce/0x160
Freed by task 67:
kfree+0x154/0x420
trigger_kthread_fn+0xfd/0x160
Do the reference first and publish once it has succeeded, so that nothing
which can fail runs after the trigger becomes findable.
Cc: stable@vger.kernel.org
Fixes: 7ab0fc61ce73 ("tracing: Move histogram trigger variables from stack to per CPU structure")
Reported-by: Sashiko AI <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/linux-trace-kernel/20260907092944.3950E1F00A3D@smtp.kernel.org/
Link: https://patch.msgid.link/20260907124420.607097-2-donggeunyoo.kernel@gmail.com
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Acked-by: Tom Zanussi <zanussi@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
hist_register_trigger() puts the trigger on the global named_triggers
list in cmd_ops->init(), and only then sets the trace clock:
if (data->cmd_ops->init) {
ret = data->cmd_ops->init(data);
if (ret < 0)
goto out;
}
if (hist_data->enable_timestamps) {
ret = tracing_set_clock(file->tr, hist_data->attrs->clock);
if (ret) {
hist_err(tr, HIST_ERR_SET_CLOCK_FAIL, errpos(clock));
goto out;
}
The clock string is not checked anywhere before that call, so a named
trigger using common_timestamp with an unknown clock fails after it has
already become findable. event_hist_trigger_parse() then frees it
without taking it off the list, and the next lookup by name reads the
freed object:
~# cd /sys/kernel/tracing/events/sched/sched_switch
~# echo 'hist:name=foo:keys=common_pid:ts=common_timestamp:clock=bogus' > trigger
bash: echo: write error: Invalid argument
~# echo 'hist:name=foo:keys=common_pid' > trigger
BUG: KASAN: slab-use-after-free in find_named_trigger+0xac/0xc0
Read of size 8 at addr ffff88800915d760 by task init/1
find_named_trigger+0xac/0xc0
hist_register_trigger+0xc1/0x900
event_hist_trigger_parse+0x3146/0x6af0
event_trigger_write+0xce/0x160
Freed by task 63:
kfree+0x154/0x420
trigger_kthread_fn+0xfd/0x160
Set the clock before the trigger is registered, so that nothing which
can fail runs after it is published, the way commit 6f86bdeab633
("tracing: Fix bad hist from corrupting named_triggers list") moved the
registration below the rest of the setup.
tracing_set_filter_buffering() is reference counted, so the init failure
path has to drop the reference that the clock block now takes first.
Cc: stable@vger.kernel.org
Fixes: a4072fe85ba3 ("tracing: Add a clock attribute for hist triggers")
Link: https://patch.msgid.link/20260907091415.554535-1-donggeunyoo.kernel@gmail.com
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Correct "availabe" to "available", reported by scripts/checkpatch.pl using
the misspelling list in scripts/spelling.txt. Only touches comments, no
code changes.
Link: https://patch.msgid.link/20260907062608.13924-1-hemanth.selam@gmail.com
Assisted-by: Cursor:claude-opus-5
Signed-off-by: Hemanth Selam <hemanth.selam@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The .percent and .graph modifiers exist only for histogram values, but a
value carrying either of them has been rejected since v6.3. The example
in Documentation/trace/histogram.rst,
# echo 'hist:keys=prev_comm:vals=hitcount.percent:nohitcount' > \
events/sched/sched_switch/trigger
returns -EINVAL.
parse_field() sets the two flags only when the field is neither a key nor
a variable, that is, only on a value:
} else if (strncmp(modifier, "percent", 7) == 0) {
if (*flags & (HIST_FIELD_FL_VAR | HIST_FIELD_FL_KEY))
goto error;
*flags |= HIST_FIELD_FL_PERCENT;
__create_val_field() then rejects a value for carrying them, so no field
can reach hist_trigger_print_val(), where both are implemented.
commit e0213434fe3e ("tracing: Do not let histogram values have some
modifiers") added the check after a value with .buckets oopsed in
hist_field_name(). That happens because .buckets and .log2 make
create_hist_field() build a nested field in operands[0] which
hist_field_name() then walks into. The percent and graph flags do not
create an operand and are not read by hist_field_name(); they are only
used when printing a value.
Stop rejecting the two flags on a value. The check for variables is left
alone, where they are unreachable anyway because parse_field() rejects a
variable carrying them first.
With the two flags removed, the trigger above installs and prints as
documented:
{ prev_comm: rcu_preempt } hitcount (%): 0.00
{ prev_comm: init } hitcount (%): 99.98
Totals:
Hits: 237896
Cc: stable@vger.kernel.org
Fixes: e0213434fe3e ("tracing: Do not let histogram values have some modifiers")
Link: https://patch.msgid.link/20260907052113.430818-1-donggeunyoo.kernel@gmail.com
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
print_entries() uses n_entries both as the number of sort entries and as
its own return value, so the -ENOMEM it stores when the stats allocation
fails overwrites the count that the cleanup still needs:
n_entries = tracing_map_sort_entries(map, ...);
if (n_entries < 0)
return n_entries;
...
if (!stats) {
n_entries = -ENOMEM;
goto out;
}
...
out:
tracing_map_destroy_sort_entries(sort_entries, n_entries);
tracing_map_destroy_sort_entries() takes an unsigned int and loops up to
it, so -ENOMEM arrives as 4294967284. It walks an array of at most
map->max_elts pointers and calls destroy_sort_entry(), which dereferences
and frees, on whatever lies past the end.
Reading the hist file of a trigger with a .percent value, with that
allocation forced to fail:
BUG: KASAN: vmalloc-out-of-bounds in tracing_map_destroy_sort_entries+0xa0/0xb0
Read of size 8 at addr ffffc90000045000 by task init/1
tracing_map_destroy_sort_entries+0xa0/0xb0
hist_show+0x6f7/0x1df0
seq_read_iter+0x2b8/0x1190
vfs_read+0x176/0xa40
The buggy address belongs to a 4-page vmalloc region starting at
ffffc90000041000 allocated at tracing_map_sort_entries+0x5c/0xd50
A few pages further the fault is fatal. The registers at the oops confirm
the bound: the loop's end pointer less the array start, over the pointer
size, is 4294967284.
Return the error in a separate variable and leave n_entries holding the
count, the way tracing_map_sort_entries() does on its own error path.
The stats block is only entered for a value carrying .percent or .graph,
which __create_val_field() has rejected since v6.3, so this cannot be
reached in mainline as it stands. It becomes reachable again with
"tracing: hist: let values keep the percent and graph modifiers", so it
should be applied first.
Cc: stable@vger.kernel.org
Fixes: abaa5258ce5e ("tracing: Add .percent suffix option to histogram values")
Link: https://patch.msgid.link/20260907060323.480728-1-donggeunyoo.kernel@gmail.com
Reported-by: sashiko-bot@kernel.org
Closes: https://lore.kernel.org/all/20260907053113.1CED91F00A3A@smtp.kernel.org/
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Writing a hist trigger whose value or variable carries a modifier that is
not allowed there leaks the fields that were built for it.
__create_val_field() takes the field from parse_expr() and stores it in
hist_data->fields[] only after the modifier checks have run:
hist_field = parse_expr(hist_data, file, field_str, flags, var_name,
&n_subexprs);
...
if (hist_field->flags & HIST_FIELD_FL_VAR) {
if (hist_field->flags & (...))
goto err;
} else {
if (hist_field->flags & (...))
goto err;
}
hist_data->fields[val_idx] = hist_field;
Both checks jump past that store, and the err label returns without
freeing anything. The error unwinds to create_hist_data(), which calls
destroy_hist_data() -> destroy_hist_fields(), and that reaches a field
only by walking fields[]. A field that never got there is unreachable.
commit e0213434fe3e ("tracing: Do not let histogram values have some
modifiers") set ret to -EINVAL and fell through to the store, which left
the field owned by fields[] and freed along with the rest of hist_data.
Splitting the check into a value case and a variable case replaced that
fall-through with a goto that skips it.
With CONFIG_DEBUG_KMEMLEAK, 200 writes of
# echo 'hist:keys=prev_pid:vals=next_pid.log2' > \
events/sched/sched_switch/trigger
each correctly rejected with -EINVAL, leave 332 unreferenced objects
(63744 bytes) reported at create_hist_field(); 200 install and remove
cycles of a valid trigger leave none. A '.log2' field is two
allocations, since create_hist_field() puts the plain field in
operands[0] of the log2 field, and both are reported.
Use destroy_hist_field() rather than __destroy_hist_field() so that
operands[0] is freed as well. It returns early for HIST_FIELD_FL_VAR_REF,
which is what an operand owned by hist_data->var_refs[] needs; the
rejected field itself is never a var ref, because a var ref never carries
a modifier flag.
Cc: stable@vger.kernel.org
Fixes: e30fbc618e97 ("tracing/histograms: Allow variables to have some modifiers")
Link: https://patch.msgid.link/20260907034948.240387-1-donggeunyoo.kernel@gmail.com
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
create_var_ref() allocates a VAR_REF hist_field and then calls
init_var_ref() to fill it in. When that fails the field is leaked.
commit 656fe2ba85e8 ("tracing: Use hist trigger's var_ref array to destroy
var_refs") made destroy_hist_field() return early for
HIST_FIELD_FL_VAR_REF, since var refs are freed by walking the trigger's
var_refs[] array instead. create_var_ref() adds the field to that array
only after init_var_ref() has succeeded, so on this path the field is in
neither place and nothing frees it. The call was correct when it was
written, before var refs were taken out of destroy_hist_field().
init_var_ref() cannot free it either. The caller owns the field, so
init_var_ref() undoes only its own string allocations and leaves the
field alone. Freeing it there would leave create_var_ref() passing freed
memory to destroy_hist_field(), which reads its flags.
Call __destroy_hist_field(), which frees the field without consulting
the flag.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260906133352.3815019-1-donggeunyoo.kernel@gmail.com
Fixes: 656fe2ba85e8 ("tracing: Use hist trigger's var_ref array to destroy var_refs")
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Using the same variable three or more times in one hist trigger leaks the
variable reference and its strings when the trigger is removed.
commit 656fe2ba85e8 ("tracing: Use hist trigger's var_ref array to destroy
var_refs") made a trigger's var_refs[] array the only owner of a var ref:
destroy_hist_field() returns early for HIST_FIELD_FL_VAR_REF, so the field
expressions never destroy one. One entry, freed once, no count needed.
commit 8bcebc77e85f ("tracing: Fix histogram code when expression has same
var as value") then made repeated references share one object and added a
count of them. Only the increment side exists, since those expressions
still return early and never drop a reference, so __destroy_hist_field()
sees how many references were created rather than how many are left. It
frees when the decremented count is 0 or 1, so two references work and
three or more leak.
Sharing kept one array entry per object, and create_var_ref() searches and
appends within a single trigger, so nothing outside it holds the object.
Removing a trigger whose variables are still referenced is already refused
by check_var_refs() with -EBUSY. Drop the count and free unconditionally.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260906124025.3550596-1-donggeunyoo.kernel@gmail.com
Fixes: 8bcebc77e85f ("tracing: Fix histogram code when expression has same var as value")
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
When a graph entry does not fit in the trace_seq, print_graph_entry()
saves it in the iterator's fgraph_data and reprints it on the next read.
The entry has already been consumed from the ring buffer by then, so the
copy is all that is left of it.
The copy is sized with iter->ent_size, which no longer describes the
saved entry but whatever entry the iterator has moved on to. The
argument count is derived from the same field, so a 72 byte entry saved
and then reprinted ahead of a 48 byte return entry loses its arguments.
Record the size next to the failure flag, so that the two are always set
together, and restore it before reprinting.
Cc: stable@vger.kernel.org
Fixes: ff5c9c576e75 ("ftrace: Add support for function argument to graph tracer")
Link: https://patch.msgid.link/20260906034406.1335316-1-donggeunyoo.kernel@gmail.com
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
FGRAPH_MAX_INDEX has no user, and it expands to FGRAPH_INDEX_SIZE and
FGRAPH_RET_INDEX, neither of which is defined anywhere in the tree. It
was added in that form by commit 91c46b0aa917 ("function_graph:
Implement fgraph_reserve_data() and fgraph_retrieve_data()"), which
introduced the current data word layout under new names, so anything
referencing it would have failed to build ever since.
Remove it.
Link: https://patch.msgid.link/20260905211922.1196366-1-donggeunyoo.kernel@gmail.com
Signed-off-by: Donggeun Yoo <donggeunyoo.kernel@gmail.com>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
The alloc_obj_fields array is never modified. Make the array itself
const, not just the strings it points to.
Signed-off-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260906223358.25717-1-brads@mainlining.org
|
|
dup_task_struct() copies the parent's task_struct, including ret_stack.
ftrace_graph_init_task() clears the copied function graph state, but it
currently runs after copy_exec_state().
For non-CLONE_VM forks, copy_exec_state() allocates a new task_exec_state.
If that allocation fails, copy_process() reaches bad_fork_free and
free_task() calls ftrace_graph_exit_task(). Since the child still carries
the parent's ret_stack pointer, the unwind frees the parent's active
function graph return stack. The parent subsequently accesses freed memory
from function_graph_enter_regs().
KASAN reports:
[ 22.190920] ==================================================================
[ 22.195899] BUG: KASAN: slab-use-after-free in function_graph_enter_regs+0xa76/0xb90
[ 22.200747] Write of size 8 at addr ff110000054dc0a8 by task repro/1
[ 22.205134]
[ 22.210770] CPU: 0 UID: 0 PID: 1 Comm: repro Not tainted 7.2.0-07732-g9328b3b03bdc-dirty #3 PREEMPT(lazy)
[ 22.212576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 22.213750] Call Trace:
[ 22.215271] <TASK>
[ 22.216242] ? ftrace_stub_direct_tramp+0x10/0x10
[ 22.217774] dump_stack_lvl+0x4e/0x70
[ 22.220531] print_report+0x157/0x4b4
[ 22.223202] ? fixup_red_left+0x9/0x30
[ 22.224407] ? complete_report_info+0x83/0x110
[ 22.226679] ? function_graph_enter_regs+0xa76/0xb90
[ 22.228084] kasan_report+0xce/0x100
[ 22.230109] ? function_graph_enter_regs+0xa76/0xb90
[ 22.232860] ? stack_trace_save+0x4/0xd0
[ 22.234156] function_graph_enter_regs+0xa76/0xb90
[ 22.236090] ? kasan_save_stack+0x30/0x50
[ 22.237752] ? __pfx_function_graph_enter_regs+0x10/0x10
[ 22.238694] ? ring_buffer_lock_reserve+0x345/0xf80
[ 22.239628] ? stack_trace_save+0x4/0xd0
[ 22.242121] ? stack_trace_save+0x4/0xd0
[ 22.243588] ftrace_graph_func+0xda/0x160
[ 22.245362] ? ftrace_stub_direct_tramp+0x10/0x10
[ 22.246520] 0xffffffffa0000095
[ 22.250528] ? stack_trace_save+0x9/0xd0
[ 22.251757] ? ring_buffer_unlock_commit+0x11d/0x5c0
[ 22.253152] stack_trace_save+0x9/0xd0
[ 22.254264] kasan_save_stack+0x30/0x50
[ 22.273631] kasan_save_track+0x14/0x30
[ 22.276763] kasan_save_free_info+0x3b/0x70
[ 22.278296] __kasan_slab_free+0x43/0x70
[ 22.280157] kmem_cache_free+0xbf/0x3b0
[ 22.282963] ? ftrace_stub_direct_tramp+0x10/0x10
[ 22.284001] free_task+0xa2/0x160
[ 22.285699] ? ftrace_stub_direct_tramp+0x10/0x10
[ 22.286752] copy_process+0x2aae/0x7bc0
Initialize the child function graph state immediately after
dup_task_struct(), before the first fallible operation.
Cc: stable@vger.kernel.org
Fixes: 6b1c66c9cca9 ("exec_state: relocate dumpable information")
Reviewed-by: Bradley Morgan <include@grrlz.net>
Link: https://patch.msgid.link/20260822195321.962383-2-Jeremy.Jean@oss.cyber.gouv.fr
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Object kfunc calls embed metadata owned by the program BTF. A sleepable
program can remain active under Tasks Trace RCU after its last reference is
dropped, while program teardown releases the BTF through ordinary RCU. The
invocation can then dereference freed metadata in bpf_obj_new().
Move btf_put() into __bpf_prog_put_rcu(), the callback that frees the
program. When teardown is deferred, that callback runs after the program's
own grace period (Tasks Trace RCU for sleepable programs and ordinary RCU
otherwise), so the BTF outlives every active invocation. The non-deferred
path invokes the callback synchronously, so load-error cleanup stays direct.
Fixes: 958cf2e273f0 ("bpf: Introduce bpf_obj_new")
Signed-off-by: Sanghyun Park <sanghyun.park.cnu@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Sun Jian <sun.jian.kdev@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260901014829.3504342-2-sanghyun.park.cnu@gmail.com
|
|
destroy_user_event() destroys the event's fields before attempting to
remove the trace event call. If user_event_set_call_visible() fails,
e.g. because the event is still enabled and trace_remove_event_call()
returns -EBUSY, the event is left registered with an irreversibly
destroyed field list. Any subsequent interaction with the event then
operates on an empty field list while it is still fully visible in
tracefs.
Move the field destruction after the call removal, and splice the
field list back onto the event when the removal fails so the event
remains in a consistent state.
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260904115223.2976446-1-bsdhenrymartin@gmail.com
Fixes: 7f5a08c79df35 ("user_events: Add minimal support for trace_event into ftrace")
Signed-off-by: Henry Martin <bsdhenrymartin@gmail.com>
Reviewed-by: Beau Belgrave <beaub@linux.microsoft.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
put_pwq() bounces the release to pwq_release_worker to get out from
under pool->lock. workqueue_init() creates that kthread_worker well
after workqueue_init_early() has built the system workqueues, so a pwq
allocation failing partway through apply_wqattrs_prepare() unwinds
through apply_wqattrs_cleanup() and dereferences a NULL worker.
There is no way back from that. Running the release inline is not an
option, that is what the bounce exists for, and workqueue_init_early()
already BUG_ON()s a system workqueue it failed to allocate.
BUG_ON() the missing worker instead of oopsing inside
kthread_queue_work().
Suggested-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/all/144b3dc986135ebe34f671cdd02bd8aa@kernel.org/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
cgrp_dead_tasks and cgrp_dead_tasks_iwork are independent per-CPU variables
and are independently accessed via a this_cpu_ptr().
Having a custom struct with those two members makes it possible to have only
one per-CPU accessor and access the second member via an offset. This makes
the code a bit more compact and is a micro-optimization.
Besides that, it allows cgrp_dead_tasks_iwork_fn() to access the list
pointer via the passed iwork pointer. This not only eliminates the
this_cpu_ptr() but also makes it possible to be invoked on a different CPU.
This does not happen as of today but might if I can flush the irq_work items
from another CPU during CPU-hotplug events.
Create struct cgroup_dead with the per-CPU variables as members. Access the
struct with a per-CPU accessor and use container_of() in the irq_work
callback.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Michal Koutný <mkoutny@suse.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
cpu_pm_notify() walks the notifier chain lockless under only
rcu_read_lock(). That only works if a removed notifier block is not
freed until every concurrent walker is done with it.
The chain used to be an atomic_notifier, whose unregister ends in
synchronize_rcu(). Commit b2f6662ac08d ("PM: cpu: Make notifier chain
use a raw_spinlock_t") switched it over to a raw_notifier, and in
doing so replaced that with raw_notifier_chain_unregister(), which
does not synchronize. The grace period quietly went away, so a driver
that frees the memory holding its notifier block right after
cpu_pm_unregister_notifier() returns can race with a concurrent
walker:
cpu1 (idle exit) cpu2 (driver remove)
---------------- --------------------
cpu_pm_notify(CPU_PM_EXIT)
rcu_read_lock()
nb = rcu_dereference_raw(*nl)
cpu_pm_unregister_notifier(&od->nb)
unlink, no grace period
remove() frees od (devm)
nb->notifier_call(nb, ...)
/* use after free */
The rcu_read_lock() on cpu1 does not stop cpu2 from freeing the
block, and nothing else does. The read side is the idle exit path,
so this can hit on any system where a driver with a cpu_pm notifier
gets unbound.
Add the grace period back. It is only needed when a notifier was
actually removed, so wait on success and return -ENOENT without
waiting otherwise. The kerneldoc gets back the "may sleep" note that
the same commit dropped.
Fixes: b2f6662ac08d ("PM: cpu: Make notifier chain use a raw_spinlock_t")
Cc: All applicable <stable@vger.kernel.org>
Signed-off-by: Bradley Morgan <include@grrlz.net>
[ rjw: Subject tweak ]
Link: https://patch.msgid.link/20260813074343.26460-3-include@grrlz.net
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
|
|
The KUnit test suite turned insert_extent() and sort_idmaps() into global
symbols and declared them in include/linux/user_namespace.h. That header
is pulled in nearly everywhere through linux/cgroup.h and the btrfs
self-tests carry a static insert_extent() of their own, so any config
with both test suites enabled fails to build:
fs/btrfs/tests/inode-tests.c:16:13: error: conflicting types for 'insert_extent'
include/linux/user_namespace.h:212:12: note: previous declaration of 'insert_extent'
A global symbol needs a prefix. Rename them to
uid_gid_map_insert_extent() and uid_gid_map_sort(). It's the better name
anyway.
No functional changes.
Fixes: f6f1a2496ba5 ("userns: Add KUnit test suite for uid_gid_map")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202609110843.okbPe7GM-lkp@intel.com/
Closes: https://lore.kernel.org/oe-kbuild-all/202609110937.zHKPPXJk-lkp@intel.com/
Link: https://patch.msgid.link/20260911-suggerieren-suchaktion-fliehen-87b9cdf1d169@brauner
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Rearming a queued timer with nonzero slack can leave the timerqueue out
of order. remove_and_enqueue_same_base() checks the new soft expiry
against its neighbours' hard expiries, then stores the new hard expiry
in the node without requeueing it.
For example, with A at 10 and B at 20, rearming A at 11 with slack 30
passes the neighbour check but leaves A's hard expiry of 41 before B's
20. The same function also caches the soft expiry in base->expires_next
when updating or inserting the first timer, giving next-event selection
an earlier deadline than the queue head's hard expiry.
Set the timer expiry before handling the queue. Use its stored hard
expiry for the in-place ordering check and both updates to
base->expires_next.
The early update is safe because remove_and_enqueue_same_base() runs
with base->cpu_base->lock held. The lock keeps the queue stable while
hrtimer_can_update_in_place() checks the new expiry against both
neighbours. If the check fails, timerqueue_linked_del() removes the node
without comparing expiry values before it is reinserted.
Fixes: eddffab8282e3 ("hrtimer: Keep track of first expiring timer per clock base")
Fixes: 343f2f4dc5425 ("hrtimer: Try to modify timers in place")
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Assisted-by: LLM
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260910143442.2018-1-parri.andrea@gmail.com
|
|
When an access goes past the struct and the last member is a flexible
array, btf_struct_walk() folds the offset back into a single element with
(off - moff) % t->size, but never checks that the element type has a size.
BTF takes an empty struct, so this in program BTF
/* event could be empty */
struct event {
#ifdef HAVE_TIMESTAMP
__u64 ts;
#endif
};
struct batch {
int nr;
struct event events[];
};
divides by zero at prog load time. Getting there needs a PTR_TO_BTF_ID that
is not MEM_ALLOC, e.g. a plain read of a local kptr stashed in a map from a
sleepable program.
Oops: divide error: 0000 [#1] SMP KASAN PTI
RIP: 0010:btf_struct_walk+0x53f/0x1570
Call Trace:
<TASK>
btf_struct_access+0x42a/0xcd0
check_ptr_to_btf_access+0x4dc/0x1160
check_mem_access+0x3a45/0x8740
check_load_mem+0x36a/0xd10
do_check_common+0x3ef0/0xb210
bpf_check+0x6d3b/0x8580
bpf_prog_load+0xf7c/0x2720
__sys_bpf+0xa83/0x3690
__x64_sys_bpf+0xc7/0x150
x64_sys_call+0x1f3f/0x27e0
do_syscall_64+0xe5/0x610
entry_SYSCALL_64_after_hwframe+0x76/0x7e
</TASK>
Reject a zero-sized element type. The fixed array path in the same function
already bails out on the same thing:
btf_struct_walk()
...
/* skip empty array */
if (moff == mtrue_end)
continue;
msize /= total_nelems;
Fixes: 9c5f8a1008a1 ("bpf: Support variable length array in tracing programs")
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260910122316.186384-1-jiayuan.chen@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
