| Age | Commit message (Collapse) | Author |
|---|
|
bpf_loop() declares its nr_loops argument as ARG_ANYTHING. Privileged
programs may pass pointer values to such arguments, so check_func_arg()
lets a pointer-valued R1 reach the helper-specific checks.
Since commit bb124da69c47 ("bpf: keep track of max number of bpf_loop
callback iterations"), the verifier marks R1 precise and reads its upper
bound to limit callback simulation. Precision backtracking only accepts
scalar registers, so passing a pointer instead triggers the "backtracking
misuse" verifier warning. Kernels with panic_on_warn enabled subsequently
panic.
Introduce ARG_SCALAR for helper arguments that only accept scalar values
and use it for bpf_loop() nr_loops. Generic helper argument validation then
rejects pointers before loop inlining and precision processing.
Fixes: bb124da69c47 ("bpf: keep track of max number of bpf_loop callback iterations")
Reported-by: syzbot+7b47f87674e9a1569110@syzkaller.appspotmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260905014735.1452988-2-memxor@gmail.com
Closes: https://lore.kernel.org/bpf/6a9ad24c.b5d4176b.238c3e.0001.GAE@google.com/
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
|
|
dispatch_one() tests ops flags and bypass state against the root scheduler
in both places where it decides to keep running @prev: the early keep of a
@prev with slice left tests the root's bypass state, and the keep-last at
the end tests the root's SCX_OPS_ENQ_LAST and bypass state. Both are
properties of the scheduler @prev belongs to, and put_prev_task_scx(), which
acts on the outcome, reads them from that scheduler. When @prev belongs to a
sub-scheduler the two sides disagree.
The keep-last case is visible. The root set SCX_OPS_ENQ_LAST, so a lone
@prev of a sub-scheduler is not kept and is enqueued with SCX_ENQ_LAST to a
sub-scheduler that never opted in. This trips the WARN_ON_ONCE in
put_prev_task_scx() for the missing flag, and the sub-scheduler queues the
task like any other and triggers no follow-up scheduling event, which can
lead to stalls.
Test SCX_OPS_ENQ_LAST and bypass state on @prev's sched in both places and
charge SCX_EV_DISPATCH_KEEP_LAST to it. Read the sched at each decision, as
the dispatch in between can drop the rq lock.
Fixes: 88234b075c3f ("sched_ext: Introduce scx_task_sched[_rcu]()")
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
dispatch_one() uses the root scheduler for everything it does, including the
two decisions to keep running @prev, which are wrong when @prev belongs to a
sub-scheduler. The function has to deal with @prev's scheduler too. Rename
the root's local from sch to root_sch for clarity and to make room for it.
No functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Andrea Righi <arighi@nvidia.com>
|
|
When a preempt_count mismatch is detected in handle_softirqs(), the
error message currently prints the softirq action handler (h->action).
However, as Thomas Gleixner pointed out, the action handler for a given
softirq vector is always a known, fixed function (e.g., run_timer_softirq
for TIMER_SOFTIRQ). Printing it provides zero diagnostic value. The real
issue typically lies in the functions invoked *within* that handler, not
the handler entry point itself.
Therefore, as suggested by Sebastian Andrzej Siewior, remove h->action
from the print statement entirely to clean up the error message.
Suggested-by: Thomas Gleixner <tglx@linutronix.de>
Suggested-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Enlin Mu <enlin.mu@linux.dev>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Link: https://patch.msgid.link/20260825011509.8505-1-enlin.mu@linux.dev
|
|
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
|
|
Use newly added mark_arg_precision() helper in check_mem_size_reg().
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-10-0f5a360ff15d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Stop verification if mark_chain_precision() fails when called from
loop_flag_is_zero(). No functional change intended for the paths where
backtracking succeeds.
Fixes: 1ade23711971 ("bpf: Inline calls to bpf_loop when callback is known")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-9-0f5a360ff15d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
btf_check_func_arg_match() accepts a NULL register for an
ARG_PTR_TO_BTF_ID argument tagged __arg_nullable and skips
check_reg_type() and check_func_arg_reg_off() without marking the
register precise. Hence a checkpoint created on such a path would
prune against arbitrary scalar value.
Fixes: e2b3c4ff5d18 ("bpf: add __arg_trusted global func arg tag")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-7-0f5a360ff15d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_kfunc_arg() allows bpf_register_is_null() for nullable arguments
w/o marking the underlying scalar register precise. Hence a checkpoint
created on such a path would prune against arbitrary scalar value.
Fixes: 3bda08b63670 ("bpf: Allow NULL buffers in bpf_dynptr_slice(_rw)")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-5-0f5a360ff15d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_mem_reg() allows bpf_register_is_null() for nullable arguments
w/o marking the underlying scalar register precise. Hence a checkpoint
created on such a path would prune against arbitrary scalar value.
The argument may live on the stack rather than in a register when a
call has more than MAX_BPF_FUNC_REG_ARGS arguments, hence the new
mark_arg_precision() helper.
Fixes: e5069b9c23b3 ("bpf: Support pointers in global func args")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-3-0f5a360ff15d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_func_arg() allows bpf_register_is_null() for nullable arguments
w/o marking the underlying scalar register precise. Hence a checkpoint
created on such a path would prune against arbitrary scalar value.
check_helper_call() enforces second parameter of the
bpf_get_local_storage() to be zero, w/o marking the underlying scalar
register precise. Hence a checkpoint created on such a path would
prune against arbitrary scalar value.
Grouping these two into one patch, as they share the same fixes tag.
Fixes: b5dc0163d8fd ("bpf: precise scalar_value tracking")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-1-0f5a360ff15d@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Bottom-Half (BH) workqueues execute work items in softirq context.
To prevent softirqs from starving user and kernel threads, bh_worker()
enforces execution limits (i.e., BH_WORKER_JIFFIES and BH_WORKER_RESTARTS).
When keep_working() is still true but either the time slice or restart
count is exhausted, bh_worker() yields execution and re-raises the
softirq via kick_bh_pool().
Currently, there is no observability into when a BH worker hits these
limits and is forced to yield.
Add the workqueue_bh_budget_yield tracepoint, emitted when bh_worker() exits
the processing loop with pending work items remaining. It records, the
worker pool ID, pool CPU, number of loop restarts consumed, a boolean flag
indicating whether the yield was due to a time slice timeout, and a boolean
flag indicating whether this is a high-priority BH pool.
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
When a worker pool fails to create a new worker thread (e.g., typically
under severe memory pressure where memory reclaim paths depend on pending
work items), send_mayday() signals distress to the workqueue's rescuer
thread. The rescuer then takes over processing the pending work items via
assign_rescuer_work().
While pwq->stats[PWQ_STAT_MAYDAY] and pwq->stats[PWQ_STAT_RESCUED] track
these occurrences cumulatively, there is currently no event-driven
mechanism to observe exactly when mayday distress occurs or which work
items require rescue.
Add two new tracepoints namely workqueue_mayday and workqueue_rescued to
make distress and rescuer execution easily observable (e.g., via Ftrace
or eBPF).
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
Instead of hard coding 30 for the number of bits used for the static
buffer ids in two places, create a macro. This way if it changes in the
future, it will change in all the locations that use it.
Link: https://patch.msgid.link/20260904151641.17eae0aa@gandalf.local.home
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
When a concurrency-managed per-CPU work item runs continuously without
sleeping for longer than wq_cpu_intensive_thresh_us, wq_worker_tick() marks
the worker as WORKER_CPU_INTENSIVE and kicks it out of concurrency
management so that pending work items on the pool are not starved.
While CONFIG_WQ_CPU_INTENSIVE_REPORT logs rate-limited warnings and
pwq->stats[PWQ_STAT_CPU_INTENSIVE] maintains a cumulative counter, there is
currently no tracepoint emitted at the moment of this transition.
Therefore, add the workqueue_cpu_intensive tracepoint, recording the
work_struct pointer and callback function pointer, workqueue name,
executing CPU, and the runtime duration consumed in microseconds.
This enables eBPF profilers, bpftrace, and Ftrace to immediately detect and
attribute CPU-hogging work items in real time.
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
The comment about returning an error if the read fails on the first
iteration is slightly incorrect. It makes it sound like the only reason it
could fail on a later iteration is if the subbuf order changed. That is
incorrect, it could also fail if the length passed in was not a multiple
of the subbuf size. Fix the comment.
Link: https://lore.kernel.org/all/20260904143527.40e73d36@gandalf.local.home/
Link: https://patch.msgid.link/20260904144902.506862a1@gandalf.local.home
Fixes: dae8dda341d2 ("tracing: Fix subbuf resize races with trace_pipe_raw readers")
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Although ring_buffer_per_cpu::nr_pages is defined as unsigned long, it
is capped to 32-bits in a few places, limiting the operations possible
on a very large buffer. Use `unsigned long` where appropriate and
prevent truncation of values using nr_pages (or nr_subbufs).
While at it, subbuf_size must be at least `unsigned int`.
Note that persistent, remote and user-mapped ring buffers are capping
the number of pages to 30 bits already, making "int" safe in many
places.
Link: https://patch.msgid.link/20260904164450.1345852-5-vdonnefort@google.com
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Static ring buffers (i.e. persistent, user-mapped and remote) rely on
the bpage::id field. The number of pages for those ring buffers must fit
into that variable. Enforce this limit on ring buffer creation or
user-mapping.
While at it, prevent nr_pages underflow when allocating a persistent
buffer.
Link: https://patch.msgid.link/20260904164450.1345852-4-vdonnefort@google.com
Fixes: be68d63a139b ("ring-buffer: Add ring_buffer_alloc_range()")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Concurrent subbuffer resizes may crash trace_pipe_raw readers or leak
uninitialized memory to userspace due to stale size values.
Modify ring_buffer_alloc_read_page() to handle the resizing of an
existing buffer_data_read_page if necessary and add a new
ring_buffer_read_page_size(). This new function enables ring-buffer
buffer_data_read_page users to not call the racy
ring_buffer_subbuf_size_get(). This makes the spare_size member of
ftrace_buffer_info redundant.
Finally, handle buffer_data_read_page/reader_page order discrepancy in
ring_buffer_read_page(). On a mismatch simply copy manually the data to
the buffer_data_read_page.
Link: https://lore.kernel.org/all/20260817140812.2C7D41F00A3A@smtp.kernel.org/
Link: https://patch.msgid.link/20260904164450.1345852-3-vdonnefort@google.com
Fixes: bce761d75745 ("ring-buffer: Read and write to ring buffers with custom sub buffer size")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
Callback frame constructors initialize map-typed argument registers with
__mark_reg_known_zero() and then restore map_ptr. This clears map_uid,
which is the only field distinguishing inner maps that share an
inner_map_meta template.
When a timer callback invokes bpf_for_each_map_elem() on a second inner
map, both the saved first map and the second map value can reach the nested
callback as the same template with map_uid zero. bpf_timer_init() then
accepts pairing the timer from the second map with the first map.
The runtime records the first map in the timer without taking a reference.
Freeing that map does not find the timer stored in the second map, so a
later timer callback dereferences the freed map.
Copy map_uid from the same caller register as map_ptr when constructing
for-each, timer/workqueue, and task-work callback arguments. The existing
identity check can then reject mismatched inner maps while allowing a
callback value to be paired with its actual map.
Fixes: 3e8ce29850f1 ("bpf: Prevent pointer mismatch in bpf_timer_init.")
Fixes: 69c087ba6225 ("bpf: Add bpf_for_each_map_elem() helper")
Fixes: 5c8fd7e2b5b0 ("bpf: bpf task work plumbing")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904104203.345917-8-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_map_kptr_access() permits a scalar store into an untrusted kptr
field only when the register is known to contain zero. Unlike other
verifier checks whose outcome depends on a scalar value, it does not mark
that register precise.
A state checkpoint reached with an imprecise zero can therefore prune a
second path that reaches the store with an arbitrary nonzero scalar. The
program can write attacker-controlled bits into the kptr field and load
them back as a PTR_TO_BTF_ID.
Call mark_chain_precision() before accepting a known-zero register. This
forces state equivalence to compare its scalar range and makes the verifier
visit and reject a path carrying a nonzero value.
Fixes: 61df10c7799e ("bpf: Allow storing unreferenced kptr in map")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260904104203.345917-6-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
rhtab_map_update_existing() and rhtab_delete_elem() call
bpf_obj_free_fields() when replacing or deleting a value. These map
operations can run from BPF programs in NMI context, where releasing a
referenced kptr or another complex field is not generally safe.
Array and hash maps avoid that problem by cancelling only the asynchronous
fields which can be stopped safely in the caller context. Other ownership
state remains attached to the allocation until its memory allocator
destructor performs the final cleanup.
Use bpf_obj_cancel_fields() for the corresponding rhtab paths as well. This
cancels timers, workqueues, and task work while allowing rhtab_mem_dtor() to
release referenced kptrs when the allocation is eventually destroyed.
Fixes: 6905f8601298 ("bpf: Allow special fields in resizable hashtab")
Signed-off-by: Nuoqi Gui <gnq25@mails.tsinghua.edu.cn>
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
[ kkd: Rebased, used direct helper calls, and rewrote the commit log ]
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904104203.345917-4-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
rhtab_map_update_elem() initializes special fields after obtaining an
element from bpf_mem_cache_alloc(). The allocator can return a fresh,
zeroed unit, or recycle one from its RCU-pending lists before the
registered destructor has run.
A BPF program can retain a map-value pointer after deleting its element
and initialize and arm a timer through that pointer. If the deleted unit
is recycled, check_and_init_map_value() clears the only pointer to the
timer. Neither a later deletion nor rhtab_mem_dtor() can then cancel it,
and the callback can run with its key and value pointing into freed memory.
Do not reinitialize special fields on insertion. Fresh allocator units are
already zeroed. For recycled units, the special fields are ownership state
that must remain visible to the eventual destructor. copy_map_value()
already skips those fields, matching the non-preallocated hash-map path and
the lifecycle established by commit 275c30bcee66 ("bpf: Don't reinit map
value in prealloc_lru_pop").
Fixes: 6905f8601298 ("bpf: Allow special fields in resizable hashtab")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Yuan Chen <chenyuan@kylinos.cn>
[ kkd: Split out the fix and rewrote the commit log ]
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904104203.345917-2-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Add a new Kconfig option CONFIG_BPF_JIT_KASAN that automatically enables
generic KASAN (Kernel Address SANitizer) memory access checks for
JIT-compiled BPF programs as well, when both KASAN (and more
specifically, generic KASAN with KASAN_VMALLOC) and JIT compiler are
enabled. This new Kconfig is not a user selectable one: it is
automatically enabled if KASAN is enabled on a compatible platform. When
enabled, the JIT compiler will emit shadow memory checks before memory
loads and stores to detect use-after-free or out-of-bounds accesses at
runtime. The option is gated behind HAVE_EBPF_JIT_KASAN, as it needs
proper arch-specific implementation.
Acked-by: Andrey Konovalov <andreyknvl@gmail.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Signed-off-by: Alexis Lothoré (eBPF Foundation) <alexis.lothore@bootlin.com>
Link: https://lore.kernel.org/r/20260903-kasan-v9-2-2407fe99255a@bootlin.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
In order to prepare to emit KASAN checks in JITed programs, JIT
compilers need to be aware about whether some load/store instructions
are targeting the bpf program stack, as those should not be monitored
(we already have guard pages for that, and it is difficult anyway to
correctly monitor any kind of data passed on stack).
To support this need, make the BPF verifier mark the instructions
depending on whether they could access or not memory other than stack.
As different states in the verifier could lead to different memory types
for the same access, just marking an instruction as accessing stack only
is not enough (it could be some other memory type in another verifier
state), so the algorithm rather sets by default any load/store
instruction as stack only, and if _any_ state leads to any memory access
type other than PTR_TO_STACK, it overrides this setting. It also takes
care about shifting back the instruction marking in adjust_insn_aux_data
if the verifier patches instructions. However, if the verifier generates
new BPF_ST/BPF_STX/BPF_LDX while patching some instructions, those new
ones are systematically marked as non-stack-accessing: this may
over-instrument a few memory accessing instructions, but it allows
making sure that we will not miss accidentally any.
Signed-off-by: Alexis Lothoré (eBPF Foundation) <alexis.lothore@bootlin.com>
Link: https://lore.kernel.org/r/20260903-kasan-v9-1-2407fe99255a@bootlin.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Since commit e645535a954a ("tracing: Add option to use memmapped
memory for trace boot instance") changed trace_array_get_by_name() to
trace_array_create_systems(), enable_instances() does not reuse the
same name instance. Therefore, if an administrator mistakenly specifies
multiple `trace_instance=` options with duplicate names, all are
created but only the first is accessible via tracefs.
Check whether an instance with the same name already exists before
creating a new one, and reject duplicates with a warning.
Link: https://patch.msgid.link/178847790399.283263.5313150997200138426.stgit@devnote2
Fixes: e645535a954a ("tracing: Add option to use memmapped memory for trace boot instance")
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull probes fixes from Masami Hiramatsu:
- Protect kprobe_blacklist with RCU
RCU-protect kprobe_blacklist and use kfree_rcu() to prevent UAF races
during module unloading and enable safe atomic lookups.
- Fix multi-probe field use-after-free
Duplicate field and type strings on trace_probe_event to prevent UAF
when freeing primary probe
- Fix probe BTF member lookup:
Check the containing inner struct/union kflag when resolving
anonymous members to ensure correct bitfield offset calculation
Prevent unnamed bitfields from being pushed to anon_stack in
btf_find_struct_member(), avoiding false lookup errors
Fix code block indentation in get_bitoffset_of_field()
- uprobes error pointer safety
Guard free_trace_uprobe() with IS_ERR_OR_NULL() to avoid crashing
during automatic cleanup when an error pointer is returned
* tag 'probes-fixes-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
kprobes: Protect kprobe_blacklist with RCU
tracing/probes: Fix use-after-free on field name/type of events with multiple probes
tracing/probes: Fix code indent in get_bitoffset_of_field()
tracing/probes: Fix BTF kflag check for anonymous struct member access
tracing/probes: Fix anon_stack check for unnamed bitfields in btf_find_struct_member
uprobes: guard trace cleanup against error pointers
|
|
A local kptr load of an object containing a graph node is marked MEM_RCU
and NON_OWN_REF while protected by RCU. When the last RCU read-side critical
section ends, invalidate_rcu_protected_refs() removes MEM_RCU and marks the
pointer PTR_UNTRUSTED, but leaves NON_OWN_REF set.
The stale flag lets graph kfunc argument checks continue treating the
pointer as a live borrowed reference. In particular, bpf_rbtree_remove()
can accept a pointer after its protection ended and return it as a new
owning reference, even though the object may already have been freed.
Clear NON_OWN_REF when an RCU-protected pointer is demoted. A spin lock also
provides implicit RCU protection, so invalidate non-owning references before
demoting RCU-protected pointers when releasing the lock. Otherwise the
demotion would clear the flag before invalidate_non_owning_refs() can find
and invalidate those aliases.
The demoted pointer remains available for fault-protected reads. Exempt such
reads from the allocated-object reference-state assertion; writes through a
fault-prone pointer are already rejected, and bpf_may_fault_on_deref() makes
the surviving loads use BPF_PROBE_MEM.
Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904084325.52250-6-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
bpf_refcount_acquire() is fallible for a borrowed reference because the
object may have reached a zero refcount. The verifier therefore keeps
KF_RET_NULL on the return value unless the argument is an owning reference.
An RCU-protected load of a local kptr is marked MEM_ALLOC, but it only
receives NON_OWN_REF when the pointee contains a graph node. A refcounted
object without a graph node consequently looks like an owning reference
even though the loaded register has no acquired reference state. If the
program drops the last real reference while remaining in the RCU critical
section, refcount_inc_not_zero() returns NULL while the verifier treats the
result as non-NULL.
Only classify the argument as owning when it is backed by a verifier-tracked
reference. This retains the non-NULL return for pointers from bpf_obj_new(),
bpf_kptr_xchg(), or an earlier successful acquisition, while requiring a
NULL check for borrowed RCU kptrs.
Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Ning Ding <dingning04@gmail.com>
[ kkd: Rewrote commit log ]
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904084325.52250-4-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
map_kptr_match_type() treats perm_flags as the set of register type flags
that a kptr field permits. Adding MEM_PERCPU to that set for
BPF_KPTR_PERCPU does not require the source register to carry it, however.
The subset test consequently accepts both a plain bpf_obj_new() allocation
and a referenced kernel pointer into a __percpu_kptr map field.
Loads from the field are always marked MEM_PERCPU. Consumers then treat the
stored value as the cookie returned by bpf_percpu_obj_new(): per-CPU pointer
helpers relocate it, and map teardown selects the per-CPU free path. A plain
allocation can therefore provide an arbitrary kernel read/write, while a
kernel pointer can be relocated into an invalid address or sent through a
missing destructor.
Require the source MEM_PERCPU flag to match the destination field kind.
This preserves valid bpf_percpu_obj_new() stores and rejects both the
program-BTF and kernel-BTF variants.
Fixes: 36d8bdf75a93 ("bpf: Add alloc/xchg/direct_access support for local percpu kptr")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260904084325.52250-2-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
check_cond_jmp_op() accepts "if rA <op> rB" as a NULL check for a
nullable pointer rA when rB is a scalar known to be zero,
lifts PTR_MAYBE_NULL from rA in the corresponding branch and does not
mark rB precise. Consider the following program:
r0 = bpf_get_prandom_u32();
r6 = 1; /* the r6 == 0 path is explored first */
if (r0 == 0) goto 1f;
r6 = 0;
1:
r0 = bpf_map_lookup_elem(map, &0); /* absent, NULL at runtime */
if (r0 == r6) goto 2f; /* taken as a NULL check for r0 */
*(u8 *)(r0 + 0); /* verifier: map value; runtime: zero */
2:
return 0;
The r6 == 0 path is explored first and the dereference is accepted.
The r6 == 1 path is pruned at the checkpoint recorded for (1),
so the comparison is never verified with a non-zero r6. At runtime a
failed lookup returns NULL, NULL != 1 takes the non-NULL edge and the
program dereferences a pointer that is zero.
Fixes: 2f4cb53eed44 ("bpf: detect non null pointer with register operand in JEQ/JNE.")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260904083325.2083493-7-eddyz87@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
Consider the following program:
r1 = map_value; /* low 32 bits are zero at runtime */
r6 = 0xdead000000000000;
if w1 != 0 goto l1;
l0: r1 += r6;
r2 = *(u64 *)(r1 + 0);
exit;
l1: r6 = 0;
goto l0;
At the moment is_branch_taken() reports the jump as always taken,
because it does not distinguish between BPF_JMP and BPF_JMP32
comparisons when processing 'if w1 != 0 ...'.
Fixes: cac616db39c2 ("bpf: Verifier track null pointer branch_taken with JNE and JEQ")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260904083325.2083493-5-eddyz87@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
check_cond_jmp_op() copies the compared registers into
env->{false,true}_reg{1,2} before collect_linked_regs() runs and copies
those snapshots back into both branch states afterwards.
collect_linked_regs() records at most LINKED_REGS_MAX members of a
linked registers group in the jump history and calls clear_scalar_id()
for every member that does not fit. The compared register is not exempt
from that.
As a consequence, sync_linked_regs() might adjust ranges for more
registers than bpf_bt_sync_linked_regs() can propagate precision to.
Collect the linked registers before the snapshots are taken instead.
This might lead to some unnecessary clear_scalar_id's, but from
previous testing situations with many linked registers are
extremely rare.
Fixes: ec1d77cb0ee9 ("bpf: Use bpf_verifier_env buffers for reg_set_min_max")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260904083325.2083493-3-eddyz87@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
reg_not_null() decides that a register holds a non-NULL value by
looking at its type alone. For pointer types that allow arithmetic the
type only guarantees a non-NULL base, in case of an unbound offset
the runtime offset value might still add up to NULL.
Consider the followng program:
r6 = bpf_map_lookup_elem(map, &0); /* present */
if (r6 == 0) return 0;
r7 = bpf_map_lookup_elem(map, &1); /* absent, NULL at runtime */
r8 = r7;
r8 -= r6; /* pointer - pointer: unknown scalar, -r6 */
r8 <<= 1;
r8 >>= 1; /* any non-negative offset is accepted by */
/* check_reg_sane_offset_ptr() */
r6 += r8; /* verifier: map value; runtime: zero */
if (r7 != r6) return 0;
*(u8 *)(r7 + 0); /* r7 is inferred non-NULL, both are zero */
At runtime both registers are zero, the comparison is true and the
load faults with NULL pointer dereference.
Require the offset to be within +-BPF_MAX_VAR_OFF in reg_not_null().
Fixes: cac616db39c2 ("bpf: Verifier track null pointer branch_taken with JNE and JEQ")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260904083325.2083493-1-eddyz87@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
|
|
The sole caller, create_new_namespaces(), always passes a valid uts
namespace pointer from the task's nsproxy. The other copy_*_ns()
helpers, e.g. copy_time_ns() and copy_pid_ns(), don't perform such a
check either. Drop the BUG_ON() for consistency with them.
Signed-off-by: Shaojie Sun <sunshaojie@kylinos.cn>
Link: https://patch.msgid.link/20260903064828.126643-3-sunshaojie@kylinos.cn
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Commit 0a36bad01731 ("release_task: kill the no longer needed
get/put_pid(thread_pid)") removed the reference around proc_flush_pid().
It assumed that free_pids(post.pids) at the end of release_task() would
keep thread_pid alive until then.
That assumption is wrong. __change_pid() only records a detached PID in
post.pids when pid_has_task() is false for every PIDTYPE. If another task
still uses the exiting task's PID as its process group or session ID,
__unhash_process() removes the exiting task's PIDTYPE_PID link but leaves
the PID out of post.pids. release_task() therefore holds no reference to
it after dropping tasklist_lock.
The other task can then remove the remaining PIDTYPE links. Its
free_pids() call schedules delayed_put_pid(), and the RCU callback can free
the PID before the first release_task() reaches proc_flush_pid().
An unprivileged reproducer races wait4(-1) against setsid() to trigger this
ordering. Three of three fresh v7.2 KASAN boots reported:
BUG: KASAN: slab-use-after-free in
proc_invalidate_siblings_dcache+0x3e2/0x3f0
Read of size 8 by task h7_pid_reaper/1921
Call Trace:
proc_invalidate_siblings_dcache
release_task
wait_consider_task
__do_wait
do_wait
kernel_wait4
Freed by task 0:
kmem_cache_free
put_pid
delayed_put_pid
rcu_core
Last potentially related work creation:
__call_rcu_common
free_pids
ksys_setsid
KASAN identified a 144-byte object from the pid cache and located the bad
read 80 bytes into the freed object, matching pid->inodes. With an
explicit reference, three of three fresh boots completed without a KASAN
report. The concurrent RCU callback dropped its reference while
proc_flush_pid() was protected, and the balancing put_pid() performed the
final free afterward.
Take a reference before __unhash_process() clears p->thread_pid and release
it after proc_flush_pid() completes.
A tested source reproducer is available privately on request. No
controlled read or write, information leak, or privilege escalation is
claimed. The mainline patch applies directly to v6.19.y and newer;
v6.16.y through v6.18.y need a context-adjusted backport.
Fixes: 0a36bad01731 ("release_task: kill the no longer needed get/put_pid(thread_pid)")
Reported-by: syzbot+0aee5e8066eddbbe7397@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=0aee5e8066eddbbe7397
Reported-by: syzbot+e8b3520b53e78e90034e@syzkaller.appspotmail.com
Link: https://syzkaller.appspot.com/bug?extid=e8b3520b53e78e90034e
Cc: stable@vger.kernel.org # see patch description, needs adjustments for 6.16.y-6.18.y
Signed-off-by: Daehyeon Ko <4ncienth@gmail.com>
Link: https://patch.msgid.link/20260831001221.3755948-1-4ncienth@gmail.com
Acked-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
The kobjects of a livepatch are released via kobject_put(). When
CONFIG_DEBUG_KOBJECT_RELEASE is enabled, kobject_put() does not release
the kobject synchronously but schedules a delayed release with a random
delay of up to 4 seconds (see kobject_release() in lib/kobject.c).
klp_free_patch_finish() only waits for the release of the patch kobject:
klp_free_patch_finish():
kobject_put(&patch->kobj);
wait_for_completion(&patch->finish);
patch->finish is completed by the patch kobject's release callback. If
the patch kobject was never added to sysfs, or if some child kobjects
were initialized but never added to sysfs (e.g. when klp_enable_patch()
fails after klp_init_patch_early()), those un-added children do not hold
a reference on the patch kobject. kobject_add() is what takes the parent
reference, so the patch kobject can be released first, completing
patch->finish while the child releases are still pending.
The caller then unloads the livepatch module, which destroys the static
klp_object and klp_func structures. The delayed child release callbacks
later access this freed memory, causing a use-after-free.
Fix it by making every child kobject hold an explicit reference on its
parent from the moment the object is initialized: klp_init_object_early()
takes a reference on the patch kobject and klp_init_func_early() takes a
reference on the object kobject. Unlike the reference taken by
kobject_add(), these references also exist for objects that are never
added to sysfs, and the release callbacks drop them unconditionally.
This guarantees the patch kobject is released only after all child
kobjects have been released, so patch->finish cannot be completed before
the static structures are safe to free.
Because kobj->parent is set only by kobject_add(), add explicit
back-pointers, obj->patch and func->obj, so the release callbacks can
find the parent. Dynamic objects and nop functions are freed by their
release callbacks; save the parent pointer before freeing and drop the
parent reference afterwards.
Reported-by: sashiko-bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260809094046.50ED31F000E9@smtp.kernel.org/
Suggested-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Tested-by: Petr Mladek <pmladek@suse.com>
Link: https://patch.msgid.link/20260830054857.64758-2-laoar.shao@gmail.com
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
When a preempt_count mismatch is detected in handle_softirqs(), the
error message currently prints the softirq action handler (h->action).
However, as Thomas Gleixner pointed out, the action handler for a given
softirq vector is always a known, fixed function (e.g., run_timer_softirq
for TIMER_SOFTIRQ). Printing it provides zero diagnostic value. The real
issue typically lies in the functions invoked *within* that handler, not
the handler entry point itself.
Therefore, as suggested by Sebastian Andrzej Siewior, remove h->action
from the print statement entirely to clean up the error message.
Suggested-by: Thomas Gleixner <tglx@linutronix.de>
Suggested-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Enlin Mu <enlin.mu@unisoc.com>
Signed-off-by: Enlin Mu <enlin.mu@linux.dev>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://patch.msgid.link/20260825011509.8505-1-enlin.mu@linux.dev
|
|
irq_affinity_setup() ignores the return value of cpulist_parse(). This
can result in using a "half parsed" cpulist. E.g. on a system with 8
cores, the parameter "irqaffinity=0,1,6-10,4" would result in using only
CPU 0 and 1, since the parsing fails at 6-10. Other boot parameters
which take a cpulist (e.g. isolcpus=, workqueue.unbound_cpus=) already
handle this by falling back to a default.
Check the return value of cpulist_parse() and let
init_irq_default_affinity() set the default in early_irq_init()
later, if parsing fails.
Signed-off-by: Lukas Beckmann <lbckmnn@mailbox.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Radu Rendec <radu@rendec.net>
Link: https://patch.msgid.link/20260822160502.4665-1-lbckmnn@mailbox.org
|
|
On PREEMPT_RT, FUTEX_CMP_REQUEUE_PI can trigger a KASAN report
(slab-out-of-bounds) in futex_requeue_pi_complete() invocation of
rcuwait_wake_up().
The futex_q used by futex_wait_requeue_pi() is allocated on the waiter's
stack. An early wakeup can race with a PI requeue as follows:
waiter requeue task
------ ------------
futex_wait_requeue_pi()
futex_do_wait()
schedule()
futex_requeue
futex_proxy_trylock_atomic()
futex_requeue_pi_prepare()
Q_REQUEUE_PI_NONE -> Q_REQUEUE_PI_IN_PROGRESS
* timeout/ signal wakes waiter *
futex_requeue_pi_wakeup_sync()
Q_REQUEUE_PI_IN_PROGRESS -> Q_REQUEUE_PI_WAIT
requeue_pi_wake_futex
futex_requeue_pi_complete()
cmpxchg Q_REQUEUE_PI_WAIT -> Q_REQUEUE_PI_LOCKED
rcuwait_wait_event()
if (atomic_read(&q->requeue_state) != Q_REQUEUE_PI_WAIT)
break /* no schedule() */
/* q.pi_state->owner == current */
futex_private_hash_put()
/* return from syscall */
rcuwait_wake_up(&q->requeue_wait)
/* q is gone */
futex_requeue_pi_complete() publishes Q_REQUEUE_PI_LOCKED before
calling rcuwait_wake_up(). The waiter observes this state in
rcuwait_wait_event() before invoking schedule() in rcuwait_wait_event().
Here, the waiter is free leave the syscall before requeue task can
complete the wake.
To address this race skip rcuwait_wake_up() in the Q_REQUEUE_PI_LOCKED
case.
This state is only published by requeue_pi_wake_futex(), which saves
q->task before futex_requeue_pi_complete() and wakes the waiter via
wake_up_state().
This wake is intended to wake the waiter from its futex_do_wait() sleep.
If the waiter is still sleeping there, it can not get into the
Q_REQUEUE_PI_WAIT state (and require this removed wake).
Should the waiter be woken up from futex_do_wait() by other means (as in
this example) and sleep in futex_requeue_pi_wakeup_sync() then the
wake_up_state() from requeue_pi_wake_futex() will wake it, too.
Should the waiter task terminate before wake_up_state() had a chance to
wake the task then the task pointer does not become invalid because the
futex_hash_bucket::lock is held and the task pointer is RCU protected.
[bigeasy: Updated comment and commit message]
Fixes: 07d91ef510fb1 ("futex: Prevent requeue_pi() lock nesting issue on RT")
Signed-off-by: Yao Kai <yaokai34@huawei.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260901135453.3121948-3-bigeasy@linutronix.de
|
|
There is rt_mutex_{pre|post}_schedule() around
rt_mutex_wait_proxy_lock() to ensure that sched_submit_work()/
sched_update_worker() is invoked before we schedule out and block on
rt_mutex while waiting for it become available.
The reason is that blocking on rt_mutex assigns a pi_waiter for the PI
chain and sched_submit_work() will also assign a pi_waiter if it blocks
on lock but a this point we already have a waiter assigned.
We can't skip sched_submit_work() entirely because I/O relies on the
fact that I/O queue is flushed while it blocks on a sleeping lock.
Therefore sched_submit_work() is moved before we block on the lock.
Sleeping lock in this context means mutex or rw_semaphore not spinlock_t
on PREEMPT_RT. Because the mutex abstraction on PREEMPT_RT uses the same
abstraction as the futex proxy lock, the futex code ended up using
rt_mutex_{pre|post}_schedule(), too.
Using it is/ was just to keep the task_struct::sched_rt_mutex assertion
happy. Futex proxy lock is used only in the syscall context of a task.
At this point it never got any I/O that needs to be flushed and it can't
be a workqueue that needs to notify that it will be scheduled out.
Therefore sched_submit_work() does nothing here.
By mistake futex_wait_requeue_pi() -> rt_mutex_wait_proxy_lock() did not
get the rt_mutex_{pre|post}_schedule() annotation. This was not noticed
because in this callchain the lock is (usually) not contended and so
rt_mutex_slowlock_block() does not schedule, triggering the assert.
Adding rt_mutex_pre_schedule() here looks wrong (as noted by PeterZ)
because at this point there is a pi_waiter recorded and invoking
sched_submit_work() with a possible lock contention would be wrong.
Add rt_mutex_futex_{pre|post}_schedule() which toggles the
sched_rt_mutex assert and does not involve sched_submit_work(). Add
asserts here to ensure that sched_submit_work() would do nothing. Use it
only in futex proxy lock case which is rt_mutex_wait_proxy_lock().
Remove it from futex_lock_pi().
Fixes: d14f9e930b90 ("locking/rtmutex: Use rt_mutex specific scheduler helpers")
Reported-by: Yao Kai <yaokai34@huawei.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260901135453.3121948-2-bigeasy@linutronix.de
Closes: https://lore.kernel.org/all/20260717084922.4153317-2-yaokai34@huawei.com
|
|
Provide devm_{atomic,blocking}_notifier_chain_register()
Base for merging into subsystem trees that want to benefit from the devm
variants of notifier_chain_register functions created by Eliav Farber
for the 7.4 merge window.
|
|
check_ld_abs() models a failed BPF_LD_ABS or BPF_LD_IND in a
subprogram as an implicit return with R0 set to zero. It calls
prepare_func_exit() to explore this synthesized path.
When the load is reached directly from a synchronous callback,
prepare_func_exit() enforces the callback return contract and marks R0
precise. R0 is not derived from a real instruction on this path, so
precision backtracking reaches the callback call with R0 still requested
and triggers the "callback unexpected regs" verifier bug. A privileged
program loader can therefore cause a verifier warning and an -EFAULT
BPF_PROG_LOAD.
These legacy packet-load instructions are deprecated. Reject them from
callbacks rather than complicating their implicit-return model. Check all
active frames before constructing the implicit return so nested static
subprograms cannot hide the callback context.
Global functions are verified independently with a fresh frame zero, so
an active-frame check cannot identify a global function called from a
callback. Also check the complete subprogram call graph during stack-depth
validation and reject a function containing a legacy load when any caller
is a callback. This covers global and static descendants without making
has_ld_abs transitive, preserving its per-function BTF return-type check.
Ordinary uses outside callbacks remain supported.
Fixes: ee861486e377 ("bpf: Fix ld_{abs,ind} failure path analysis in subprogs")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://lore.kernel.org/bpf/20260903152147.C0E241F00A3A@smtp.kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20260903214758.2727663-8-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
The faultable variants of bpf_get_stack() and bpf_get_task_stack() pass
may_fault=true into the common stack collection code. Resolving user-space
build IDs may then call build_id_parse_file() and block on filesystem
reads.
Neither helper prototype sets might_sleep. Since prototype selection uses
the sleepability of the whole program, the verifier can still allow these
helpers from a non-sleepable region within that program, such as an
explicit RCU or preemption-disabled region. The task-stack helper can also
be called from a non-sleepable timer callback of a sleepable program.
Mark both faultable prototypes as sleepable. The existing helper context
check then rejects these calls while continuing to allow them in genuinely
sleepable contexts.
Fixes: d4dd9775ec24 ("bpf: wire up sleepable bpf_get_stack() and bpf_get_task_stack() helpers")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260903214758.2727663-6-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
When bpf_btf_find_by_name_kind() finds a type in module BTF, it
returns a new BTF object fd through __btf_new_fd(). This reaches
anon_inode_getfd(), which can sleep while allocating or expanding the
current task fd table.
The helper prototype does not set might_sleep, so the verifier allows
the helper in non-sleepable contexts such as BPF timer callbacks. The
fd allocation can then sleep in softirq context and install the fd into
the interrupted task.
Mark the helper as sleepable. This preserves calls from the main body
of a sleepable syscall program while rejecting calls from its
non-sleepable regions.
Fixes: 3d78417b60fb ("bpf: Add bpf_btf_find_by_name_kind() helper.")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://lore.kernel.org/bpf/20260903155150.D57251F000E9@smtp.kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260903214758.2727663-4-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
bpf_rbtree_add() invokes its comparator while the caller holds the root
lock. The native insertion code retains raw parent and link pointers across
the callback, so the verifier prohibits unlocking, consuming tree nodes,
or changing RCU state from that callback.
in_rbtree_lock_required_cb() only checks the innermost verifier frame.
Static subprogram calls are permitted while holding a spin lock, and such a
call pushes a frame without in_callback_fn set. Consequently, all callback
restrictions disappear in the nested frame. The subprogram can unlock the
tree, remove and drop the node being compared, then relock. Native insertion
resumes with the stale parent pointer and links freed memory into the tree.
Walk all active frames for the rbtree callback instead. Benign static
subprograms remain permitted, while callback restrictions follow execution
into nested frames.
Fixes: a44b1334aadd ("bpf: Allow calling static subprogs while holding a bpf_spin_lock")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260903214758.2727663-2-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
mark_fastcall_pattern_for_call() must ensure that matched
"spill; call; fill" instruction series is not interrupted by a jump.
Otherwise the rewrite applied by bpf_remove_fastcall_spills_fills()
is not sound.
Record the instructions targeted by jumps in
insn_aux_data[*].jump_target when the CFG is built and use this flag
to stop growing a pattern at such an instruction. Jumps to the first
spill are fine.
Note that existing insn_aux_data[*].jmp_point field can't be reused,
as it marks subprogram return instructions.
Fixes: 5b5f51bff1b6 ("bpf: no_caller_saved_registers attribute for helper calls")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260903205820.1743087-1-eddyz87@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
bpf_convert_ctx_accesses() rewrites an atomic on an arena pointer from
BPF_STX | BPF_ATOMIC to BPF_STX | BPF_PROBE_ATOMIC, this patch adjusts
print_bpf_insn() to print such instructions as regular atomics with a
'probe_' prefix (instead of printing them as BUG_XX).
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260903171542.1438050-2-eddyz87@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
bpf_convert_ctx_accesses() rewrites an atomic on an arena pointer from
BPF_STX | BPF_ATOMIC to BPF_STX | BPF_PROBE_ATOMIC, and it runs before
bpf_opt_subreg_zext_lo32_rnd_hi32().
That pass emits an explicit zero extension for a 32-bit cmpxchg even
when bpf_jit_needs_zext() is false. This is done because on some
architectures 32-bit cmpxchg requires explicit zero extension for the
dst register. E.g. on x86-64 'lock cmpxchg' does not change the %eax
if comparison is successful, while BPF semantics declare that each
operation on a 32-bit register zero extends it's upper half.
is_cmpxchg_insn() matches BPF_MODE == BPF_ATOMIC only, so an arena
cmpxchg misses said zero extension adjustment. This patch adjusts
is_cmpxchg_insn() to match BPF_PROBE_ATOMIC alongside BPF_ATOMIC.
Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20260903171542.1438050-1-eddyz87@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
|
Sashiko reported that the code was using meta->nr_subbufs without making
sure that it matched the nr_pages + 1 on data that was assuming the two
were the same.
Add a check to the persistent ring buffer validation code to make sure
that the saved nr_subbufs matches what we expect.
Link: https://patch.msgid.link/20260903132728.7fb27d34@gandalf.local.home
Fixes: f5b95f1fa2ef3 ("ring-buffer: Validate the persistent meta data subbuf array")
Reported-by: sashiko-bot@kernel.org
Closes: https://lore.kernel.org/all/20260901164836.D962D1F000E9@smtp.kernel.org/
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
|
