| Age | Commit message (Collapse) | Author |
|---|
|
cgroup_bpf_enabled(CGROUP_SOCK_OPS).
BPF_CGROUP_RUN_PROG_XXX() macros guard __cgroup_bpf_run_filter_XXX()
with cgroup_bpf_enabled().
However, even when no SOCK_OPS prog is attached, callers still
initialise struct bpf_sock_ops_kern (memset(), etc.) or evaluate
BPF_SOCK_OPS_TEST_FLAG(), which loads tp->bpf_sock_ops_cb_flags
from a cold cacheline near the end of struct tcp_sock.
Similar to bpf_tcp_ops, let's check cgroup_bpf_enabled() before
BPF_SOCK_OPS_TEST_FLAG() and struct bpf_sock_ops_kern setup, and
rename BPF_CGROUP_RUN_PROG_SOCK_OPS{,_SK}() with __ prefix.
Since all direct callers of tcp_call_bpf() pass 0 and NULL for
nargs and args, they can be folded into the new tcp_call_bpf()
macro.
All callers of tcp_call_bpf_{2,3}arg() check BPF_SOCK_OPS_TEST_FLAG()
and do not need the return value. These are replaced with the new
tcp_call_bpf_flag() macro.
Now, bpf_skops_hdr_opt_len() and bpf_tcp_ops_hdr_opt_len() are
just NOPs without bpf prog attached.
$ llvm-objdump -S -D --disassemble=tcp_established_options vmlinux
...
; asm goto(ARCH_STATIC_BRANCH_ASM("%c0 + %c1", "%l[l_yes]")
; size = MAX_TCP_OPTION_SPACE - remaining;
ffffffff825127ea: b9 28 00 00 00 movl $0x28, %ecx
ffffffff825127ef: 29 c1 subl %eax, %ecx
; return size;
ffffffff825127f1: 89 c8 movl %ecx, %eax
...
ffffffff825127fb: 2e e9 6f 02 43 00 jmp 0xffffffff82942a70 <__x86_return_thunk>
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Link: https://patch.msgid.link/20261008031604.256498-6-kuniyu@google.com
|
|
The legacy SOCK_OPS guards some hooks with a per-socket flag,
tp->bpf_sock_ops_cb_flags.
In contrast, bpf_tcp_ops was initially designed without per-socket
flags so that users can simply define only the callbacks they need.
However, it turned out that even attaching a bpf_tcp_ops with a NULL
callback incurs measurable overhead in the fast path. [0]
We could reuse tp->bpf_sock_ops_cb_flags, but it is u8 and has only
1 bit left for a new opt-in callback, while not all of the 7 existing
opt-in hooks are in the fast path and really need a flag guard.
Moreover, tp->bpf_sock_ops_cb_flags has two bpf helper functions to
update it, both of which require sock_owned_by_me(sk):
* bpf_sock_ops_cb_flags_set()
* bpf_setsockopt(TCP_BPF_SOCK_OPS_CB_FLAGS)
Both helpers only overwrite the field, which leads to reading and
modifying the flags in the BPF prog and then writing them back
via the helper. This prevents use from the fast path (tc or
cgroup_skb) hooks where bh_lock_sock() alone cannot prevent races
with process context.
Let's add a new u32 field, tp->bpf_tcp_ops_flags, at the end of the
tcp_sock_read_txrx cacheline group, along with a new kfunc,
bpf_tcp_ops_set_flags().
bpf_tcp_ops_set_flags() takes bitmasks of flags to enable and
disable and updates tp->bpf_tcp_ops_flags atomically via
try_cmpxchg() without relying on lock_sock().
The kfunc is exposed to bpf_tcp_ops, BPF_PROG_TYPE_CGROUP_SOCKOPT,
and BPF_PROG_TYPE_CGROUP_SKB.
The first argument is struct tcp_sock * so that bpf_tcp_sock() etc
is required for cgroup hooks, but not for bpf_tcp_ops where struct
sock * is promoted to struct tcp_sock * automatically.
The fast-path callbacks will be guarded by the new flags in a later
patch after updating the existing selftest to avoid breaking bisection.
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Link: https://lore.kernel.org/netdev/CAMB2axMwBuz3X4Uwn5uzZUqm91EbiMnbQX832wUFOQ44fOwDnQ@mail.gmail.com/ #[0]
Link: https://patch.msgid.link/20261008031604.256498-3-kuniyu@google.com
|
|
Pull to receive 069403dff337 ("sched_ext: Fix use-after-free of a
destroyed user DSQ by a deferred reenqueue") and point its new cancel
sweep in free_dsq_irq_workfn() at dsq->pcpu_user, which 92344fad5b54
("sched_ext: Skip per-CPU data allocation for built-in DSQs") renamed
here, so that for-next builds.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs fixes from Christian Brauner:
"This contains fixes for the current development cycle.
All of them came out of a review of the mount code that started with a
bug report. The review modeled the corner cases of mount propagation,
unmounting and mount reference counting and turned up a lot of bugs.
Most of them years old. Most fixes come with a selftest.
- Rework connected mounts.
A mount that is unmounted together with its parent can stay
attached to the parent to keep its mountpoint covered. That happens
when the mountpoint is removed with rmdir(), unlink() or rename(),
when a detached tree is dissolved, and for locked mounts in any
umount that isn't synchronous, including the teardown of their
mount namespace. The parent then owns the child and drops it on its
own final mntput(). So any reference from the child's superblock
back to one of its ancestors becomes a cycle that is never freed.
A loop device backed by an image on a tmpfs and mounted on that
same tmpfs is enough. Remove the directory the tmpfs is mounted on
from the host, let the container's mount namespace exit, and the
loop device, the tmpfs and the filesystem on the loop device are
leaked for good. The same works with autofs, zram, ecryptfs,
binfmt_misc, fuse passthrough, zloop, a mass storage gadget and md,
and the selftests have reproducers for them. This has been possible
since v4.1. It is also why "put_mnt_ns(): leave mounts connected"
was reverted in -rc5. Keeping every mount of a dying mount
namespace connected made these cycles trivial to create.
Every unmounted mount is now detached from its parent. Where the
mountpoint has to stay covered the mount leaves a cover on the
parent instead, allocated together with the mount. A lookup on the
unmounted parent that hits a cover finds an empty immutable
directory or file on the private nullfs instance. Nothing leads
from a cover to another mount, so no unmounted mount owns another
one and no cycle can form.
This is visible to userspace. A formerly connected mount can no
longer be reached through its unmounted parent and ".." inside it
leads nowhere, as for every other lazily unmounted mount.
With that the private nullfs instance becomes reachable from
userspace, so it now refuses mounts on top, is mounted read-only,
and refuses fsnotify marks and file locks. Its inodes are shared by
every holder and a watch or a lock would otherwise reach across
users. may_decode_fh() now also decides its subtree check under a
single mount_lock hold, as a racing umount could otherwise let it
decode into what a locked child covered.
- umount:
* Don't silently unmount busy mounts.
Since v4.13 propagate_umount() takes down propagated copies of
the victim with children as long as each child is an overmount or
another copy of the victim, but propagate_mount_busy() only ever
checked copies without children or with just an overmount. A
container that moved a tree beneath its copy of a host mount lost
that tree from under its open file descriptor to a plain umount()
on the host. propagate_mount_busy() now applies the same rules,
walking each chain of copies once.
* Don't let a migrating task hide its reference from umount().
mnt_get_count() sums the per-cpu counters under mount_lock but
the mntget() and mntput() fast paths don't take it. A task that
takes a reference on a cpu the sum has already passed and drops
it after migrating to one the sum hasn't reached yet hides the
reference it held to begin with, and umount() succeeds with the
file still open. Gets and puts now live in separate per-cpu
counters and all puts are summed before all gets with a full
barrier in between, the way srcu_readers_active_idx_check() does
it. mntget() is unchanged and mntput() gains an smp_wmb().
* Check each submount for references right before unmounting it.
shrink_submounts() and mark_mounts_for_expiry() checked all their
victims up front. Unmounting the first could move a busy
overmount to where the next victim's propagated copy is looked up
and it was then unmounted without a check.
* Never expire a locked mount.
A shrinkable mount moved beneath a locked mount with
MOVE_MOUNT_BENEATH takes over the lock, and umount() of an
unlocked ancestor expired it and revealed what it covered. That
umount() now fails with EBUSY as it does for any other locked
child. A lazy umount still takes the whole tree.
- Overmounts and locked mounts:
* Unhash a dentry before detaching the mounts on it.
unlink(), rmdir() and rename() detach the mounts on the victim
but only d_delete() it once its inode is unlocked, a window that
includes an expedited RCU grace period. In between, a lookup from
a mount namespace in which the dentry is a mountpoint found it
hashed, positive and uncovered. Drop the dentry first, as
d_invalidate() already does.
* Don't reveal overmounted entries in refwalk.
A refwalk that had grabbed the dentry before the unlink never
rechecked it the way rcuwalk does with d_seq and mount_lock.
Without any artificial widening three walkers read the covered
file 27 times in a minute. step_into() now fails an unhashed
dentry marked DCACHE_CANT_MOUNT with -ESTALE and the walk is
retried.
* Keep covered mounts covered in OPEN_TREE_NAMESPACE.
Creating such a mount namespace only takes a user namespace and
the copy followed bind mount rules: no children without
AT_RECURSIVE and no unbindable mounts with it. An unprivileged
user could see what mounts covered in the source, such as the
parts of /proc and /sys that container runtimes mask. If the
caller doesn't own the source mount namespace a non-recursive
copy of a mount with something mounted below the requested
directory is now refused and a recursive copy includes unbindable
mounts, the way unshare() copies.
* Keep the lock on a mount that a propagated copy is moved beneath.
MNT_LOCKED moved to any mount that ended up beneath a locked
mount, propagated copies included. A host mount and umount on a
directory covered by a locked mount in a less privileged mount
namespace left that cover unlocked for the namespace's owner to
remove. Only mounts the caller places beneath take over the lock
now.
* Handle mount locking for automounts correctly. Which copies to
lock was decided by the mount namespace of the task that
triggered the automount. A task in a user namespace that
triggered one on a host mount through a file descriptor got the
host's own automount locked while its own copy stayed unlocked
and could have nosuid, nodev and noexec cleared. Use the owner of
the mount namespace the mount lands in.
- Use-after-free and crashes:
* Refuse an automount below a mount that is in no namespace.
The private clones overlayfs uses for its layers have the
MNT_NS_INTERNAL error pointer as their namespace, which
finish_automount() let through and count_mounts() dereferenced. A
fanotify filesystem mark on an overlayfs lower layer hands out
file descriptors on such a clone. With debugfs as the lower layer
opening "tracing" oopses with namespace_sem held for writing and
every mount operation on the system blocks from then on.
* Reset the old parent's ->overmount in mnt_change_mountpoint().
When propagate_umount() moved an overmount off a mount that a
file descriptor kept alive, MOVE_MOUNT_BENEATH through that
descriptor later followed the stale pointer into the freed
overmount.
* statmount() with STATMOUNT_BY_FD and pivot_root() read the parent
of a mount that may be unmounted and only held by a file
descriptor, while the parent's final mntput() can free it.
statmount() now reads it under mount_lock and pivot_root() first
checks that both mounts are in the caller's mount namespace.
* Queue a mount only once for mount notifications. A mount
reparented by one umount_tree() and taken down by the next under
the same namespace_sem hold, as in shrink_submounts(), was queued
twice. That cut the mounts queued in between out of notify_list
while it still pointed at them, and once they were freed every
later mount operation walked freed memory.
* Don't let a pseudo dentry become the root of a mount. A bind
mount of a bpf token file did that with a DCACHE_NORCU dentry,
which is freed without an RCU grace period while lockless path
walks may still look at it. Refuse to clone such a mount.
* Don't inherit MNT_UMOUNT in clone_mnt(). A bind mount of a lazily
unmounted nsfs or pidfs mount through its file descriptor started
out flagged as unmounted. Among other things __detach_mounts()
then dropped the namespace's reference on it, the mount outlived
its namespace and mount_setattr() through the descriptor read the
freed namespace. A recursive bind mount of such a mount also
copied the unmounted stack still attached to it. That now fails
with EINVAL, copying the mount itself still works.
* Remove the fsnotify marks of a mount namespace in free_mnt_ns()
instead of the RCU callback that frees the namespace, where
taking the group mutexes meant sleeping in softirq context.
- Propagation and copies:
* Keep a copied mount unbindable. Since v6.17 clone_mnt() didn't
copy the unbindable flag, so every mount namespace created with
CLONE_NEWNS had bindable copies of all unbindable mounts. This
had been fixed once before.
* Refuse MOVE_MOUNT_SET_GROUP on an unbindable mount. It made the
mount an unbindable slave, a state nothing else can produce, or
silently dropped the unbindable flag. CRIU applies MS_UNBINDABLE
after restoring sharing and isn't affected.
* Check a recursive bind mount for mount namespace loops.
Recursively bind mounting a tree from another mount namespace
could put a mount of a namespace's file inside that same
namespace, which then pins itself and all its mounts. Repeating
it leaks without limit, the reproducer took Shmem from 380 kB to
65916 kB. The copy is now checked with check_for_nsfs_mounts()
before it is grafted, as move_mount() does.
* Look at the topmost mount for a mount namespace file.
attach_recursive_mnt() never looked at the topmost mount of the
source's chain of overmounts. If that was the chain's only mount
namespace file an existing mount at a propagated destination got
buried below the root of the nsfs file where no path walk reaches
it.
* Don't put a mountpoint on a dentry that's being removed.
attach_recursive_mnt() makes a mountpoint of the source's root
without its inode lock, so a racing rmdir() of that directory
could leave a mount on it that nothing ever detaches.
d_set_mounted() now checks cant_mount() as well.
- nullfs:
* Take no inode lock for readdir of an immutable directory.
The root of every empty mount namespace is the same nullfs
directory and iterate_dir() held its i_rwsem across
->iterate_shared(). A reader whose buffer faults on a FUSE mount
of its own holds it for as long as its server wants, and with an
exclusive locker queued behind it every lookup that misses the
dcache, every create and every mount in that directory waits. One
user of an empty mount namespace stalls all others. Directories
with the new FOP_IMMUTABLE flag skip the lock.
* Refuse to reconfigure internal superblocks through fspick(),
MS_REMOUNT or the read-only remount that a synchronous umount()
of the root does. For nullfs only root in the initial user
namespace could do it, but the superblock is shared by every
mount namespace and the flags showed up in statfs() for all of
them.
* Don't update the access time on nullfs and refuse F_SET_RW_HINT
on an immutable inode.
- unshare: Free an nsproxy that was never installed with
nsproxy_free() when set_cred_ucounts() fails. put_nsproxy() dropped
active references that were never taken, which triggered a warning
and hid the caller's own namespaces from listns().
- Smaller changes: mount_setattr() checks the target before it walks
the tree to allocate peer group ids, unshare() puts the old
fs_struct before the old namespaces, dissolve_on_fput() drops the
file's reference to the tree itself, disconnect_mount() is
simplified and the documentation of the propagated unmount rule is
brought up to date"
* tag 'vfs-7.3-rc7.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (58 commits)
namespace: simplify disconnect_mount()
selftests/filesystems: test covered mounts
namespace: rework connected mounts
nullfs: add an empty immutable regular file
selftests/filesystems: check that reading the root of an empty mount namespace stalls nobody
selftests/filesystems: add a helper that holds a readdir in a page fault
readdir: take no inode lock on an immutable directory
nullfs: refuse file locks
fsnotify: let a filesystem refuse marks on its objects
namespace: nothing is mounted on or written through knullfs
namespace: keep the private nullfs instance in knullfs
fhandle: decide the subtree check under mount_lock
selftests/filesystems: check that an automount below an overlay layer is refused
selftests/filesystems: check the atime of the empty mount namespace root
selftests/filesystems: check that a lock lands on the right mount and stays
namespace: keep the lock on a mount that a propagated copy is moved beneath
namespace: never expire a locked mount
nullfs: don't update the access time
namespace: handle mount locking for automounts correctly
namespace: refuse an automount below a mount that is in no namespace
...
|
|
perf coordinates fixes:
cxl/acpi: Convert QTG _DSM latencies to picoseconds
cxl: Account for link width in latency calculation
cxl/port: Convert PCIe link latency to nanoseconds
GPF path fixes:
cxl/pci: Update only the Port GPF timeout fields
cxl/pci: Program the Port GPF timeouts before caching the DVSEC
cxl/pci: Use the Device GPF DVSEC for restricted endpoints
dport fixes:
cxl/port: Attach endpoints to a dport under the lock that pins it
cxl/core: Hold the dport host lock across dport lookup and use
cxl/port: Clear cached dport pointers when a dport is removed
|
|
cxl_pci_get_latency() derives bandwidth from the per-lane speed returned
by pcie_link_speed_mbps(), ignoring the negotiated link width. This
overestimates the FlitLatency contribution for multi-lane links.
Sections 2.11.3 and 2.11.4 of the Intel CXL Memory Device Software Guide
describe the link bandwidth as the product of the negotiated speed and
width.
Replace pcie_link_speed_mbps() with pcie_link_bandwidth_mbps() and
migrate both CXL callers (the only users of the API). The new helper
obtains speed and width from a single Link Status read. The CXL
bandwidth calculation no longer needs a separate width read and
multiplication.
While at it, make error handling more robust by converting PCI config
read errors to negative errno values and checking for zero bandwidth
before calculating latency.
Fixes: 4d07a05397c8 ("cxl: Calculate and store PCI link latency for the downstream ports")
Cc: stable@vger.kernel.org
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Reviewed-by: Alison Schofield <alison.schofield@intel.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> # pci/pci.c
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Link: https://patch.msgid.link/20261009045452.1107921-3-den@valinux.co.jp
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
|
|
Pack bpf_reg_state to bring it's size from 88 bytes to 80.
Consequently, bpf_func_state changes from 1048 to 960.
Thus bpf_func_state moves from 2K to 1K allocator bucket.
1K bucket is the one it fit before the SCEV series.
With this optimization SCEV increases memory consumption on a subset
of Meta internal programs by 3%, without this optimization the
increase is 48%.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-36-bcba9bc53ac9@gmail.com
|
|
For loops like this:
int arr[10];
for (int i = 0; i < 10; i++)
arr[i] = i;
use(arr[i]);
SCEV-based loop widening converts `i` from being enumerated 10 times
as a constant value 0, 1, ..., 9 to being enumerated once as a range
[0..9]. From the point of view of the main verification pass, this
replaces constant-offset stack access with varying-offset
stack access.
The verifier tracks stack values accessed through varying offsets much
less precisely than those accessed via constant offsets.
For example, the read at use(arr[i]) would yield STACK_MISC.
Precision might be lost for the following operations:
- spills (BPF_ST/STX to the stack): stored value can no longer be
recovered when read back, e.g. after the loop;
- fills (BPF_LDX from the stack): a varying-offset read of a spilled
pointer yields a SCALAR_VALUE, so the pointer is lost.
- calls that construct objects on the stack (dynptr, iter, irq_flag,
res_spin_lock): the stack argument is required to have a
constant offset.
This commit adds logic to avoid widening registers used by
such instructions:
- During the main SCEV computation stage, collect_store_base_regs():
- inspects instructions accessing the stack (spill, fill, call to a
function listed in bpf_needs_fixed_stack_off), and
- if there is a SCEV expression for a stack base address register,
records which registers the expression depends on in
the bpf_loop->store_base_reg field.
- `store_base_reg`s accumulated for nested loops are propagated
to outer loops as well.
- During the loop-widening stage, bpf_compute_loop_iters() refuses to
widen a loop when some of the registers the loop changes appear in
store_base_regs.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-35-bcba9bc53ac9@gmail.com
|
|
Loop-bound computation and register widening
===========================================
During the main verification pass, use SCEV expressions and register
values at the entry to the loop to estimate the number of iterations
the loop might execute. Use this number to widen loop induction variables.
For example:
0: r7 = 5;
1: r6 = 0;
2: while (r6 < 3) { // latch
3: if (r6 == 1)
4: r7 = 10;
5: r6 += 1;
6: }
8: use(r7);
- Scalar evolution expressions (SCEVs) at the latch are:
- r6 = (linear r6 1)
- r7 = (any r7 10)
- For this pre-condition loop, this corresponds to the linear inequality
`r6_entry + n < 3`. With r6_entry = 0, the inequality is `n < 3`.
- Hence infer that r6's range at line 2 is [0,3], including the final
failed test. The body starts with r6 in [0,2].
- r7's (any r7 10) expression means that its initial value 5 from line 0
is joined with 10 from line 4: it is widened to [5,10] within
and after the loop.
Currently such inference is supported for:
- Reducible loops with one backedge and complete backedge/exit lists.
- A linear latch dominating the backedge, with one successor exiting
the loop. Additional exits are allowed; they can shorten execution.
- 64-bit arithmetic and comparisons whose continuation condition
normalizes to signed/unsigned < or <=, or !=.
- Initial counter, latch offset, slope and bound evaluable as constants
at loop entry, yielding a finite, representable iteration bound.
- Values with an eligible type at loop entry, e.g. SCALAR_VALUE or
PTR_TO_STACK, but not PTR_TO_CTX.
In order for widening to proceed, every register live at the loop
entry has to have a SCEV that can be widened. Currently, the following
expressions are supported:
- (linear <self> <step>): entry value + iteration# * constant step.
- (reg <self>): loop invariant; keep its entry value unchanged.
- (any ...): join the entry value with immediates or invariant registers;
types must match, and immediates require SCALAR_VALUE.
Main verification loop integration
==================================
Before proceeding with the usual instruction processing, do_check()
takes the following steps at loop entries (H):
- Checks whether the loop can be widened and obtains its
iteration bound (scev.c:bpf_compute_loop_iters()).
- If it can, saves an unwidened checkpoint E0, which will be used as
a base for deriving widened values throughout this invocation of
the loop.
- Widens registers in the current state using E0 and SCEV expressions
computed for H (scev.c:bpf_widen_scev_regs()).
- Pushes a per-frame loop-stack record to the current verifier state.
For terminating loops, the record stores E0 and the iteration bounds.
- Lets is_state_visited() create a new checkpoint W at the same
header, now containing the widened values, and proceeds with
normal checking.
- On an exit edge, pops the loop record and verifies the exit
path normally.
is_state_visited() treats a loop proven to terminate like an
iterator-based loop: RANGE_WITHIN comparison attempts to establish
convergence against a checkpoint whose exploration is still
in progress.
For example:
r0 = 0;
H: r0 += 1;
if (r0 != 3) goto H;
exit;
first H: prove H_count = 3; save E0(r0 = 0)
widen r0 to [0,2]; push the loop record
save W(r0 = [0,2])
body: r0 becomes [1,3]; fork at the conditional
exit: r0 = 3; pop the loop record and check exit
backedge: r0 = [1,2]; clamp, then match with W and prune
Clamping
--------
Independent register ranges lose correlations between induction
variables. For example:
r6 = 0; r7 = 0;
do { r6 += 1; r7 += 2; } while (r6 < 3);
use(r7);
header: r6 in [0,2], r7 in {0,2,4}
body: r6 in [1,3], r7 in {2,4,6}
backedge: r6 in [1,2], r7 still in {2,4,6}
The latch narrows r6 but does not propagate this refinement to r7.
To account for this, before comparing states on a backedge,
scev.c:bpf_clamp_scev_regs() recomputes r7's range from E0 and the
saved iteration bound, then intersects the current range {2,4,6}
with the recomputed {0,2,4}, arriving at {2,4}.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-34-bcba9bc53ac9@gmail.com
|
|
Assign algebraic expressions to loop variables (registers and stack
spills), describing how their values evolve across iterations. These
summaries provide symbolic input for the subsequent widening patch.
The algorithm draws on ideas from the following papers:
- "Symbolic Evaluation of Chains of Recurrences for Loop Optimization"
Robert A. van Engelen, 2000
- "The CR# Algebra and its Application in Loop Analysis and Optimization"
Robert A. van Engelen, 2004
The analysis proceeds in two phases:
- Compute expressions for one symbolic iteration: start each variable
with a reference to its input value, apply instruction effects and
join expressions where control-flow paths merge.
- Convert the resulting backedge updates to recurrences at the header,
then substitute these recurrences into expressions at the latch.
Analyze loops innermost first. Within each loop, visit blocks in
topological order (reverse postorder, with backedges ignored and nested
loops collapsed). Treat each nested loop as an opaque operation:
preserve its invariant values, forget the others and continue at its
exits, without expanding its iterations.
For example, using 64-bit arithmetic:
0: r7 = 5;
1: r6 = 0;
2: do { // header
3: if (r6 == 1)
4: r7 = 10;
5: r6 += 1;
6: } while (r6 < 3); // latch
Expressions use symbolic inputs r6 and r7; the concrete values from
lines 0-1 are supplied later by the main verifier.
First phase:
------------
At each instruction, an environment maps register numbers to their
current expressions. Transfer functions update this mapping, while
joins merge the environments arriving along different paths.
For illustration, follow each branch separately to the backedge:
- At 3, take the path skipping 4: r6 = r6, r7 = r7.
- At 5, transfer(r6 += 1): r6 = r6 -> r6 = (+ r6 1).
- At 2, the first path contributes r6 = (+ r6 1), r7 = r7.
- At 4, on the other path from 3, transfer(r7 = 10):
r7 = r7 -> r7 = 10.
- At 5, transfer(r6 += 1): r6 = r6 -> r6 = (+ r6 1).
- At 2, join the two paths' contributions:
join((+ r6 1), (+ r6 1)) = (+ r6 1) for r6;
join(r7, 10) = (any r7 10) for r7.
Equal expressions stay unchanged; different expressions become ANY
alternatives. These summarize one iteration, without unrolling the loop.
Second phase:
-------------
The update r6 = (+ r6 1) says that each iteration adds 1 to r6.
Repeating this update n times gives r6_entry + n, a linear recurrence.
- At header 2, r6 = (+ r6 1) becomes (linear r6 1), meaning
r6_entry + n, where n counts iterations from zero.
- At latch 6, substitute this recurrence into (+ r6 1):
(+ (linear r6 1) 1) simplifies to (linear (+ r6 1) 1),
meaning r6_entry + 1 + n.
- At both points, r7 retains (any r7 10): either its entry value
or 10. ANY records alternatives, not numeric ranges or the path
conditions selecting them. Unchanged variables retain their inputs.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-33-bcba9bc53ac9@gmail.com
|
|
SCEV integration logic might create two consecutive checkpoints at the
same instruction w/o processing any instructions in between.
This happens for loop-entry checkpoint -> regular checkpoint
sequences, where the loop-entry checkpoint has to remain unchanged
but the regular checkpoint is used for widening.
This commit adapts bpf_mark_chain_precision() to support this.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-32-bcba9bc53ac9@gmail.com
|
|
SCEV needs information about the program's basic-block structure.
Piggyback on compute_predecessors() and set
insn_aux_data->bb_end for an instruction if:
- it has multiple successors, or
- its sole successor has multiple predecessors, or
- it is a jump instruction.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-31-bcba9bc53ac9@gmail.com
|
|
A utility function counting the number of STACK_SPILL slots in a
bpf_stat_state instance. For use by SCEV widening logic.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-30-bcba9bc53ac9@gmail.com
|
|
When dealing with loops like:
for (ptr = ...; ptr < end_ptr; ptr += 8)
...
Scalar evolution needs to know if ptr and end_ptr identify a same
memory location (possibly with different offsets). If they do,
it is sound to attempt to infer iteration bounds for such a loop.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-29-bcba9bc53ac9@gmail.com
|
|
SCEV widening needs an explicit checkpoint preserving the original
loop-entry state, independently of ordinary state-cache heuristics.
This commit extracts checkpoint creation logic from
bpf_is_state_visited() into a separate utility function.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-28-bcba9bc53ac9@gmail.com
|
|
Utility functions to take a union of two registers' scalar values:
merge the circular 32-bit and 64-bit bounds, tnums and base/step
of two registers, then synchronize and validate the result.
Leave id invalidation and type validation to the caller.
Caller should use bpf_reg_union_compatible() to ensure
that final union result is sensible.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-27-bcba9bc53ac9@gmail.com
|
|
Provide a helper to set a register to a known scalar value for use outside
verifier.c. To be used by SCEV widening logic.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-26-bcba9bc53ac9@gmail.com
|
|
A utility function for setting a register's range and step information.
Used by SCEV widening and clamping logic further in the series.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-25-bcba9bc53ac9@gmail.com
|
|
SCEV needs to process loops from innermost to outermost, summarizing
results for nested loops. Add a pass to compute the loop structure.
Use a non-recursive adaptation of the algorithm from:
"A New Algorithm for Identifying Loops in Decompilation" by Wei et al.
Record the loop hierarchy per instruction:
- The field insn_aux_data->loop_header records the innermost loop
header containing it.
- The field insn_aux_data->loop records additional information about
the loop if this instruction is a loop header (exits, backedges,
irreducibility).
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-24-bcba9bc53ac9@gmail.com
|
|
SCEV needs to find an exit condition that dominates a loop's backedge;
such conditions bound every loop iteration. Compute the immediate
dominator tree as a prerequisite for identifying such conditions.
Use the iterative reverse-postorder algorithm from:
"A Simple, Fast Dominance Algorithm" by Cooper et al.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-23-bcba9bc53ac9@gmail.com
|
|
Dominator intersections and SCEV worklist scheduling need to compare
instructions by their DFS postorder rank. Save per-instruction
postorder numbers alongside the existing postorder sequence.
This commit modifies bpf_compute_postorder() to actually do the DFS
traversal, instead of scheduling traversal of all siblings at once.
Consider a graph:
A -> B
A -> C
C -> B
Old result: C, B, A
DFS result: B, C, A
Fixes: efcda22aa541 ("bpf: compute instructions postorder per subprogram")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-21-bcba9bc53ac9@gmail.com
|
|
Extend scalar register tracking with a linear "base + step * k" description:
each scalar carries two u16 fields, base and step (invariant: base < step),
recording that its value is some point on the line base + step * k.
A fresh scalar starts as base=0, step=1, i.e. any integer.
This lets the verifier reason about strided values, e.g. an index multiplied
by an element size, which neither the min/max bounds nor the tnum can capture
for a non-power-of-2 stride. Such reasoning is necessary to allow
array access with a variable offset through PTR_TO_BTF_ID, e.g.
for expressions like p->arr[i].field.
reg_bounds_sync() uses the description in two ways:
- __reg_bound_offset() clears the low bits of var_off implied by the number
of trailing zeros shared by base and step.
- __update_bounds_from_step() intersects the 64-bit range with the line via
cnum64_intersect_linear(), snapping smin/smax to actual points on it.
This is the last step of the sync, as refining the range from var_off
can move the bounds off the line.
range_within() is extended to check whether the cur register's equation
describes a subset of the points described by the old register's equation.
Scalar ALU operations update base/step as follows:
- ADD of a constant (including pointer + scalar): keep step, shift base;
- MUL by a constant: scale step and base by the constant;
- LSH by a constant: scale step and base by the corresponding power of two;
- sign extension: keep the register unchanged if all values fit the target
type, reset to base=0, step=1 otherwise;
- All other operations reset to base=0, step=1.
Branch (eq/neq) checks do not take into account base/step yet.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-19-bcba9bc53ac9@gmail.com
|
|
For use in SCEV logging to make large widened bounds more readable.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-17-bcba9bc53ac9@gmail.com
|
|
When windening SCEV expressions like *fp-8 -> (any *fp-8 *fp-16),
fp-8 might be not yet allocated. Expose a utility function such
that widening logic can handle this.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-16-bcba9bc53ac9@gmail.com
|
|
Expose the existing helpers verifier.c:rev_opcode() and
verifier.c:flip_opcode() as bpf_rev_opcode() and bpf_flip_opcode()
and declare them in bpf_verifier.h. The helpers are used by SCEV
logic while analyzing loop conditions.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-15-bcba9bc53ac9@gmail.com
|
|
Add cnum64_intersect_linear(), which intersects a cnum64 interval with the
set of integers congruent to 'base' modulo 'step', i.e. the points
base + step * k. It tightens the interval by rounding its signed minimum up
and its signed maximum down to the nearest such point, and returns an empty
cnum when no point falls within the interval.
The arithmetic is done on the signed bounds [smin, smax]. The line is defined
over the integers, while a cnum64 is an arc on the u64 circle, so a u64-modular
computation would misplace the line for negative values whenever 'step' does
not divide 2^64. Working from the signed bounds keeps the residues consistent,
e.g. it can reason about a range like [-9, 3] with step 3.
Also add imod() to bpf_verifier.h: a mathematical modulo returning an
always-non-negative residue (unlike C's '%', which follows the sign of the
dividend). It is used by cnum64_intersect_linear() and by the base/step
tracking added in subsequent patches.
This is a building block for tracking scalar registers whose value is known
to lie on a line base + step * k.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-14-bcba9bc53ac9@gmail.com
|
|
Add cnum32_union() and cnum64_union() to compute a smallest circular
range containing both inputs.
Do so by enumerating the following configurations in a rotated
frame (a.base at the origin):
0 UT_MAX
|---------------------------------------------------|
[= a ==============================] |
[= b tail =] [= b main ===================>
[= union ===========================================]
0 UT_MAX
|---------------------------------------------------|
[= a =====================] |
[= b tail ======] [= b main =====>
[= union tail ============] [= union main =>
0 UT_MAX
|---------------------------------------------------|
[= a =======] |
[= b tail =============] [= b main =====>
[= union tail =========] [= union main =>
0 UT_MAX
|---------------------------------------------------|
[= a =========================] |
| [= b ====================] |
[= union ==================================] |
0 UT_MAX
|---------------------------------------------------|
[= a =====================================] |
| [= b ========] |
[= union =================================] |
0 UT_MAX
|---------------------------------------------------|
[= a ============] |
| [= b =======] |
Two possible covering arcs:
[= ab ======================================] |
[= ba tail ======] [= ba main =========>
Pick the smaller one.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-13-bcba9bc53ac9@gmail.com
|
|
Used by the SCEV widening logic later in the series.
The verifier checks alignment for some memory accesses.
When SCEV widens a pointer or an index variable, it must retain
alignment information in order to allow array access.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-12-bcba9bc53ac9@gmail.com
|
|
Stack liveness currently analyzes global callees in the context of their
callers. The resulting read/write marks describe only the current body,
but freplace can replace it after the caller has been verified.
A no-op callee taking a stack pointer can be replaced with one writing
through that pointer, leaving the call's may_write marks incomplete.
SCEV could then preserve a stack counter recurrence across a call that
resets it, invalidating a loop termination proof.
Instead, derive global call effects from BTF argument types instead,
same is it done on the main verification pass.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-10-bcba9bc53ac9@gmail.com
|
|
SCEV needs to know which registers might be stack pointers at a
particular instruction. This information is used to avoid widening
registers if that would hinder further verification.
See the patch [1] for additional information.
liveness.c:compute_subprog_args() already tracks this information.
This commit modifies it to save the information in insn_aux_data for
further usage.
[1] "bpf: avoid widening registers that hinder exact stack-slot tracking"
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-7-bcba9bc53ac9@gmail.com
|
|
SCEV needs live stack information in order to skip tracking dead
slots. Liveness tracks this per function instance, but SCEV is not
callchain-sensitive and needs a summary across calling contexts.
For each instruction, union the current-frame liveness masks from all
analyzed function instances. Represent 8-byte slots as a summary of
constituent halves. Store the summary in insn_aux_data.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-6-bcba9bc53ac9@gmail.com
|
|
SCEV needs may-write information to invalidate stack expressions after
indirect writes. Liveness tracks this per function instance,
but SCEV is not callchain-sensitive and needs a summary across
calling contexts.
For each instruction, union the current-frame may_write masks from all
analyzed instances. Represent 8-byte slots as a summary of constituent
halves. Store the summary in insn_aux_data and expose it via a helper.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-5-bcba9bc53ac9@gmail.com
|
|
Return kfunc stack effects through arg_access_info while preserving the
existing BTF size lookup and read/def classification. Retain __sz/__szk
handling and conservatively mark pointer arguments as possible writes.
Remove the temporary signed-size adapter from liveness. Add liveness
assertions for privileged and unprivileged __uninit outputs.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-3-bcba9bc53ac9@gmail.com
|
|
Adjust bpf_helper_stack_access_bytes() to return results as a struct
arg_access_info. The result is based on bpf_func_proto->argX_type
value as in the following table:
| Argument flags | allow_uninit_stack | Meaning |
|----------------------+--------------------+----------------------|
| MEM_WRITE | true | may_read, must_write |
| MEM_WRITE+MEM_UNINIT | true | must_write |
| MEM_WRITE | false | may_read, must_write |
| MEM_WRITE+MEM_UNINIT | false | may_read, may_write |
| 0 | true/false | may_read |
MEM_WRITE now produces must_write when allow_uninit_stack is true:
the verifier discards previous stack knowledge over the affected
range. Previously, liveness only treated MEM_UNINIT this way.
Also use resolve_map_arg_type() to account for bpf_map_peek_elem()
treating its buffer as an input for bloom filters.
Annotate ringbuf dynptr submit and discard arguments with MEM_WRITE.
Their existing OBJ_RELEASE handling already invalidates the dynptr, but
the prototypes need to describe the write as well as the release.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-2-bcba9bc53ac9@gmail.com
|
|
The following patches in the series add support for may_write property
inference. To properly track this property there is a need to
distinguish three ways the helpers/kfuncs affect stack:
- read;
- write and destroy all previous stack tracking information for
affected slots;
- both read and write the stack slots, potentially preserving some of
the tracking information.
The later property can't be encoded by signed access_byte convention.
Hence, introduce arg_access_info with a size and separate may_read,
may_write and must_write bits. Convert the liveness access recording
paths to use it. Proper may_write calculation and support will be
added in the following patches.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261009-scev-minimal-rebase-v3-1-bcba9bc53ac9@gmail.com
|
|
The term "KHO scratch" is vague and overloaded.
For one, it does not accurately describe what the memory is for. From
KHO's perspective, it is memory passed by the previous kernel that is
guaranteed to have no allocations and hence it is safe to allocate from.
For another, the term is overloaded. KHO also knows to discover other
ranges of memory that don't have any preservations and are safe to
allocate from. This is done by kho_extend_scratch(). These areas, while
called "scratch", are completely different from the chunk of memory
passed by the previous kernel for early boot allocations.
Rename "KHO scratch" to "KHO boot memory", or "KHO bootmem" in short.
Update all function names, variable names, comments, and documentation
to use this.
Call the discovered areas "noprsrv", matching what memblock would see
them as. Rename kho_extend_scratch() to reflect this.
This patch largely has no functional changes. The only functional
changes are renames of debugfs files from "scratch_phys" and
"scratch_len" to "bootmem_phys" and "bootmem_len" respectively.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260922041321.233986-4-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
The term "KHO scratch" is vague. It does not accurately describe what
the memory is for. Generally, scratch is for things that don't contain
anything useful. This is not true.
For memblock, KHO_SCRATCH is the _only_ useful memory during a KHO boot.
It is memory passed by the previous kernel that is guaranteed to have no
preserved pages, and so is the only thing safe to allocate from early in
boot.
In addition, KHO now has the ability to discover other "scratch" areas.
These aren't directly passed by the previous kernel. Instead, they are
found by searching the KHO preserved memory map.
Memblock does not care about the difference between the two. From
memblock's perspective, both are memory areas with no preservations and
so they are safe to allocate from before KHO fully initializes.
Rename MEMBLOCK_KHO_SCRATCH to MEMBLOCK_KHO_NOPRSRV. This name clearly
shows what the memory is for from memblock's perspective. Update
memblock functions and comments that refer to scratch to use noprsrv.
There are no functional changes apart from the flag name change in
debugfs.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260922041321.233986-3-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
This config is only a proxy for CONFIG_KEXEC_HANDOVER. It is only
enabled by KHO. No point in the indirection. Use CONFIG_KEXEC_HANDOVER
directly.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
Link: https://patch.msgid.link/20260922041321.233986-2-pratyush@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux into HEAD
KVM/vfio: Use file-based reference counting for KVM
This series switches the KVM-VFIO interface and external consumers over to
standard file-based reference counting, eliminating all external KVM symbol
exports.
Currently, VFIO integrates with KVM by looking up kvm_get_kvm_safe() and
kvm_put_kvm() dynamically using symbol_get(), manually tracking module
reference counts and storing a put_kvm function pointer in struct
vfio_device.
In the ARM64-on-s390 architecture, a second concurrent KVM module
(kvm-arm64) is introduced alongside native KVM to host hardware-accelerated
ARM64 guests. Having exported global symbols (like kvm_get_kvm/kvm_put_kvm)
creates symbol conflicts and prevents clean coexistence of two KVM modules.
Additionally, the file based counting simplifies the code and reuses
the existing fs refcounting.
Instead of passing raw KVM pointers and managing module symbols manually, the
interface now passes the underlying VM file descriptor throughout VFIO and
associated architecture subsystems. To safely extract the KVM instance from a
file, an architecture-namespaced helper mechanism is introduced that verifies
the file belongs to the expected KVM implementation before accessing its
internal state. A back-pointer from the KVM instance to its associated file is
maintained across its lifecycle so subsystems can safely acquire file
references on demand. Finally, with VFIO, architecture page tracking, and
device hooks converted to use file references, the remaining KVM reference-
counting exports are restricted strictly to internal KVM modules.
|
|
Rename swiotlb_size_or_default() to swiotlb_default_pool_size() to make
clear that it returns the current size of the default SWIOTLB pool.
This includes changes made by command-line options or sizing heuristics,
not just the compiled-in default.
No functional change.
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Link: https://lore.kernel.org/r/20260924060756.1325156-2-aneesh.kumar@kernel.org
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
|
|
Auto-boot for an always-on remote processor registers an asynchronous
request_firmware_nowait() whose callback, rproc_auto_boot_callback(),
calls rproc_boot() first and only releases the fetched image at the
very end of the callback. rproc_boot() does not take this image as an
argument: it fetches the very same image again with a synchronous
request_firmware(), so every auto-boot reads the firmware image twice
and allocates the buffer twice; on kernels with the sysfs fallback
enabled, the uevent round-trip is repeated as well.
The asynchronous request exists only so that rproc_add() does not
block on the filesystem read; the image it fetches is never used.
The core already defers rproc_boot() to a worker for detached
processors (attach_work). Reuse that worker for offline processors
too and drop the asynchronous firmware request: rproc_boot() fetches
the image exactly once and dispatches between a firmware boot and
an attach based on the processor state. The work is renamed to
boot_work to match its widened role.
commit 400e64df6b23 ("remoteproc: add framework for controlling remote
processors") noted back in 2011 that "we must wait until it completes
before we try to unregister the device". rproc_del() now does exactly
that: it waits for the boot work with cancel_work_sync(), which also
closes the theoretical window in which a pending attach work could
outlive the rproc instance.
The changed fallback behaviour only matters for legacy
configurations. udev dropped its userspace firmware loader back in
2014, as recorded in
Documentation/driver-api/firmware/fallback-mechanisms.rst, and the
kernel has documented since commit 02c399306826 ("firmware_loader:
enhance Kconfig documentation over FW_LOADER") that "Linux no longer
relies on or uses a fallback mechanism in userspace". Auto-boot now
uses the same synchronous fallback semantics as every other explicit
boot source (sysfs, cdev).
Signed-off-by: Yonghao Zhang <hyz3367@gmail.com>
Link: https://lore.kernel.org/r/20261001081056.38293-1-hyz3367@gmail.com
Signed-off-by: Mathieu Poirier <mathieu.poirier@linaro.org>
|
|
A ring created with IORING_SETUP_R_DISABLED is exempt from its register
restrictions until it is enabled. This lets the creator register the
restrictions and then enable the ring with IORING_REGISTER_ENABLE_RINGS,
whether or not that opcode is in the allowed set.
Since commit ed82f35b926b ("io_uring: allow registration of per-task
restrictions"), a ring also inherits restrictions from the task that
creates it, and the same exemption applies to those. A task whose
register opcodes are restricted can therefore create a disabled ring,
issue any register opcode on it, and then enable it.
Mark rings whose restrictions were inherited from the task, and keep
checking register opcodes for them while the ring is disabled. Enabling
such a ring now requires IORING_REGISTER_ENABLE_RINGS to be in the
task's allowed set. Per-ring restrictions registered on a disabled ring
work as before.
Fixes: ed82f35b926b ("io_uring: allow registration of per-task restrictions")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Eugene Gvozdetsky <gvozdet@gmail.com>
Link: https://patch.msgid.link/20261009-iouring-task-restrict-fix-v1-1-a49daf55a12c@gmail.com
[axboe: amend with inheritance flag fixup]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Move the following functions and declarations from the public header
include/linux/dmaengine.h into the driver-private header
drivers/dma/dmaengine.h, since they are only used by DMA engine drivers
and not by DMA engine consumers:
- dma_async_device_register()
- dmaenginem_async_device_register()
- dma_async_device_unregister()
- dma_async_device_channel_register()
- dma_async_device_channel_unregister()
- is_slave_direction()
- dmaengine_get_direction_text()
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20261001143955.2195063-1-Frank.Li@oss.nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
|
|
Add a dmaengine_desc_set_callback() inline helper that sets the
callback, callback_result, and callback_param fields of a
dma_async_tx_descriptor together in a single call.
Using this helper instead of open-coding the three field assignments
ensures all three related fields are always updated consistently and
makes call sites more readable.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20261002195410.213610-1-Frank.Li@oss.nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
|
|
Add dmaengine_get_provider_device() API and kernel doc for
dmaengine_get_dma_device() to describe the difference.
dmaengine_get_dma_device() is used for DMA mapping because some DMA
engines provide per-channel iommu mapping.
dmaengine_get_provider_device() is used to find a DMA engine channel
by a filter function.
In most cases they return the same value, but they have different
semantic meanings.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20261002202341.253152-1-Frank.Li@oss.nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
|
|
Add a new inline helper dmaengine_get_max_xor() that returns the maximum
number of XOR source buffers supported by the DMA channel's device.
Returns 0 if the channel is NULL or does not support XOR operations.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20261005-dmaengine_prep_dma_pq-v3-5-a4a241f26fa7@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
|
|
Add two new inline helpers to query DMA channel capabilities without
directly accessing the internal dma_device pointer:
dmaengine_get_cap_mask(chan) - returns a const pointer to the
capability mask of the DMA device backing the channel.
dmaengine_has_cap(chan, tx_type) - returns true if the channel
supports the given transaction type.
These complement the existing dma_has_cap() macro which operates on
a dma_cap_mask_t directly.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20261005-dmaengine_prep_dma_pq-v3-4-a4a241f26fa7@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
|
|
Add dmaengine_get_copy_align(), dmaengine_get_xor_align(),
dmaengine_get_pq_align(), and dmaengine_get_fill_align() inline
helpers to retrieve the alignment requirement of a DMA channel as
a power-of-2 exponent.
These complement the existing dmaengine_is_*_aligned() test helpers
by providing direct access to the alignment value, which is useful
when callers need to propagate or compare alignment constraints.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20261005-dmaengine_prep_dma_pq-v3-3-a4a241f26fa7@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
|
|
DMA engine consumers typically hold a struct dma_chan pointer and should
not need to reach into struct dma_device directly to check alignment
requirements. Add dmaengine_is_copy_aligned(), dmaengine_is_xor_aligned(),
dmaengine_is_pq_aligned() and dmaengine_is_fill_aligned() as channel-based
counterparts to the existing is_dma_*_aligned() family.
Each wrapper calls dmaengine_check_align() directly with the alignment
value from chan->device, keeping struct dma_device opaque to consumers.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20261005-dmaengine_prep_dma_pq-v3-2-a4a241f26fa7@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
|
|
Add dmaengine_prep_dma_(pq|pq_val|interrupt|xor)() as an inline
wrapper around the device_prep_dma_(pq|pq_val|interrupt|xor)() callback,
consistent with how other dmaengine prep helpers like
dmaengine_prep_dma_memcpy() are structured.
Provide a clean public API that hides the direct callback access and add
proper NULL-checks on chan, chan->device, and the callback pointer.
Assisted-by: LLM
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20261005-dmaengine_prep_dma_pq-v3-1-a4a241f26fa7@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
|
