| Age | Commit message (Collapse) | Author |
|---|
|
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
# Conflicts:
# Documentation/scheduler/index.rst
# arch/arm64/configs/defconfig
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git
# Conflicts:
# include/linux/uio.h
# lib/iov_iter.c
|
|
# New commits in timers/core:
348f54c435bf ("selftests/timers: clocksource-switch: Fix unchecked open()/read()")
b03638013add ("selftests: timers: Measure the CPU timers on the clock they count")
a252cb93e156 ("selftests: timers: Count what tick is worth in the drift estimate")
3945c4a3ea15 ("time/kunit: Add time64_to_tm() case beyond 32-bit day count")
2927f7ca7844 ("time: Prevent time64_to_tm() day truncation on 32-bit")
bc5b66c300b8 ("timekeeping: Use READ_ONCE/WRITE_ONCE() for ktime_sec to prevent tearing")
bb41ece16463 ("timers/migration: Mark racy updates to tmigr_event::ignore field")
1159ad0a6aaa ("timers: Mark racy updates to hlist_node::pprev field")
5dffe33bfdaf ("hrtimer: Apply READ_ONCE() to lockless base->running loads")
ad80926d7803 ("hrtimer: Mark the hrtimer_sleeper structure's ->task field __private")
15f84398330c ("hrtimer: Update hrtimer_resolution only if value changes")
7eed1771a6e8 ("futex: Use accessor for hrtimer_sleeper ->task field in requeue")
9cd2f4e304d9 ("rtmutex: Use accessor for hrtimer_sleeper ->task field")
3990d196954a ("net: pktgen: Use accessor for hrtimer_sleeper ->task field")
9b7bcd671e9c ("timers: Use accessor for hrtimer_sleeper ->task field in sleep_timeout.c")
fa3d486086d6 ("futex: Use accessor for hrtimer_sleeper ->task field in waitwake.c")
851c30277918 ("io-uring/rw: Use accessor for hrtimer_sleeper ->task field")
66b29025b676 ("wait: Use accessor for hrtimer_sleeper ->task field")
cefc1a24ace5 ("aio: Use accessor for hrtimer_sleeper ->task field")
d166a1cd9016 ("hrtimer: Mark data-racy accesses to hrtimer_sleeper ->task field")
7b07d15ed1d7 ("posix-timers: Handle exit in do_exit() completely")
54ad1e0ea42c ("posix-cpu-timers: Prevent enqueueing when PF_EXITING is set")
760ad335d600 ("posix-cpu-timers: Use PF_EXITING to indicate exit")
8e9ed3b66f40 ("posix-cpu-timers: Move inlines out of public header")
24adce0b86c9 ("posix-timers: Move POSIX timer group exit related code out of do_exit()")
7117334f022b ("posix-timers: Move posixtimer_exec_cleanup() out of exec.c")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
|
|
|
|
Signed-off-by: Christian Brauner <brauner@kernel.org>
|
|
* for-7.4/io_uring-dmabuf:
io_uring/rsrc: add dmabuf backed registered buffers
io_uring/rsrc: add regbuf import flags
io_uring/rsrc: add uncloneable regbuf flag
io_uring/rsrc: extend buffer update
io_uring/rsrc: introduce buf registration structure
nvme-pci: add SGL support for the dmabuf path
nvme-pci: implement dma-buf backed requests
nvme-pci: rename nvme_pci_sgl_set_data to nvme_pci_dma_iter_set_sgl
block: add dma-buf support for raw bdev
block: introduce dma map backed bio type
block: always adjust bi_offset on bio_advance_iter
iov_iter: add iterator type for dmabuf maps
dma-buf: introduce initial file I/O infrastructure
|
|
Implement dmabuf backed registered buffers. To register them, the user
should specify IO_REGBUF_TYPE_DMABUF for the regitration and pass the
desired dmabuf fd and a file for which it should be registered.
>From there, it can be used with io_uring read/write requests
IORING_OP_{READ,WRITE}_FIXED) as normal. The requests should be issued
against the file specified during registration, and otherwise they'll be
failed. The user should also be prepared to handle spurious -EAGAIN by
reissuing the request.
Internally, dmabuf registered buffers is an optin feature for io_uring
request opcodes and they should pass a special flag on import to use it.
Suggested-by: David Wei <dw@davidwei.uk>
Suggested-by: Vishal Verma <vishal1.verma@intel.com>
Suggested-by: Tushar Gohad <tushar.gohad@intel.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/4aa230df871ee275536f367087b44ebfebb88cc1.1791975048.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We'll have special registered buffer types that can't be used with all
opcodes and need special handling. Add separate flags to control
registered buffer import, which will be used to specify what kind of
buffers the caller can handle.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/eb798e045bc245ebb7b0b42ba09a34b92f8b216f.1791975048.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
It's hard to implement cloning if the internal structure needs to be
mutable and/or relies on other ring resources. In preparation to such
buffer types, add a flag indicating that the buffer can't be cloned. It
might be possible to add cloning in the future for them, but that would
likely need reallocating the structure and reacquiring resources in case
by case manner.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/c73c459578981e64a352e46c0db39ad46621c63a.1791975048.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We need to pass more information to buffer registration than we can fit
into a single struct iovec. This patch allows users to optionally pass
struct io_uring_regbuf_desc. Apart from having more space for future use
cases, it also introduces registration types.
Currently, the type can be either of IO_REGBUF_TYPE_UADDR, which mirrors
the iovec path, or IO_REGBUF_TYPE_EMPTY for leaving a buffer table slot
empty. The next patch introduces a dmabuf backed type, and can be useful
for other extensions like splicing a list of user addresses (i.e.
iovec[]), interoperability with zcrx, kernel allocated memory like was
brough up by Cristoph. Note, the type only represents a registration
option, which is distinct from how io_uring internally stores it.
The flags field is not used yet but always useful to have, e.g. we can
encode read-only / write-only restrictions using it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/0d60f1ebab80ef594a78ea27c2edd5ad33faa640.1791975048.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
In preparation to following changes, instead of passing an iovec for
buffer registration introduce a new structure. It'll be moved to uapi
later, but for now it's initialised early from a user provided iovec.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/178bbcbc7a0a9cf017284e8b15d5434fb546fcaf.1791975048.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
* for-7.4/io_uring: (21 commits)
io_uring/io-wq: don't call try_to_wake_up() twice in io_wq_worker_wake()
io_uring: cleanup the io_uring_attr_pi definition
io_uring: cleanup __io_prep_rw
io_uring/memmap: fix non-compound alloc fallback with memcg accounting
io_uring: wait for in-flight requests on ring release
io_uring: drop registered files and buffers at release time
io_uring: run cancelations synchronously on ring release
io_uring/cancel: cancel and wait for all requests on process exit
io_uring/notif: count pending zerocopy notifications per ring
io_uring/uring_cmd: only cancel requests of the given task
io_uring: put request files before posting the completions
io_uring/rw: don't reap io-wq IOPOLL completions while io-wq has a reference
io_uring: post io-wq completions from the last request reference
io_uring/io-wq: put the request file before posting a completion
io_uring/io-wq: release raw spinlock before calling wake_up()
io_uring/rsrc: fix accounting for cloned buffers with huge pages
io_uring: check for room before posting the dummy skip CQE
io_uring/fdinfo: ignore IORING_CQE_F_32 in last CQ array slot
nvme/ioctl: call io_uring_cmd_set_res32() in ->end_io()
io_uring/cmd: split io_uring_cmd_set_res() from io_uring_cmd_done()
...
|
|
io_wq_worker_wake() does an unconditional wake_up_process(). This means
that __set_notify_signal() makes no real sense. Change it to use
set_tsk_thread_flag(TIF_NOTIFY_SIGNAL) instead to avoid test_and_set_bit()
and another wake_up_state(TASK_INTERRUPTIBLE) if NOTIFY_SIGNAL was not set.
Perhaps io_wq_worker_wake() can rely on __set_notify_signal() solely, but I
don't understand this code enough and this patch doesn't change the current
behaviour.
Signed-off-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Christian Brauner (Amutable) <brauner@kernel.org>
Link: https://patch.msgid.link/asunjCm5Ovj_YDSp@redhat.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Don't spin on local work the exiting task can't run.
For a ring still R_DISABLED when the task got its tctx node, the
submitter is set later and may be another task, so DEFER_TASKRUN
local work stays pending forever and the exiting task spins in
do_exit().
- Don't charge the SQ/CQ rings or provided buffer rings to
RLIMIT_MEMLOCK. The region API conversion started charging them,
which the rings had been exempt from.
- Don't exempt disabled rings from task restrictions on register
- Populate the CONNECT filter fields for a 24-byte sockaddr_in6
* tag 'io_uring-7.3-20261010' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/cancel: don't spin on local work the task can't run
io_uring/net: populate CONNECT filter fields for 24-byte sockaddr_in6
io_uring/register: don't exempt disabled rings from task restrictions
io_uring/kbuf: don't charge provided buffer rings to RLIMIT_MEMLOCK
io_uring: do not charge the SQ/CQ rings to RLIMIT_MEMLOCK
|
|
io_uring_cancel_generic() refuses to sleep while any ring on the exiting
task's tctx has local work pending. It assumes the task is that ring's
submitter and that the next io_uring_try_cancel_requests() pass will run
the work. This does not hold for a ring that was still
IORING_SETUP_R_DISABLED when the task got its tctx node from
io_uring_create() or io_ringfd_register(). The submitter is set later by
IORING_REGISTER_ENABLE_RINGS, possibly from another task, and only the
submitter may run local work on an IORING_SETUP_DEFER_TASKRUN ring. To
the exiting task the work stays pending forever, the loop never reaches
schedule(), and the task trips the WARN_ON_ONCE() once and then spins in
do_exit().
An unprivileged user can set this up at will by creating a disabled
ring, having a child enable it and stop, posting to the ring with
IORING_OP_MSG_RING and exiting with a request in flight that only the
child can complete. The exiting thread burns a CPU until the child runs
or dies, and with panic_on_warn set the WARN panics the kernel.
Fix by skipping the sleep only when io_allowed_defer_tw_run() allows the
task to run the work. This is reasonable since work it cannot run cannot
complete any of its requests either, so nothing is lost by sleeping.
Drop the WARN_ON_ONCE() since handing a disabled ring to another thread
is a supported scenario, not a bug state.
Fixes: 360cd42c4e95 ("io_uring: optimise io_req_local_work_add")
Reported-by: AutonomousCodeSecurity@microsoft.com
Signed-off-by: Cen Zhang (Microsoft Security FORGE Labs) <cenzhang@linux.microsoft.com>
Link: https://patch.msgid.link/20261008191602.45101-1-cenzhang@linux.microsoft.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_connect_bpf_populate() fills connect.port and connect.v6_addr for
AF_INET6 only when addr_len >= sizeof(struct sockaddr_in6), i.e. 28
bytes.
The IPv6 connect paths accept SIN6_LEN_RFC2133 (24 bytes), which
already holds sin6_port and sin6_addr. io_connect() passes addr_len to
the socket layer unchanged.
At addr_len 24 the filter sees family AF_INET6 with port and address
zero, while the kernel connects to the real destination. A filter that
denies an IPv6 port or address is bypassed. Denying connect.port == P
rejects [::1]:P with -EACCES at addr_len 28 but connects at addr_len 24.
Use SIN6_LEN_RFC2133 as the threshold. sin6_scope_id stays zero when
absent.
The threshold mismatch was found by an AI coding assistant (Claude Code)
comparing this hook against the net/ipv6 length checks on my
instructions; it also drafted the reproducer and this patch. I verified
the bypass and the fix in a VM on 7.3-rc6 and that 7.2 has the same
code.
Fixes: 899bea8248ce ("io_uring/net: allow filtering on IORING_OP_CONNECT")
Cc: stable@vger.kernel.org # 7.2
Signed-off-by: Natasha Klaus <natalie.klaus@runtimeverification.com>
Link: https://patch.msgid.link/20261009133807.730724-1-natalie.klaus@runtimeverification.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A ring created with IORING_SETUP_R_DISABLED is exempt from its register
restrictions until it is enabled. This lets the creator register the
restrictions and then enable the ring with IORING_REGISTER_ENABLE_RINGS,
whether or not that opcode is in the allowed set.
Since commit ed82f35b926b ("io_uring: allow registration of per-task
restrictions"), a ring also inherits restrictions from the task that
creates it, and the same exemption applies to those. A task whose
register opcodes are restricted can therefore create a disabled ring,
issue any register opcode on it, and then enable it.
Mark rings whose restrictions were inherited from the task, and keep
checking register opcodes for them while the ring is disabled. Enabling
such a ring now requires IORING_REGISTER_ENABLE_RINGS to be in the
task's allowed set. Per-ring restrictions registered on a disabled ring
work as before.
Fixes: ed82f35b926b ("io_uring: allow registration of per-task restrictions")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Eugene Gvozdetsky <gvozdet@gmail.com>
Link: https://patch.msgid.link/20261009-iouring-task-restrict-fix-v1-1-a49daf55a12c@gmail.com
[axboe: amend with inheritance flag fixup]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Just like with the SQ and CQ rings, the provided buffer rings were also
not charged against the memlock limit prior to the below mentioned
commit. Restore that behavior.
Fixes: ef62de3c4ad5 ("io_uring/kbuf: use region api for pbuf rings")
Reported-by: Hengyu Liang <hengyul@cs.unc.edu>
Cc: stable@vger.kernel.org
Tested-by: Hengyu Liang <hengyul@cs.unc.edu>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Commit 8078486e1d53 ("io_uring: use region api for SQ") and commit
81a4058e0cd0 ("io_uring: use region api for CQ") made io_uring_setup()
allocate the rings with io_create_region().
However, io_create_region() charges the memory to RLIMIT_MEMLOCK, and
the rings had been exempt from that limit since commit 26bfa89e25f4
("io_uring: place ring SQ/CQ arrays under memcg memory limits"). As of
now, a user without CAP_IPC_LOCK gets ENOMEM from io_uring_setup() when
their rings exceed the limit, which is 8 MiB by default. PostgreSQL
developers have already hit this in their io_uring tests [1].
The issue can be reproduced with a simple liburing program, run as an
unprivileged user:
#include <liburing.h>
#include <stdio.h>
int main(void)
{
static struct io_uring ring[64];
int i;
for (i = 0; i < 64; i++)
if (io_uring_queue_init(4096, &ring[i], 0) < 0)
break;
printf("%d rings\n", i);
return 0;
}
Before those commits (v6.13), it prints "64 rings". After those commits
(v6.14), it prints "21 rings".
This patch makes io_create_region() take the user to charge, and passes
no user for the SQ/CQ rings.
Link: https://www.postgresql.org/message-id/flat/667f7dcf-404b-4898-a426-57ca6e0617f6@gmail.com [1]
Fixes: 8078486e1d53 ("io_uring: use region api for SQ")
Fixes: 81a4058e0cd0 ("io_uring: use region api for CQ")
Cc: stable@vger.kernel.org
Signed-off-by: Hengyu Liang <hengyul@cs.unc.edu>
Link: https://lore.kernel.org/io-uring/20261006125732.3425762-1-hengyul@cs.unc.edu/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Switch on attr_type_mask once, and also do away with the local variable
for attr_ptr.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20261005141954.1353351-2-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a task_work add use-after-free with SQPOLL.
The sqpoll thread could pop and complete the last request while
io_req_normal_work_add() was still looking at them after the mpscq
push.
Use the same approach as DEFER_TASKRUN to protect from that, holding
an RCU read lock across the add, and have exit wait for an RCU grace
period for SQPOLL rings as well.
- CQE32 ring fixes: correct the free entry check for 32b CQEs, zero the
big_cqe for aux CQEs, and only post the dummy skip CQE on CQE_MIXED
rings
- Mark the source filter table as COW when cloning bpf filters, so
registering another filter on the source doesn't modify the shared
table in place
- Initialize the task context before running the BPF loop
- Requeue zcrx multishot receives stopped by a local resource
- End a TX_TIMESTAMP multishot cmd when the CQ is full (lollipopkit)
* tag 'io_uring-7.3-20261002' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring: fix task_work add use-after-free with SQPOLL
io_uring/cmd_net: end TX_TIMESTAMP multishot when the CQ is full
io_uring/zcrx: requeue multishot receives stopped by a local resource
io_uring: initialize task context before running the BPF loop
io_uring: zero big_cqe for aux CQEs on CQE32 rings
io_uring: fix free entry check for 32b CQEs on CQE32 rings
io_uring: only post the dummy skip CQE on CQE_MIXED rings
io_uring/bpf_filter: mark source as COW when cloning filters
|
|
io_region_allocate_pages() first tries io_mem_alloc_compound(). If that
fails, it uses alloc_pages_bulk_node() to allocate separate pages.
The bulk allocator does not support memcg accounting. __GFP_ACCOUNT is
unconditionally supplied, so when memcg_kmem_online() is true, it only
allocates one page. For a multi-page request, io_uring treats this short
result as failure and returns ENOMEM, even when the memcg has enough
space for all pages.
When this happens, fill the rest of the request with order 0 allocations
via alloc_page(), which support memcg accounting. This maintains the
existing fast path, while fixing the incorrect ENOMEM return when memcg
is used.
Signed-off-by: David Wei <dw@davidwei.uk>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Fixes: 1e21df691ffa ("io_uring/memmap: implement kernel allocated regions")
Link: https://patch.msgid.link/20261001181752.537767-1-dw@davidwei.uk
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The hrtimer_sleeper structure's ->task field is used as a flag to indicate
that the associated hrtimer has expired. This means that the hrtimer
handler can be storing to this field while other code is loading from it
to check for expiry. Note that additional races appear for hrtimers that
can be restarted, which could be argued to be a user error. However, that
is no reason to let the compiler introduce additional confusion, and to
this end, the hrtimer_sleeper_task_get() was introduced, use of which also
has the benefit of avoiding open-code access to hrtimer_sleeper innards.
Therefore, apply this accessor to the io_hybrid_iopoll_delay() function.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260919001428.3133388-4-paulmck@kernel.org
|
|
For SQPOLL, the sqpoll thread runs task_work off its own list and
doesn't need to be told about it, so it can pop and complete a request
as soon as mpscq_push() has published it. If that was the last request
holding the ring alive, io_ring_exit_work() then frees the ctx, stops
the thread, and with it the tctx, while io_req_normal_work_add() is
still looking at both after the push:
BUG: KASAN: slab-use-after-free in io_req_normal_work_add+0x439/0x510
Read of size 4 at addr ff11000000ca0000 by task repro/55
(...)
BUG: KASAN: slab-use-after-free in queue_work_on+0x25/0x70
Write of size 8 at addr ff11000000c3bd00 by task repro/55
Hold an RCU read lock across the add, like io_req_local_work_add() already
does, and have io_ring_exit_work() wait for a grace period for SQPOLL rings
as well before freeing the ctx.
Reported-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://lore.kernel.org/io-uring/20260924205056.3759980-2-Jeremy.Jean@oss.cyber.gouv.fr/
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
With cancelations now run at release time, what's left in-flight on
the ring afterwards is mostly I/O that has already been issued to a
device and just needs to finish. Until that happens, the files from
those requests pin the files they were using. Wait for those.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Registered file and buffer tables are only torn down from
io_ring_ctx_free(), which only runs when all requests have completed and
the ctx refs have hit zero. Before this, every registered file remains
open.
The resource nodes are reference counted and any request using a
registered file or buffer holds its own reference to the node, so this
only releases what is actually unused. The rest follow as those requests
complete or get canceled.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_uring_release() just marks the ring as dying and punts everything
else to exit_work, including the cancelation of requests that are easily
cancelable right away Until that work has run, and the task_work it
generates has as well, those requests keep their files pinned. An
application that closes its ring and then expects the files it had been
using to be closed as well may be confused by this currently not being
the case.
Re-factor the cancelation pass out of io_ring_exit_work() and run it
directly from release, before punting the rest.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
When a task exits, io_uring only cancels and waits for the few request
types that must not outlive it, and leaves everything else to be torn
down whenever the ring itself goes away. This happens from a kernel
workqueue, and may be some time after the final fput of the ring has
been completed. This sometimes causes application issues, where a task
that was doing IO on a file in /mnt, for example, will leave /mnt busy
for a brief period of time after close(ring_fd) is done.
If the whole thread group is exiting, no thread is left to reap
completions or care about those requests. For this case, tear everything
down. This is basically what exec does today.
Two types of requests are ignored, as they never pin any files and will
always complete on their own. One is armed timeouts, which may still be
required to trigger if a ring is being shared, and the other is SEND_ZC
notifications, which can take almost an unbounded time to complete.
io_uring_try_cancel_requests() now takes flags to manage that behavior.
While in there, fix up the wait loop for the exit case. A task that is
exiting because of a fatal signal still has that signal pending, which
turns the interruptible sleep into a busy loop. Use short
uninterruptible sleeps in that case instead, task_work still gets run in
between.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
SEND_ZC notifications are io_kiocbs like any other as far as request
and task accounting goes, but they are special in that they cannot be
canceled and only complete once the network stack is done with the
data. With an unresponsive peer, that can take a while.
The exit and release paths are about to start waiting for requests to
finish so that the files they pin get released in a timely manner, and
notifications neither pin files nor finish in a timely manner. Keep a
per-ring count of them so those paths can leave them out.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_uring_try_cancel_uring_cmd() ignores the task it is asked to cancel
on behalf of as soon as cancel_all is true, and cancels every cancelable
command on the ring instead. That differs from how other requests are
matched, where a valid task always restricts matching to that task's
requests and cancel_all simply lifts the REQ_F_INFLIGHT restriction.
So far the only caller passing both a task and cancel_all is exec, where
tearing down another task's commands on a shared ring is somewhat
impolite. The exit path is about to do the same for every exiting
process though, so bring it in line. Match on the task if one is given,
and cancel everything only for ring teardown, which passes NULL. This
leaves cancel_all as useless, hence drop it.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The batch flush posts the CQEs first and drops the files afterwards,
with a plain fput() that defers the release to task_work. For the
submitting task that runs on the way back to userspace, but with SQPOLL
the sqpoll thread only gets to it after the io_uring task_work is done.
Userspace sees the CQE in between, and closing the file and exec'ing
it fails with ETXTBSY as the request still holds it.
Drop the files synchronously before the CQEs are posted, for requests
that the flush is about to free. Ring files stay deferred, releasing
one takes its uring_lock and may wait on its requests.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
IOPOLL requests issued from io-wq are reaped by io_do_iopoll() on the
submitter, with the same window as the task_work completions: the CQE
is posted while the worker may still hold its reference, and the file
is only dropped once it does.
Leave such a request on the iopoll list until the worker has dropped
its reference, the next pass then reaps it and the put from
io_free_batch_list() is the last one. Also move the reference handling
for the task_work case into a helper.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io-wq holds a reference on a request for the duration of the issue, as
the completion may run before the issue returns. If it does, the CQE is
posted from that completion while the worker still holds its reference,
and the file is only put once the worker drops it. That means the actual
file put happens potentially long after CQE posting.
Post the completion with the last put for refcounted requests.
io_free_req() no longer marks the request as CQE_SKIP, that is now done
by whoever posted the CQE while another reference was still held.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Once io-wq is done with the request, put any potential file that request
had pinned upfront. This ensures any file references are fully put by
the time io-wq posts a completion. Registered files are held by the
resource node and don't need this.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
SOCKET_URING_OP_TX_TIMESTAMP arms an EPOLLERR apoll multishot and, on
each error queue edge, posts one 32 byte aux CQE per timestamp skb. Aux
CQEs have no overflow backing, so io_uring_cmd_post_mshot_cqe32() fails
once the CQ ring is full. The loop then stops, splices the unprocessed
skbs back onto sk_error_queue and returns -EAGAIN.
-EAGAIN is IOU_RETRY, so the request goes idle until the next poll
event. io_arm_apoll() forces EPOLLET and draining the CQ does not re-run
the command, so the spliced-back timestamps stay queued until an
unrelated later timestamp produces a new edge.
End the multishot when a CQE cannot be posted, as multishot poll and
recv already do. There is no partial result to report, so complete with
-ENOBUFS: no CQ space is left for the timestamp CQEs. This command does
not use provided buffers, so the value cannot be confused with a buffer
ring running empty. The terminal CQE still reaches userspace because
request completions can go to the overflow list. The application sees a
CQE without IORING_CQE_F_MORE, which it already has to handle, and the
first issue of the re-armed request delivers the queued timestamps.
A timestamp that cannot be extracted keeps its current behaviour.
This was found with LLM assistance while reviewing io_uring multishot
handlers for the pattern behind a RECV_ZC stall reported earlier (see
Link), and confirmed with a reproducer on v7.3-rc2 and an A/B run of
this patch.
Fixes: 9e4ed359b8ef ("io_uring/netcmd: add tx timestamping cmd support")
Link: https://lore.kernel.org/all/010001a0cd914529-23df2f94-bbe0-49cd-ae5d-05922356fa12-000000@email.amazonses.com/
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-6-sol
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: lollipopkit <a@lolli.tech>
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/20260927074206.65427-1-a@lolli.tech
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A multishot RECV_ZC can go idle with unread TCP data when a
receiver-local resource runs out. An empty copy-fallback niov freelist
returns -ENOMEM; a full CQ returns -ENOSPC. After partial progress the
stop reason is lost: io_zcrx_copy_chunk() and io_zcrx_recv_skb() report
the copied bytes, and tcp_read_sock() drops an error from a later call
once earlier data was consumed. The edge-triggered request is then not
requeued, so it cannot consume the remaining data until another socket
event arrives.
Record in io_zcrx_recv_skb() when a walk stops before consuming the
length it was offered, and requeue if the pass still returned data. The
failed chunk is first on the next pass. The resource is usually still
exhausted by then, so that pass fails at once and ends the request with
an error CQE, which the application must handle by re-arming. The
request does not spin, and the stall becomes a visible error. Keep the
existing skb limit and SOCK_DONE handling.
Fixes: 11ed914bbf94 ("io_uring/zcrx: add io_recvzc request")
Fixes: bc57c7d36c4c ("io_uring/zcrx: add copy fallback")
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-6-sol
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Junyuan Feng <a@lollipopkit.com>
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/010001a0cd914529-23df2f94-bbe0-49cd-ae5d-05922356fa12-000000@email.amazonses.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io-wq will hold acct->lock, which is a raw spinlock, over a call to
wake_up(), which uses a normal spinlock. This isn't valid nesting under
CONFIG_PREEMPT_RT, as normal spinlocks are sleeping locks there, while
raw spinlocks operate like classic spinning locks. This triggers a
kernel warning:
BUG: sleeping function called from invalid context at kernel/locking/spinlock_rt.c:46
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 985654, name: iou-wrk-983605
preempt_count: 1, expected: 0
RCU nest depth: 0, expected: 0
CPU: 3 PID: 985654 Comm: iou-wrk-983605 Tainted: G O 6.1.83-rt28-g19631eb82f21
stack:0 ppid:977479 flags:0x00000008
tgid:977553 prio:120 preempt:0x100000001 rcu_read_lock_nesting:0
used_cpu 3 wake_cpu 3 on_cpu 3 on_rq 1 migrate_dis 0
arrive:17423426040675 queued:0 prev_sum:33955300 sum_exec:33955300
Call trace:
dump_backtrace.part.0+0xdc/0xec
show_stack+0x1c/0x30
dump_stack_lvl+0xac/0xc4
dump_stack+0x14/0x30
__might_resched+0x13c/0x170
rt_spin_lock+0x34/0xc0
__wake_up_common_lock+0x68/0xd0
__wake_up+0x1c/0x24
io_worker_handle_work+0x5b0/0x600
io_wqe_worker+0xf4/0x310
ret_from_fork+0x10/0x20
Store whether we need to call wake_up(), and call it safely outside of
the acct->lock.
Signed-off-by: Xiaochuan Li <chuanx2070@163.com>
Link: https://patch.msgid.link/20260923172032.11057-1-chuanx2070@163.com
[axboe: rewrite commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Submitting SQEs through an io_uring BPF loop can trigger a NULL pointer
dereference in io_submit_sqes(), as shown by the following arm64 report:
[ 2049.380301] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 2049.388788] pc : io_submit_sqes+0x48/0x6c4
[ 2049.389153] lr : bpf_io_uring_submit_sqes+0x10/0x1c
[ 2049.396310] Call trace:
[ 2049.396531] io_submit_sqes+0x48/0x6c4 (P)
[ 2049.396893] bpf_io_uring_submit_sqes+0x10/0x1c
[ 2049.397293] bpf_prog_1a2a95cb199202b5+0x3c/0x68
[ 2049.397699] bpf__io_uring_bpf_ops_loop_step+0x5c/0x94
[ 2049.398149] io_run_loop+0x70/0x2e4
[ 2049.398461] __arm64_sys_io_uring_enter+0xf0/0x710
[ 2049.398882] invoke_syscall+0x54/0x10c
[ 2049.399414] el0_svc_common.constprop.0+0x40/0xe0
[ 2049.399829] do_el0_svc+0x1c/0x28
[ 2049.400125] el0_svc+0x38/0x1d0
[ 2049.400409] el0t_64_sync_handler+0xa0/0xe4
[ 2049.400779] el0t_64_sync+0x198/0x19c
io_uring_enter() dispatches to io_run_loop() before reaching the normal
submission path's io_uring_add_tctx_node() call. The loop only checks
whether the caller is allowed to run task work; it does not initialize
current->io_uring or associate the task with the ring. A BPF call to
bpf_io_uring_submit_sqes() then reaches io_get_task_refs(), which assumes
that current->io_uring is valid.
A ring created with IORING_SETUP_R_DISABLED, IORING_SETUP_SINGLE_ISSUER
and IORING_SETUP_DEFER_TASKRUN can be enabled by a different task that
has never used io_uring. Enabling the ring makes that task the submitter
without allocating its io_uring task context. Its first BPF-driven
submission of a pending SQE can therefore cause a kernel Oops.
Call io_uring_add_tctx_node() in io_run_loop() before invoking the loop
and propagate any initialization error. Do this before acquiring
uring_lock, since task context setup may acquire the same mutex. This
also establishes the task-to-ring association used for cancellation.
Fixes: 033af2b3eb19 ("io_uring: introduce callback driven main loop")
Signed-off-by: Yao Kai <yaokai34@huawei.com>
Link: https://patch.msgid.link/20260923014310.591292-1-yaokai34@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
FD_PREPARE() releases the fd as soon as the file expression fails. The
fd either holds the descriptor or the error. The separate field is
redundant. Make fd and file plain members and struct fd_prepare simpler.
Callers of FD_PREPARE() get a const pointer to the guard. Simplify all
users as they can now easily access the trivial struct.
No functional changes.
Link: https://patch.msgid.link/20260917-work-file-fd_prepare-v1-4-b87534ca49f3@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
io_wq_exit_workers() cancels task work for queued worker creation with
task_work_cancel_match(). That has a plain load of task->task_works.
io_queue_worker_create() adds the new entry and tests IO_WQ_BIT_EXIT to
cancel it if the workqueue is already on its way out.
That must be ordered. task_work_add() has a full barrier via cmpxchg().
But io_wq_exit_start() sets the bit with set_bit() which doesn't have
any memory ordering.
So afaict, on weakly ordered architectures the exiting task may load
task_works before the store of the bit is visible. The other side tests
the bit before the store is visible as well. So the entry remains queued
with worker_refs and the exiting task waits on worker_done indefinitely.
If the task still has rings then io_uring_del_tctx_node() provides the
barrier via test_and_set_bit() in io_wq_set_exit_on_idle(). When it
has closed all rings though that barrier is gone.
Add the barrier after the set_bit().
Fixes: 71a85387546e ("io-wq: check for wq exit after adding new worker task_work")
Cc: stable@vger.kernel.org
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260921-work-coredump-fixes-v3-4-8e4adb1619e6@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Compound pages in registered buffers are accounted per ring, tracked in
ctx->hpage_acct and charged when a ring takes its first reference to a
given page. Cloning a buffer takes that reference in the destination ring
without charging for it, and io_buffer_unmap() only uncharges when it
drops the last reference to the buffer itself. Those two don't line up
once a ring drops its last reference to a compound page without also
being the last user of the buffer, and the charge for it never gets
dropped.
Charge compound pages on clone when they're new to the destination ring,
and uncharge them on unmap whenever the ring's reference count for the
page drops to zero, regardless of whether it's the last user of the
buffer. Regular pages are only charged by the registering ring, so those
still get uncharged by whoever releases the buffer last.
Fixes: df0a52537c0f ("io_uring/rsrc: add huge page accounting for registered buffers")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919221725.3706704-1-benquike@gmail.com
[axboe: split from original patch, and correct commit message, there is
no double un-accounting here, just a leak of the accounting]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Posting the dummy skip CQE at the end of a CQE_MIXED ring needs 3 free
slots, one for the dummy and two for the 32b CQE it makes room for. With
fewer than that, the dummy gets posted and the 32b allocation then fails
right after and goes to overflow, having wasted a slot for nothing.
Check up front that there's room for both.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A cqe32 entry spans two CQ array slots, so the last CQ array slot can't
contain a cqe32 entry. If the CQ tail points at the last CQ array slot
and the kernel wants to write a cqe32 entry, it uses io_fill_nop_cqe()
to pad the last CQ array slot with a dummy entry and make the tail wrap
around.
However, malicious userspace can directly set IORING_CQE_F_32 on the
last CQ array slot, causing __io_uring_show_fdinfo() to read the second
cqe32 half from beyond the CQ array. Change __io_uring_show_fdinfo() to
explicitly ignore the IORING_CQE_F_32 flag in this case.
This is not a real bugfix, just tightening the code a bit, because:
1. the number of CQE slots is always a power of 2, see
io_uring_fill_params
2. the ring_region region consists of:
- a 64-byte header
- pow(2, N) CQE slots (each 0x10 bytes)
- optionally, the SQ array
3. the ring_region size must be page-aligned because it is shared memory
Together, these properties imply that the last CQE slot can't be close
before the end of a page, so the "out-of-bounds" data is in memory that
is anyway accessible to userspace.
Reported-by: Dominik Maier <dmnk+artist@google.com>
Fixes: 82ceb7fcc5ff ("io_uring/fdinfo: handle mixed sized CQEs")
Cc: stable+noautosel@kernel.org # no impact due to memory layout
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Signed-off-by: Jann Horn <jannh@google.com>
Link: https://patch.msgid.link/20260911-uring-fdinfo-tighten-v2-1-fa24d517035e@google.com
[axboe: fix wrap condition check]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Merge io_uring patches from the 7.3 kernel release, as some later
patches in this branch will depend on them.
* io_uring-7.3:
io_uring: zero big_cqe for aux CQEs on CQE32 rings
io_uring: fix free entry check for 32b CQEs on CQE32 rings
io_uring: only post the dummy skip CQE on CQE_MIXED rings
io_uring/bpf_filter: mark source as COW when cloning filters
io_uring/rw: keep CQE flags on iopoll requests when adding kbuf flags
io_uring/net: don't overconsume buffers when using MSG_TRUNC
io_uring/net: let io_recv_buf_select return the length of the buffer region
io_uring/rw: end write accounting from ->ki_complete
|
|
io_fill_cqe_aux() only clears big_cqe if cqe32 is set, which is only the
case for CQE_MIXED. On a CQE32 ring every CQE is 32b, and the extra 16b
are left holding whatever was in that slot before.
Clear it for CQE32 rings as well, like it was before CQE_MIXED support
was added.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
On CQE32 rings, len is in units of 32b CQEs when io_cqe_cache_refill()
checks for free entries, so requiring cqe32 + 1 of them means insisting
on 2 free CQEs where 1 is enough, and a ring with a single free entry
needlessly goes to overflow.
Check for a free CQE directly in that case, and keep the cqe32 + 1 check
for CQE_MIXED where the units are 16b.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_cqe_cache_refill() posts a dummy skip CQE when a 32b CQE would
straddle the end of the ring, so the real one can start at offset 0. That
only makes sense on CQE_MIXED rings, where slots are 16b and a 32b CQE
needs two of them back to back. On a CQE32 ring every slot is already
32b, but io_fill_cqe_aux32() still passes cqe32 == true, and the dummy
then gets written at cqes[off] rather than cqes[off << 1], inside another
CQE.
Limit the dummy to CQE_MIXED rings.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_bpf_filter_clone() shares the filter table between source and
destination and marks the destination as needing a copy before it can be
modified, but leaves the source as is. Registering another filter on the
source then modifies the shared table in place, and the change shows up
in everything that cloned it.
Mark the source as COW as well.
Fixes: ed82f35b926b ("io_uring: allow registration of per-task restrictions")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919112523.3872581-1-benquike@gmail.com
[axboe: edited patch and commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a deadlock in the write path with superblock freezing
- Fix an issue where a provided buffer ring would overconsume when
using MSG_TRUNC
- Keep the CQE flags on iopoll requests when adding kbuf flags
* tag 'io_uring-7.3-20260911' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/rw: keep CQE flags on iopoll requests when adding kbuf flags
io_uring/net: don't overconsume buffers when using MSG_TRUNC
io_uring/net: let io_recv_buf_select return the length of the buffer region
io_uring/rw: end write accounting from ->ki_complete
|
|
In preparation for setting the io_uring NVMe passthru CQE results from
the blk-mq request completion rather than the task work callback, split
out functions io_uring_cmd_set_res{,32}() from __io_uring_cmd_done().
io_uring_cmd_done{,32}() now call io_uring_cmd_set_res{,32}() and then
__io_uring_cmd_done(). This allows __io_uring_cmd_done() to be made
CQE-size-agnostic, with 3 fewer arguments.
Signed-off-by: Caleb Sander Mateos <csander@purestorage.com>
Reviewed-by: Joanne Koong <joannelkoong@gmail.com>
Link: https://patch.msgid.link/20260909155848.2069290-3-csander@purestorage.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
