| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- A batch of fixes from Chris:
- Charge registered buffer arrays, rsrc nodes, tctx nodes and
personalities to memcg, and free the XArray nodes a failed store
leaves behind.
- Clear bpf_ctx while submitting SQEs from a bpf-ops program, as the
socket code otherwise skips both the capability check and
lock_sock() for a setsockopt from the SQ.
- Free the task context after an RCU grace period, closing a UAF
when a foreign task frees the last request.
- Stop io_ring_exit_work() from waiting on an SQPOLL thread held in
get_signal(), and drop sqd->lock while the thread waits for
in-flight requests, as either one could pin every iou_exit slot
- io-wq fixes:
- Don't create new workers once the exit has begun, which could
leave an exiting worker in the thread group under exec
- save the worker type before dropping its reference (KASAN UAF)
- Defer io-wq enqueues from a foreign task to task_work, as an
IOPOLL reap by another task could race io_wq_put_and_exit()
- Keep the drain active when the deferred request being started is
itself a drain request, and don't attach a new ring to an SQPOLL
thread that is already canceling
- Fix the multishot recv length limit accounting, and count cancelled
timeouts in cq_timeouts so an armed off timeout doesn't fire early
- Advance the buffer ring head when a pickup empties it, and keep the
recvmsg buffer select length within MAX_RW_COUNT
- Mask the socket type in the bpf filter population so SOCK_CLOEXEC or
SOCK_NONBLOCK can't bypass a type filter, and enforce the per-task
register restrictions for blind registration
- Complete a non-blocking connect() that fails with -EAGAIN rather than
retrying it through poll and io-wq, matching connect(2)
- napi list traversal cleanup, and remove a useless min_not_zero()
* tag 'io_uring-7.4-20261029' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (21 commits)
io_uring/sqpoll: don't attach to a thread that is canceling
io_uring/net: fix multishot recv length limit accounting
io_uring/timeout: count cancelled timeouts in cq_timeouts
io_uring/napi: clean up list traversal with list_for_each_entry_safe()
io_uring: defer foreign io-wq enqueues to task_work
io_uring/io-wq: save worker type before dropping its reference
io_uring/cancel: drop sqd->lock while SQPOLL waits
io_uring/sqpoll: don't wait for the thread to exit
io_uring: free the task context after a grace period
io_uring/bpf-ops: clear bpf_ctx while submitting SQEs
io_uring: free XArray nodes left by a failed store
io_uring: charge tctx nodes and personalities
io_uring/rsrc: charge buffer arrays and nodes to memcg
io_uring/io-wq: no new workers once the exit has begun
io_uring: keep the drain active when a deferred drain request is started
io_uring/net: don't retry a non-blocking connect() that fails with -EAGAIN
io_uring/register: enforce per-task restrictions in io_uring_register_blind()
io_uring/net: mask socket type with SOCK_TYPE_MASK in io_socket_bpf_populate()
io_uring/net: Remove useless min_not_zero() usage
io_uring/net: keep recvmsg buffer select length within MAX_RW_COUNT
...
|
|
io_sq_offload_create() refuses to attach a ring to an SQPOLL thread
that has already gone, by checking sqd->thread under sqd->lock. Since
the thread drops the lock while io_uring_cancel_generic() waits for its
requests, that check can run while the thread is on its way out and
still see it. The new ring is then added to ctx_list, io_uring_setup()
succeeds, the thread finishes canceling and exits without ever having
looked at the ring, and io_uring_enter() fails with -EOWNERDEAD.
Only a thread that exits on a fatal signal can be caught like this, as
the stop bit is set once the last ring has let go of the sqd, when no fd
is left to attach through. A fatal signal is group wide and attaching
needs the same thread group, so the attacher is on its way out as well.
Still, make the check see it: set the stop bit before canceling,
whatever the reason for the exit, and test it when attaching.
Reported-by: Karl Mehltretter <kmehltretter@gmail.com>
Fixes: cb0306c52ffa ("io_uring/cancel: drop sqd->lock while SQPOLL waits")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_recv_finish() debits sr->mshot_total_len with min_t(int, sel->val,
sr->mshot_total_len). sel->val is the cumulative byte count of the
request: on a bundle retry the receive path adds sr->done_io back to the
return value, and the bundle branch in io_recv_finish() recovers the new
bytes with this_ret = sel->val - sr->done_io. The debit therefore uses
the wrong value after a bundle retry, and the int cast truncates a limit
above INT_MAX.
For an IORING_OP_RECV multishot request with optlen above INT_MAX, the
limit becomes negative when cast to int, so the first completion debits
the entire limit and ends the multishot with IORING_RECV_MSHOT_DONE
while the socket still holds data that covers the limit. With 120 bytes
queued, a 64-byte buffer and optlen 0x80000000, the first 64-byte CQE
lacks IORING_CQE_F_MORE and 56 bytes remain unread; the same setup with
optlen 0x7fffffff posts 64 and 56 bytes with F_MORE.
After a bundle retry, the bytes already debited for the previous
attempts are debited again, so the limit is exhausted early. With 1400
bytes queued, optlen 1000 and buffers of 600, 100, 100, 50, 50 and 200
bytes, the first attempt delivers 700 bytes and retries; the second
delivers 200 more, but the debit accounts for 900 and finishes the
request with a terminal 900-byte CQE while 500 bytes remain unread. The
limit could have been reached.
Debit only the bytes received by this invocation, sel->val -
sr->done_io, and compare as unsigned so a limit above INT_MAX is
accounted correctly. Non-bundle multishot requests keep sr->done_io at
zero, so their accounting is unchanged.
Fixes: 0ebc9a7ecf6a ("io_uring/net: Support multishot receive len cap")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Cheng Yongkang <teel4res@gmail.com>
Link: https://patch.msgid.link/179328314078.648427.10086632105828535293@gmail.com
[axboe: fix-up near-by incorrect comment while in there]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_timeout_cancel() posts the cancelled timeout's -ECANCELED CQE without
advancing ctx->cq_timeouts. io_timeout_fn() and io_kill_timeout()
advance it for every timeout CQE they post, and io_flush_timeouts()
computes the number of non-timeout completions as
seq = cached_cq_tail - cq_timeouts
to compare against each off timeout's target_seq. The cancelled
timeout's CQE is therefore counted as a regular completion, so every
cancellation makes an already armed off timeout fire one completion
early.
A ring with a keep-alive timeout of off=4 completes it after two NOPs
when one off=100 victim is cancelled through IORING_OP_ASYNC_CANCEL, and
before any NOP when two victims are cancelled. The cancel CQE should
count, but the cancelled timeouts' -ECANCELED CQEs must not: the
expected counts are three and two NOPs respectively.
Advance cq_timeouts when io_timeout_extract() succeeds, under
timeout_lock and before the -ECANCELED completion is queued, matching
io_timeout_fn() and io_kill_timeout(). The -ENOENT and -EALREADY paths
post no new timeout CQE and keep their accounting. Both the
ASYNC_CANCEL and the TIMEOUT_REMOVE path go through io_timeout_cancel().
Fixes: bfe68a221905 ("io_uring: off timeouts based only on completions")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Cheng Yongkang <teel4res@gmail.com>
Link: https://patch.msgid.link/179328314078.648427.13761781546953594668@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Switch both loops to the standard list_for_each_entry_safe() pattern and
remove the explanatory comment.
Signed-off-by: Anvik Loitongbam <permissiontogallop@gmail.com>
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
Link: https://patch.msgid.link/20261029144745.23792-1-permissiontogallop@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
An IOPOLL ring without DEFER_TASKRUN can be reaped by a task other than
the one that submitted the I/O. If a polled request is marked
REQ_F_REISSUE, io_free_batch_list() calls io_queue_iowq() with the
original tctx. Ordinary task exit can tear down tctx->io_wq after
draining only the tracked requests. The live request pins the task, but
not its io-wq, so the foreign task can race io_wq_put_and_exit() and
enqueue through a freed or NULL queue.
Forward foreign userspace enqueues to the owning task before accessing
tctx->io_wq. Use a dedicated task_work callback so reissues still go
straight to io-wq, and fail them if task_work is running in cancellation
mode. Preserve the existing PF_KTHREAD cancellation path. This also
covers the other callers of io_queue_iowq() without changing the
worker-owned enqueue path.
Signed-off-by: Kyle Zeng <kylebot@openai.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_wq_cancel_tw_create() inspects cb->func after calling
io_worker_cancel_cb(). For an existing worker, that call can drop the
last worker reference and wake io_worker_exit(). If the canceling task
is preempted, the worker can be unlinked and freed by kfree_rcu() before
the callback function is inspected. KASAN reports a slab-use-after-free
read in this path.
Save whether the callback belongs to an unpublished continuation worker
while its reference is still held, then use the saved result for the
conditional free. This preserves the existing accounting and frees only
the worker allocated for the canceled continuation.
Fixes: e6db6f9398da ("io_uring/io-wq: only free worker if it was allocated for creation")
Cc: stable@vger.kernel.org
Signed-off-by: Kyle Zeng <kylebot@openai.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
An exiting SQPOLL thread calls io_uring_cancel_generic() with sqd->lock
held, and keeps it across cond_resched() and the sleep until nothing is
in flight anymore. A zero-copy send notification can't be canceled and
stays pending as long as the peer doesn't read, so the lock is held for
as long as a user cares to. io_ring_exit_work() needs it to park the
thread, and so do IORING_REGISTER_IOWQ_AFF and IOWQ_MAX_WORKERS. With
64 such rings sharing the thread, every iou_exit slot sleeps on the lock
and no ring gets freed for anyone.
Drop the lock around cond_resched() and the sleep, like the main loop of
the thread does. Each walk of ctx_list is still under one hold, and
nothing is kept across the unlock. The thread drops its io-wq, frees its
tctx and clears sqd->thread under one hold after the loop, so whoever
holds the lock and finds sqd->thread set can still use them.
Fixes: 63809137ebb5 ("io_uring: flush notifiers after sendzc")
Signed-off-by: Chris Mason <mason@kernel.org>
[axboe: adapted to the schedule_timeout() based loop]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_sq_thread_stop() wakes the SQPOLL thread and waits for it to exit.
The thread belongs to the thread group that created the ring, so a
SIGSTOP or the cgroup freezer holds it in get_signal(), and the wake up
doesn't change that. io_ring_exit_work() then sleeps in that wait until
someone continues or kills the process. iou_exit has 64 slots, so 64
closed SQPOLL rings of a stopped process keep every other ring from
being freed, for everyone.
Don't wait. Set the stop bit, wake the thread, and let it exit when it
gets to run again. The sqd can now outlive its last ring, so give it a
second refcount with one reference for the rings and one for the thread,
and free it with the last one. The thread can't reach a freed ring,
rings leave ctx_list before dropping their reference, and the io-wq hash
is refcounted.
Fixes: 16efa4fce3b7 ("io_uring: allow IO worker threads to be frozen")
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A task other than the owner can free a request, io_ring_exit_work()
running the local task_work of a DEFER_TASKRUN ring for example, or
io_do_iopoll() reaping completions. io_put_task() then takes the else
branch:
percpu_counter_sub(&tctx->inflight, 1);
if (unlikely(atomic_read(&tctx->in_cancel)))
wake_up(&tctx->wait);
put_task_struct(tctx->task);
The first line is what lets the owner leave io_uring_cancel_generic(),
and on exec it frees the tctx right after. Nothing holds that off for
the remaining three lines, the task reference only covers a tctx freed
along with the task_struct. It used to work out because
io_uring_del_tctx_node() took the uring_lock that the freeing side holds
across io_put_task(), until that moved to tctx_lock.
Free the tctx with kfree_rcu(), and enter an RCU read section in
io_put_task() before the count is dropped.
Fixes: 5623eb1ed035 ("io_uring/tctx: add separate lock for list of tctx's in ctx")
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
bpf_io_uring_submit_sqes() calls io_submit_sqes() from inside the loop
program, with current->bpf_ctx set, and requests issued inline run with
it set too. The socket code uses has_current_bpf_ctx() to recognize
calls from bpf_setsockopt() and friends, where the caller holds the
socket lock and has already been vetted, and skips both.
So a SOCKET_URING_OP_SETSOCKOPT from the SQ passes sockopt_capable()
without the capability, and runs without lock_sock(). A task with no
capability can set SO_MARK, IP_TRANSPARENT or TCP_REPAIR, and two
threads changing IP_OPTIONS on one socket, one through the ring, double
free in no time. SOCK_DESTROY via IORING_OP_SENDMSG ends up in
tcp_abort() without the lock the same way.
Clear the run context around io_submit_sqes(). The SQEs come from
userspace and must be issued like those of any other ring. They still
run inside the program's RCU tasks trace read section, so a request
that sleeps there holds up that grace period, but other request types
do that already.
Fixes: 890819248a86 ("io_uring/bpf-ops: add kfunc helpers")
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A failed xa_store() can leave nodes without entries in the tree, and
erasing every entry doesn't free those. An XArray needs xa_destroy()
before it goes away, even an empty one. That was dropped for tctx->xa in
commit ca6484cd308a ("io_uring: no need to call xa_destroy() on empty
xarray") over a lockdep report from freeing the tctx in softirq context,
and ctx->personalities never had it.
Now that the nodes are charged, a store can fail at memory.max and the
leaked nodes stay charged. Call xa_destroy() in io_uring_clean_tctx(),
which the task runs itself in process context before the tctx is freed,
in the error path of __io_uring_add_tctx_node(), and for
ctx->personalities in io_ring_ctx_free().
Fixes: ca6484cd308a ("io_uring: no need to call xa_destroy() on empty xarray")
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The io_tctx_node allocated the first time a task touches a ring, and
the XArray nodes behind tctx->xa and ctx->personalities, are plain
GFP_KERNEL. tctx->xa is indexed by the ring pointer, so nearly every
entry needs nodes of its own, and a task gets one entry per ring it has
used. 642 threads on 4000 rings held 2.3GB with 59MB charged against a
64MB memory.max.
Allocate the node with GFP_KERNEL_ACCOUNT and set XA_FLAGS_ACCOUNT on
both XArrays.
Fixes: 0f2122045b94 ("io_uring: don't rely on weak ->files references")
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The io_mapped_ubuf of a registered buffer has one bio_vec per page,
unless the pages coalesce into huge pages. Huge pages only count against
RLIMIT_MEMLOCK once per ring, so a buffer that maps the same 2MB folio
over and over in a way that defeats coalescing gets a full bvec array
per registration while only the one folio counts as locked. 1GB of that
registered in 100 slots took several times a 64MB memory.max, with 2.6MB
charged.
Allocate the io_mapped_ubuf and the io_rsrc_node with
GFP_KERNEL_ACCOUNT. An object reused from the ring caches stays charged
to whoever allocated it first, same as the io_kiocb cache.
Fixes: de2939388be5 ("io_uring: improve registered buffer accounting for huge pages")
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_wq_exit_start() sets IO_WQ_BIT_EXIT, but worker creation task_work
queued before that stays queued until io_wq_exit_workers() cancels it.
In between the task runs its task_work while canceling requests, and a
worker can still get created.
That's a problem for exec. begin_new_exec() calls io_uring_task_cancel()
after de_thread(), when it's supposedly the only thread left, and
io_wq_exit_workers() only waits for ->worker_done, which a worker
completes before do_exit(). So the exiting worker is still in the thread
group, using the mm, files and sighand, while exec replaces them. Easy
to hit with a signal pending when create_io_thread() fails and the retry
gets queued right before execve().
Cancel the queued creation task_work as soon as the exit bit is set.
Pending work that isn't otherwise canceled now ends with -ECANCELED when
the io-wq is destroyed, rather than depending on whether the task
happened to run its task_work first.
Fixes: 685fe7feedb9 ("io-wq: eliminate the need for a manager thread")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
[axboe: the barrier is already upstream, see 0e5663856b69, only the
cancel is new]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
ecb6817b769f ends the drain when the dispatched drain request has posted
its CQE and nothing else is deferred. But the deferred list is processed
in between, and if the request it starts is itself a drain request, the
drain is over before that one has even run. A request submitted after
that point then runs ahead of it:
1. P0 in flight, D1 and D2 (both IOSQE_IO_DRAIN) deferred behind it
2. P0 completes, D1 is started
3. D1 completes, drain_done is set, D2 is started, the list is empty
4. IO_RING_F_DRAIN_ACTIVE is cleared with D2 still in flight
Anything submitted now bypasses the drain. The same happens with D2 as
the head of a link, as those aren't tracked in ->drain_req at all.
Have io_queue_deferred() report whether it started a drain request, and
don't end the drain if it did. Starting non-drain requests and ending it
is still fine, they were only waiting on the drain request that just
completed.
Fixes: ecb6817b769f ("io_uring: end the drain when the drain request completes")
Reported-by: Karl Mehltretter <kmehltretter@gmail.com>
Link: https://lore.kernel.org/io-uring/20261027043323.98916-1-kmehltretter@gmail.com/
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring dma-buf support from Jens Axboe:
"This adds native dma-buf support to the I/O path, exposed through
io_uring. It's been in development for a while and should be ready to
go now.
- dma-buf file I/O infrastructure: a dma_buf_io_ctx created through a
new ->init_dma_buf_io_ctx file operation, and a dma_buf_io_map
holding the device DMA mapping. The two are split so a mapping can
be invalidated and re-established underneath a live context, with
the map counting in-flight requests so it can be torn down safely.
- A new iov_iter type for dmabuf maps, and a dma map backed bio type
for the block layer, with raw block device support on top.
- nvme-pci backend implementing dma-buf backed requests, including
SGL support.
- io_uring dmabuf backed registered buffers, built on a reworked
buffer registration structure with an extended update path, regbuf
import flags, an uncloneable flag for buffer types that can't be
cloned, and cloning support for registered dma-bufs"
* tag 'for-7.4/io_uring-dmabuf-20261023' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/dma-buf: don't fail fixed buffers without dma-buf support
nvme-pci: detach dma-bufs only on removal
block: reject dma-bufs larger than 4GiB
io_uring/dma-buf: allow cloning registered dma-bufs
dma-buf-io: document dma_buf_io_init_map
dma-buf-io: fix typos in kernel docs
block: fix indention for dma-buf checks
io_uring/rsrc: add dmabuf backed registered buffers
io_uring/rsrc: add regbuf import flags
io_uring/rsrc: add uncloneable regbuf flag
io_uring/rsrc: extend buffer update
io_uring/rsrc: introduce buf registration structure
nvme-pci: add SGL support for the dmabuf path
nvme-pci: implement dma-buf backed requests
nvme-pci: rename nvme_pci_sgl_set_data to nvme_pci_dma_iter_set_sgl
block: add dma-buf support for raw bdev
block: introduce dma map backed bio type
block: always adjust bi_offset on bio_advance_iter
iov_iter: add iterator type for dmabuf maps
dma-buf: introduce initial file I/O infrastructure
|
|
IORING_OP_CONNECT on an O_NONBLOCK AF_UNIX stream socket retries a
connect that fails with -EAGAIN because the listener's backlog is full.
Polling the client socket doesn't help: an unconnected stream socket
reports EPOLLOUT and EPOLLHUP right away, before the backlog drains. So
io_uring retries the connect until the apoll retry limit is hit, then
punts the request to io-wq. With a full backlog, io_connect() runs 128
times in the submitting task and once more in a new io-wq worker. That
worker stays in the task's thread group, so a later
unshare(CLONE_NEWUSER) fails with -EINVAL.
None of this changes the result. __sys_connect_file() adds the socket's
own O_NONBLOCK to every attempt, so the attempt from io-wq fails with
-EAGAIN as well. That -EAGAIN becomes the CQE result, and connect(2)
returns the same -EAGAIN.
Let's complete the request with -EAGAIN right away when the socket has
O_NONBLOCK set, like connect(2) does. io_accept() already completes
-EAGAIN this way for IORING_ACCEPT_DONTWAIT. -EINPROGRESS still waits
for the connection to complete. A blocking socket still ends up in
io-wq, where the connect sleeps until the listener accepts.
Fixes: f8e85cf255ad ("io_uring: add support for IORING_OP_CONNECT")
Signed-off-by: Daan De Meyer <daan@amutable.com>
Link: https://patch.msgid.link/20261027-b4-io-uring-connect-eagain-v1-1-f4e3aaaa49ee@amutable.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
When io_uring_register(2) is invoked with fd == -1, it dispatches
directly to io_uring_register_blind() without checking
current->io_uring_restrict.
As a result, a task with per-task register restrictions installed via
IORING_REGISTER_RESTRICTIONS can still invoke blind registration opcodes
such as IORING_REGISTER_SEND_MSG_RING even when those opcodes are not in
the task's allowed register_op bitmap.
Check current->io_uring_restrict->register_op in
io_uring_register_blind() before dispatching blind registration opcodes.
Fixes: ed82f35b926b ("io_uring: allow registration of per-task restrictions")
Assisted-by: ARTIST LLM
Signed-off-by: Dominik Maier <dmnk@google.com>
Link: https://patch.msgid.link/20261022232121.2538616-1-dmnk@google.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
When an IORING_OP_SOCKET request is prepared in io_socket_prep(),
sock->type retains creation flags such as SOCK_CLOEXEC and SOCK_NONBLOCK
until __sys_socket_file() masks them off with SOCK_TYPE_MASK during
execution.
However, io_socket_bpf_populate() copies sock->type into
bctx->socket.type without masking off ~SOCK_TYPE_MASK. As a result, a
cBPF filter inspecting bctx->socket.type against socket types such as
SOCK_DGRAM or SOCK_STREAM can be bypassed by ORing SOCK_CLOEXEC or
SOCK_NONBLOCK into the socket type.
Mask sock->type with SOCK_TYPE_MASK when populating bctx->socket.type.
Fixes: cff1c26b4223 ("io_uring/net: allow filtering on IORING_OP_SOCKET data")
Assisted-by: ARTIST LLM
Signed-off-by: Dominik Maier <dmnk@google.com>
Link: https://patch.msgid.link/20261022232120.2537530-1-dmnk@google.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
In the else arm, sel->val is necessarily 0 so the min_not_zero() actually
always returns kmsg->msg.msg_inq. Remove the min_not_zero() usage and
replace it with kmsg->msg.msg_inq.
Signed-off-by: Clément Léger <cleger@meta.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20261023144953.3584118-1-cleger@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_recvmsg() takes the transfer length from the ring provided buffer when
sqe->len is 0, and such a buffer can be up to 4G. min_ret is an int, so 2G
or more makes it negative and the (ret < min_ret) block is skipped, and the
oversized length reaches the socket layer unchanged: an empty socket then
completes with 0 instead of arming poll, and a short read is reported as
complete. Clamp the length to MAX_RW_COUNT, as io_recv() does.
Fixes: c7fb19428d67 ("io_uring: add support for ring mapped supplied buffers")
Signed-off-by: Yichong Chen <chenyichong@uniontech.com>
Link: https://patch.msgid.link/20261023112209.87979-1-chenyichong@uniontech.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_ring_buffers_peek() flags the request with REQ_F_BL_EMPTY if the
buffers it picked were the last ones available, comparing head to tail.
When the loop breaks because max_len was exhausted the buffer was picked,
but head is left pointing at it, so the flag isn't set if it was the last
buffer. A bundle receive then retries, finds no buffer, and the request,
which has received data, is marked as failed on account of the -ENOBUFS.
That severs a link, and ends a multishot.
Fixes: 35c8711c8fc4 ("io_uring/kbuf: add helpers for getting/peeking multiple buffers")
Signed-off-by: Yichong Chen <chenyichong@uniontech.com>
Link: https://patch.msgid.link/20261023091711.83481-1-chenyichong@uniontech.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring updates from Jens Axboe:
"The bulk of this is a big robustness pass over cancelation, io-wq,
timeouts, and poll. Outside of that:
- Large series from Josef hardening the cancelation, io-wq, timeout,
poll, and wait paths. Fixes for lost wakeups and lost cancels as
io-wq arms poll/timeouts, work stranding on worker exit, CQE32
resize handling, napi entry timeouts, and a pile of smaller
correctness issues across the board
- Rework of how io-wq posts completions, posting from the last
request reference and dropping the request file before the
completion, and running ring release cancelations synchronously -
canceling and waiting for all in-flight requests and dropping
registered files and buffers at release time
- Don't charge the SQ/CQ rings or provided buffer rings to
RLIMIT_MEMLOCK, which the region API conversion had started doing
- Split io_uring_cmd_set_res() out from io_uring_cmd_done() so a
command can set the result from its ->end_io() handler, used by
nvme
- CQE32 ring fixes for the free entry check, aux CQE big_cqe zeroing,
and the dummy skip CQE
- Timeout fixes for sequence handling and multishot updates
- net fixes for MSG_TRUNC buffer overconsumption and keeping recv
buffer select length within MAX_RW_COUNT
- Wake real-time waiters from hard interrupt context
- Don't spin on local work an exiting task can't run, reachable by an
unprivileged user
- Various other fixes and cleanups"
* tag 'for-7.4/io_uring-20261021' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (75 commits)
io_uring: end the drain when the drain request completes
io_uring/futex: return a wait's data to the cache when it fails to arm
io_uring/wait: wake real-time waiters from hard interrupt context
io_uring/waitid: return -EAGAIN for a non-blocking pidfd
io_uring/net: keep recv buffer select length within MAX_RW_COUNT
io_uring/waitid: prevent cancel reference wrap
io_uring/poll: don't loop on HUP with multiple references
io_uring/timeout: don't lose a cancel as io-wq arms a timeout
io_uring/timeout: don't flush with the sequence behind the last flush
io_uring/timeout: don't put a sequence target behind the last flush
io_uring/poll: avoid retry wakeup loss when dropping ownership
io_uring: look up the file before queueing to io-wq
io_uring/timeout: don't end a linked timeout with 1
io_uring/timeout: no ABS update of multishot timeouts
io_uring/net: don't retry recv on uncommitted buffers
io_uring: don't pass restart error codes to userspace
io_uring/openclose: keep the lock in io_pipe_fixed()
io_uring/sqpoll: let the thread sleep on disabled rings
io_uring/kbuf: handle index wrap in last buffer check
io_uring/zcrx: fix buffer offsets in areas above 4GB
...
|
|
io_attach_dmabuf_map() is called for every fixed read/write, but returns
-EOPNOTSUPP when CONFIG_DMA_SHARED_BUFFER is disabled before checking
whether the request uses a dma-buf at all. So READ_FIXED and WRITE_FIXED
fail for ordinary registered buffers. Check REQ_F_DMABUF first.
Fixes: 62e4bf5b25d4 ("io_uring/rsrc: add dmabuf backed registered buffers")
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://patch.msgid.link/20261022132027.2293931-1-anuj1072538@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull timer core updates from Thomas Gleixner:
"Updates for the core timekeeping and timers subsystem:
- Futher cleanups for the POSIX timer exit/exec related code to
simplify the code flow and to make it more robust against future
changes.
- Prevent tearing of timekeeper::ktime_sec which can be accessed
without the sequence count protection.
The timekeeper is updated by memcpy() with an uneven sequence count
to force readers into the retry path. On 64-bit machine the
ktime_sec member can be accessed atomically without the sequence
count protection. That is only correct when the update side
memcpy() guarantees atomicity at the 64-bit granularity. That's not
guaranteed due to debug options, compilers and instruction
availability. If the memcpy() does not provide the atomicity then
the unprotected readers can observe store tearing.
To prevent this write the ktime_sec member before the memcpy() so
that a non-atomic memcpy() will write the same data. A similar fix
has been done for the xtime_sec member, but the ktime_sec
equivalent was missed.
- Prevent time64_to_tm() day truncation on 32-bit kernels
The function receives a 64-bit seconds value but stores the
quotient of the computation in a long, which truncates the result
for large input values. Cure this by using div_s64_rem() which
stores the quotient in s64 and only leaves the seconds-within-day
reminder in a long, which is guaranteed to fit. This fix comes with
a corresponding kunit test to prevent further mishaps.
- Mark data-racy accesses to the hrtimer_sleeper::task member.
The recent updates to KCSAN flagged these benign races. Annotate
them with new helper functions, convert all existing usage sites
and mark the task member private after that so that future users
which fail to use the helpers can be detected automatically.
- Address similar benign KCSAN flagged issues in the hrtimer, timer
and timer migration code.
- Fixes for a few selftest issues"
* tag 'timers-core-2026-10-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (26 commits)
selftests/timers: clocksource-switch: Fix unchecked open()/read()
selftests: timers: Measure the CPU timers on the clock they count
selftests: timers: Count what tick is worth in the drift estimate
time/kunit: Add time64_to_tm() case beyond 32-bit day count
time: Prevent time64_to_tm() day truncation on 32-bit
timekeeping: Use READ_ONCE/WRITE_ONCE() for ktime_sec to prevent tearing
timers/migration: Mark racy updates to tmigr_event::ignore field
timers: Mark racy updates to hlist_node::pprev field
hrtimer: Apply READ_ONCE() to lockless base->running loads
hrtimer: Mark the hrtimer_sleeper structure's ->task field __private
hrtimer: Update hrtimer_resolution only if value changes
futex: Use accessor for hrtimer_sleeper ->task field in requeue
rtmutex: Use accessor for hrtimer_sleeper ->task field
net: pktgen: Use accessor for hrtimer_sleeper ->task field
timers: Use accessor for hrtimer_sleeper ->task field in sleep_timeout.c
futex: Use accessor for hrtimer_sleeper ->task field in waitwake.c
io-uring/rw: Use accessor for hrtimer_sleeper ->task field
wait: Use accessor for hrtimer_sleeper ->task field
aio: Use accessor for hrtimer_sleeper ->task field
hrtimer: Mark data-racy accesses to hrtimer_sleeper ->task field
...
|
|
Since requests are drained based on allocated requests, the drain stays
active after the drain request has posted its CQE, until it is freed and
a new request finds the defer list empty. A drain request run by io-wq
is freed only after the worker drops its reference, so a request
submitted after the drain completed can still be deferred behind it,
and get issued much later than the application expects.
A drain request dispatched from the defer list runs alone. Track it,
and end the drain when its CQE is flushed and nothing else is deferred.
Have io-wq complete drain requests through task_work, so that this
happens under ->uring_lock before the CQE is visible. Linked drain
requests keep waiting until the link is freed.
Reported-by: Wei Gao <wegao@suse.com>
Link: https://lore.kernel.org/io-uring/20261018070154.16944-1-wegao@suse.com/
Fixes: 8fb7aee05591 ("io_uring: drain based on allocates reqs")
Cc: stable@vger.kernel.org
Tested-by: Wei Gao <wegao@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_futex_wait() takes its io_futex_data from ctx->futex_cache, and a
completed wait gives it back in io_futex_complete(). A wait that fails
to arm kfree()s it instead.
The usual failure is -EAGAIN from futex_wait_setup(): the futex word no
longer holds the expected value. That is routine for a waiter that
re-arms with the value it last read, since the word changes whenever
there is something to wake for. Each such failure takes an object out
of the cache for good, so the cache drains and from then on waits
allocate.
The cache exists so that arming a wait doesn't have to go to the slab
allocator, and that matters most to the waiters that re-arm the most:
event loops, often real-time ones. A kmalloc() or kfree() that misses
the per-CPU slab takes the node's list_lock, shared with every other
user of that kmalloc size on the system, and may go on to the page
allocator.
On PREEMPT_RT list_lock is a sleeping lock. A real-time task that
finds it held waits for the holder, which may be any task on any CPU,
to get to run and release it. Priority inheritance boosts the holder
only up to the waiter's priority, so the waiter still sits behind
whatever outranks that on the holder's CPU, such as threaded interrupt
handlers. Re-arming a futex wait should not depend on any of that.
Put the data back in the cache on that path too, while the submit lock
still protects it.
In an event loop re-arming one IORING_OP_FUTEX_WAIT per message under
load, 0.39 arms per message failed with -EAGAIN, and io_futex_wait() did
0.39 kmalloc()s and kfree()s per message. With this change, 100,000
waits that all fail with -EAGAIN do one kmalloc() in total.
Fixes: 194bb58c6090 ("io_uring: add support for futex wake and wait")
Assisted-by: LLM
Signed-off-by: Austin Schuh <austin.linux@gmail.com>
Link: https://patch.msgid.link/20261019231243.802569-1-austin.linux@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull file table updates from Christian Brauner:
"This annotates struct fdtable's fd array with __counted_by_ptr so
KASAN and the fortified functions can bounds check it against max_fds
and adds a KUnit suite for fdtable. A follow-up uses __struct_size()
in the test since sparse doesn't know __builtin_dynamic_object_size().
FD_PREPARE() is simplified. It was built as an ACQUIRE-style guard on
infrastructure that was never needed. The ACQUIRE machinery and
CLASS_INIT() go away, the variable is declared with __cleanup()
directly, struct fd_prepare loses its err field since the fd either
holds the descriptor or the error, and callers get a const pointer to
the guard behind a UNIQUE_ID() name so they can't shadow it. Code
generation is identical"
* tag 'vfs-7.4-rc1.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
file: make struct fd_prepare const and kill its err field
cleanup: remove CLASS_INIT()
file: declare the FD_PREPARE() variable with __cleanup() directly
file: simplify FD_PREPARE()
vfs: use __struct_size() in the fdtable KUnit test
vfs: Add KUnit tests for fdtable
vfs: Annotate struct fdtable's fd field with __counted_by_ptr
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull signal handling updates from Christian Brauner:
"This stops TIF_NOTIFY_SIGNAL from interrupting work that can't be
restarted and fixes a bunch of coredump and signal bugs.
TIF_NOTIFY_SIGNAL kicks a task out of an uninterruptible sleep so it
returns to userspace, runs its task work and goes back to sleep. That
breaks coredumps: dump_interrupted() only lets fatal signals interrupt
a dump and writes to a regular file ignore TIF_NOTIFY_SIGNAL. But the
pipe and the socket paths check signal_pending() Adeep down in code
that is shared with regular syscalls and bail. So a crashing process
with io_uring requests in flight loses most of its coredump.
In the selftest an unfixed kernel delivers 357324 of at least 5111808
bytes. And io_uring isn't the only source. klp_send_signals(),
bpf_task_work_schedule_signal(), landlock's tsync, kthread_stop() and
the printk kunit test raise the bit as well, some of them from irq
context against any task. So this can't be fixed per subsystem and the
state has to be per-task and ambient.
Add PF_NO_NOTIFY_SIGNAL and no_notify_signal_save()/restore() helpers
modelled on memalloc_nofs_save(). Inside such a section
signal_pending() ignores TIF_NOTIFY_SIGNAL. Coredumps, the smb send
path, which hand-rolled its own version of this, and the pid namespace
reaper, which busy-looped when the bit kept getting raised, are
converted.
The second part is a set of fixes that fell out of looking at the
coredump code with kres and modelling it in TLA+:
- coredump_finish() woke the parked threads without holding rcu. A
parked thread can exit on a spurious wakeup and be freed before
wake_up_process() takes its pi_lock.
- Only SIGKILL and the freezers interrupt a dump now and the three
freezers agree. Power management and cgroup v1 freezing aborted a
dump while a cgroup v2 freeze either waited for the dump or cut it
short depending on where the core went.
- io-wq set its exit bit without a barrier against the task work that
creates workers. On weakly ordered architectures exit could hang on
worker_done.
- retarget_shared_pending() slingshotted a shared signal to the dumper
when a zapped sibling restored its signal mask. That set
TIF_SIGPENDING on the dumper and truncated the core.
- A tracer could clear the signal mask of a user worker with
PTRACE_SETSIGMASK and inject a coredump signal. An io-wq worker of
an SQPOLL ring dumping core deadlocks the whole group.
PTRACE_SETSIGMASK is refused for user workers now and
__set_task_blocked() enforces the fork-time mask.
- exec cancels io_uring requests before de_thread() so a worker
creation queued as task work can't add a thread to a thread group
that de_thread() just took down.
- No io threads are created from a PF_SIGNALED or PF_POSTCOREDUMP
creator anymore. Such a worker was invisible to zap_process() but
still decremented threads_remaining on exit so the dumper could
leave coredump_wait_inactive() early.
Also included are a comment that do_each_pid_task() needs
tasklist_lock, a READ_ONCE() in pid_alive() and a regression test for
fork() into a pid namespace whose ancestor lost its init"
* tag 'kernel-7.4-rc1.signal' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (21 commits)
pid: use READ_ONCE() in pid_alive()
selftests: pid_namespace: test fork with dead ancestor
pid: document that do_each_pid_task() needs tasklist_lock
signal: enforce the user worker signal mask in __set_task_blocked()
fork: use SIG_KERNEL_ONLY_MASK for the user worker signal mask
fork: don't create io threads once PF_POSTCOREDUMP is set
fork: move the coredump and exec checks into create_io_thread()
exec: cancel io_uring requests before de_thread()
selftests/coredump: expect PTRACE_SETSIGMASK to be refused on a user worker
selftests/coredump: test a user worker as the coredumping thread
ptrace: refuse to change the signal mask of a user worker
selftests/coredump: test shared signal retargeting during a dump
signal: only SIGKILL and the freezers interrupt a coredumping task
signal: don't retarget shared signals in a dying thread group
io-wq: order the exit bit against worker creation task work
coredump: hold RCU while releasing parked threads
pid_namespace: prevent TIF_NOTIFY_SIGNAL from interrupting the reaper
smb: prevent TIF_NOTIFY_SIGNAL from interrupting
selftests/coredump: test that TIF_NOTIFY_SIGNAL doesn't truncate a coredump
coredump: prevent TIF_NOTIFY_SIGNAL from interrupting coredumps
...
|
|
Since commit cebf123c634a ("io_uring: implement our own schedule timeout
handling"), a CQ wait with a timeout sleeps on io_uring's own hrtimer
instead of an hrtimer sleeper. On PREEMPT_RT, an hrtimer that isn't
marked for hard interrupt expiry expires in the per-CPU ktimers thread,
which runs at SCHED_FIFO 1. A real-time task waiting in io_uring_enter()
is therefore not woken at its deadline while any other real-time task is
runnable on its CPU: the wakeup waits until every one of them has blocked.
hrtimer sleepers, which schedule_hrtimeout_range_clock() used to give
io_uring, don't have this problem: __hrtimer_setup_sleeper() marks the
timer hard when the sleeping task is in a real-time or deadline class.
Both of io_uring's callbacks are fine in hard interrupt context:
io_cqring_timer_wakeup() sets a flag and calls wake_up_process(), and
io_cqring_min_timer_wakeup() only reads ring state, under RCU, before
doing the same or re-arming the timer. Apply the sleeper rule to them.
Measured on a Jetson Orin Nano running 6.18.39-rt with this change
backported: a SCHED_FIFO 40 task waits in io_uring_enter() for 40,000
deadlines 500us apart, while a SCHED_FIFO 20 task runs bursts of up to
1ms on the same CPU. Before, 6,647 wakeups were more than 100us late,
the latest by 1,040us, and 2,758 deadlines passed without a wakeup at
all. After, none were more than 100us late and the latest was 18us.
This mirrors epoll: epoll_pwait2() waits on an hrtimer sleeper, so its
timeout is already hard for real-time tasks, and on the same setup it
was never more than 17us late.
Fixes: cebf123c634a ("io_uring: implement our own schedule timeout handling")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Austin Schuh <austin.linux@gmail.com>
Link: https://patch.msgid.link/20261017171726.601443-1-austin.linux@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
IORING_OP_WAITID on a pidfd opened with PIDFD_NONBLOCK completes at
once with res 0 and an all-zero siginfo when the child has not changed
state, although WNOHANG was not requested. waitid(2) with the same
arguments fails with -EAGAIN. An application that submits the same
request it would pass to waitid(2) and treats res 0 as "the child
changed state" sees si_signo 0 and si_pid 0 instead.
kernel_waitid_prepare() turns an O_NONBLOCK pidfd into WNOHANG in
wo->wo_flags. kernel_waitid() then maps the resulting 0 return to
-EAGAIN when the caller did not ask for WNOHANG itself, as added in
commit ba7d25f3dff6 ("exit: support non-blocking pidfds").
io_waitid() uses kernel_waitid_prepare() and __do_wait() directly and
does not do that mapping.
Do the same check in io_waitid() before completing the request. Only
the inline path needs it: with WNOHANG set, __do_wait() never returns
-ERESTARTSYS, so such a request is never armed on the wait queue.
Reproduced on x86_64 under QEMU: with a PIDFD_NONBLOCK pidfd of a
running child and options = WEXITED, waitid(2) fails with -EAGAIN and
IORING_OP_WAITID completed with res 0 before the change; it completes
with res -EAGAIN after it.
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Assisted-by: LLM
Signed-off-by: Nickolai Zeldovich <nickolai@csail.mit.edu>
Link: https://patch.msgid.link/20261018025824.799245-1-nickolai@csail.mit.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Since commit dcbd1c054848 ("io_uring/net: let io_recv_buf_select return
the length of the buffer region"), io_recv_buf_select() returns the
selected length as an int. For a non-bundle recv with sqe->len == 0 the
length comes from the ring provided buffer, which can be up to 4G, so a
buffer of 2G or more makes io_recv() fail the request with a bogus
negative result (e.g. -2147483648) instead of receiving into it. Clamp
the length to MAX_RW_COUNT, which import_ubuf() limits the transfer to
anyway.
Fixes: dcbd1c054848 ("io_uring/net: let io_recv_buf_select return the length of the buffer region")
Cc: stable@vger.kernel.org
Signed-off-by: Tamaki Yanagawa <ty@000ty.net>
Assisted-by: Claude:claude-opus-5-5
Link: https://patch.msgid.link/20261018042227.1105-1-ty@000ty.net
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_waitid packs the cancellation state into bit 31 of ->refs and uses
bits 30:0 as a 31-bit reference counter:
#define IO_WAITID_CANCEL_FLAG BIT(31)
#define IO_WAITID_REF_MASK GENMASK(30, 0)
__io_waitid_cancel() marks a request canceled, then increments the low
bits of ->refs to claim completion ownership. When another path owns the
request, the increment remains even though cancellation returns false.
The request remains discoverable by cancellation, so repeated failed
attempts can wrap the 31-bit reference field to zero. A later
cancellation can claim ownership and complete the request a second time.
Use atomic_fetch_or() when setting the cancel flag and inspect the old
flag value. The first cancel attempt continues to the existing ownership
increment, preserving its wake and issuer handoff semantics. Later
attempts see the cancel flag and return without incrementing the
reference field.
This makes repeated failed cancellation attempts idempotent while
leaving the first-cancel state transitions unchanged.
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Junyeong Lee <jylee5498@gmail.com>
Link: https://patch.msgid.link/20261019152100.63565-1-jylee5498@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_poll_check_events() drops down to one reference when a multishot
request sees HUP with more than one, so that the loop goes around once
more. It did so by subtracting one less than it had. With the cmpxchg
that only takes the local count down: the exchange fails, the next pass
reads the unchanged count, and goes through the same motions again. The
loop then only ends once io_poll_issue() stops the request.
Drop the extra references from the counter instead, and loop once more
without trying the exchange. The next pass sees one reference, like it
did before, and ends unless a wake up came in.
Fixes: 937d38dcacd6 ("io_uring/poll: avoid retry wakeup loss when dropping ownership")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
One problem that was brought up is that the file table size is often a
limiting factor when working with dma-bufs, which is quite an old
problem. With a single io_uring people can work it around by closing
the dma-buf fd after registration, but for more complex cases we need
a way to dynamically register it into another ring. We also need to
deduplicate internal registrations as each creates a new attachment and
related structures, which could involve large allocations and iommu
mappings. Solve it by enabling io_uring buffer cloning.
I made dma-bufs registered for io_uring not cloneable between rings
because the implementation could modify parts of struct io_buf_dma
without any locking and sharing it would've created races. That was an
optimisation that got dropped at some point for simplicity, so we can
safely enable cloning back as long as IORING_SETUP_IOPOLL rejection
is honoured. Kill the uncloneable flag as unused as well.
Reported-by: William Power <william.power@intel.com>
Reported-by: Tushar Gohad <tushar.gohad@intel.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/b76cf21ee0e2362489c864e43781b20d95fd75f7.1792142588.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
An IORING_OP_TIMEOUT run by io-wq, because of IOSQE_ASYNC, has the same
problem as poll, futex and waitid. io_wq_submit_work() looks at
IO_WQ_WORK_CANCEL once, before it issues the request. A cancel in between
flags the work, gets -EALREADY, and searches ->timeout_list in vain,
while the worker then adds the request and starts the timer:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
= -EALREADY
io_timeout_cancel() = -ENOENT
io_timeout()
adds the request to ->timeout_list
hrtimer_start()
= IOU_ISSUE_SKIP_COMPLETE
Nobody looks at the flag again, and the timeout runs its course. At ring
exit, io_uring_try_cancel_requests() cancels io-wq work and then kills
the timeouts the same way, so an exiting task can get stuck for the full
duration of such a timeout.
Look at the flag under ->timeout_lock, which the search takes as well,
and fail the request if it is set. If the search came first, the flag is
visible. If it comes later, it finds the request on the list.
Fixes: ce35a47a3a02 ("io_uring: add IOSQE_ASYNC")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_flush_timeouts() measures the events since the last flush as the
distance of the current sequence from ->cq_last_tm_flush. A fired
timeout bumps ->cq_timeouts before its CQE is posted, so until it is
the sequence can be behind the last flush, and the distance wraps:
last flush: seq 100, cq_last_tm_flush = 100
Y: IORING_OP_TIMEOUT, off 50, target_seq 150
pure timeout A fires, cq_timeouts = 1
task_work posts A's CQE, cached_cq_tail = 101
pure timeout B fires before the flush, cq_timeouts = 2
io_flush_timeouts(): seq = 99, events_got = 0xffffffff
Y, and every sequence timeout behind it, is ended with -ETIME although
no event happened. Two timeouts expiring close together is all it takes,
a completion with IOSQE_CQE_SKIP_SUCCESS after a single one does it too.
Nothing but those in flight CQEs separates the two, so count from the
last flush when the sequence is behind it, as io_timeout() does.
Fixes: f010505b78a4 ("io_uring: flush timeouts that should already have expired")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_timeout_fn() and io_kill_timeout() bump ->cq_timeouts when a timeout
fires, but its CQE is only posted later from task_work. Until then the
sequence io_timeout() computes is one below the last flush per fired
timeout. Now that ->cq_last_tm_flush is kept while sequence timeouts are
pending, a timeout armed in that window with a small offset gets a target
behind the base. Its distance from the base wraps, and it's never
completed by events, only by its timer. With DEFER_TASKRUN that window
is as wide as the application makes it, as submission runs before the
local task_work.
Nothing but those in flight CQEs separates the two, so use the last
flush as the base for the target when the sequence is behind it.
Fixes: c6c0b3eba88c ("io_uring/timeout: don't skip over reached sequences")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
If a wakeup sets IO_POLL_RETRY_FLAG between the owning task consuming
the existing instance of IO_POLL_RETRY_FLAG and the loop check
of io_poll_check_events(), then we stall on processing the request
until an unrelated wakeup comes along.
Using atomic_cmpxchg instead of atomic_sub_return to guarantee both the
count v and poll_refs are 0 and no wake-ups are missed.
Cc: stable@vger.kernel.org
Fixes: a26a35e9019f ("io_uring: make poll refs more robust")
Signed-off-by: Juanlu Herrero <juanlu@fastmail.com>
Assisted-by: LLM
Suggested-by: Anand Natarajan <anand@meta.com>
Link: https://patch.msgid.link/20261013-pu-cmpxchg-v1-1-4ddc10fd8a03@fastmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_prep_async_work() decides by the file of a request what kind of work
it is for io-wq. I/O on regular files and block devices is bounded work,
limited to min(SQ entries, 4 * CPUs) workers, for anything else that may
block indefinitely the limit is RLIMIT_NPROC. Writes to a regular file
are hashed by the inode, so that they are run one after the other instead
of having a bunch of workers fight for the inode lock.
Since the file is only assigned when a request is issued, a request that
is queued without an attempt to issue it, REQ_F_FORCE_ASYNC, doesn't have
one at that point. io_prep_async_work() takes that for "not a regular
file or block device":
- Reads and writes end up as unbound work. 32 reads with IOSQE_ASYNC from
a slow block device on a ring with 4 SQ entries get 32 workers, not 4.
- Writes aren't hashed. Four buffered writes to one file get four
workers, three of which do nothing but wait for the lock.
The late assignment is for the benefit of links, where a request may use
a fixed file that one of its predecessors installs. By the time a request
gets to io_queue_iowq() these are done, it is as good as issued. Assign
the file there.
io_assign_file() moves up and is forced inline. With a third caller the
compiler may stop inlining it, and every request that is issued would
pay for a call.
IORING_OP_SPLICE is left as it is, unbound and not hashed. The file of
the request is where it writes to, which says nothing about how long it
takes: a splice from an empty pipe or a socket into a regular file waits
for its input. As bounded work, four of these on a ring with 4 SQ entries
keep all other bounded work of the task from running, and as hashed work
one of them blocks all writes to that file.
REQ_F_FORCE_ASYNC is not only set for IOSQE_ASYNC. What changes, and
brings back what was the case before the file assignment was deferred:
- Requests with IOSQE_ASYNC or IOSQE_IO_DRAIN, and all requests that are
submitted while a drain is active: reads and writes on regular files
and block devices count against the limit for bounded workers, including
the one set with IORING_REGISTER_IOWQ_MAX_WORKERS, so fewer of them run
at the same time.
- Of these, writes to one regular file are run one at a time, unless they
are O_DIRECT and the filesystem has FOP_DIO_PARALLEL_WRITE.
- IORING_OP_FALLOCATE and IORING_OP_FTRUNCATE, which always go to io-wq,
are run one at a time per file, and in turn with these writes.
- The descriptor or the fixed file slot is looked up when the request is
queued, not when a worker gets around to it, so closing or replacing it
in between no longer has an effect on the request.
Fixes: 6bf9c47a3989 ("io_uring: defer file assignment")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-43-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A linked timeout that fires cancels the request it is linked to, and
completes with -ETIME if that worked out:
ret = io_try_cancel(req->tctx, &cd, 0);
...
io_req_set_res(req, ret ?: -ETIME, 0);
io_try_cancel() returns 0 or an error, except for requests that are parked
on ->futex_list or ->waitid_list. io_futex_cancel() and io_waitid_cancel()
return the number of requests that they have cancelled, and
io_try_cancel() hands it on.
So the linked timeout of an IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAITV or
IORING_OP_WAITID that it has cancelled completes with 1, each time.
The waitid test of liburing checks for that 1, and has to be changed to
expect -ETIME, which is what the linked timeout of any other request
completes with.
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Fixes: 194bb58c6090 ("io_uring: add support for futex wake and wait")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-42-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
IORING_TIMEOUT_MULTISHOT can't be combined with IORING_TIMEOUT_ABS, prep
rejects that, as the request is armed again with the same value each time
it fires. IORING_TIMEOUT_UPDATE doesn't check and takes an absolute time
for a multishot timeout. The first expiry is as asked for, but
io_timeout_complete() then arms the timer with ->mode of the original
request, i.e. it uses the absolute time as the interval. With
CLOCK_MONOTONIC the next expiry is an uptime away, with CLOCK_REALTIME
more than fifty years.
Fail such an update with -EINVAL, and do so before the timeout is
touched.
Fixes: ea97f6c8558e ("io_uring: add support for multishot timeouts")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-41-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Commit 41b70df5b38b ("io_uring/net: commit partial buffers on retry")
commits ring provided buffers when a receive is going to be retried after
a partial transfer, so that another request using the same buffer group
can't be handed the memory the first one keeps receiving into. It commits
the number of bytes received so far. That does the job for a single
buffer which is consumed as a whole, but the request hangs on to all it
has mapped:
- For a buffer ring with IOU_PBUF_RING_INC only these bytes are consumed.
The rest of the buffer stays at the head of the ring.
- For a bundle only the buffers which hold these bytes are consumed, the
others that were peeked stay in the ring.
In both cases the next receive with that buffer group gets memory that
the retried request is still going to write to, and whoever comes last
overwrites the data of the other. What the retry fills isn't committed at
all, as the buffer list isn't known any more at that point. So the kernel
hands it out again later on, while the CQE tells the application that it
was used, and for an incremental buffer that it is done with.
The remainder can't be committed in advance: if the request ends up
receiving less, there is no way to tell the application that more was
consumed than the CQE accounts for. Complete the request with what it has
got instead of retrying it. MSG_WAITALL is no longer honoured for
receives into incremental buffers or multi buffer bundles, they can
return a short count, with the request marked as failed like for other
short MSG_WAITALL transfers.
Fixes: 41b70df5b38b ("io_uring/net: commit partial buffers on retry")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-40-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
-ERESTARTSYS and friends are instructions for the system call exit code,
which either restarts the call or turns them into -EINTR. They are not
supposed to be seen by applications. A request isn't a system call,
nobody looks at what it returns before that is put into a CQE.
The read/write and the networking opcodes convert these codes to -EINTR
by hand. The others don't. An io-wq worker that is blocked in the open of
a FIFO, a tty or some other device, or in a splice from an empty pipe,
and that gets interrupted because the request is cancelled, completes it
with -512:
IORING_OP_OPENAT, IOSQE_ASYNC // FIFO without a writer
IORING_OP_ASYNC_CANCEL // -EALREADY, worker is signalled
CQE of the open: res == -512
There are more candidates, like whatever a file system or driver returns
from ->flush(), ->fsync(), ->fallocate() or ->uring_cmd(), or breaking a
lease or delegation.
Take care of it in io_issue_sqe(), which all of them go through. The
helper is moved from rw.c as is. The cost is a compare and a branch that
is not taken per request.
Fixes: 15b71abe7b52 ("io_uring: add support for IORING_OP_OPENAT")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-39-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_pipe_fixed() installs both ends of the pipe in the fixed file table,
drops ->uring_lock, and copies the slot numbers to the application. If
that faults, it takes the lock again and removes whatever is in these
slots.
If the request is run by io-wq the lock really is dropped in between, and
the slots are visible to everybody else. By the time the error path gets
to them, they may have been closed and reused, by IORING_OP_CLOSE and a
direct open or accept, or by IORING_REGISTER_FILES_UPDATE. Then a file is
removed from the table that has nothing to do with the failed request.
Do the copy with the lock held. Faulting in user memory under
->uring_lock is fine, the registration paths do it all the time.
Fixes: 53db8a71ecb4 ("io_uring: add support for IORING_OP_PIPE")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-38-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
__io_sq_thread() doesn't submit anything as long as the ring has
IORING_SETUP_R_DISABLED set. The check that keeps the SQPOLL thread from
going to sleep while the SQ ring holds entries doesn't know about that.
If an application fills in SQEs first and enables the ring later, the
thread finds that there is work each time it is about to sleep, goes
around, doesn't submit it, and so on. It uses up a CPU until the ring
gets enabled.
Entries in the SQ ring of a disabled ring are no reason to stay awake.
io_register_enable_rings() clears the flag and then wakes the thread, and
the flag is looked at after prepare_to_wait(), so the thread can't miss
that.
Fixes: 0298ef969a11 ("io_uring: clean R_DISABLED startup mess")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-37-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_ring_buffer_select() flags the request with REQ_F_BL_EMPTY if it takes
the last buffer of the ring. Head and tail are 16-bit counters, but
"head + 1" is computed as an int. If head is 65535 and tail has wrapped
to 0, that compares 65536 to 0 and the flag isn't set.
A bundle receive relies on the flag to tell that there is no point in
trying to get more data. Without it, it goes for another round, finds no
buffer, and the request, which has received data, is marked as failed on
account of the -ENOBUFS. That severs a link, and ends a multishot.
Fixes: c3f9109dbc9e ("io_uring/kbuf: flag request if buffer pool is empty after buffer pick")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-36-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_zcrx_queue_cqe() computes the offset of the buffer in its area from
net_iov_idx(), which is an unsigned int, so the shift is done in 32 bits
and only the result is widened to the u64 that goes into the CQE. An area
can be up to 1TB in size. For every niov beyond the first 4GB the CQE
points the application at a buffer 4GB, or a multiple of that, below the
one which has the data. It reads the wrong data, and then returns a
buffer it doesn't own via the refill queue while the real one is never
given back.
Fixes: 11ed914bbf94 ("io_uring/zcrx: add io_recvzc request")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-35-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
__io_napi_busy_loop() caps the busy poll time to what is left until the
timeout of the wait. The difference is negative if the timeout has passed
already, which is what an application gets if it uses
IORING_ENTER_ABS_TIMER with a time that isn't in the future any more. The
min_t(u64, ...) turns that into a huge value, so the cap has no effect
and the wait, which should return right away, spins for the full busy
poll time first.
Fixes: d5cce407e4f5 ("io_uring/napi: postpone napi timeout adjustment")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-34-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
