| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a task_work add use-after-free with SQPOLL.
The sqpoll thread could pop and complete the last request while
io_req_normal_work_add() was still looking at them after the mpscq
push.
Use the same approach as DEFER_TASKRUN to protect from that, holding
an RCU read lock across the add, and have exit wait for an RCU grace
period for SQPOLL rings as well.
- CQE32 ring fixes: correct the free entry check for 32b CQEs, zero the
big_cqe for aux CQEs, and only post the dummy skip CQE on CQE_MIXED
rings
- Mark the source filter table as COW when cloning bpf filters, so
registering another filter on the source doesn't modify the shared
table in place
- Initialize the task context before running the BPF loop
- Requeue zcrx multishot receives stopped by a local resource
- End a TX_TIMESTAMP multishot cmd when the CQ is full (lollipopkit)
* tag 'io_uring-7.3-20261002' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring: fix task_work add use-after-free with SQPOLL
io_uring/cmd_net: end TX_TIMESTAMP multishot when the CQ is full
io_uring/zcrx: requeue multishot receives stopped by a local resource
io_uring: initialize task context before running the BPF loop
io_uring: zero big_cqe for aux CQEs on CQE32 rings
io_uring: fix free entry check for 32b CQEs on CQE32 rings
io_uring: only post the dummy skip CQE on CQE_MIXED rings
io_uring/bpf_filter: mark source as COW when cloning filters
|
|
For SQPOLL, the sqpoll thread runs task_work off its own list and
doesn't need to be told about it, so it can pop and complete a request
as soon as mpscq_push() has published it. If that was the last request
holding the ring alive, io_ring_exit_work() then frees the ctx, stops
the thread, and with it the tctx, while io_req_normal_work_add() is
still looking at both after the push:
BUG: KASAN: slab-use-after-free in io_req_normal_work_add+0x439/0x510
Read of size 4 at addr ff11000000ca0000 by task repro/55
(...)
BUG: KASAN: slab-use-after-free in queue_work_on+0x25/0x70
Write of size 8 at addr ff11000000c3bd00 by task repro/55
Hold an RCU read lock across the add, like io_req_local_work_add() already
does, and have io_ring_exit_work() wait for a grace period for SQPOLL rings
as well before freeing the ctx.
Reported-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://lore.kernel.org/io-uring/20260924205056.3759980-2-Jeremy.Jean@oss.cyber.gouv.fr/
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
SOCKET_URING_OP_TX_TIMESTAMP arms an EPOLLERR apoll multishot and, on
each error queue edge, posts one 32 byte aux CQE per timestamp skb. Aux
CQEs have no overflow backing, so io_uring_cmd_post_mshot_cqe32() fails
once the CQ ring is full. The loop then stops, splices the unprocessed
skbs back onto sk_error_queue and returns -EAGAIN.
-EAGAIN is IOU_RETRY, so the request goes idle until the next poll
event. io_arm_apoll() forces EPOLLET and draining the CQ does not re-run
the command, so the spliced-back timestamps stay queued until an
unrelated later timestamp produces a new edge.
End the multishot when a CQE cannot be posted, as multishot poll and
recv already do. There is no partial result to report, so complete with
-ENOBUFS: no CQ space is left for the timestamp CQEs. This command does
not use provided buffers, so the value cannot be confused with a buffer
ring running empty. The terminal CQE still reaches userspace because
request completions can go to the overflow list. The application sees a
CQE without IORING_CQE_F_MORE, which it already has to handle, and the
first issue of the re-armed request delivers the queued timestamps.
A timestamp that cannot be extracted keeps its current behaviour.
This was found with LLM assistance while reviewing io_uring multishot
handlers for the pattern behind a RECV_ZC stall reported earlier (see
Link), and confirmed with a reproducer on v7.3-rc2 and an A/B run of
this patch.
Fixes: 9e4ed359b8ef ("io_uring/netcmd: add tx timestamping cmd support")
Link: https://lore.kernel.org/all/010001a0cd914529-23df2f94-bbe0-49cd-ae5d-05922356fa12-000000@email.amazonses.com/
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-6-sol
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: lollipopkit <a@lolli.tech>
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/20260927074206.65427-1-a@lolli.tech
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A multishot RECV_ZC can go idle with unread TCP data when a
receiver-local resource runs out. An empty copy-fallback niov freelist
returns -ENOMEM; a full CQ returns -ENOSPC. After partial progress the
stop reason is lost: io_zcrx_copy_chunk() and io_zcrx_recv_skb() report
the copied bytes, and tcp_read_sock() drops an error from a later call
once earlier data was consumed. The edge-triggered request is then not
requeued, so it cannot consume the remaining data until another socket
event arrives.
Record in io_zcrx_recv_skb() when a walk stops before consuming the
length it was offered, and requeue if the pass still returned data. The
failed chunk is first on the next pass. The resource is usually still
exhausted by then, so that pass fails at once and ends the request with
an error CQE, which the application must handle by re-arming. The
request does not spin, and the stall becomes a visible error. Keep the
existing skb limit and SOCK_DONE handling.
Fixes: 11ed914bbf94 ("io_uring/zcrx: add io_recvzc request")
Fixes: bc57c7d36c4c ("io_uring/zcrx: add copy fallback")
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-6-sol
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Junyuan Feng <a@lollipopkit.com>
Reviewed-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/010001a0cd914529-23df2f94-bbe0-49cd-ae5d-05922356fa12-000000@email.amazonses.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Submitting SQEs through an io_uring BPF loop can trigger a NULL pointer
dereference in io_submit_sqes(), as shown by the following arm64 report:
[ 2049.380301] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 2049.388788] pc : io_submit_sqes+0x48/0x6c4
[ 2049.389153] lr : bpf_io_uring_submit_sqes+0x10/0x1c
[ 2049.396310] Call trace:
[ 2049.396531] io_submit_sqes+0x48/0x6c4 (P)
[ 2049.396893] bpf_io_uring_submit_sqes+0x10/0x1c
[ 2049.397293] bpf_prog_1a2a95cb199202b5+0x3c/0x68
[ 2049.397699] bpf__io_uring_bpf_ops_loop_step+0x5c/0x94
[ 2049.398149] io_run_loop+0x70/0x2e4
[ 2049.398461] __arm64_sys_io_uring_enter+0xf0/0x710
[ 2049.398882] invoke_syscall+0x54/0x10c
[ 2049.399414] el0_svc_common.constprop.0+0x40/0xe0
[ 2049.399829] do_el0_svc+0x1c/0x28
[ 2049.400125] el0_svc+0x38/0x1d0
[ 2049.400409] el0t_64_sync_handler+0xa0/0xe4
[ 2049.400779] el0t_64_sync+0x198/0x19c
io_uring_enter() dispatches to io_run_loop() before reaching the normal
submission path's io_uring_add_tctx_node() call. The loop only checks
whether the caller is allowed to run task work; it does not initialize
current->io_uring or associate the task with the ring. A BPF call to
bpf_io_uring_submit_sqes() then reaches io_get_task_refs(), which assumes
that current->io_uring is valid.
A ring created with IORING_SETUP_R_DISABLED, IORING_SETUP_SINGLE_ISSUER
and IORING_SETUP_DEFER_TASKRUN can be enabled by a different task that
has never used io_uring. Enabling the ring makes that task the submitter
without allocating its io_uring task context. Its first BPF-driven
submission of a pending SQE can therefore cause a kernel Oops.
Call io_uring_add_tctx_node() in io_run_loop() before invoking the loop
and propagate any initialization error. Do this before acquiring
uring_lock, since task context setup may acquire the same mutex. This
also establishes the task-to-ring association used for cancellation.
Fixes: 033af2b3eb19 ("io_uring: introduce callback driven main loop")
Signed-off-by: Yao Kai <yaokai34@huawei.com>
Link: https://patch.msgid.link/20260923014310.591292-1-yaokai34@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_fill_cqe_aux() only clears big_cqe if cqe32 is set, which is only the
case for CQE_MIXED. On a CQE32 ring every CQE is 32b, and the extra 16b
are left holding whatever was in that slot before.
Clear it for CQE32 rings as well, like it was before CQE_MIXED support
was added.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
On CQE32 rings, len is in units of 32b CQEs when io_cqe_cache_refill()
checks for free entries, so requiring cqe32 + 1 of them means insisting
on 2 free CQEs where 1 is enough, and a ring with a single free entry
needlessly goes to overflow.
Check for a free CQE directly in that case, and keep the cqe32 + 1 check
for CQE_MIXED where the units are 16b.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_cqe_cache_refill() posts a dummy skip CQE when a 32b CQE would
straddle the end of the ring, so the real one can start at offset 0. That
only makes sense on CQE_MIXED rings, where slots are 16b and a 32b CQE
needs two of them back to back. On a CQE32 ring every slot is already
32b, but io_fill_cqe_aux32() still passes cqe32 == true, and the dummy
then gets written at cqes[off] rather than cqes[off << 1], inside another
CQE.
Limit the dummy to CQE_MIXED rings.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919203516.2581409-1-benquike@gmail.com
[axboe: split from Hui's original patch, edited commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_bpf_filter_clone() shares the filter table between source and
destination and marks the destination as needing a copy before it can be
modified, but leaves the source as is. Registering another filter on the
source then modifies the shared table in place, and the change shows up
in everything that cloned it.
Mark the source as COW as well.
Fixes: ed82f35b926b ("io_uring: allow registration of per-task restrictions")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
Link: https://patch.msgid.link/20260919112523.3872581-1-benquike@gmail.com
[axboe: edited patch and commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Fix a deadlock in the write path with superblock freezing
- Fix an issue where a provided buffer ring would overconsume when
using MSG_TRUNC
- Keep the CQE flags on iopoll requests when adding kbuf flags
* tag 'io_uring-7.3-20260911' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/rw: keep CQE flags on iopoll requests when adding kbuf flags
io_uring/net: don't overconsume buffers when using MSG_TRUNC
io_uring/net: let io_recv_buf_select return the length of the buffer region
io_uring/rw: end write accounting from ->ki_complete
|
|
io_do_iopoll() assigns the result of io_put_kbuf() to the request's CQE
flags upon completion. This overwrites any CQE flags that may have been
set by the opcode-specific layer. (For example, if __io_uring_cmd_done()
had set IORING_CQE_F_32, it would be cleared.) Switch the = to an |= so
the kbuf flags are added to the existing CQE flags rather than replacing
them. io_req_rw_complete() does the same with the io_put_kbuf() result.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Reported-by: sashiko-bot@kernel.org
Link: https://sashiko.dev/#/message/20260827191705.D53C91F000E9%40smtp.kernel.org
Signed-off-by: Caleb Sander Mateos <csander@purestorage.com>
Reviewed-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://patch.msgid.link/20260902210200.2336720-1-csander@purestorage.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
When a recv/recvmsg is issued with MSG_TRUNC and the incoming packet is
larger than the provided buffer, the net layer returns the full length
of the packet rather than the number of bytes actually copied into the
buffer. As a result, io_uring advances more of the provided buffer ring
than was actually filled. Use the actual filled region size to consume
the buffer, but still return the full size to preserve MSG_TRUNC
semantics.
Take care with multishot, because that seems to already truncate the
consumption based on the available payload size.
This was reported in https://github.com/axboe/liburing/issues/1619.
Fixes: ae98dbf43d75 ("io_uring/kbuf: add support for incremental buffer consumption")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260728191454.1850326-1-krisman@suse.de
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260902230041.1320658-3-krisman@suse.de
[axboe: fold in size_t unsigned fix]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
In preparation to using this field as an upper limit to truncation,
return the size of the allocated region.
Fixes: ae98dbf43d75 ("io_uring/kbuf: add support for incremental buffer consumption")
Cc: stable@vger.kernel.org
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260902230041.1320658-2-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Commit b000145e9907 moved both the fsnotify calls and the write
accounting out of the kiocb completion handler and into the
io_req_rw_complete() task_work. However, only the fsnotify part actually
needed to move as it may sleep. Ending the write accounting is just a
percpu_up_read() on the superblock writers sem.
Deferring it is a problem, because it makes dropping SB_FREEZE_WRITE
protection depend on the ring owner getting to running task_work. But
the task may be blocked in freeze_super(), causing it to never get to
that:
task io-wq worker
--------------------------------------------------------------
io_write()
io_kiocb_start_write() (takes sb_writers, hidden from
lockdep by __sb_writers_release)
write_iter() -> -EIOCBQUEUED
ioctl(FS_IOC_SHUTDOWN)
bdev_freeze()
freeze_super()
percpu_down_write() <- waits for the reader above
io_write()
kiocb_start_write()
percpu_down_read() <- queued
behind the
writer
<bio completes>
io_complete_rw()
queues io_req_rw_complete() <- never runs, task is in D state
End the write from io_complete_rw() instead, and leave only the fsnotify
calls in task_work.
Reported-by: syzbot+2eb3d983669d3e49d4fa@syzkaller.appspotmail.com
Cc: stable@vger.kernel.org
Fixes: b000145e9907 ("io_uring/rw: defer fsnotify calls to task context")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
"A few smaller fixes for io_uring that should go into the 7.3-rc1
kernel, all three headed to stable as well. This contains:
- A few fixes around cancellation and teardown for waitid
- Cap the user size for the query interface copy-out"
* tag 'io_uring-7.3-20260828' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/waitid: avoid siginfo copy during ring teardown
io_uring/waitid: honor task_work cancellation
io_uring/query: cap user size passed to copy_struct_to_user
|
|
During ring teardown, io_ring_exit_work() cancels outstanding requests
from a kworker with a NULL tctx. The waitid cancellation path eventually
reaches io_waitid_finish(), which copies the stored siginfo to the
userspace pointer supplied with the request.
Ring-wide teardown does not run in the task context that submitted the
request, so it must not access that task's userspace pointer. Depending
on the address and mm state, the copy may fail with -EFAULT, but the
uaccess itself is inappropriate from the teardown kworker.
Use a no-copy cancellation callback when io_waitid_remove_all() is
called without an owning task context. Complete the request with
-ECANCELED while releasing the waitid state without touching siginfo.
Keep the existing siginfo handling for explicit async cancellation and
task-scoped cancellation.
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Cc: stable@vger.kernel.org
Signed-off-by: Hui Su <sh_def@163.com>
Link: https://patch.msgid.link/20260818103336.1922818-3-sh_def@163.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_waitid_cb() may run through the fallback task_work path when
task_work_add() can no longer queue work to the originating task. The
fallback runs from a kworker and io_uring marks such task work as
canceled through tw.cancel.
io_waitid_cb() currently ignores tw.cancel and calls __do_wait().
waitid is task-context dependent: __do_wait() performs child lookup
relative to current, and the retry path also uses
current->signal->wait_chldexit. If the callback runs from the fallback
kworker, current is therefore not the task that submitted the request.
Honor tw.cancel before entering __do_wait(). Complete the request with
-ECANCELED and skip the siginfo copy, since canceled task work may run
without the submitting task's userspace execution context.
Keep the existing siginfo handling for normal waitid completion and
explicit cancellation.
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Cc: stable@vger.kernel.org
Signed-off-by: Hui Su <sh_def@163.com>
Link: https://patch.msgid.link/20260818103336.1922818-2-sh_def@163.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_handle_query_entry() clamps hdr.size for the inbound copy_from_user()
but keeps the original user value as usize. copy_struct_to_user() uses
that usize and, when it is larger than the kernel result, clear_user()s
the trailing bytes.
As hdr.size is a __u32, a query can request nearly 4 GiB of zeroing,
including on the error path where res_size stays 0. The interface is
reachable without a ring via IORING_REGISTER_QUERY.
Reject sizes larger than PAGE_SIZE, as recommended for copy_struct_*
interfaces.
Fixes: c265ae75f900 ("io_uring: introduce io_uring querying")
Cc: stable@vger.kernel.org # 6.18+
Signed-off-by: Laxman Acharya Padhya <acharyalaxman8848@gmail.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260821103317.91437-1-acharyalaxman8848@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
Pull fuse updates from Miklos Szeredi:
- Improve performance of the io-uring transport by introducing buffer
pools and zero-copy (Joanne)
- Fix lots of bugs (Baokun Li)
- Fix io-uring initialization issues (Joanne, Bernd)
- More prep work for large folios (Joanne)
- Don't limit buffered read to 128k (Jim Harris)
- Fix zeroing of page end (dirtied with mmap) on file size extension
(Jimmy Zuber)
- Improve performance in certain cases with wake_up_sync() when queuing
request (Xuewen Yan)
- Misc fixes and cleanups (Xuewen Yan)
* tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse: (35 commits)
fuse: zero the partial EOF page when extending a file
io_uring: Add missing include for ITER_SOURCE and ITER_DEST
fuse: Fix the condition to enable over-io-uring
fuse: invalidate the correct range after O_APPEND direct write
selftests/fuse: test post-EOF page zeroing when a file is extended
fuse: wake one waiter per freed slot when raising max_background
fuse: use min_not_zero() in fuse_init_server_timeout()
fuse: copy request headers via a stack buffer for io-uring
fuse: give wakeup hints to the scheduler for synchronous requests
fuse: check for NULL root inode in fuse_fill_super_submount
fuse: reject a duplicate fd= mount option
cuse: wait for pending RCU callbacks on module exit
fuse: fix invalidate lock leak on open O_TRUNC DAX failure
fuse: fix invalidate lock leak on setattr writeback failure
fuse: wait for FR_FINISHED on abort_on_kill to prevent use-after-free
fuse: make dentry_tree_work static
docs: fuse: document io-uring buffer pool and zero-copy uapi
fuse: add zero-copy over io-uring
fuse: support registered buffer pools in io-uring
fuse: add io-uring buffer pools
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block updates from Jens Axboe:
- NVMe updates via Keith:
- Enable Clang context analysis for the nvme host driver, adding
context annotations across core, fabrics, rdma, tcp and pci
- nvmet reservation state exposed through a new namespace-level
debugfs directory, plus ABI documentation for the host sysfs and
target configfs interfaces
- nvme-tcp host memory disclosure fixes on the read path: reject a
read that transferred too few bytes, don't accept C2HData based
on blk_rq_payload_bytes() alone, and fix the R2T case for a read
command
- Parallelize nvme-rdma I/O queue allocation and startup (Surabhi)
- Apple nvme fixes and quirks: page aligned admin queue buffers,
destroy the admin queue on removal, and various DMA/NVMMU
correctness fixes
- A large pile of nvmet and host fixes for out-of-bounds reads,
refcount/resource leaks, and NULL derefs across auth, zns,
passthru, pci-epf, rdma and configfs
- Various other fixes and cleanups
- MD updates via Yu Kuai:
- llbitmap reshape support, the large series wiring exact bitmap
mapping and reshape lifecycle through raid5 and raid10, growing
the page cache in place, and remapping checkpointed bits as
reshape progresses
- raid5 fixes for lockless max_nr_stripes and recovery_offset
accesses, a reshape deadlock with more failed devices than max
degraded, and bitmap batch counter consistency
- Atomic write handling for raid1/raid10, and removal of the
REQ_NOWAIT support from raid1/10/456
- raid5-ppl use-after-free fix in ppl_do_flush()
- A batch of smaller fixes across md core and the bitmap code
- s390/dasd ESE full-track write support and the surrounding
infrastructure, plus enabling CONTEXT_ANALYSIS for s390/block
- RWF_DONTCACHE support for block devices, built on new task-context
bio completion infrastructure, and wiring it up for the iomap and
buffer dropbehind writeback paths
- Async io_uring zone reset all, plus zone management command cleanups
allowing REQ_NOWAIT and tightening conventional zone rejection
- Block integrity refactoring: lift BIP_CHECK_FLAGS to the shared
header, handle nogenerate/noverify properly in fs-integrity, and drop
the blk-integrity.h include from bdev.c
- Split out a new blk_plug.h header
- ublk improvements: add UBLK_F_IO_DESC_SIZE, split request validation
from io_desc init, reject non-power-of-2 zone sizes in SET_PARAMS,
and a series of hardening fixes around map/unmap and auto buf reg
- null_blk cleanups and configfs serialization fixes
- nbd queue freeze removal on the setup paths, and a new
pre_defined_connections module parameter for pre-created devices
- blk-cgroup fixes for the race between policy activation and blkg
destruction, and accounting per-cpu stats over possible CPUs across
blk-stat, iolatency, iocost and kyber
- Various dio fixes: leak on metadata mapping error, validate user
space vectors during extraction, and set dma_alignment from the
backing file for loop and zloop direct I/O
- bio cleanups
- Various other fixes and cleanups all over
* tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (241 commits)
nbd: add pre_defined_connections module parameter for pre-created devices
nbd: remove queue freeze for newly created nbd from netlink path
nbd: factor out a nbd_genl_foreach_sock
nbd: skip queue freeze when setting size at device startup
nbd: remove queue freeze in nbd_add_socket
nbd: clear queue limits on disconnect
nbd: disallow NBD_SET_SOCK on an active device
nbd: simplify find_fallback() by removing redundant logic
blk-mq: add missing call to srcu_barrier() in blk_mq_free_tag_set()
block: mtip32xx: synchronize ioctls with device removal
ublk: avoid teardown retry loop on xarray allocation failure
null_blk: fix UBSAN shift-out-of-bounds when zone_size is 0 or overflows
block: don't include blk-integrity.h in bdev.c
xfs: avoid double deferrals for RWF_DONTCACHE writes
loop: Fix recently introduced lock inversion
block: set QUEUE_FLAG_DYING unconditionally in blk_mark_disk_dead()
swim3: Add missing MODULE_DESCRIPTION
selftests: ublk: add SET_PARAMS validation test
selftests: ublk: add helper for SET_PARAMS
ublk: reject non-power-of-2 zone sizes in SET_PARAMS
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring update from Jens Axboe:
"On top of the usual cleanups and fixes, the bigger items in here are:
- zcrx work, most of it centered around adding dynamic area
provisioning, plus a bunch of prep and cleanups leading up to it:
scale refilling with large pages, coalesce same-niov RQEs on
refill, separate the RQ head/tail cache lines and cache the RQ
tail, and rework the area creation locking.
- Fix the futex inflight accounting so that only private futex waits
are marked inflight, and don't mark wake requests as inflight at
all.
- Drop the custom iov copy in the buffer select prep and msg header
copy paths, using the generic helpers instead.
- Fix a folio size overflow in io_vec_fill_bvec(), and account the
pages a compound region really uses in the memmap path.
- Fix an iovec leak in uring_cmd when the async cmd isn't recycled,
skip the blocking task work for io_uring_cmd_issue_blocking(), and
don't skip completion for a synchronous multishot cmd
- Defer eventfd signaling when queued from a wakeup handler
- Fix io-wq worker accounting when canceling creation callbacks
- Annotate remote tasks for kcoverage"
* tag 'for-7.3/io_uring-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (30 commits)
io_uring: Add missing include for ITER_SOURCE and ITER_DEST
io_uring/uring_cmd: don't skip completion for a synchronous multishot cmd
io_uring/memmap: account the pages a compound region really uses
io_uring/zcrx: add dynamic area provisioning
io_uring/zcrx: lock area creation with pp_lock
io_uring/zcrx: keep array of areas
io_uring/zcrx: move freelist lock to struct zcrx
io_uring/zcrx: unmap under netdev lock
io_uring/zcrx: split dmabuf unmap and release
io_uring/zcrx: don't pass ifq_reg to area creation
io_uring/zcrx: add helper for deriving area token
io_uring/zcrx: don't reload skb_shinfo
io_urint/zcrx: narrow var scope in io_zcrx_recv_skb()
io_uring/zcrx: constify area_reg on import
io_uring/zcrx: coalesce same-niov RQEs on refill
io_uring/zcrx: cache RQ tail
io_uring/zcrx: add RQ iterator
io_uring/zcrx: move RQ head/tail to separate cache lines
io_uring/zcrx: scale refilling with large pages
io_uring/io-wq: fix worker accounting when canceling creation callbacks
...
|
|
Rename IO_IMU_DEST and IO_IMU_SOURCE to IO_BUF_DEST and IO_BUF_SOURCE
and export it so subsystems may use it.
This is needed by the io_buffer_register_bvec() path for callers who may
need the buffer to be both readable and writable.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Link: https://patch.msgid.link/20260612184840.4058966-5-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Add io_buffer_register_bvec() for registering a bvec array.
This is a preparatory patch for fuse-over-io-uring zero-copy.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260612184840.4058966-4-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Split the main initialization logic in io_buffer_register_request() into
a helper function.
This is a preparatory patch for supporting kernel-populated buffers in
fuse io-uring, which will be reusing this logic.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260612184840.4058966-3-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Currently, io_buffer_register_bvec() takes in a request. In preparation
for supporting kernel-populated buffers in fuse io-uring (which will
need to register bvecs directly, not through a struct request), rename
this to io_buffer_register_request().
A subsequent patch will commandeer the "io_buffer_register_bvec()"
function name to support registering bvecs directly.
Rename io_buffer_unregister_bvec() to a more generic name,
io_buffer_unregister(), as both io_buffer_register_request() and
io_buffer_register_bvec() callers will use it for unregistration.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260612184840.4058966-2-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
If IORING_URING_CMD_MULTISHOT is set, io_uring_cmd() treats any
non-negative return from ->uring_cmd() as the driver having taken
ownership of the request and returns IOU_ISSUE_SKIP_COMPLETE. But
nothing guarantees that the driver did so, and any handler that just
completes the command inline and returns 0 or a positive result then
leaves the request orphaned, leaking the io_kiocb, the async data,
and the file reference.
The special case isn't needed. ublk returns -EIOCBQUEUED for the
multishot fetch command, which is passed through as-is, and the poll
driven socket timestamp command returns -EAGAIN. Kill it, a multishot
handler that wants to hang on to the request must return -EIOCBQUEUED
or -EAGAIN like any other command.
Fixes: 620a50c92700 ("io_uring: uring_cmd: add multishot support")
Cc: stable@vger.kernel.org
Reported-by: syzbot+a4ccdd7ebf452e4d4701@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/6a7a0194.b50370da.49fe0.0056.GAE@google.com/
Link: https://lore.kernel.org/all/20260811115125.1831170-1-vasilisalmpanis@gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_mem_alloc_compound() allocates get_order(size) pages, which rounds a
region size that is not a power of two up to the next order. The pages
past the region are part of the same allocation and cannot be used for
anything else, but io_create_region() accounts reg->size >> PAGE_SHIFT,
so they are never charged against RLIMIT_MEMLOCK. For a ring with 4096
SQ entries and the default CQ size the region is 37 pages while the
allocation is 64.
Account the tail pages together with the region, and fall back to the
exact sized bulk allocation when they do not fit the limit, so a user
close to their limit still gets the region rather than an error.
io_free_region() gives the same amount back, the compound case being the
one that set IO_REGION_F_SINGLE_REF.
Counting how many 4096 entry rings an unprivileged user can create under
a given RLIMIT_MEMLOCK, before and after:
limit (pages) before after
256 2 2
300 2 2
350 3 2
400 3 3
512 5 4
Five rings under a 512 page limit really pin 640 pages.
Fixes: dfbbfbf19187 ("io_uring: introduce concept of memory regions")
Link: https://lore.kernel.org/all/87ik5ncj8d.fsf@mailhost.krisman.be/
Signed-off-by: Ali Ahmet Memis <ali@iusegentoo.com>
Link: https://patch.msgid.link/20260806180044.275543-1-ali@iusegentoo.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
It's not always possible for the user to predict during registration how
much memory zcrx will need to sustain the traffic. Allow to dynamically
add more areas with a new ctrl code ZCRX_CTRL_ADD_AREA.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/43495ca686713b3f0014c9f47c4c0276d704da4d.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Protect __zcrx_create_area() with pp_lock. It's not needed for now,
nobody can take the lock in parallel, but we'll need it for dynamic area
creation for avoiding races with the device dying.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/a667399971120227a28d54ebabc50e707f717bf3.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Currently, we have only a one area per zcrx instance, and struct
io_zcrx_ifq stores a single pointer. To prepare for adding more areas,
replace it with an array of areas.
We'll be creating them at runtime, and the array is protected by 3
locks: ->pp_lock, ->alloc_lock and ->rq.lock. It takes all of them when
switching arrays, and readers should hold either of them.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/d186adfa0ea26f651804cffef77d477f257eb404.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
freelist_lock, which protects slow path allocations, is currently stored
in struct io_zcrx_area. Once we add support for multiple queues, we'll
need a lock in the zcrx ctx, move it there.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/be7feadff1e210861433f99fe44170aa27ec5b33.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Make sure we unmap areas while closing a queue.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/1c41f349f8bcbcaafb17a9c81d4157aa9d30de93.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Until now unmapping and destroying dmabuf were the same thing. To keep
it consistent with non-dmabuf, split it into two separate helpers. Unmap
destroys mappings and attachements as it should, and release only
putting down the dmabuf fd reference.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/bd8c1fd8dac85ce0cf839b389ecd6050761965ca.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We might want to create an area without having an instance of struct
io_uring_zcrx_ifq_reg. Extract a helper that doesn't have the ifq
registration structure as an argument but takes the buf length
explicitly.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/76751ee4747abb4893058fcaaa36895dfe24a297.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Add zcrx_area_id_to_token() to deduplicate the way the area token is
calculated out of the area index.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/c7752161065a1f7273ce3de9683804dd105c6d53.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Keep skb_shinfo in a local variable so that it doesn't reload it on
every iteration of the loop.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/1e4c864a5ea639803c866aea70b8d5bf558f189f.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A preparation patch that limits scopes of a couple variables in
io_zcrx_recv_skb() and rename them, it makes it easier to reason about
the code.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/182c140502e7be534caf10360714a4d78f2ce5f4.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_import_area() doesn't modify its struct io_uring_zcrx_area_reg
argument, add const to enforce that, it'll make later modifications
easier.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/2cd4a3e43875576587c900ebdf8f1fb4af266e4a.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
With large rx piages I often see >10 sequential RQEs referring to the
same niov. Instead of putting them one by one, count such RQEs during
parsing and batch refcounting for the niov.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/71f11d5de47553f187737e0e892ab783b03f53b1.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The RQ tail is updated by the user space. Cache it to reduce cache line
bouncing. Refilling now tries to exhaust the previous batch of rqes, but
since it could be too low, the iterator is allowed to recalculate the
rqes to process once after synching the tail value.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/b42d2ed8b0e697110b646573b5da2a5f8e85f92e.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Add a iterator structure and helper functions for the refill queue
processing to avoid polluting io_zcrx_ring_refill() with extra state
and logic once it's extended in following patches.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/5d2933ea7bcf9c19c44e43afb3638d7afb362b3c.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
RQ head and tail are currently put into the same cache line, which can
cause false sharing problems when refill is run on another CPU. Put them
into separate cache lines.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Link: https://patch.msgid.link/f769116a3f5267f06cb9a9a819d482ddea2e0852.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_zcrx_ring_refill() caps the loop by mixing the max number of
allocated netmems and the number of available RQEs together, which
caps the number of entries to process the pp cache size. As a result,
when niovs are heavily fragmented, the refilling logic allocates only a
small number of niovs per call on average and sometimes even none.
Keep a separate counter for the number of processed RQ entries, which is
capped by a roughly calculated from the page size value to keep the
cache full. And separately break if it allocates enough niovs.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/143cf439299728759eb0a840c866363fe99293f0.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
create_worker_cb() reserves an io-wq worker slot only after its
task-work callback runs. If the callback is canceled before then,
io_worker_cancel_cb() still decrements acct->nr_workers. When an
existing worker retires with its creation callback pending, that
worker has already decremented the same account's worker count.
The resulting undercount permits worker creation beyond the account's
configured limit. On an AST2600 OpenBMC system, an unchanged sensor
daemon reached 4,291 threads with the original kernel. With an
equivalent downstream fix, 25 passive samples under its normal
workload showed 6-9 threads.
Decrement nr_workers only when the canceled callback is not
create_worker_cb(). Continuation callbacks still release their reserved
slot, and both callback types retain the existing running-count,
reference-count, and create-state cleanup.
Fixes: 1d5f5ea7cb7d ("io-wq: remove worker to owner tw dependency")
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Vishnu Razdan <vrazdan@openai.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260811-vrazdan-io-wq-b4-submit-v1-1-719ced16c921@openai.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
ddir was only ever forwarded to io_compat_msg_copy_hdr, which
never looked at it. Drop it.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-4-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Similar to commit f4eaf8eda89e ("io_uring/rsrc: Drop io_copy_iov in
favor of iovec API"), avoid the custom copy of a single iovec and just
rely on the iovec api. This lets the compat and native paths share the
buffer-select length lookup
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-3-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Similar to commit f4eaf8eda89e ("io_uring/rsrc: Drop io_copy_iov in
favor of iovec API"), avoid custom copy and just rely on the iovec api.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-2-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_req_local_work_add() signals the CQ ring eventfd inline when it is the
one to push the first entry onto ->work_list. For DEFER_TASKRUN rings that
add is frequently done from a waitqueue wakeup handler, where an
arbitrary waitqueue lock is held.
eventfd_signal_mask() only refuses to recurse when current->in_eventfd
is set, but that bit is set by eventfd_signal_mask() itself. If the wake
chain starts somewhere else, signal goes out inline and can feed back
into epoll.
Add IOU_F_TWQ_IN_WAKE, set it on the task_work add done from the three
waitqueue callbacks, and use it to force io_eventfd_signal() down the
existing call_rcu_hurry() deferral instead of signaling inline.
Fixes: 21a091b970cd ("io_uring: signal registered eventfd to process deferred task work")
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20260813133843.2933127-1-4ncienth@gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_import_reg_vec() converts the estimated number of bio_vec entries
into iovec-sized storage when struct bio_vec is larger than struct
iovec. The conversion still multiplies nr_segs by sizeof(struct
bio_vec) in size_t without checking for overflow.
On 32-bit kernels, a registered buffer large enough to make
io_estimate_bvec_size() return 357913942 segments wraps the byte count
from 0x100000008 to 8. io_vec_realloc() then reserves only the input
iovecs plus one extra slot while io_vec_fill_bvec() writes the full
bio_vec array.
Check both the multiplication and the rounding addition before
deriving the replacement iovec count.
Fixes: b4e41050b212 ("io_uring/rsrc: raise registered buffer 1GB limit")
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://patch.msgid.link/20260812203042.720348-1-Jeremy.Jean@oss.cyber.gouv.fr
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
