| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring update from Jens Axboe:
"On top of the usual cleanups and fixes, the bigger items in here are:
- zcrx work, most of it centered around adding dynamic area
provisioning, plus a bunch of prep and cleanups leading up to it:
scale refilling with large pages, coalesce same-niov RQEs on
refill, separate the RQ head/tail cache lines and cache the RQ
tail, and rework the area creation locking.
- Fix the futex inflight accounting so that only private futex waits
are marked inflight, and don't mark wake requests as inflight at
all.
- Drop the custom iov copy in the buffer select prep and msg header
copy paths, using the generic helpers instead.
- Fix a folio size overflow in io_vec_fill_bvec(), and account the
pages a compound region really uses in the memmap path.
- Fix an iovec leak in uring_cmd when the async cmd isn't recycled,
skip the blocking task work for io_uring_cmd_issue_blocking(), and
don't skip completion for a synchronous multishot cmd
- Defer eventfd signaling when queued from a wakeup handler
- Fix io-wq worker accounting when canceling creation callbacks
- Annotate remote tasks for kcoverage"
* tag 'for-7.3/io_uring-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (30 commits)
io_uring: Add missing include for ITER_SOURCE and ITER_DEST
io_uring/uring_cmd: don't skip completion for a synchronous multishot cmd
io_uring/memmap: account the pages a compound region really uses
io_uring/zcrx: add dynamic area provisioning
io_uring/zcrx: lock area creation with pp_lock
io_uring/zcrx: keep array of areas
io_uring/zcrx: move freelist lock to struct zcrx
io_uring/zcrx: unmap under netdev lock
io_uring/zcrx: split dmabuf unmap and release
io_uring/zcrx: don't pass ifq_reg to area creation
io_uring/zcrx: add helper for deriving area token
io_uring/zcrx: don't reload skb_shinfo
io_urint/zcrx: narrow var scope in io_zcrx_recv_skb()
io_uring/zcrx: constify area_reg on import
io_uring/zcrx: coalesce same-niov RQEs on refill
io_uring/zcrx: cache RQ tail
io_uring/zcrx: add RQ iterator
io_uring/zcrx: move RQ head/tail to separate cache lines
io_uring/zcrx: scale refilling with large pages
io_uring/io-wq: fix worker accounting when canceling creation callbacks
...
|
|
It's not always possible for the user to predict during registration how
much memory zcrx will need to sustain the traffic. Allow to dynamically
add more areas with a new ctrl code ZCRX_CTRL_ADD_AREA.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/43495ca686713b3f0014c9f47c4c0276d704da4d.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
"Notification" is too long and the abbreviated version is used in
several places, which is inconsistent and more ambiguous for users.
Rename it to event, which is easier to keep consistent. To keep the
change small, only change uapi/ + do necessary fix ups, and the rest of
internals can be adjusted in the next release.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/f95ca6717da3c8d3649a1a7f0d883a563f545052.1784726895.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
ZCRX_NOTIF_NO_BUFFERS tells when page pool fails to allocate memory from
zcrx. "No buffers" could be more confusing, rename it to
ZCRX_NOTIF_ALLOC_FAIL.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/29bd4fc069bc89691868beba0627ffbe570c2722.1784726895.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Keep zcrx statistics generic and don't stick "notif" to its uapi
definitions. Stats dosn't need to be bound to notification details,
it makes it cleaner and more readable.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/6a39676b6f71b67d3f89c6ebab7a3739873834a3.1784726895.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Add support for an optional stats struct embedded in the refill queue
region, allowing userspace to monitor copy-fallback in real-time.
Userspace queries the stats struct size and alignment via
IO_URING_QUERY_ZCRX_NOTIF (notif_stats_size / notif_stats_alignment),
then provides a stats_offset in zcrx_notification_desc pointing to a
location within the refill queue region.
The kernel updates the stats counters in-place on every copy-fallback
event.
Signed-off-by: Clément Léger <cleger@meta.com>
[pavel: rename io_uring_zcrx_notif_stats]
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/f6af5a21015efea4b733b9d77aba22c637788fe4.1779189667.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Add a ZCRX_NOTIF_COPY notification type to signal userspace when a
received fragment could not be delivered using zero-copy and was
instead copied into a buffer.
Signed-off-by: Clément Léger <cleger@meta.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/3d54bcd8bf10b3a1e88beb0cd39c40c3937bea4f.1779189667.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
There are currently no easy ways for the user to know if zcrx is out of
buffers and page pool fails to allocate. Add uapi for zcrx to communicate
it back.
It's implemented as a separate CQE, which for now is posted to the creator
ctx. To use it, on registration the user space needs to pass an instance
of struct zcrx_notification_desc, which tells the kernel the user_data
for resulting CQEs and which event types are expected / allowed.
When an allowed event happens, zcrx will post a CQE containing the
specified user_data, and lower bits of cqe->res will be set to the event
mask. Before the kernel could post another notification of the given
type, the user needs to acknowledge that it processed the previous one
by issuing IORING_REGISTER_ZCRX_CTRL with ZCRX_CTRL_ARM_NOTIFICATION.
The only notification type the patch implements is
ZCRX_NOTIF_NO_BUFFERS, but we'll need more of them in the future.
Co-developed-by: Vishwanath Seshagiri <vishs@meta.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Vishwanath Seshagiri <vishs@meta.com>
Link: https://patch.msgid.link/35cd307a03a43583838a2e151fc641c69abd786f.1779189667.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Allow creating a zcrx instance without attaching it to a net device.
All data will be copied through the fallback path. The user is also
expected to use ZCRX_CTRL_FLUSH_RQ to handle overflows as it normally
should even with a netdev, but it becomes even more relevant as there
will likely be no one to automatically pick up buffers.
Apart from that, it follows the zcrx uapi for the I/O path, and is
useful for testing, experimentation, and potentially for the copy
receive path in the future if improved.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/674f8ad679c5a0bc79d538352b3042cf0999596e.1774261953.git.asml.silence@gmail.com
[axboe: fix spelling error in uapi header and commit message]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Split out zcrx uapi into a separate file. It'll be easier to manage it
this way, and that reduces the size of a not so small io_uring.h. Since
there are users that expect that zcrx definitions come with io_uring.h,
it includes the new file.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
