io_uring: don't split 128b SQEs at the SQPOLL cap

If an SQPOLL thread serves more than one ring, __io_sq_thread() submits
no more than IORING_SQPOLL_CAP_ENTRIES_VALUE entries per ring and round,
for fairness. On a ring with IORING_SETUP_SQE_MIXED the cap may fall
between the two entries of a 128b SQE. io_init_req() finds that just one
entry is left of what it was asked to submit and fails the request with
-EINVAL, although the second entry is sitting in the SQ ring.

In the next round the second entry is taken for an SQE of its own, so on
top of the failed request the application gets whatever its payload
amounts to when it is read as an SQE.

The application has done nothing wrong, and it can't know where the cap
is going to be applied. For SQPOLL, let the request have its second entry
if the SQ ring holds it, in excess of the number of entries that were to
be submitted.

Fixes: 1cba30bf9fdd ("io_uring: add support for IORING_SETUP_SQE_MIXED")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-24-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

1 files changed, 16 insertions, 6 deletions

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 01187f00943a..7092efc22dd2 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1818,17 +1818,27 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req,
 		 * A 128b op on a non-128b SQ requires mixed SQE support as
 		 * well as 2 contiguous entries.
 		 */
-		if (!(ctx->flags & IORING_SETUP_SQE_MIXED) || *left < 2 ||
+		if (!(ctx->flags & IORING_SETUP_SQE_MIXED) ||
 		    (unsigned)(sqe - ctx->sq_sqes) >= ctx->sq_entries - 1)
 			return io_init_fail_req(req, -EINVAL);
+		if (*left >= 2) {
+			/*
+			 * A 128b operation on a mixed SQ uses two entries, so
+			 * we have to increment the head and cached refs, and
+			 * decrement what's left.
+			 */
+			current->io_uring->cached_refs++;
+			(*left)--;
+		} else if (!(ctx->flags & IORING_SETUP_SQPOLL) ||
+			   !__io_sqring_entries(ctx)) {
+			return io_init_fail_req(req, -EINVAL);
+		}
 		/*
-		 * A 128b operation on a mixed SQ uses two entries, so we have
-		 * to increment the head and cached refs, and decrement what's
-		 * left.
+		 * else: the SQPOLL thread caps the number of entries it submits
+		 * in one go, which is none of the application's business. Take
+		 * the 2nd entry on top.
 		 */
-		current->io_uring->cached_refs++;
 		ctx->cached_sq_head++;
-		(*left)--;
 	}
 
 	if (unlikely(sqe_flags & ~SQE_COMMON_FLAGS)) {