diff options
| author | Pavel Begunkov <asml.silence@gmail.com> | 2022-03-21 22:02:23 +0000 |
|---|---|---|
| committer | Jens Axboe <axboe@kernel.dk> | 2022-04-24 17:34:16 -0600 |
| commit | b605a7fabb607adbe4ea7cc97b69ae6e0555e7b2 (patch) | |
| tree | 3ddff450eeba7d9f5655f3179c1d5b3222f18a1f /fs/io_uring.c | |
| parent | a538be5be328229d4da3343d4d6514bb4d5c3d5d (diff) | |
| download | lwn-b605a7fabb607adbe4ea7cc97b69ae6e0555e7b2.tar.gz lwn-b605a7fabb607adbe4ea7cc97b69ae6e0555e7b2.zip | |
io_uring: move poll recycling later in compl flushing
There is a new (req->flags & REQ_F_POLLED) check in
__io_submit_flush_completions() for poll recycling, however
io_free_batch_list() is a much better place for it. First, we prefer it
after putting the last req ref just to avoid potential problems in the
future. Also, it'll enable the recycling for IOPOLL and also will place
it closer to all other req->flags bits clean up requests.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/31dfe1dafda66ba3ce36b301884ec7e162c777d1.1647897811.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'fs/io_uring.c')
| -rw-r--r-- | fs/io_uring.c | 18 |
1 files changed, 9 insertions, 9 deletions
diff --git a/fs/io_uring.c b/fs/io_uring.c index fddfbf0f9310..c6746b416816 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2635,6 +2635,15 @@ static void io_free_batch_list(struct io_ring_ctx *ctx, if (!req_ref_put_and_test(req)) continue; } + if ((req->flags & REQ_F_POLLED) && req->apoll) { + struct async_poll *apoll = req->apoll; + + if (apoll->double_poll) + kfree(apoll->double_poll); + list_add(&apoll->poll.wait.entry, + &ctx->apoll_cache); + req->flags &= ~REQ_F_POLLED; + } io_queue_next(req); if (unlikely(req->flags & IO_REQ_CLEAN_FLAGS)) io_clean_op(req); @@ -2673,15 +2682,6 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) if (!(req->flags & REQ_F_CQE_SKIP)) __io_fill_cqe_req(req, req->result, req->cflags); - if ((req->flags & REQ_F_POLLED) && req->apoll) { - struct async_poll *apoll = req->apoll; - - if (apoll->double_poll) - kfree(apoll->double_poll); - list_add(&apoll->poll.wait.entry, - &ctx->apoll_cache); - req->flags &= ~REQ_F_POLLED; - } } io_commit_cqring(ctx); |