diff options
| author | Pavel Begunkov <asml.silence@gmail.com> | 2022-07-07 15:13:17 +0100 |
|---|---|---|
| committer | Jens Axboe <axboe@kernel.dk> | 2022-07-24 18:39:17 -0600 |
| commit | e8375e43ca2d67f31b2408092eee4f91e1e140f4 (patch) | |
| tree | 366ecb0bfbd0c068d73620267699b6dc96ec2212 /io_uring/poll.c | |
| parent | b21a51e26e9aed66159c2c18651da22d585d2998 (diff) | |
| download | lwn-e8375e43ca2d67f31b2408092eee4f91e1e140f4.tar.gz lwn-e8375e43ca2d67f31b2408092eee4f91e1e140f4.zip | |
io_uring: consolidate hash_locked io-wq handling
Don't duplicate code disabling REQ_F_HASH_LOCKED for IO_URING_F_UNLOCKED
(i.e. io-wq), move the handling into __io_arm_poll_handler().
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/0ff0ffdfaa65b3d536131535c3dad3c63d9b7bb0.1657203020.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'io_uring/poll.c')
| -rw-r--r-- | io_uring/poll.c | 12 |
1 files changed, 7 insertions, 5 deletions
diff --git a/io_uring/poll.c b/io_uring/poll.c index 77b669b06046..76592063abe7 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -523,8 +523,12 @@ static int __io_arm_poll_handler(struct io_kiocb *req, * io_poll_can_finish_inline() tries to deal with that. */ ipt->owning = issue_flags & IO_URING_F_UNLOCKED; - atomic_set(&req->poll_refs, (int)ipt->owning); + + /* io-wq doesn't hold uring_lock */ + if (issue_flags & IO_URING_F_UNLOCKED) + req->flags &= ~REQ_F_HASH_LOCKED; + mask = vfs_poll(req->file, &ipt->pt) & poll->events; if (unlikely(ipt->error || !ipt->nr_entries)) { @@ -618,8 +622,7 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) * apoll requests already grab the mutex to complete in the tw handler, * so removal from the mutex-backed hash is free, use it by default. */ - if (!(issue_flags & IO_URING_F_UNLOCKED)) - req->flags |= REQ_F_HASH_LOCKED; + req->flags |= REQ_F_HASH_LOCKED; if (!def->pollin && !def->pollout) return IO_APOLL_ABORTED; @@ -876,8 +879,7 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) * If sqpoll or single issuer, there is no contention for ->uring_lock * and we'll end up holding it in tw handlers anyway. */ - if (!(issue_flags & IO_URING_F_UNLOCKED) && - (req->ctx->flags & (IORING_SETUP_SQPOLL | IORING_SETUP_SINGLE_ISSUER))) + if (req->ctx->flags & (IORING_SETUP_SQPOLL|IORING_SETUP_SINGLE_ISSUER)) req->flags |= REQ_F_HASH_LOCKED; ret = __io_arm_poll_handler(req, poll, &ipt, poll->events, issue_flags); |