diff options
| author | Pavel Begunkov <asml.silence@gmail.com> | 2021-03-19 17:22:30 +0000 |
|---|---|---|
| committer | Jens Axboe <axboe@kernel.dk> | 2021-04-11 17:41:58 -0600 |
| commit | 33f993da9829738da3e088fb5d3128880a4137ba (patch) | |
| tree | aea8567125c2dc92d86b77eeee73a5307149c8d4 /fs/io_uring.c | |
| parent | 493f3b158a1e445e24d567847045baf5a723d206 (diff) | |
| download | lwn-33f993da9829738da3e088fb5d3128880a4137ba.tar.gz lwn-33f993da9829738da3e088fb5d3128880a4137ba.zip | |
io_uring: optimise io_uring_enter()
Add unlikely annotations, because my compiler pretty much mispredicts
every first check, and apart jumping around in the fast path, it also
generates extra instructions, like in advance setting ret value.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'fs/io_uring.c')
| -rw-r--r-- | fs/io_uring.c | 14 |
1 files changed, 7 insertions, 7 deletions
diff --git a/fs/io_uring.c b/fs/io_uring.c index 12e2ec7cfba4..13a64b1db3be 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -9183,31 +9183,31 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, size_t, argsz) { struct io_ring_ctx *ctx; - long ret = -EBADF; int submitted = 0; struct fd f; + long ret; io_run_task_work(); - if (flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP | - IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG)) + if (unlikely(flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP | + IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG))) return -EINVAL; f = fdget(fd); - if (!f.file) + if (unlikely(!f.file)) return -EBADF; ret = -EOPNOTSUPP; - if (f.file->f_op != &io_uring_fops) + if (unlikely(f.file->f_op != &io_uring_fops)) goto out_fput; ret = -ENXIO; ctx = f.file->private_data; - if (!percpu_ref_tryget(&ctx->refs)) + if (unlikely(!percpu_ref_tryget(&ctx->refs))) goto out_fput; ret = -EBADFD; - if (ctx->flags & IORING_SETUP_R_DISABLED) + if (unlikely(ctx->flags & IORING_SETUP_R_DISABLED)) goto out; /* |