| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring dma-buf support from Jens Axboe:
"This adds native dma-buf support to the I/O path, exposed through
io_uring. It's been in development for a while and should be ready to
go now.
- dma-buf file I/O infrastructure: a dma_buf_io_ctx created through a
new ->init_dma_buf_io_ctx file operation, and a dma_buf_io_map
holding the device DMA mapping. The two are split so a mapping can
be invalidated and re-established underneath a live context, with
the map counting in-flight requests so it can be torn down safely.
- A new iov_iter type for dmabuf maps, and a dma map backed bio type
for the block layer, with raw block device support on top.
- nvme-pci backend implementing dma-buf backed requests, including
SGL support.
- io_uring dmabuf backed registered buffers, built on a reworked
buffer registration structure with an extended update path, regbuf
import flags, an uncloneable flag for buffer types that can't be
cloned, and cloning support for registered dma-bufs"
* tag 'for-7.4/io_uring-dmabuf-20261023' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
io_uring/dma-buf: don't fail fixed buffers without dma-buf support
nvme-pci: detach dma-bufs only on removal
block: reject dma-bufs larger than 4GiB
io_uring/dma-buf: allow cloning registered dma-bufs
dma-buf-io: document dma_buf_io_init_map
dma-buf-io: fix typos in kernel docs
block: fix indention for dma-buf checks
io_uring/rsrc: add dmabuf backed registered buffers
io_uring/rsrc: add regbuf import flags
io_uring/rsrc: add uncloneable regbuf flag
io_uring/rsrc: extend buffer update
io_uring/rsrc: introduce buf registration structure
nvme-pci: add SGL support for the dmabuf path
nvme-pci: implement dma-buf backed requests
nvme-pci: rename nvme_pci_sgl_set_data to nvme_pci_dma_iter_set_sgl
block: add dma-buf support for raw bdev
block: introduce dma map backed bio type
block: always adjust bi_offset on bio_advance_iter
iov_iter: add iterator type for dmabuf maps
dma-buf: introduce initial file I/O infrastructure
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block updates from Jens Axboe:
- NVMe updates via Keith:
- Apple M4 (t8132) ANS2 NVMe support
- nvme-multipath sysfs attributes for path_state, iopolicies, and
fail_on_ctrl_loss, plus path state evaluation fixes
- nvme-tcp parallelized I/O queue allocation and startup, and using
the caller's network namespace
- nvmet passthru UAF fix and rework, PR preempt use-after-free
fixes, auth hardening, and traffic based keep alive support
- DH-HMAC-CHAP secret attributes readable by root only
- Various other fixes and cleanups
- swim driver overhaul and fixes, plus floppy fixes, via a pull from
Denis
- Block device buffered and direct I/O rework from Tal: use
iomap_dirty_folio, take i_rwsem for the write fallback and splice
read, honor IOCB_NOWAIT in the buffered read path, and reject atomic
write fallback to buffered I/O
- Large zone revalidation rework: serialize revalidation, handle
capacity changes by dropping and retrying zone write plugs, propagate
read-only and offline conditions, and treat offline and read-only
zones as dead
- ublk robustness work around canceled FETCH rounds and going live over
canceled io commands, dropping the default I/O scheduler, and a batch
of selftest additions
- blk-iocost flush cost support and zone append accounting
- rust block fixes for Send/Sync bounds on GenDisk and TagSet, and
import style cleanups
- Atomic write fixes capping size by the PI buffer constraints and
failing short atomic pins
- Skip the redundant flush for O_DSYNC direct writes, and only use
REQ_FUA when the device supports it
- Store GPT attributes as a raw value
- Remove support for swapping loop fds
- Various other fixes and cleanups all over
* tag 'for-7.4/block-20261021' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (144 commits)
block: restore the iterator when rejecting a misaligned bio
blk-mq: clear stale inactive flag on hctx reuse
block, bfq: initialize active lists for all actuators
zloop: print device that had errors
ublk: pin module while release work is pending
blk-mq: drop the tag_set argument from blk_mq_free_rq_map/tags
blk-mq: share one tags_srcu instance across all tag sets
blk-iocost: fix stale comment in ioc_rqos_throttle()
blk-iocost: account zone append completions in latency stats
blk-iocost: charge zone appends as writes
blk-iocost: add flush cost support with the flushiops model parameter
ublk: use global index for reported zones
selftests: ublk: add test for going live over canceled io commands
selftests: ublk: stop a batch thread when PREP_IO_CMDS fails
selftests: ublk: move the control command helpers into ctrl.c
ublk: refuse to go live after an io command was canceled
ublk: let STOP_DEV cancel the server's commands before its release
ublk: allow a NULL io->cmd in the cancel callback
ublk: reset the FETCH round under ub->mutex
ublk: reset the FETCH round in release also without a disk
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/modules/linux
Pull module updates from Petr Pavlu:
- Ignore local labels and mapping symbols during module load
These symbols were already ignored during kallsyms address-to-symbol
lookups. Filter them out earlier during module loading to reduce
lookup overhead and prevent them from appearing in /proc/kallsyms.
- Fix potentially incorrect access of SetOnce<T> when setting Rust
module parameters.
- Fix a potential read of an uninitialized count when printing taint
information for unloaded modules.
- Allocate the kmod argument array using kmalloc_array() in preparation
for conversion to type-aware allocators.
- Clean up includes in linux/kmod.h
Add explicit includes of linux/umh.h to files that use usermode
helpers. Then clean up the #include list in linux/kmod.h, notably
removing the compatibility linux/umh.h include.
- Remove the error-injection.h include from linux/module.h
Add explicit includes of linux/error-injection.h to files that use
the ALLOW_ERROR_INJECTION() macro and currently rely on it being
provided indirectly via linux/module.h. Then remove the unnecessary
linux/error-injection.h include from linux/module.h.
- Replace __MODULE_STRING() with __stringify() in various drivers in
preparation for removing the former macro.
* tag 'modules-7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/modules/linux:
drivers: Replace __MODULE_STRING() with __stringify()
module: Fix uninitialized count read of unloaded taint tracking record
rust: module_param: Fix potentially incorrect access of `SetOnce<T>`
module: Remove the error-injection.h include from linux/module.h
drm/xe: Include error-injection.h for ALLOW_ERROR_INJECTION()
drm/i915: Include error-injection.h for ALLOW_ERROR_INJECTION()
syscalls: Include error-injection.h for ALLOW_ERROR_INJECTION()
net: Include error-injection.h for ALLOW_ERROR_INJECTION()
block: Include error-injection.h for ALLOW_ERROR_INJECTION()
module: Bring includes in linux/kmod.h up to date
time/jiffies: Include linux/sysctl.h for proc_int_u2k_conv_uop(), ...
umh, treewide: Explicitly include linux/umh.h where needed
modules/kmod: Allocate argv with kmalloc_array()
module/kallsyms: Ignore local labels and mapping symbols during module load
module/kallsyms: Refactor current symbol filtering into is_core_symbol()
module/kallsyms: Rename is_mapping_symbol() to is_ignored_kernel_symbol()
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
Pull cgroup updates from Tejun Heo:
- A write to cgroup.subtree_control or cgroup.type could hang in D
state for as long as a task pinning a dying controller took to exit,
as syzbot reported. The wait is now interruptible.
- cpuset: the deadline bandwidth check ran on cpusets that weren't
valid partition roots, failing cpuset writes with -EBUSY while
deadline tasks existed. The CPU count the fair class takes from
cpuset assumed a shared hierarchy that cgroup1 may not have.
- Finding a cgroup's effective controller state no longer walks up the
hierarchy, for per-I/O users such as nvmet. io.stat reads of cgroups
without block devices skip the stat flush.
- cpuset v1-only code moved to the v1 file and the exclusive flag
dropped from v2, with documentation corrections for partition roots
and boot-isolated CPUs.
- Dead cgroups fail writes with -ENODEV consistently, plus small
cleanups.
- Selftest fixes, most visibly for tests that skipped on machines with
many mounts, and a new pids.events notification test.
* tag 'cgroup-for-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup: (27 commits)
selftests/cgroup: test pids.events notification
selftests: cgroup: make test_memcg_sock check flushed sock stats
selftests: cgroup: skip test_memcg_sock under cgroup.memory=nosocket
selftests/cgroup: Free the paths returned by cg_control()
selftests: memcg: Don't close an uninitialized fd
cgroup: Track the effective css in each cgroup
cgroup/cpuset: Move cpuset_update_flag() to cpuset-v1.c
selftests: cgroup: Preserve command arguments in with_stress.sh
selftests: cgroup: Enable the cpu controller in test_cpu
cgroup/cpuset: Use cpuset_v2() in cpuset_num_cpus()
selftests/cgroup: add settings file with a longer timeout
cgroup/cpuset: Remove CS_CPU_EXCLUSIVE handling code from v2
cgroup/cpuset: Run SCHED_DEADLINE shrink test on valid partition root only
docs: cgroup-v2: Describe effective CPUs of partition roots
docs: cgroup-v2: Describe boot-isolated CPUs in cpuset.cpus.isolated
cgroup/cpuset: Remove redundant sched domain rebuild from update_prstate()
selftests/cgroup: don't crash on a truncated mount entry in cg_find_root()
selftests/cgroup: read the entire file in read_text()
blk-cgroup: skip rstat flush for cgroups without blkgs
cgroup: use assign_bit() where applicable
...
|
|
The dma-buf offset is set in the 32-bit bvec_iter bi_offset. Reject
buffers that can carry an offset that would wrap.
Fixes: 839231d19393 ("block: add dma-buf support for raw bdev")
Signed-off-by: Keith Busch <kbusch@kernel.org>
Link: https://patch.msgid.link/20261021184658.644645-2-kbusch@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
bio_iov_iter_get_pages() can extract some iovecs before encountering a
misaligned one. It releases the pages and rejects the entire bio, but
leaves the supplied iterator advanced.
An iomap direct-I/O caller may subsequently reexpand the iterator's
byte count and fall back to buffered I/O. The reexpanded count then
includes bytes before the iterator's current segment, potentially
causing iterate_iovec() to read beyond the imported iovec array.
In a reproducer using an initial empty iovec followed by 10-byte and
1015-byte segments, writev() returned 1015 instead of 1025 and KASAN
reported an 8-byte read beyond the nine-element iovec allocation.
Saving and restoring the iterator on the rejected-bio path makes
writev() return 1025 and eliminates the KASAN report.
Restore the entire iterator rather than reverting a byte count:
advancing past an empty segment changes its position without consuming
any bytes.
Leave successful and partially accepted bio paths unchanged.
Fixes: 14b007e17881 ("block: validate user space vectors during extraction")
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reported-by: Haocheng Yu <l1za0.sec@gmail.com>
Closes: https://lore.kernel.org/20261018081203.2431035-1-l1za0.sec@gmail.com
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Tal Zussman <tz2294@columbia.edu>
Cc: Daehyeon Ko <4ncienth@gmail.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Hannes Reinecke <hare@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20261020170250.014BD1F000C9@smtp.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
When the last CPU mapped to an hctx goes offline,
blk_mq_hctx_notify_offline() sets BLK_MQ_S_INACTIVE. Updating the
number of hardware queues exits every hctx and parks it on
q->unused_hctx_list. A later update can reuse a parked hctx through
blk_mq_alloc_and_init_hctx(), at any queue index, and nothing clears
the flag.
If the reused hctx is mapped to an online CPU, blk_mq_get_tag() keeps
returning BLK_MQ_NO_TAG for it. __blk_mq_alloc_requests() sleeps and
retries, expecting CPU hotplug to move the task to another CPU, so a
task that stays on that CPU never gets a tag. Onlining the original CPU
doesn't recover it either, because blk_mq_hctx_notify_online() only
clears the flag on the hctx that CPU now maps to.
Clear BLK_MQ_S_INACTIVE in blk_mq_init_hctx() so a reused hctx starts
like a freshly allocated one. A reused hctx has no cpuhp instances at
that point, so the offline callback can't set the flag again before
blk_mq_add_hw_queues_cpuhp() registers it. Clearing the flag in
blk_mq_exit_hctx() is not enough, because a parked hctx keeps its cpuhp
instances until blk_mq_remove_hw_queues_cpuhp(). Preserve the other
hctx state flags.
Reproduced with null_blk on v7.3 in a four-vCPU arm64 QEMU/KVM guest:
modprobe null_blk nr_devices=0
mkdir /sys/kernel/config/nullb/nullb0
cd /sys/kernel/config/nullb/nullb0
echo 4 > submit_queues; echo 1 > memory_backed; echo 16 > size
echo 1 > power
echo 0 > /sys/devices/system/cpu/cpu3/online
echo 2 > submit_queues
echo 4 > submit_queues
Offlining CPU3 marks hctx3 inactive. After the two updates it comes back
as hctx1 and is mapped to CPU0:
# cat /sys/block/nullb0/mq/1/cpu_list
0
# cat /sys/kernel/debug/block/nullb0/hctx1/state
INACTIVE
A 4K O_DIRECT read pinned to CPU0 hangs in D state, while the same read
on CPU1 and CPU2 completes. Onlining CPU3 again leaves hctx1 inactive
and the CPU0 read still hangs. With this change hctx1 state is empty and
the reads on CPU0, CPU1 and CPU2 complete, before and after onlining
CPU3.
Fixes: bf0beec0607d ("blk-mq: drain I/O when all CPUs in a hctx are offline")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Nirmoy Das <nirmoyd@nvidia.com>
Link: https://patch.msgid.link/20261020125129.430600-1-nirmoyd@nvidia.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull iomap updates from Christian Brauner:
"This contains the iomap changes for this cycle.
The bulk is lazy bounce buffering for checksummed reads. XFS bounce
buffers every direct read on a device that requires stable pages for
checksumming so that userspace modifying the buffer while the read is
in flight can't corrupt the data and fail the checksum.
That burns DRAM bandwidth and CPU for copies that are almost never
needed and, since the bounce buffer was a bio_vec, produced smaller
and unaligned I/Os with 4k pages, i.e. 1 MB minus 4k.
Now the read goes to the user buffer first and only a checksum failure
allocates a bounce buffer and reads again. Integrity verification
moves out of iomap into the filesystem so it can handle errors its own
way, T10 PI handling for direct I/O moves into ->submit_io() to match
buffered I/O, and a sysfs knob restores the old behaviour for the rare
application that really does modify in-flight buffers.
The block side gains bio_prepare_reissue(), exports
fs_bio_integrity_{alloc,free}(), caps atomic writes by the PI buffer
size and fixes an integer overflow in max_integrity_io_size() and a PI
verification bug with an initial bvec offset. The old read side bounce
buffering in the block layer is removed. This is a branch shared with
the block tree and may also reach you through the block pull request.
Three error handling regressions are fixed:
- A failed ->writeback_submit() left wpc->wb_ctx pointing at an
already completed ioend which was then submitted a second time and
corrupted ip->i_ioend_list on XFS
- iomap_fiemap() lost an iteration error when it flushed the pending
extent and marked a truncated list as complete
- A failed direct I/O bio lost its error when the sub-block tail
zeroing succeeded. An XFS error tag to inject the writeback failure
comes with it.
iomap_dio_hole_iter() respects the block size alignment when
iov_iter_zero() stops short at a page boundary, which btrfs with 8k
blocks on 4k pages hit as an assertion since it disables page faults
during direct I/O. The homegrown deferral of error completions to task
context is replaced with bio_complete_in_task(). Plus an unused
parameter, two typo fixes and a kernel-doc fix"
* tag 'vfs-7.4-rc1.iomap' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (34 commits)
fs/iomap: document ret parameter of iomap_iter_continue
iomap: follow the alignment requirement for iomap_dio_hole_iter()
iomap: don't lose a failed direct I/O bio's error when zeroing the tail
iomap: don't lose a fiemap iteration error when emitting the last extent
xfs: add an error tag to inject a ->writeback_submit() failure
iomap: don't resubmit an ioend after ->writeback_submit() failed
iomap: use bio_complete_in_task for buffered simple dio read failures
iomap: use bio_complete_in_task for buffered write failures
iomap: use bio_complete_in_task for buffered read failures
block,iomap: remove the old read side bounce buffering support
xfs: log a message at mount time when using integrity protection
xfs: add error injection for lazy bounce buffering
xfs: add support for lazy direct read bounce buffering
iomap,xfs: move integrity verification to the file system
xfs: use BIO_COMPLETE_IN_TASK for bounce buffered read I/Os
iomap: better read bounce buffering support
block,iomap: fix protection information verification with initial bvec offset
xfs: move PI generation into xfs_submit_zoned_bio
iomap,xfs: move T10 PI handling for direct I/O into ->submit_io
iomap: add a IOMAP_IOEND_INTEGRITY flag
...
|
|
Commit 2d31c684a053 ("block, bfq: inject I/O to underutilized
actuators") extended bfqd->active_list from a single list_head to an
array indexed by actuator, but bfq_init_queue() still only initialized
the first two entries:
INIT_LIST_HEAD(&bfqd->active_list[0]);
INIT_LIST_HEAD(&bfqd->active_list[1]);
For devices with more than two independent access ranges, the remaining
list heads stay zeroed by kzalloc() and are never set up as empty lists.
When BFQ later adds a bfq_queue to one of those lists, list_add() writes
through head->next, which is NULL, corrupting the list or crashing the
kernel.
Initialize every actuator's active list using bfqd->num_actuators,
which is already set up earlier in bfq_init_queue().
Fixes: 2d31c684a053 ("block, bfq: inject I/O to underutilized actuators")
Signed-off-by: Li RongQing <lirongqing@baidu.com>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/20261019051137.1715-1-lirongqing@baidu.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
Pull SCSI fixes from James Bottomley:
"All fixes in drivers (or the tape ULD) and one core fix to bsg (the
block ioctl equivalent of sg_io).
The biggest is the race in auth updates fix in iscsi configfs, but the
next biggest one is in bsg but the generic pieces of that are mostly
threading reply_len"
* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
scsi: bsg: Pass reply_len to bsg_setup_queue() to fit UFS ARPMB replies
scsi: libsas: Fix SMP IO deadlock during HA resume
scsi: libiscsi: Validate the task named by a management response
scsi: libiscsi: Validate the ITT reflected in a Reject PDU
scsi: libiscsi: Reject a negative task index from parse_pdu_itt
scsi: target: iscsi: Fix race in discovery auth updates
scsi: target: Disable interrupts while holding delayed_cmd_lock
scsi: st: Fix use-after-free race between st_remove() and st_open()
scsi: st: Save the visited block for the partition the drive is in
scsi: xen/scsifront: Check for a NULL shadow entry on a backend response
|
|
Commit 9ad8e5af3279 ("blk-mq: Pass tag_set to blk_mq_free_rq_map/tags")
added the argument so that both helpers could reach the tag set's
tags_srcu. Now that the SRCU instance is global, nothing uses it.
Remove it.
No functional change.
Signed-off-by: Qiliang Yuan <odys.yuan@gmail.com>
Reviewed-by: Ming Lei <tom.leiming@gmail.com>
Acked-by: Paul E. McKenney <paulmck@kernel.org>
Link: https://patch.msgid.link/20261001-bug-blk-mq-global-tags-srcu-v1-2-e8415da10d1b@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Freed request pages and flush queues are handed to call_srcu() so that
concurrent tag iterators never see them disappear. Each tag set embeds
its own srcu_struct for this, which blk_mq_free_tag_set() has to drain
with srcu_barrier() before cleanup_srcu_struct(). The barrier waits for
a full SRCU grace period even when the callbacks are long done, adding
1-3 ms to every tag set teardown. Drivers that free tag sets under a
lock make it worse: ublk does it under the global ublk_ctl_mutex, so
deleting devices from many threads serializes on that grace period.
The callbacks free memory that only blk-mq owns. They touch neither
the tag set nor driver code, so they don't need a per-set domain.
Replace the per-set srcu_struct with a single global one. Freeing a tag
set no longer waits for a grace period, and each tag set no longer
allocates per-CPU SRCU state. The read side is unchanged. The only cost
is that a long-running iterator can now delay these frees for other tag
sets too, which only postpones freeing memory.
On 7.3-rc5 in a 16-vCPU VM, with 1 queue and depth 16 per ublk device,
median of 3 runs (5 for the 16-thread null_blk case):
before after
ublk DEL_DEV p50, serial 3.45 ms 0.17 ms
ublk delete 1000 devices, serial 19.5 s 14.9 s
ublk delete 4000 devices, 16 thr 19.0 s 2.9 s
null_blk destroy 500 devs, serial 9.97 s 9.25 s
null_blk destroy 256 devs, 16 thr 5.15 s 4.30 s
Fixes: ad0d05dbddc1 ("blk-mq: Defer freeing of tags page_list to SRCU callback")
Signed-off-by: Qiliang Yuan <odys.yuan@gmail.com>
Reviewed-by: Ming Lei <tom.leiming@gmail.com>
Acked-by: Paul E. McKenney <paulmck@kernel.org>
Link: https://patch.msgid.link/20261001-bug-blk-mq-global-tags-srcu-v1-1-e8415da10d1b@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
There is a weird indention I botched while rebasing, fix it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/de91f480f8507d44ddf99c193ef63dec0f02c749.1792142187.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The comment says that priority-inversion IOs are "punted to
@ioc->aux_iocg", but no aux_iocg field ever existed in struct ioc.
The comment was introduced already stale by commit da437b95db83
("blk-iocost: grab ioc->lock for debt handling"). Update it to
describe the current use_debt / iocg->abs_vdebt mechanism.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260922043856.2020116-5-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
ioc_rqos_done() only accounts READ and WRITE completions, so zone
append completions are excluded from the latency window: the vrate
feedback loop cannot see ZA-induced latency, leaving it unable to
respond to device saturation caused by zone appends. Similarly,
calc_size_vtime_cost_builtin() does not classify zone append as a
write operation.
Charging (the cost model) and feedback (the latency window) are
separate mechanisms, so this is not covered by the previous patch
that prices zone appends.
Treat zone append as WRITE for both latency accounting and cost
classification.
Fixes: 0512a75b98f8 ("block: Introduce REQ_OP_ZONE_APPEND")
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260922043856.2020116-4-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Zone append is a primary write operation for zoned devices; zoned xfs
and btrfs use it for data writes. It is priced at zero, so the zone
append portion of zoned workloads runs outside the controller: a
1%-weight cgroup issued 16000 appends at zero cost on a zoned null_blk.
Treat REQ_OP_ZONE_APPEND the same as REQ_OP_WRITE so it goes through
the existing seq/rand classification and charging.
Fixes: 0512a75b98f8 ("block: Introduce REQ_OP_ZONE_APPEND")
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260922043856.2020116-3-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The builtin linear cost model defines coefficients only for READ and
WRITE, so the flush component of IOs is priced at zero: standalone
flushes (the dataless REQ_OP_WRITE | REQ_PREFLUSH bios from
blkdev_issue_flush()) and the pre-flush the block layer issues ahead of
data-bearing REQ_PREFLUSH bios are both free. A cgroup limited to 1%
weight could issue an unbounded number of flushes without being
throttled: an fsync loop produced ~510k flushes in 12s with
cost.usage staying at zero, monopolizing the device while iocost
reported no activity. On ext4, a write+fsync workload showed the same
gap: the write component was charged but the flush component was not.
A flush is not like a write and the write coefficients say nothing
about what it costs, so instead of pricing it off them, add a flushiops
entry to io.cost.model, following the existing iops parameters: it sets
the rate at which flushes are charged, LCOEF_FLUSH =
VTIME_PER_SEC / flushiops. The linear model cannot express how the
cost of a flush depends on the preceding writes, so this is a
per-request policy charge: a user-tunable knob rather than a hardware
property. A bio with REQ_PREFLUSH is charged one flush on top of its
data cost, and a bio with REQ_FUA one more flush on devices without
native FUA support. Zero (the default and the builtin profiles) means
no charge, so nothing changes until the parameter is configured.
Also skip the iocg->cursor update for dataless bios: they only reach it
once priced, and their bi_sector is not a data position, so setting
the cursor from it would misclassify the following IOs.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260922043856.2020116-2-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Commit c0e473a0d226 ("block: fix race between set_blocksize and read
paths") added shared inode locking around ordinary buffered writes,
but missed the direct-I/O fallback path.
Move the lock into blkdev_buffered_write() so both paths serialize
with set_blocksize().
Also correct the comment claiming that this path takes invalidate_lock;
it only takes i_rwsem.
Fixes: c0e473a0d226 ("block: fix race between set_blocksize and read paths")
Signed-off-by: Julian Sun <sunjunchao@bytedance.com>
Link: https://patch.msgid.link/20261015091105.1315605-3-sunjunchao@bytedance.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blkdev_write_iter() rejects NOWAIT buffered writes, but does not
enforce this restriction when direct I/O falls back to buffered I/O.
Successful fallback waits for writeback, violating NOWAIT semantics.
Reject NOWAIT requests in blkdev_buffered_write() with -EAGAIN.
Fixes: c35fc7a5abae ("block_dev: support RFW_NOWAIT on block device nodes")
Signed-off-by: Julian Sun <sunjunchao@bytedance.com>
Link: https://patch.msgid.link/20261015091105.1315605-2-sunjunchao@bytedance.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
bsg-lib allocates a 96-byte (SCSI_SENSE_BUFFERSIZE) reply buffer per
job, but struct ufs_rpmb_reply is 104 bytes: a 40-byte struct
ufs_bsg_reply followed by a 64-byte EHS. When an Advanced RPMB command
returns an EHS, ufshcd_advanced_rpmb_req_handler() copies it right after
the 40-byte header and writes 8 bytes past the end of the buffer.
Let each BSG transport pass its reply buffer size to bsg_setup_queue()
and have ufs_bsg request sizeof(struct ufs_rpmb_reply), so the full
reply reaches user space. Other transports keep SCSI_SENSE_BUFFERSIZE.
Fixes: 6ff265fc5ef6 ("scsi: ufs: core: bsg: Add advanced RPMB support in ufs_bsg")
Cc: stable@vger.kernel.org
Signed-off-by: Stanley Jhu <stanleyjhu@google.com>
Link: https://patch.msgid.link/20261013132708.1669415-1-stanleyjhu@google.com
Signed-off-by: Martin K. Petersen (Oracle) <mkp@kernel.org>
|
|
Add a simple proxy implementation of init_dma_buf_io_ctx() forwarding
the call to a new struct block_device_operations operation. Also reject
dma-buf backed iterators for buffered IO.
Reviewed-by: Christoph Hellwig <hch@lst.de>
[pavel: reject dma-buf without O_DIRECT]
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/c01c4709dc55a12fb0f2fe1ec847b5f1f0c92789.1791975048.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Premapped buffers don't require a generic bio_vec since these have
already been dma mapped. Repurpose the bi_io_vec space to store dmabuf
maps as they are mutually exclusive.
The bio splitting differs from the normal path because it's already
pre-mapped and for the block layer it's just an offset into the dma-buf.
The actual segmentation is only available to the importer driver and not
the block layer, however, it doesn't contain alignment gaps and we don't
have to check it. For the same reason we can't precisely split by the
number of segments, but we use ->min_seg_shift stored in the map to
calculate the minimum number of bytes a request consisting of
lim->max_segments full segments can cover and split by that. It's
stricter and can add extra splitting. E.g. for a {4K, 4G} segmentation,
the min segment size is 4K, and we'll split it into bios of
(4K * lim->max_segments) bytes each, but it should be good enough for
now to cover the most popular use cases.
Suggested-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/39b964cf7510f42755188563da63d3d5acaf2aa9.1791975048.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Since commit 2729a60bbfb9 ("block: don't silently ignore metadata for
sync read/write"), blkdev_direct_IO() sends every IOCB_HAS_METADATA
request to __blkdev_direct_IO(). Remove the now unreachable metadata
mapping in __blkdev_direct_IO_async() and the unmap in its completion
handler.
No functional change.
Assisted-by: Claude:claude-fable-5
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Link: https://patch.msgid.link/20261009-blkdev-fixes-v6-9-307939c387df@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
bio_iov_iter_align_down() drops trailing bvecs with unpin_user_page(),
but a bvec built by iov_iter_extract_bvecs() can span several pages of
one folio, each with its own pin. All but the first pin leak.
The partially trimmed bvec has the same problem. Shrinking bv_len does
not release the pins for the pages cut off by the trim, and
__bio_release_pages() only unpins the pages bv_len still covers at
completion.
Both issues occur only with a logical block size above PAGE_SIZE and a
large folio backing the user buffer. On a device with a 64K logical
block size, an O_DIRECT pwritev() from a hugetlb mapping that ends 16K
past a block boundary leaks one huge page per call, whether the
remainder is its own bvec or the tail of a larger one.
Unpin all pages of a dropped bvec with bvec_unpin(), and unpin the
pages trimmed off the last bvec as well. Move bvec_unpin() up and split
its page count into a helper so both sites share it.
Fixes: 20a0e6276edb ("block: align the bio after building it")
Assisted-by: Claude:claude-fable-5
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Tested-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Link: https://patch.msgid.link/20261009-blkdev-fixes-v6-8-307939c387df@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blkdev_direct_write() turns an -EBUSY from page cache invalidation into
a 0 return, so an IOCB_ATOMIC write is retried in full through
blkdev_buffered_write(), with no atomicity guarantee.
Skip the buffered fallback in blkdev_write_iter() for IOCB_ATOMIC, as
it already does for IOCB_NOWAIT, so the -EBUSY case returns -EAGAIN and
the caller retries, matching __iomap_dio_rw().
ext4 has the same fallback and only warns in it. For block devices the
fallback can be skipped before any I/O is submitted, so fail early
instead.
Fixes: caf336f81b3a ("block: Add fops atomic write support")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260802-blkdev-fixes-v1-0-a82fc549fd74%40columbia.edu?part=2
Assisted-by: Claude:claude-fable-5
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: John Garry <john.garry@linux.dev>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Link: https://patch.msgid.link/20261009-blkdev-fixes-v6-6-307939c387df@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
On a partial page pin, __blkdev_direct_IO_simple() and
__blkdev_direct_IO_async() submit what was pinned with REQ_ATOMIC set
and leave the rest to the buffered fallback, tearing an IOCB_ATOMIC
write.
This can be triggered deterministically. A 16K pwritev2(RWF_ATOMIC)
whose last page is PROT_NONE, on a scsi_debug device with atomic_wr=1,
completes short with only three of the four pages written, violating
RWF_ATOMIC semantics.
Make bio_iov_iter_get_pages() release the pins and return -EINVAL when
a REQ_ATOMIC bio doesn't cover the whole iterator, since an atomic
write is submitted as a single bio and a short one would be torn. That
covers iomap as well, where a partially unmapped buffer could trip the
WARN_ON_ONCE() in iomap_dio_bio_iter_one(). The async block device path
currently sets REQ_ATOMIC after pinning, so set it before, and move
REQ_NOWAIT along with it.
Fixes: caf336f81b3a ("block: Add fops atomic write support")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260802-blkdev-fixes-v1-0-a82fc549fd74%40columbia.edu?part=2
Assisted-by: Claude:claude-fable-5
Reviewed-by: John Garry <john.garry@linux.dev>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Link: https://patch.msgid.link/20261009-blkdev-fixes-v6-5-307939c387df@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blkdev_read_iter() takes inode_lock_shared() unconditionally around
filemap_read(). Unlike blkdev_write_iter(), it does not reject
IOCB_NOWAIT for buffered I/O, so a non-blocking read, or the buffered
tail of a short IOCB_NOWAIT direct read, blocks behind set_blocksize()
holding i_rwsem across sync_blockdev().
A preadv2(RWF_NOWAIT) issued while another thread changes the block
size with a dirty page cache blocks for as long as sync_blockdev()
takes, 4 to 6 seconds on a scsi_debug device with delay=5.
Use inode_trylock_shared() for IOCB_NOWAIT and return the bytes the
direct path already read, or -EAGAIN if none, when the lock is
contended, preserving NOWAIT semantics.
Fixes: c0e473a0d226 ("block: fix race between set_blocksize and read paths")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260802-blkdev-fixes-v1-0-a82fc549fd74%40columbia.edu?part=2
Assisted-by: Claude:claude-fable-5
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Link: https://patch.msgid.link/20261009-blkdev-fixes-v6-4-307939c387df@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
def_blk_fops wires ->splice_read directly to filemap_splice_read(),
which allocates folios based on mapping_min_folio_order() without any
lock against set_blocksize(). A splice from a block device can race
set_blocksize() raising the minimum folio order and insert a folio that
is too small for the mapping. blkdev_read_iter() wraps filemap_read()
in inode_lock_shared() for this reason, but the splice path was missed.
Splicing from a block device while toggling the block size between 512
bytes and 64K with BLKBSZSET hits this within seconds on a
CONFIG_DEBUG_VM kernel:
page dumped because: VM_BUG_ON_FOLIO(folio_order(folio) < mapping_min_folio_order(mapping))
kernel BUG at mm/filemap.c:858!
Oops: invalid opcode: 0000 [#1] SMP NOPTI
RIP: 0010:__filemap_add_folio+0x51c/0x570
Call Trace:
filemap_add_folio+0x64/0x140
page_cache_ra_order+0x1dd/0x3d0
filemap_get_pages+0x153/0x760
filemap_splice_read+0x13f/0x300
splice_file_to_pipe+0xc0/0xd0
do_splice+0x6a8/0x890
__do_splice+0xb0/0x210
__x64_sys_splice+0x80/0x100
do_syscall_64+0x10e/0x520
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Take inode_lock_shared() around filemap_splice_read(), like the read
path does.
Fixes: 3c20917120ce ("block/bdev: enable large folio support for large logical block sizes")
Assisted-by: Claude:claude-fable-5
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Tested-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Link: https://patch.msgid.link/20261009-blkdev-fixes-v6-3-307939c387df@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Commit c0e473a0d226 ("block: fix race between set_blocksize and read
paths") closed a race between set_blocksize() and block device I/O: with
large sector size support, set_blocksize() can change i_blkbits and the
mapping's minimum folio order while a concurrent reader still holds a
folio of the old, smaller order, leading to crashes. In particular, it
made blkdev_write_iter() wrap buffered writes in inode_lock_shared().
However, the direct I/O fallback path was missed in that conversion.
blkdev_write_iter() passes blkdev_buffered_write() as an argument to
direct_write_fallback() with no lock held. A direct write that completes
only partially then finishes as a buffered write with no protection.
This can cause a BUG by racing partial direct writes against
ioctl(BLKBSZSET). Writer threads issue O_DIRECT pwritev() with a
two-segment iovec whose second segment is an unreadable PROT_NONE
mapping. The direct path then writes the first segment, fails to pin the
second, and returns short, entering the fallback. A second thread keeps
toggling the second segment's protection so that some fallbacks get past
fault_in_iov_iter_readable() and reach the page cache, a third thread
populates the page cache with folios of the current block size via
pread() and readahead(), and a fourth thread toggles the block size
between 512 bytes and 64K with BLKBSZSET. The minimum folio order only
moves with block sizes above PAGE_SIZE, i.e. with
CONFIG_TRANSPARENT_HUGEPAGE raising BLK_MAX_BLOCK_SIZE to 64K.
On a CONFIG_DEBUG_VM kernel this yields the following BUG:
page dumped because: VM_BUG_ON_FOLIO(folio_order(folio) < mapping_min_folio_order(mapping))
kernel BUG at mm/filemap.c:858!
Oops: invalid opcode: 0000 [#1] SMP KASAN NOPTI
RIP: 0010:__filemap_add_folio+0x860/0x8d0
Call Trace:
filemap_add_folio+0xc9/0x1f0
__filemap_get_folio_mpol+0x240/0x660
iomap_write_begin+0xa87/0xd70
iomap_file_buffered_write+0x304/0x6a0
blkdev_write_iter+0x255/0x510
do_iter_readv_writev+0x23d/0x3c0
vfs_writev+0x211/0x7d0
do_pwritev+0x121/0x190
do_syscall_64+0x121/0x630
entry_SYSCALL_64_after_hwframe+0x77/0x7f
The same workload also trips WARN_ON_ONCE(pos >= folio_pos(folio) +
fsize) in iomap_trim_folio_range().
Fix this by calling blkdev_buffered_write() in the fallback path under
inode_lock_shared(), matching the plain buffered-write branch. With the
fix the same workload runs clean.
A short IOCB_NOWAIT direct write reaches the same fallback. Taking
i_rwsem there can now block behind set_blocksize(), and the fallback
already blocks on writeback of the data it copied in
direct_write_fallback(). blkdev_write_iter() already rejects a purely
buffered IOCB_NOWAIT write with -EOPNOTSUPP, so do not enter the
fallback for IOCB_NOWAIT at all: return the bytes the direct path
already wrote, or -EAGAIN if none, and let the caller retry.
The reproducer used was written by an LLM, and is available at [1].
[1] https://gist.github.com/tzussman/69d06bc57d42a42989eb038b1b5aeb74
Fixes: 3c20917120ce ("block/bdev: enable large folio support for large logical block sizes")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260730-blk-dontcache-v7-0-3e8e6850068d%40columbia.edu?part=5
Assisted-by: Claude:claude-fable-5
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Tested-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Link: https://patch.msgid.link/20261009-blkdev-fixes-v6-2-307939c387df@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
With CONFIG_BUFFER_HEAD=n, block devices are written back through iomap,
but def_blk_aops uses filemap_dirty_folio, which only sets PG_dirty. It
does not set the per-block dirty bits in the folio's iomap_folio_state,
so iomap_writeback_folio() finds no dirty range, submits no I/O and
clears PG_dirty, resulting in data loss.
Other iomap users set .dirty_folio to iomap_dirty_folio, which marks the
folio's blocks dirty before calling filemap_dirty_folio().
This is only observable with block size < folio size. With a single
block there is no iomap_folio_state to get out of sync and
iomap_writeback_folio() marks the whole folio dirty itself. For a
page-aligned device, this may require using the BLKBSZSET ioctl to set
the block size, which requires CAP_SYS_ADMIN. A device whose size is not
page aligned already gets a sub-page block size from
set_init_blocksize(), so no ioctl and no privilege is needed.
To reproduce, on a device with a sub-page block size, write a known
pattern with O_DIRECT, mmap the same range, store to it, msync() and
fsync(), then read it back with O_DIRECT. A reproducer is available at
[1].
[1] https://gist.github.com/tzussman/18ab05cba4b3fdc79cce0a69d1fd05b4
Fixes: 925c86a19bac ("fs: add CONFIG_BUFFER_HEAD")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260730-blk-dontcache-v7-0-3e8e6850068d%40columbia.edu?part=5
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Link: https://patch.msgid.link/20261009-blkdev-fixes-v6-1-307939c387df@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block fixes from Jens Axboe:
- NVMe fixes via Keith:
- Fix an out-of-bounds write in nvmet_auth_challenge(), where
sizeof() on a void pointer undercounted the challenge header and
let a short AUTH_RECEIVE buffer pass the check
- nvme-multipath fixes for an ANA log bounds check underflow, the
command effects log lifetime for multipath heads, and only
setting BLK_FEAT_ZONED after the zone info is known.
- nvmet fixes for ns->enabled teardown ordering, rejecting I/O
after the percpu ns reference is killed, device path preservation
on allocation failure, and too-short SGL segments in pci-epf
- nvme-tcp: revert the per-socket dynamic lockdep keys, and delay
the socket reclassification
- A DMA pool alignment quirk for the Micron 4100AT
- Controller state/reset race fixes, and -Wformat-security
workarounds
- blk-mq: set RQF_USE_SCHED when the operation is known, and allow
cached requests to be used for flush operations
- Reject polled dio with user integrity metadata
- Save the IRQ state in blkg_tryget_closest()
- Set the zone write granularity in virtio_blk
- ublk selftest fixes
* tag 'block-7.3-20261002' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (23 commits)
virtio_blk: set the zone write granularity
nvme-multipath: set BLK_FEAT_ZONED only after the zone info is known
nvme: fix command effects log lifetime for multipath heads
nvmet: don't allow I/O admission after percpu ns reference is killed
nvmet: defer setting ns->enabled to false in nvmet_ns_disable()
nvmet: copy the hostid into the ctrl before creating PR pc_refs
nvmet-auth: fix out-of-bounds write in nvmet_auth_challenge()
nvmet: pci-epf: reject too-short SGL segments
nvme-multipath: fix underflow in ANA log bounds checks
nvme: work around all -Wformat-security warnings
nvme: work around -Wformat-security warning
nvme: do not reset controllers in NVME_CTRL_NEW state
nvme-tcp: delay nvme_tcp_reclassify_socket()
Revert "nvme-tcp: lockdep: use dynamic lockdep keys per socket instance"
drbd: remove unused drbd_nl_mcgrps[] array
blk-mq: allow cached requests to be used for flush operations
blk-mq: set RQF_USE_SCHED when the operation is known
block: reject polled dio with user integrity metadata
selftests: ublk: fix unused_result error
blk-cgroup: save IRQ state in blkg_tryget_closest()
...
|
|
block/blk-core.c uses ALLOW_ERROR_INJECTION(), which is defined in
asm-generic/error-injection.h. It currently relies on this header being
included indirectly through linux/module.h -> linux/error-injection.h.
Add the missing include in preparation for removing the
linux/error-injection.h include from linux/module.h.
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
|
|
blk_time_get_ns() caches the current time in the task's plug. The idea
is that the cache lives for one plug and is dropped when the plug is
flushed. __blk_flush_plug() and finish_task_switch() do that.
blk_add_rq_to_plug() also flushes, when the plug has too many requests
or the last request is at least BLK_PLUG_FLUSH_SIZE. But it calls
blk_mq_flush_plug_list() directly and never drops the cached time. So
a task that keeps submitting without sleeping stamps every request
after that with the time first cached in the plug. start_time_ns and
io_start_time_ns are stale and the disk stats charge each request for
the whole time since the plug began.
Drop the cached time after the flush, same as __blk_flush_plug() does.
Fixes: da4c8c3d0975 ("block: cache current nsec time in struct blk_plug")
Cc: stable@vger.kernel.org
Signed-off-by: Vasileios Almpanis <vasileios.almpanis@virtuozzo.com>
Link: https://patch.msgid.link/20260930-blk-plug-ts-flush-v1-1-bf8ca7bc5cd6@virtuozzo.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
Pull SCSI fixes from James Bottomley:
"Mostly small driver fixes. The biggest fix is the one to the block
zone handling which might trip for real or virtual hardware if the
number of zones is > 2^32"
* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
scsi: megaraid_sas: Protect megasas_get_ctrl_info() in megasas_resume()
scsi: sd_zbc: Reject disks with too many zones
scsi: block: Fix zones_cond out-of-bounds write on zone report
scsi: leapraid: Avoid -Wformat-security warning
scsi: devinfo: Add BLIST_SKIP_IO_HINTS for EMC Symmetrix
scsi: libiscsi_tcp: Check the data direction of a Data-In PDU
scsi: ufs: pltfrm: Add quirk for R-Car S4 lacking lanes-per-direction
scsi: ufs: core: Keep internal commands dispatchable during error handling
|
|
The cached request lookups reject a request whose op_is_flush() state does
not match the incoming operation. That check is from when rq->elv and
rq->flush shared a union, so a request could not be both a scheduler
request and carry flush state. Commit be4c427809b0 ("blk-mq: use the I/O
scheduler for writes from the flush state machine") made them separate
members of struct request.
A cached request now gets its scheduler state when it is claimed by an
operation, so one claimed for a flush is set up exactly as a freshly
allocated request. Drop the check so a cached batch can mix flush and
non-flush operations.
Signed-off-by: Keith Busch <kbusch@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260922172711.1573186-2-kbusch@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The cached requests are allocated for one operation but can be handed
out for another. A passthrough command has RQF_USE_SCHED cleared, so
using those flags for a subsequent read/write bio will insert it into
the scheduler without ->prepare_request() and frees it without
->finish_request(). For kyber, this leaks the domain token acquired at
dispatch and stalls the queue.
Don't set RQF_USE_SCHED based on the first operation the batch happened
to be allocated for. Instead, set it after the request is claimed by an
operation. Introduce a helper function, blk_mq_rq_late_init() for this,
and absorb blk_mq_rq_time_init() since the two run together.
Fixes: 4b6a5d9cea91 ("block: enable batched allocation for blk_mq_alloc_request()")
Link: https://lore.kernel.org/linux-block/20260916151655.2588-1-15815827059@163.com/
Reported-by: Henry Hu <huhai@kylinos.cn>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260922172711.1573186-1-kbusch@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
IOCB_HAS_METADATA leaves iocb->private as a uio_meta, while
iocb_bio_iopoll() expects a bio. Since 2729a60bbfb9, metadata I/O
always uses __blkdev_direct_IO(), which never publishes the bio for
polling, so IOPOLL calls bio_poll() on the uio_meta.
Reproducer: io_uring IOPOLL plus a PI attribute on an integrity device
(scsi_debug dif=1 dix=1). KASAN reports:
Oops: general protection fault, probably for non-canonical
address 0xdffffc0000000004
KASAN: null-ptr-deref in range
[0x0000000000000020-0x0000000000000027]
RIP: 0010:bio_poll+0x9d/0x4b0
Call Trace:
iocb_bio_iopoll+0x5b/0x80
io_uring_classic_poll+0xf3/0x140
io_do_iopoll+0x4e3/0xc00
io_iopoll_check+0x275/0x560
__do_sys_io_uring_enter+0x2e5/0x720
Return -EOPNOTSUPP when HIPRI is combined with user metadata.
Fixes: 2729a60bbfb9 ("block: don't silently ignore metadata for sync read/write")
Assisted-by: LLM
Signed-off-by: Yang Xiuwei <yangxiuwei@kylinos.cn>
Link: https://patch.msgid.link/20260923093732.1504291-1-yangxiuwei@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Non-root io.stat reads flush the blkcg rstat subtree before walking
->blkg_list. This takes the subsystem rstat lock and checks for pending
updates on every possible CPU, even when there is no per-device state to
print.
Blkg creation is hierarchical. A descendant blkg implies an ancestor
blkg on the same device, so an empty ->blkg_list means the subtree cannot
contribute any output. Return early in that case.
A concurrent first blkg may be deferred to the next read. This is
consistent with the existing non-atomic flush and list walk.
This is not rare: 340 of 969 io.stat files were empty on a 316-CPU host.
In a 16-vCPU guest, the median open/read/close time over 35 batches of
20,000 reads changed as follows:
before after
empty io.stat 7.02 us 5.47 us
populated io.stat 7.84 us 7.83 us
This is a ~22% reduction for empty reads. Populated reads were
unchanged.
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
blk_revalidate_disk_zones() sizes the zones_cond array from the disk
capacity and zone size, but the index used by blk_revalidate_zone_cond()
comes from the device-driven report_zones() walk and is never checked
against the array size. A device reporting more zones than fit the array
makes blk_zone_set_cond() write out of bounds.
One way to reach this is a zone count exceeding 32 bits: both
blk_revalidate_zone_args.nr_zones and struct zoned_disk_info.nr_zones are
unsigned int, so a disk advertising more than UINT_MAX zones (e.g. 2^32 +
1024 zones of one 512-byte logical block) gets its zone count truncated to
a small value, undersizing the array while the report walk keeps counting
upward.
Check the index against the array size before storing the zone condition,
and refuse to revalidate when the zone count does not fit 32 bits.
Fixes: 6e945ffb6555 ("block: use zone condition to determine conventional zones")
Signed-off-by: ZHOU Jiaxiang <me@fxti.xyz>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/7815D1B293A8F55E+20260916135822.32584-2-me@fxti.xyz
Signed-off-by: Martin K. Petersen (Oracle) <mkp@kernel.org>
|
|
disk_zone_set_cond() is used to set a zone condition afer a reset, a
finish or a reset all operation. For a single zone reset or finish, we are
guaranteed that the target zone is a sequential one that is not offline
nor read-only (otherwise, the operation would have failed). For a reset
all operation, there is no point in calling this function for offline and
read-only zones since the condition checks in disk_zone_set_cond() will
result in nothing being done.
Simplify all this using disk_zone_is_offline_or_readonly() in
blk_zone_reset_all_bio_endio() to skip zones that are offline or
read-only. This change allows simplifying disk_zone_set_cond() by removing
the zone condition checks. This change is also consistent with the fact
that conventional zones can now have the offline or read-only condition.
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Link: https://patch.msgid.link/20260908085745.1082697-17-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
With SCSI and ATA SMR HDDs, the storage element depopulation feature can
change the condition of conventional zones to read-only (if a write head
is depopulated) or to offline (if a read head is depopulated).
However, the function blk_revalidate_zone_cond() currently does not allow
these conditions for conventional zones, causing a zone revalidation
failure.
Remove blk_revalidate_zone_cond() and move the zone condition checks for
conventional zones to blk_revalidate_conv_zone(), allowing the regular
BLK_ZONE_COND_NOT_WP condition as well as the BLK_ZONE_COND_OFFLINE and
BLK_ZONE_COND_READONLY conditions to match the conditions that can be
seen from a zoned device with depopulated storage elements.
The zone condition checks for sequential write required zones are moved
to blk_revalidate_seq_zone() without any change to the conditions allowed.
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://patch.msgid.link/20260908085745.1082697-16-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Any zone management operation targeting a zone that is in the read-only
or offline condition will fail. So there is no point in issuing such
BIO. Modify the check in submit_bio_noacct() to use the new helper
function bdev_zone_mgmt_allowed() to check that a zone is sequential (as
was checked before) and also that the zone is not offline nor read-only.
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://patch.msgid.link/20260908085745.1082697-15-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Since any write BIO targeting an offline or a read-only zone will fail,
there is no point in keeping zone write plugs for these zones.
So for any offline or read-only zone, the zone write plug should always
be treated as dead.
Do this by modifying disk_check_zone_wplug_dead() to always mark read-only
and offline zones as dead to force a removal of the zone write plug from
the disk hash table on BIO submission. blk_zone_wplug_prepare_bio() is
also modified to have the same checks to immediately fail a write BIO
targeting a read-only or offline zone. With these two changes, any newly
issued or unplugged write BIO targeting a read-only or offline zone is
immediately failed.
Finally, disk_zone_wplug_sync_state() is modified to add a call to
disk_mark_zone_wplug_dead() for the zone write plug of any read-only or
offline zone found during zone revalidation or a report zones.
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://patch.msgid.link/20260908085745.1082697-14-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
When revalidating the zones of a zoned block device, or when handling a
zone which has a zone write plug flagged with
BLK_ZONE_WPLUG_NEED_WP_UPDATE, the function
disk_zone_wplug_sync_wp_offset() is used to update the write pointer
offset of a zone write plug. However, this does not take into account the
condition of the zone, which may have changed to readonly or offline,
which in itself will always cause errors.
In order to catch such errors, rename disk_zone_wplug_sync_wp_offset() to
disk_zone_wplug_sync_state() and in addition to the zone write pointer,
also update the zone write plug condition if the zone is readonly or
offline.
This change also requires changes to how a reset all zones operation
(REQ_OP_ZONE_RESET_ALL) is handled so that the offline or read-only
condition of zone write plugs is not overwritten with an erroneous empty
condition. To do so, introduce the helper function
disk_zone_is_offline_or_readonly() to skip updating the condition of zones
that are read-only or offline and that do not have a zone write plug.
For zones that have a zone write plug, the helper function
disk_zone_wplug_is_offline_or_readonly() is used in
disk_zone_wplug_set_wp_offset() to not update a zone write plug write
pointer and condition for read-only and offline zones.
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Link: https://patch.msgid.link/20260908085745.1082697-13-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
When disk_revalidate_zone_resources() detects a capacity change, -ENODEV
is returned, failing the disk revalidation. However, since a capacity
change may happen due to a storage element removal being executed
concurrently to blk_revalidate_disk_zones(), we can simply retry the
revalidation to capture the new zone state with the new capacity without
failing the revalidation.
Retrying the revalidation is driven by disk_revalidate_zone_resources()
returning -EAGAIN when a new valid capacity is detected. And to avoid
getting stuck in an infinite loop revalidating zones, retries are limited
to 2.
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Link: https://patch.msgid.link/20260908085745.1082697-12-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
If during revalidation, we detect a capacity change for a zoned block
device, e.g. due to a storage element removal on an HDD, we can assume
that the device was reformatted, which implies that all sequential zones
are empty. For such case, we can remove and free all zone write plugs in
the gendisk hash table by marking them as dead, thus avoiding also to
leave zone write plugs for zones that are beyond the new device capacity
in the disk hash table.
Introduce the function disk_revalidate_capacity() to do this and call this
new function at the beginning of blk_revalidate_disk_zones(), so that the
zone revalidation process can re-create, if needed, any zone write plug
for sequential zones that are not empty.
The checks on the capacity and zone size that were in
blk_revalidate_disk_zones() are moved to disk_revalidate_capacity() and
if true, also trigger dropping all zone write plugs.
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260908085745.1082697-11-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The zone related fields of struct gendisk can be modified by
blk_revalidate_disk_zones() either on the first scan of the disk, or
during user triggered scans or device revalidation, if some
characteristics of the disk has changed (e.g. the disk capacity). Changes
to these fields are always done with the disk request queue frozen so that
BIO processing does not see any inconsistent state of the zones. This
implies a contract that reading these zone related fields must always be
done while holding a usage count on the request queue of the disk.
However, increasing the usage count of the disk request queue cannot be
done from the context of blk_revalidate_disk_zones() itself, as that would
prevent freezing the disk queue and result in a deadlock. This prevents
blk_revalidate_disk_zones() from consulting the zone related fields of
struct gendisk to detect, for instance, a change in the number of zones of
the disk. For such case, we want to detect the change, take appropriate
measures and revalidate exclusively revalidate the zones to avoid
concurrent revalidation calls to see the same change while corrections are
already on-going.
A simple solution to avoid this issue is to introduce a mutex to serialize
calls to blk_revalidate_disk_zones() and ensure only a single context at a
time can modify the zone related fields of a gendisk. In preparation for
handling disk capacity revalidation in blk_revalidate_disk_zones(), do so
with the mutex zone_revalidate_mutex. This mutex is initialized in
disk_init_zone_resources(), destroyed in disk_release_zone_resources() and
taken and released only in blk_revalidate_disk_zones() to serialize the
execution of this function.
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260908085745.1082697-10-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Introduce the helper function disk_for_all_zone_wplugs() to allow
executing an actor function on all hashed zone write plugs. This helper
is used to simplify the implementation of blk_zone_reset_all_bio_endio()
and queue_zone_wplugs_show().
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Link: https://patch.msgid.link/20260908085745.1082697-9-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Add a zone state array bound check to ensure that like
disk_zone_get_state(), we never attempt to access the zone state array
beyond its size.
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Link: https://patch.msgid.link/20260908085745.1082697-8-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Define the helper function disk_zone_is_seq() and use it to refactor
bdev_zone_is_seq(). disk_zone_is_seq() is also used in
blk_zone_wplug_handle_write().
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260908085745.1082697-7-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
