| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block fixes from Jens Axboe:
- NVMe fixes via Keith:
- Fix an out-of-bounds write in nvmet_auth_challenge(), where
sizeof() on a void pointer undercounted the challenge header and
let a short AUTH_RECEIVE buffer pass the check
- nvme-multipath fixes for an ANA log bounds check underflow, the
command effects log lifetime for multipath heads, and only
setting BLK_FEAT_ZONED after the zone info is known.
- nvmet fixes for ns->enabled teardown ordering, rejecting I/O
after the percpu ns reference is killed, device path preservation
on allocation failure, and too-short SGL segments in pci-epf
- nvme-tcp: revert the per-socket dynamic lockdep keys, and delay
the socket reclassification
- A DMA pool alignment quirk for the Micron 4100AT
- Controller state/reset race fixes, and -Wformat-security
workarounds
- blk-mq: set RQF_USE_SCHED when the operation is known, and allow
cached requests to be used for flush operations
- Reject polled dio with user integrity metadata
- Save the IRQ state in blkg_tryget_closest()
- Set the zone write granularity in virtio_blk
- ublk selftest fixes
* tag 'block-7.3-20261002' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (23 commits)
virtio_blk: set the zone write granularity
nvme-multipath: set BLK_FEAT_ZONED only after the zone info is known
nvme: fix command effects log lifetime for multipath heads
nvmet: don't allow I/O admission after percpu ns reference is killed
nvmet: defer setting ns->enabled to false in nvmet_ns_disable()
nvmet: copy the hostid into the ctrl before creating PR pc_refs
nvmet-auth: fix out-of-bounds write in nvmet_auth_challenge()
nvmet: pci-epf: reject too-short SGL segments
nvme-multipath: fix underflow in ANA log bounds checks
nvme: work around all -Wformat-security warnings
nvme: work around -Wformat-security warning
nvme: do not reset controllers in NVME_CTRL_NEW state
nvme-tcp: delay nvme_tcp_reclassify_socket()
Revert "nvme-tcp: lockdep: use dynamic lockdep keys per socket instance"
drbd: remove unused drbd_nl_mcgrps[] array
blk-mq: allow cached requests to be used for flush operations
blk-mq: set RQF_USE_SCHED when the operation is known
block: reject polled dio with user integrity metadata
selftests: ublk: fix unused_result error
blk-cgroup: save IRQ state in blkg_tryget_closest()
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
Pull SCSI fixes from James Bottomley:
"Mostly small driver fixes. The biggest fix is the one to the block
zone handling which might trip for real or virtual hardware if the
number of zones is > 2^32"
* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
scsi: megaraid_sas: Protect megasas_get_ctrl_info() in megasas_resume()
scsi: sd_zbc: Reject disks with too many zones
scsi: block: Fix zones_cond out-of-bounds write on zone report
scsi: leapraid: Avoid -Wformat-security warning
scsi: devinfo: Add BLIST_SKIP_IO_HINTS for EMC Symmetrix
scsi: libiscsi_tcp: Check the data direction of a Data-In PDU
scsi: ufs: pltfrm: Add quirk for R-Car S4 lacking lanes-per-direction
scsi: ufs: core: Keep internal commands dispatchable during error handling
|
|
The cached request lookups reject a request whose op_is_flush() state does
not match the incoming operation. That check is from when rq->elv and
rq->flush shared a union, so a request could not be both a scheduler
request and carry flush state. Commit be4c427809b0 ("blk-mq: use the I/O
scheduler for writes from the flush state machine") made them separate
members of struct request.
A cached request now gets its scheduler state when it is claimed by an
operation, so one claimed for a flush is set up exactly as a freshly
allocated request. Drop the check so a cached batch can mix flush and
non-flush operations.
Signed-off-by: Keith Busch <kbusch@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260922172711.1573186-2-kbusch@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The cached requests are allocated for one operation but can be handed
out for another. A passthrough command has RQF_USE_SCHED cleared, so
using those flags for a subsequent read/write bio will insert it into
the scheduler without ->prepare_request() and frees it without
->finish_request(). For kyber, this leaks the domain token acquired at
dispatch and stalls the queue.
Don't set RQF_USE_SCHED based on the first operation the batch happened
to be allocated for. Instead, set it after the request is claimed by an
operation. Introduce a helper function, blk_mq_rq_late_init() for this,
and absorb blk_mq_rq_time_init() since the two run together.
Fixes: 4b6a5d9cea91 ("block: enable batched allocation for blk_mq_alloc_request()")
Link: https://lore.kernel.org/linux-block/20260916151655.2588-1-15815827059@163.com/
Reported-by: Henry Hu <huhai@kylinos.cn>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260922172711.1573186-1-kbusch@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
IOCB_HAS_METADATA leaves iocb->private as a uio_meta, while
iocb_bio_iopoll() expects a bio. Since 2729a60bbfb9, metadata I/O
always uses __blkdev_direct_IO(), which never publishes the bio for
polling, so IOPOLL calls bio_poll() on the uio_meta.
Reproducer: io_uring IOPOLL plus a PI attribute on an integrity device
(scsi_debug dif=1 dix=1). KASAN reports:
Oops: general protection fault, probably for non-canonical
address 0xdffffc0000000004
KASAN: null-ptr-deref in range
[0x0000000000000020-0x0000000000000027]
RIP: 0010:bio_poll+0x9d/0x4b0
Call Trace:
iocb_bio_iopoll+0x5b/0x80
io_uring_classic_poll+0xf3/0x140
io_do_iopoll+0x4e3/0xc00
io_iopoll_check+0x275/0x560
__do_sys_io_uring_enter+0x2e5/0x720
Return -EOPNOTSUPP when HIPRI is combined with user metadata.
Fixes: 2729a60bbfb9 ("block: don't silently ignore metadata for sync read/write")
Assisted-by: LLM
Signed-off-by: Yang Xiuwei <yangxiuwei@kylinos.cn>
Link: https://patch.msgid.link/20260923093732.1504291-1-yangxiuwei@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blk_revalidate_disk_zones() sizes the zones_cond array from the disk
capacity and zone size, but the index used by blk_revalidate_zone_cond()
comes from the device-driven report_zones() walk and is never checked
against the array size. A device reporting more zones than fit the array
makes blk_zone_set_cond() write out of bounds.
One way to reach this is a zone count exceeding 32 bits: both
blk_revalidate_zone_args.nr_zones and struct zoned_disk_info.nr_zones are
unsigned int, so a disk advertising more than UINT_MAX zones (e.g. 2^32 +
1024 zones of one 512-byte logical block) gets its zone count truncated to
a small value, undersizing the array while the report walk keeps counting
upward.
Check the index against the array size before storing the zone condition,
and refuse to revalidate when the zone count does not fit 32 bits.
Fixes: 6e945ffb6555 ("block: use zone condition to determine conventional zones")
Signed-off-by: ZHOU Jiaxiang <me@fxti.xyz>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/7815D1B293A8F55E+20260916135822.32584-2-me@fxti.xyz
Signed-off-by: Martin K. Petersen (Oracle) <mkp@kernel.org>
|
|
bio_set_dev() associates the bio with a blkg through bio_associate_blkg().
If the blkg lookup misses, blkg_tryget_closest() takes q->queue_lock with
spin_lock_irq() and releases it with spin_unlock_irq(), which
unconditionally enables local interrupts.
Callers may call bio_set_dev() with interrupts already disabled, e.g.
dm-thin's pool_map() does so while holding pool->lock taken with
spin_lock_irq(). The nested spin_unlock_irq() then enables interrupts
while pool->lock is still held, so an I/O completion softirq can run on
the same CPU, re-acquire pool->lock (thin_endio(), or overwrite_endio()
-> complete_mapping_preparation()) and deadlock. lockdep reports this
as inconsistent SOFTIRQ-ON-W to IN-SOFTIRQ-W usage.
Commit 3a762de55b4e ("block: save irq state in blkg_lookup_create()")
fixed the same problem while the lock lived in blkg_lookup_create(), but
commit 9327a865e395 ("blk-cgroup: don't nest queue_lock under rcu in
blkg_lookup_create()") moved the locking into blkg_tryget_closest() and
reverted it to spin_lock_irq().
Save and restore the caller's IRQ state instead.
Fixes: 9327a865e395 ("blk-cgroup: don't nest queue_lock under rcu in blkg_lookup_create()")
Cc: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/aqQmqb1k57PXj8Ef@192.168.1.215
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The cloned bio needs to inherit the accumulated gaps between vectors so
that we can know if this bio can subscribe to the iova coalescing
optimization.
When cloning for a split, the gap only applies to the front bio since
that's as far as has been processed. The remaining bio can reset its
gaps to 0 since it advanced past the checked vectors, and will start its
accounting from there on the next split check.
Fixes: 2f6b2565d43c ("block: accumulate memory segment gaps per bio")
Reported-by: Eric Auger <eauger@redhat.com>
Tested-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260819154937.3903312-1-kbusch@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Zoned block devices do not support partitions. However, the partition
table is nevertheless still inspected, and any partition found ignored
with a warning in add_partition(). While this is generally not a problem,
and in fact beneficial to the user as it indicates an invalid use of a
zoned block device, scanning for a partition table on the device may
result in issuing read operations to offline zones (e.g. after a disk head
is depopulated for disks that support head management operations).
Since partitions are ignored anyway, completely disable partition scanning
for zoned gendisks by setting the flag GENHD_FL_NO_PART in __add_disk().
The existing check in add_partition() is left as-is to ensure that we
still get a warning if for whatever reason, despite GENHD_FL_NO_PART, we
still endup trying to add partitions.
Flagging zoned disks with GENHD_FL_NO_PART also has the benefit to expose
through sysfs the ext_range attribute with the value of 1 instead of the
default DISK_MAX_PARTS, thus correctly advertizing the fact that zoned
disks do not support partitions.
Fixes: 5eac3eb30c9a ("block: Remove partition support for zoned block devices")
Cc: stable@vger.kernel.org
Signed-off-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Link: https://patch.msgid.link/20260831025050.667758-1-dlemoal@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block updates from Jens Axboe:
- NVMe updates via Keith:
- Enable Clang context analysis for the nvme host driver, adding
context annotations across core, fabrics, rdma, tcp and pci
- nvmet reservation state exposed through a new namespace-level
debugfs directory, plus ABI documentation for the host sysfs and
target configfs interfaces
- nvme-tcp host memory disclosure fixes on the read path: reject a
read that transferred too few bytes, don't accept C2HData based
on blk_rq_payload_bytes() alone, and fix the R2T case for a read
command
- Parallelize nvme-rdma I/O queue allocation and startup (Surabhi)
- Apple nvme fixes and quirks: page aligned admin queue buffers,
destroy the admin queue on removal, and various DMA/NVMMU
correctness fixes
- A large pile of nvmet and host fixes for out-of-bounds reads,
refcount/resource leaks, and NULL derefs across auth, zns,
passthru, pci-epf, rdma and configfs
- Various other fixes and cleanups
- MD updates via Yu Kuai:
- llbitmap reshape support, the large series wiring exact bitmap
mapping and reshape lifecycle through raid5 and raid10, growing
the page cache in place, and remapping checkpointed bits as
reshape progresses
- raid5 fixes for lockless max_nr_stripes and recovery_offset
accesses, a reshape deadlock with more failed devices than max
degraded, and bitmap batch counter consistency
- Atomic write handling for raid1/raid10, and removal of the
REQ_NOWAIT support from raid1/10/456
- raid5-ppl use-after-free fix in ppl_do_flush()
- A batch of smaller fixes across md core and the bitmap code
- s390/dasd ESE full-track write support and the surrounding
infrastructure, plus enabling CONTEXT_ANALYSIS for s390/block
- RWF_DONTCACHE support for block devices, built on new task-context
bio completion infrastructure, and wiring it up for the iomap and
buffer dropbehind writeback paths
- Async io_uring zone reset all, plus zone management command cleanups
allowing REQ_NOWAIT and tightening conventional zone rejection
- Block integrity refactoring: lift BIP_CHECK_FLAGS to the shared
header, handle nogenerate/noverify properly in fs-integrity, and drop
the blk-integrity.h include from bdev.c
- Split out a new blk_plug.h header
- ublk improvements: add UBLK_F_IO_DESC_SIZE, split request validation
from io_desc init, reject non-power-of-2 zone sizes in SET_PARAMS,
and a series of hardening fixes around map/unmap and auto buf reg
- null_blk cleanups and configfs serialization fixes
- nbd queue freeze removal on the setup paths, and a new
pre_defined_connections module parameter for pre-created devices
- blk-cgroup fixes for the race between policy activation and blkg
destruction, and accounting per-cpu stats over possible CPUs across
blk-stat, iolatency, iocost and kyber
- Various dio fixes: leak on metadata mapping error, validate user
space vectors during extraction, and set dma_alignment from the
backing file for loop and zloop direct I/O
- bio cleanups
- Various other fixes and cleanups all over
* tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (241 commits)
nbd: add pre_defined_connections module parameter for pre-created devices
nbd: remove queue freeze for newly created nbd from netlink path
nbd: factor out a nbd_genl_foreach_sock
nbd: skip queue freeze when setting size at device startup
nbd: remove queue freeze in nbd_add_socket
nbd: clear queue limits on disconnect
nbd: disallow NBD_SET_SOCK on an active device
nbd: simplify find_fallback() by removing redundant logic
blk-mq: add missing call to srcu_barrier() in blk_mq_free_tag_set()
block: mtip32xx: synchronize ioctls with device removal
ublk: avoid teardown retry loop on xarray allocation failure
null_blk: fix UBSAN shift-out-of-bounds when zone_size is 0 or overflows
block: don't include blk-integrity.h in bdev.c
xfs: avoid double deferrals for RWF_DONTCACHE writes
loop: Fix recently introduced lock inversion
block: set QUEUE_FLAG_DYING unconditionally in blk_mark_disk_dead()
swim3: Add missing MODULE_DESCRIPTION
selftests: ublk: add SET_PARAMS validation test
selftests: ublk: add helper for SET_PARAMS
ublk: reject non-power-of-2 zone sizes in SET_PARAMS
...
|
|
Pull fscrypt updates from Eric Biggers:
"The main change this cycle is a significant simplification that's been
overdue for a while now: standardizing on a single file contents
encryption implementation in ext4 and f2fs, instead of having two.
Specifically, the original filesystem-layer file contents encryption
implementation is removed, and the blk-crypto implementation is now
used unconditionally. blk-crypto delegates either to inline crypto
hardware or to the CPU via blk-crypto-fallback. The latter is
functionally equivalent to the original filesystem-layer code.
The blk-crypto implementation already existed, but previously it was
used only when the filesystem was mounted with "-o inlinecrypt". Now,
"-o inlinecrypt" just selects whether inline crypto hardware is used.
To allow maintaining that user control over hardware use, the
blk-crypto API is extended with a new flag BLK_CRYPTO_CFG_ALLOW_HW.
Overall, this removes quite a bit of redundant code from ext4, f2fs,
and fs/crypto/. It should make things easier for ongoing filesystem
efforts such as iomap support, large folios, and btrfs encryption
(btrfs had already been planning to use blk-crypto exclusively.)
There are two small behavior changes of note:
- Direct I/O now works on encrypted files even without "-o inlinecrypt",
rather than falling back to buffered I/O. This is effectively a
bugfix, though I'll continue to keep an eye out for any user that
may have been depending on the buffered I/O fallback.
- IV_INO_LBLK_32 policies are no longer supported in certain cases
that didn't make sense and have no known uses.
This has been in linux-next since July 22 with no reported issues. All
encryption xfstests pass on ext4 and f2fs. As usual I've also been
using it on a system with an fscrypt-encrypted home directory. Of
course, the blk-crypto code paths also aren't new and were already
being used on many systems via the inlinecrypt mount option.
In addition to the main change described above, there are a few other
cleanups such as using lock guards for mutexes, improving
documentation, and removing a workaround for outdated gcc versions"
* tag 'fscrypt-for-linus' of git://git.kernel.org/pub/scm/fs/fscrypt/linux: (29 commits)
blk-crypto: Update docs for blk-crypto-fallback motivation
blk-crypto: Remove unused function blk_crypto_config_supported()
fscrypt: Update docs for data path
fscrypt: Remove unused function fscrypt_finalize_bounce_page()
f2fs: Update outdated comment in f2fs_write_begin()
fs: Update outdated comment for SB_INLINECRYPT
fscrypt: Update encryption policy version docs
fscrypt: Replace some variable-size memsets with fixed-size
fscrypt: Add safety checks to non-block-based en/decryption
fscrypt: Merge bio.c and inline_crypt.c into block.c
fscrypt: Remove unused functions and workqueue
fscrypt: Remove fs-layer zeroout code
fscrypt: Remove fscrypt_dio_supported()
fscrypt: Replace calls to fscrypt_inode_uses_inline_crypto()
fs/buffer: Remove fs-layer decryption code
f2fs: Remove fs-layer file contents en/decryption code
ext4: Further de-generalize the bio postprocessing code
ext4: Make ext4_bio_write_folio() return void
ext4: Remove fs-layer file contents en/decryption code
Documentation: fscrypt: Update docs for inlinecrypt
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs superblock updates from Christian Brauner:
- Make it possible to share a block device between multiple
filesystems.
erofs can mount read-only blob devices shared between many
superblocks, but because we only tracked a single superblock a
freeze, thaw, removal or sync on such a device was never propagated
to all the superblocks using it, and there was no way to find them.
Add an efficient table to lookup all superblocks using a given block
device.
- A bunch of pre-existing fixes fell out of this work:
A block-device freeze racing a btrfs device change could leave the
whole filesystem stuck frozen. A bdev_freeze() issued by "dmsetup
suspend" or an LVM snapshot resolves that holder to freeze the
filesystem. and bdev_thaw() resolves it again to thaw. A freeze
landing while btrfs is adding, removing or replacing a device freezes
the filesystem. The membership change then drops that link. So the
matching thaw could no longer find the superblock.
Forbid freezing a device for the duration of a membership change,
modelled on deny_write_access()/allow_write_access().
* tag 'vfs-7.3-rc1.super' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (24 commits)
super: fix dying superblock warning messages
block: reject block device inodes with i_rdev == 0 in lookup_bdev()
selftests/filesystems: add ustat() coverage
fs: look up the superblock via the device table in user_get_super()
super: make fs_holder_ops private
f2fs: open via dedicated fs bdev helpers
erofs: open via dedicated fs bdev helpers
fs: tolerate per-superblock freeze errors on shared devices
fs: look up superblocks via the device table in fs_holder_ops
ext4: open via dedicated fs bdev helpers
btrfs: open via dedicated fs bdev helpers
xfs: port to fs_bdev_file_open_by_path()
fs: add dedicated block device open helpers for filesystems
fs: maintain a global device-to-superblock table
ocfs2: don't reset s_dev on dismount
ext4: use anonymous devices for KUnit test superblocks
fs, block: move blk_mode_t and fop_flags_t into <linux/types.h>
super: take lock after last reference count
super: convert s_count to refcount_t s_passive
btrfs: deny freezing devices undergoing a replace
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull iomap updates from Christian Brauner:
"The bulk of this is the conversion of iomap to a single ->iomap_next()
callback and thus finishing the move to an iterator model.
Every iomap operation drove its iteration through a struct iomap_ops
holding ->iomap_begin() and ->iomap_end(). iomap_iter() only ever sees
those as pointers. That means every step of every iteration is an
indirect call.
This collapses both into one ->iomap_next() callback that finishes the
previous mapping and produces the next one. This lets callers inline
the iteration loop and pass its ->iomap_next() as a compile time
constant. That means the compiler can turn it into a direct and hence
inlineable call.
This also allows future callers to express custom logic to drive the
iteration forward better. xfs, btrfs, ext4, ext2, erofs, f2fs, gfs2,
hpfs, fuse, exfat, zonefs, ntfs, ntfs3 and the block device mapping
are all converted. No functional changes are intended.
This also adds a simple direct I/O path for small reads. On Gen5 NVMe
the __iomap_dio_rw() dominates 4K random reads. The same single-core
io_uring poll mode workload reaches ~3.2M IOPS against the raw block
device but only ~1.92M through ext4 or XFS.
__iomap_dio_rw(), iomap_iter(), iomap_dio_bio_iter() and kfree() were
at the top of the profile. The new path is very lightweight if no
special behavior is requested. The bio comes from a dedicated bioset
and laid out so the whole request is a single cacheline aligned
allocation. Completion runs inline.
That takes ext4 from 1.92M to 2.19M IOPS in the original workload. fio
shows around:
- 4% at libaio queue depths of 64 and up
- around 5% for io_uring
- up to 10% for io_uring poll mode at depth 256
on both ext4 and xfs.
A few other patches:
- iomap_folio_mark_uptodate() lets a filesystem that writes into the
page cache outside the iomap read and write paths keep iomap's
internal uptodate bitmap in sync, which fuse needs for
server-pushed notify stores before it can enable large folios;
- two fixes for iomap_bio_read_folio_range_sync(): a potential crash
when device integrity behavior is changed and a missing
bio_uninit().
- a folio batch release fix on iomap callback failures
- FGP_NOFS is dropped from iomap_get_folio()
- documentation fix"
* tag 'vfs-7.3-rc1.iomap' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (29 commits)
iomap: iomap_bio_read_folio_range_sync is missing a call to bio_uninit
iomap: don't free integrity payload that doesn't exist
docs: fix grammatical error in iomap docs
exfat: convert iomap ops to ->iomap_next()
fuse: convert iomap ops to ->iomap_next()
hpfs: convert iomap ops to ->iomap_next()
gfs2: convert iomap ops to ->iomap_next()
f2fs: convert iomap ops to ->iomap_next()
block: convert iomap ops to ->iomap_next()
ext2: convert iomap ops to ->iomap_next()
zonefs: convert iomap ops to ->iomap_next()
erofs: convert iomap ops to ->iomap_next()
ext4: convert iomap ops to ->iomap_next()
ntfs: convert iomap ops to ->iomap_next()
ntfs3: convert iomap ops to ->iomap_next()
btrfs: convert iomap ops to ->iomap_next()
xfs: convert iomap ops to ->iomap_next()
iomap: add ->iomap_next()
iomap: use GFP_NOWAIT when application for iomap_dio_simple allocations
iomap: decouple simple direct I/O reads from iomap_dio_rw
...
|
|
Commit 05c3e88488ed ("srcu: Queue sdp->work when the delay timer is
successfully deleted") added a check in cleanup_srcu_struct() if the
call to srcu_barrier() has been made before calling it, which
revealed a missing call to srcu_barrier() before calling
cleanup_srcu_struct(set->srcu). Fix this.
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Link: https://patch.msgid.link/20260812060510.3220294-1-m.szyprowski@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Not needed, probably since this file was split out.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Anuj Gupta <anuj20.g@samsung.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Link: https://patch.msgid.link/20260810144603.453283-1-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Disks created via blk_mq_alloc_disk_for_queue() (e.g. SCSI SD disks)
do not have GD_OWNS_QUEUE set. Currently __blk_mark_disk_dead() only
sets QUEUE_FLAG_DYING when GD_OWNS_QUEUE is set, so for such disks
blk_queue_enter() and __bio_queue_enter() cannot detect the dying
state via blk_queue_dying() and remain blocked waiting for I/O that
will never complete after surprise removal.
blk_mark_disk_dead() is the explicit "surprise removal" API -- the
caller has already decided the disk is dead. Setting QUEUE_FLAG_DYING
unconditionally here is appropriate: any in-flight I/O from other
threads should get -ENODEV immediately from blk_queue_enter()
regardless of GD_OWNS_QUEUE ownership.
For disks that already have GD_OWNS_QUEUE set, __blk_mark_disk_dead()
will set the flag again which is harmless.
Fixes: 6f8191fdf41d ("block: simplify disk shutdown")
Cc: stable@vger.kernel.org
Signed-off-by: Lianqin Hu <hulianqin@vivo.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/PUZPR06MB62247E82E66A3ED46CC3E6C7D2DC2@PUZPR06MB6224.apcprd06.prod.outlook.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blk_cgroup_congested() walks the current task's blkcg ancestor chain on
every readahead decision and, once swap is in use, on every anonymous and
shmem folio allocation. The answer is almost always "no", but finding that
out costs two loads per level on two cold cache lines, plus an out-of-line
kthread_blkcg() and an RCU read-side pair. On a fleet profile of hosts
running containers with 5-10 level hierarchies it costs about as much as
all of mutex_lock(), 99.4% of it under __folio_throttle_swaprate().
Gate the walk on a global count of blkcgs with a non-zero
congestion_count. The counter only moves on the 0 <-> 1 transitions of
each blkcg's congestion_count, so the extra atomic stays in the throttle
arm/disarm paths and never appears in steady state. When something is
throttled the counter is non-zero and the walk runs as before.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260814165712.510132-4-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
iocg_kick_delay() turns sufficiently large debt into an explicit
block-cgroup delay with blkcg_set_delay(), setting blkg->use_delay to
-1 and incrementing blkcg->congestion_count. Clearing it again depends
on iocg_kick_delay() running from the period timer, the waitq timer or
the issue path.
ioc_pd_free() removes the iocg from active_iocgs and cancels its waitq
timer, and no further bios can arrive, so once it has run nothing is
left which can reduce the debt and clear the delay. The blkcg stays
marked congested for the rest of its life.
blk_cgroup_congested() then returns true for every task in that cgroup
and its descendants: page_cache_sync_ra() cuts readahead to a single
page, page_cache_async_ra() skips it altogether, and
__folio_throttle_swaprate() takes swap_avail_lock and schedules a
throttle on anonymous folio allocation.
Clear it explicitly, after the list removal and the synchronous
hrtimer_cancel() so that neither timer processing nor an I/O path can
re-arm it. The free callback can also see policy data which was never
attached to a blkg, hence the pd->blkg check.
Fixes: 7caa47151ab2 ("blkcg: implement blk-iocost")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260814165712.510132-3-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io.latency can throttle a group which has no latency target of its own.
When a sibling misses its target, check_scale_change() scales down its
peers, and a peer that reaches queue depth one gets blkcg_use_delay()
called on it on every further scale-down, even with min_lat_nsec == 0.
iolatency_pd_offline() resets the target through
iolatency_set_min_lat_nsec(), which clears the delay only on a nonzero
to zero transition, so it never clears such a peer. Freeing the policy
data then leaves blkg->use_delay set and blkcg->congestion_count
elevated with nothing left that can drop it.
blk_cgroup_congested() then returns true for every task in that cgroup
and its descendants for as long as the cgroup lives: page_cache_sync_ra()
cuts readahead to a single page, page_cache_async_ra() skips it
altogether, and __folio_throttle_swaprate() takes swap_avail_lock and
schedules a throttle on anonymous folio allocation.
Clear the delay in iolatency_pd_free(). By then bio-held blkg
references have drained, or the queue is frozen for policy
deactivation, so check_scale_change() cannot re-arm it. The free
callback can also see policy data which was never attached to a blkg,
hence the pd->blkg check.
Fixes: d70675121546 ("block: introduce blk-iolatency io controller")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260814165712.510132-2-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Commit c807ab520fc3 ("block/mq-deadline: Add I/O priority support")
left the dd parameter unused in deadline_move_request().
Commit fde02699c242 ("block: mq-deadline: Remove support for zone
write locking") left dd unused in deadline_fifo_request() and
deadline_next_request().
Remove these unused function parameters.
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260812040729.27551-1-hongfu.li@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
In preparation to supporting dma-buf backed iterators and bios,
introduce bio_iov_iter_set() which attempts to set up the bio directly
from the given iterator. For now, it only supports bvec and expects
users to check the result and fall back to other means if fails, but
later we'll add more types.
Suggested-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/4686a0e47fc14f3f888967a80d45a6f66044f1e0.1785596451.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Consolidate bi_io_vec assignment for cloning in __bio_clone to keep any
further changes in one place.
Suggested-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/6ecfe8f9b1c6bfb8665fba7daf55d9ad7a8a3243.1785596451.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
struct bvec_iter::bi_bvec_done is used an offset in the current bvec,
let's rename it accordingly for better clarity. I also plan to use it
for non-bvec based iteration in the future like dma-buf, so drop the
"bvec" part.
Suggested-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/4e4c21858705a200bd8848ffe4080522e3eb5c1c.1786018753.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
ioc_pd_stat() reads ioc->enabled, ioc->vtime_base_rate, and
iocg->last_stat without holding ioc->lock, which trips KCSAN since
ioc_adjust_base_vrate() and iocg_flush_stat_upward() write those
fields under ioc->lock.
Commit 35198e323001 fixed the same issue in ioc_qos_prfill() and
ioc_cost_model_prfill() by adding spin_lock_irq(&ioc->lock). However,
those functions read configuration parameters (qos/model) that need
synchronized reads. In contrast, ioc_pd_stat() only reads stat
values (vrate, usage) where stale reads are harmless, so data_race()
is more appropriate — it silences the KCSAN warning without adding
lock contention during high-frequency stat reads.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260804054120.161933-1-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Check the BIP_CHECK flags before generating or verifying PI information,
otherwise this can be incorrectly called for non-PI metadata and
cause generation of incorrect metadata and crashed in the verification
handler.
The new behavior matches that of the block layer auto-generated
metadata.
Fixes: 0bde8a12b554 ("block: add fs_bio_integrity helpers")
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Kanchan Joshi <joshi.k@samsung.com>
Reviewed-by: Anuj Gupta <anuj20.g@samsung.com>
Link: https://patch.msgid.link/20260804123928.736596-4-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
To allow for users outside of bio-integrity-auto.c. Also add a little
comment explaining it.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Anuj Gupta <anuj20.g@samsung.com>
Reviewed-by: Kanchan Joshi <joshi.k@samsung.com>
Link: https://patch.msgid.link/20260804123928.736596-3-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
There is no benefit in using this helper over the simple flags check.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Anuj Gupta <anuj20.g@samsung.com>
Reviewed-by: Kanchan Joshi <joshi.k@samsung.com>
Link: https://patch.msgid.link/20260804123928.736596-2-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Add a new BLOCK_URING_CMD_ZONE_RESET_ALL uring cmd to reset all zones
for a given block device. This can be used by storage systems or
file system mkfs tools to initialize multiple devices in parallel.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125038.740388-7-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
All other arguments can be derived from cmd, so do that to simplify the
calling convention.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125038.740388-6-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Just like zone reset / finish, these only apply to sequential zones.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125038.740388-5-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blk_zone_wplug_handle_zone_mgmt now only checks that zone reset and
zone finish operations are directed to a sequential write required
zone. This has nothing to do with zone write plugs and is better
handled with other bio validity checks in submit_bio_noacct.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125038.740388-4-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Commit efae226c2ef1 ("block: handle zone management operations
completions") moved all block layer tracking of zone management
operations to the I/O completion handler. With that REQ_NOWAIT zone
management operations are just fine, so allow them.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125038.740388-3-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The switch at the end of blk_zone_plug_bio always returns, so remove
the dead extra return statement after it.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260804125038.740388-2-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
bfqg_prfill_stat_recursive() and bfq_bio_bfqg() read blkg->online
locklessly, same as blkg_rwstat_recursive_sum(). Annotate with
data_race() to silence KCSAN.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/20260803134104.138411-1-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The last_check_time field in throtl_grp was used by the
CONFIG_BLK_DEV_THROTTLING_LOW mechanism (in throtl_upgrade_check() and
the downgrade logic) to timestamp the last upgrade/downgrade check.
Commit bf20ab538c81 ("blk-throttle: remove CONFIG_BLK_DEV_THROTTLING_LOW")
removed all five of its uses in blk-throttle.c and the surrounding LOW
fields (latency_target, last_finish_time, checked_last_finish_time,
avg_idletime, idletime_threshold, bio_cnt, bad_bio_cnt,
bio_cnt_reset_time), but missed the field definition itself.
It has been a dead field since then: zero references in blk-throttle.c
and the whole tree. Remove it.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Reviewed-by: Tang Yizhou <yizhou.tang@shopee.com>
Link: https://patch.msgid.link/20260717054855.2018578-1-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
kyber_timer_fn() sums the per-cpu latency histograms with
for_each_online_cpu(). A CPU that goes offline mid-interval leaves its
bucket un-flushed; the samples are lost from the current decision and
re-appear (stale) when the CPU is onlined again.
Fixes: 6e25cb01ea20 ("kyber: implement improved heuristics")
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/20260720093726.28965-5-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
ioc_lat_stat() walks ioc->pcpu_stat with for_each_online_cpu() to
compute missed-ppm and rq_wait deltas. An offlined CPU is skipped, so
its delta is dropped from the period and its last_* watermark is not
advanced; on re-online the next collection sees a delta spanning the
whole offline interval, corrupting the latency/vrate picture.
Fixes: 7caa47151ab2 ("blkcg: implement blk-iocost")
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/20260720093726.28965-4-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
iolatency_check_latencies() and iolatency_ssd_stat() iterate a blkg's
per-cpu latency stats with for_each_online_cpu(). When a CPU that has
accumulated io.latency samples goes offline, its bucket is skipped: the
check loop (which also resets) neither sums nor clears it, and the show
path under-reports. On re-online the stranded samples are flushed into
a later check window, which can trigger a spurious throttle/scale
adjustment.
Fixes: d70675121546 ("block: introduce blk-iolatency io controller")
Fixes: 1fa2840e56f9 ("blk-iolatency: use a percentile approache for ssd's")
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/20260720093726.28965-3-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blk_stat_timer_fn() sums and resets a callback's per-cpu buckets using
for_each_online_cpu(). A CPU that goes offline with pending samples is
skipped, so its samples are neither accumulated into the window nor
cleared; they sit in the bucket until the CPU comes back online, at
which point the stale values are flushed into whatever window is then
running.
This silently corrupts the latency picture that consumers (notably
writeback throttling via wbt, and blk-mq latency tracking) base
decisions on around CPU hotplug: under-counting while the CPU is
offline, then a burst of stale data on re-online.
Fixes: 34dbad5d26e2 ("blk-stat: convert to callback-based statistics reporting")
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/20260720093726.28965-2-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Move the teardown sequence which offlines and frees per-policy
blkg_policy_data (pd) into a helper for readability.
No functional change intended.
Signed-off-by: Zheng Qixing <zhengqixing@huawei.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Tang Yizhou <yizhou.tang@shopee.com>
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Link: https://patch.msgid.link/20260802112525.3933753-5-yukuai@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
When switching IO schedulers on a block device, blkcg_activate_policy()
can race with concurrent blkcg deletion, leading to a use-after-free in
rcu_accelerate_cbs.
T1: T2:
blkg_destroy
kill(&blkg->refcnt) // blkg->refcnt=1->0
blkg_release // call_rcu(__blkg_release)
...
blkg_free_workfn
->pd_free_fn(pd)
elv_iosched_store
elevator_switch
...
iterate blkg list
blkg_get(blkg) // blkg->refcnt=0->1
list_del_init(&blkg->q_node)
blkg_put(pinned_blkg) // blkg->refcnt=1->0
blkg_release // call_rcu again
rcu_accelerate_cbs // uaf
Fix this by checking hlist_unhashed(&blkg->blkcg_node) before getting
a reference to the blkg. This is the same check used in blkg_destroy()
to detect if a blkg has already been destroyed. If the blkg is already
unhashed, skip processing it since it's being destroyed.
Fixes: f1c006f1c685 ("blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()")
Signed-off-by: Zheng Qixing <zhengqixing@huawei.com>
Reviewed-by: Tang Yizhou <yizhou.tang@shopee.com>
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Reviewed-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Link: https://patch.msgid.link/20260802112525.3933753-4-yukuai@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
When switching an IO scheduler on a block device, blkcg_activate_policy()
allocates blkg_policy_data (pd) for all blkgs attached to the queue.
However, blkcg_activate_policy() may race with concurrent blkcg deletion,
leading to use-after-free and memory leak issues.
The use-after-free occurs in the following race:
T1 (blkcg_activate_policy):
- Successfully allocates pd for blkg1 (loop0->queue, blkcgA)
- Fails to allocate pd for blkg2 (loop0->queue, blkcgB)
- Enters the enomem rollback path to release blkg1 resources
T2 (blkcg deletion):
- blkcgA is deleted concurrently
- blkg1 is freed via blkg_free_workfn()
- blkg1->pd is freed
T1 (continued):
- Rollback path accesses blkg1->pd->online after pd is freed
- Triggers use-after-free
In addition, blkg_free_workfn() frees pd before removing the blkg from
q->blkg_list. This allows blkcg_activate_policy() to allocate a new pd
for a blkg that is being destroyed, leaving the newly allocated pd
unreachable when the blkg is finally freed.
Fix these races by extending blkcg_mutex coverage to serialize
blkcg_activate_policy() rollback and blkg destruction, ensuring pd
lifecycle is synchronized with blkg list visibility.
Fixes: f1c006f1c685 ("blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()")
Signed-off-by: Zheng Qixing <zhengqixing@huawei.com>
Reviewed-by: Tang Yizhou <yizhou.tang@shopee.com>
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Reviewed-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Link: https://patch.msgid.link/20260802112525.3933753-3-yukuai@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blkcg_mutex
blkg_destroy_all() iterates q->blkg_list without holding blkcg_mutex,
which can race with blkg_free_workfn() that removes blkgs from the list
while holding blkcg_mutex.
Add blkcg_mutex protection around the q->blkg_list iteration to prevent
potential list corruption or use-after-free issues.
Reviewed-by: Tang Yizhou <yizhou.tang@shopee.com>
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Reviewed-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Link: https://patch.msgid.link/20260802112525.3933753-2-yukuai@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blkg_rwstat_recursive_sum() reads pos_blkg->online without the queue
lock that its doc comment requires, since blkcg_print_blkgs() stopped
holding it in 56cc24f59c14. Concurrent blkg_create/destroy flips
->online, tripping KCSAN. The race is harmless (RCU-protected, stale
online only causes minor stat noise). Use data_race() to annotate
the intentional lockless read.
Also update the stale doc comment that still requires the queue lock.
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260803133400.137906-1-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Block device buffered reads and writes already pass through
filemap_read() and iomap_file_buffered_write() respectively, both of
which handle IOCB_DONTCACHE. Enable RWF_DONTCACHE for block device files
by setting FOP_DONTCACHE in def_blk_fops.
For CONFIG_BUFFER_HEAD=y, writeback goes through buffer_head's
__bh_submit() which sets BIO_COMPLETE_IN_TASK on dropbehind folios. For
CONFIG_BUFFER_HEAD=n, writeback goes through iomap which handles it via
BIO_COMPLETE_IN_TASK on the ioend bio.
This support is useful for databases that operate on raw block devices,
among other userspace applications.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Reviewed-by: Jan Kara <jack@suse.cz>
Link: https://patch.msgid.link/20260730-blk-dontcache-v7-5-3e8e6850068d@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Some bio completion handlers need to run from preemptible task context,
but bio_endio() may be called from IRQ context (e.g., buffer_head
writeback). Callers need a way to ensure their callback eventually runs
from a sleepable context. Add infrastructure for that, in two forms:
1. BIO_COMPLETE_IN_TASK, a bio flag the submitter sets when it knows
in advance that its callback needs task context (e.g., dropbehind
writeback). bio_endio() sees the flag and offloads completion to a
worker automatically.
2. bio_complete_in_task(), a helper that completion callbacks can
invoke from within bi_end_io() when the deferral decision is
dynamic (e.g., fserror reporting).
Both share a per-CPU list drained by a work item on a WQ_PERCPU
workqueue. Producers push the bio onto the local CPU's list and schedule
the work item, which then dispatches each bio's bi_end_io() from task
context.
Both methods are gated on bio_in_atomic(), which returns true in any
context where a sleeping bi_end_io() is unsafe, including
non-preemptible task context.
Two CPU hotplug callbacks are used to drain remaining bios from the
departing CPU's batch, while maintaining the per-CPU behavior. The
CPUHP_AP_ONLINE_DYN callback disables the per-CPU work item while the
CPU is still online, preventing it from running on an unbound worker
later. CPUHP_BP_PREPARE_DYN then drains any bios added between disabling
the work item and CPU offline.
Link: https://lore.kernel.org/all/20260409160243.1008358-1-hch@lst.de/
Suggested-by: Matthew Wilcox <willy@infradead.org>
Suggested-by: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260730-blk-dontcache-v7-2-3e8e6850068d@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The bio-based drivers don't necessarily check the alignment split, and
stacking block drivers don't always handle a misalignment detected after
submitting the bio. Validate user vectors against the device's
dma_alignment as the bio is built from the iov_iter, rejecting
misaligned early with -EINVAL.
Cc: stable@vger.kernel.org
Fixes: 5ff3f74e145a ("block: simplify direct io validity check")
Fixes: 7eac33186957 ("iomap: simplify direct io validity check")
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Link: https://patch.msgid.link/20260720201057.1862857-6-kbusch@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Convert block iomap_ops to the new ->iomap_next() callback. The callback is
generated with the DEFINE_IOMAP_ITER_NEXT() macro, which wraps the
iomap_iter_next() helper to finish the previous mapping if needed and
produce the next one. No functional changes are intended.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Link: https://patch.msgid.link/20260729192737.3190206-15-joannelkoong@gmail.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
A failed integrity mapping holds a dio reference, so we need to go
through the full bio ending in case there were previously submitted
bio's in the sequence.
Fixes: 2729a60bbfb92 ("block: don't silently ignore metadata for sync read/write")
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Link: https://patch.msgid.link/20260720201057.1862857-3-kbusch@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blkdev_iov_iter_get_pages() can return various error values, including
EIO, EFAULT, and ENOMEM. Set the actual reported status so user space
can know why an operation failed.
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Link: https://patch.msgid.link/20260720201057.1862857-2-kbusch@meta.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
