diff options
| author | Mark Brown <broonie@kernel.org> | 2026-08-21 13:56:44 +0100 |
|---|---|---|
| committer | Mark Brown <broonie@kernel.org> | 2026-08-21 13:56:44 +0100 |
| commit | ac8420416e13c022d9495dfe9aa8388fb1c3ca39 (patch) | |
| tree | f2bda564c7b2e8402667d050edfe9fba550e7167 | |
| parent | 5a91bf467b12a0973d8a6634e8c9b7a4918828c5 (diff) | |
| parent | f3d3bb6e63ceddb85bf640fab8c2fa621b706ab1 (diff) | |
| download | linux-next-ac8420416e13c022d9495dfe9aa8388fb1c3ca39.tar.gz linux-next-ac8420416e13c022d9495dfe9aa8388fb1c3ca39.zip | |
Merge branch 'fs-next' of linux-next
223 files changed, 12371 insertions, 2479 deletions
diff --git a/Documentation/ABI/testing/sysfs-fs-f2fs b/Documentation/ABI/testing/sysfs-fs-f2fs index 1b58c029abd0..f4e6a7415cde 100644 --- a/Documentation/ABI/testing/sysfs-fs-f2fs +++ b/Documentation/ABI/testing/sysfs-fs-f2fs @@ -1002,3 +1002,13 @@ Description: It can be used to tune priority of f2fs critical task, e.g. f2fs_ck threads, limitation as below: - it requires user has CAP_SYS_NICE capability. - the range is [100, 139], by default the value is 120. + +What: /sys/fs/f2fs/<disk>/max_atc_write_bio_size +Date: June 2026 +Contact: Bart Van Assche <bvanassche@acm.org> +Description: Every time a write operation completes f2fs_write_end_io() is + called. This function may be called from an atomic context, + e.g. from inside an interrupt handler. This attribute controls + the maximum size of a write bio that is completed in atomic + (atc) context. The default value for this attribute is UINT_MAX + which means that this functionality is disabled by default. diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst index 0413dcd9ef69..28300fee22bf 100644 --- a/Documentation/block/ublk.rst +++ b/Documentation/block/ublk.rst @@ -382,17 +382,17 @@ Zero copy --------- ublk zero copy relies on io_uring's fixed kernel buffer, which provides -two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`. +two APIs: `io_buffer_register_request()` and `io_buffer_unregister`. ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call -`io_buffer_register_bvec()` for ublk server to register client request +`io_buffer_register_request()` for ublk server to register client request buffer into io_uring buffer table, then ublk server can submit io_uring IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF` -calls `io_buffer_unregister_bvec()` to unregister the buffer, which is -guaranteed to be live between calling `io_buffer_register_bvec()` and -`io_buffer_unregister_bvec()`. Any io_uring operation which supports this -kind of kernel buffer will grab one reference of the buffer until the -operation is completed. +calls `io_buffer_unregister()` to unregister the buffer, which is guaranteed +to be live between calling `io_buffer_register_request()` and +`io_buffer_unregister()`. Any io_uring operation which supports this kind of +kernel buffer will grab one reference of the buffer until the operation is +completed. ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and be trusted, because it is ublk server's responsibility to make sure IO buffer diff --git a/Documentation/filesystems/f2fs.rst b/Documentation/filesystems/f2fs.rst index b45d7a687625..1cb339d9d2f1 100644 --- a/Documentation/filesystems/f2fs.rst +++ b/Documentation/filesystems/f2fs.rst @@ -1043,6 +1043,41 @@ So, the key idea is, user can do any file operations on /dev/vdc, and reclaim the space after the use, while the space is counted as /data. That doesn't require modifying partition size and filesystem format. +Dynamic Device Aliasing Management +---------------------------------- + +In addition to static device aliasing by deleting the aliasing file, F2FS +supports dynamic management of device aliasing. This mechanism allows the system +to dynamically transition partition ownership between F2FS userdata and external +entities (e.g., zRAM, raw partition) based on system requirements without +deleting the master aliasing file or requiring unmount/remount. + +The master aliasing file is created during the initial format of the file system +and remains as a persistent control entity (ioctl gateway) in the root directory. + +- Partition Reservation (In-service to Aliased) + When a specific partition needs to be dedicated to external services (e.g., zRAM), + a user can reserve the device alias range via ioctl. The kernel resets GC victim + information for the target range, marks segments as in-use to prevent new + allocations, and triggers forced GC to migrate existing valid data out of the + range. Finally, it reserves these blocks in the SIT to effectively exclude the + device from the usable capacity. + +- Partition Release (Aliased to In-service) + When external usage concludes, the space is reclaimed not by deleting the file, + but through the release ioctl. The kernel truncates blocks associated with + the file, releasing them back to general filesystem allocation. + +.. code-block:: + + # f2fs_io dev_alias release /mnt/f2fs/vdc.file + # df -h + /dev/vdb 64G 753M 64G 2% /mnt/f2fs + + # f2fs_io dev_alias reserve /mnt/f2fs/vdc.file + # df -h + /dev/vdb 64G 33G 32G 52% /mnt/f2fs + Per-file Read-Only Large Folio Support -------------------------------------- diff --git a/Documentation/filesystems/fuse/fuse-io-uring.rst b/Documentation/filesystems/fuse/fuse-io-uring.rst index d73dd0dbd238..29f98057500d 100644 --- a/Documentation/filesystems/fuse/fuse-io-uring.rst +++ b/Documentation/filesystems/fuse/fuse-io-uring.rst @@ -11,6 +11,9 @@ and works. For generic details about FUSE see fuse.rst. This document also covers the current interface, which is still in development and might change. +For the userspace protocol, see +Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst. + Limitations =========== As of now not all requests types are supported through io-uring, userspace @@ -95,5 +98,34 @@ Sending requests with CQEs | <fuse_unlink() | | <sys_unlink() | - - +Buffer pools +============ + +Without a buffer pool, every entry needs to pass a dedicated payload buffer +large enough for the maximum payload size. A buffer pool decouples entries +from payload buffers. The server hands the kernel one contiguous buffer pool +of memory and when the kernel sends the server a request, it indicates the +offset into the pool for that request's payload. Internally, the kernel is +able to manage/optimize the buffer pool memory however it likes. + +A server may also register the pool region with io_uring as a fixed buffer. +The backing pages are then pinned once, avoiding per-request pinning and +address translation. This also allows servers to use the same registered +buffers for subsequent backing store I/O through io-uring, keeping data +in the same pinned pages without additional pinning / mapping overhead. + +Zero-copy +========= + +Zero-copy lets the server read from / write to the client's pages (pinned +user pages for direct I/O, or page-cache folios for buffered I/O) without an +intermediary payload copy. This requires CAP_SYS_ADMIN privileges. + +When a fuse request arrives for a file that opted into zero-copy, the kernel +registers the relevant pages (pinned user pages for direct i/o or underlying +page cache folios for buffered i/o) into a sparse slot in the server's +io_uring registered buffer table. The server can then operate on these pages +directly using io-uring fixed buffer operations (eg read_fixed / write_fixed) +and the kernel unregisters these pages when the request completes. +Non-page-backed args (eg op out headers) will go through the payload buffer as +normal. diff --git a/Documentation/filesystems/fuse/index.rst b/Documentation/filesystems/fuse/index.rst index 393a845214da..3dada6c4057a 100644 --- a/Documentation/filesystems/fuse/index.rst +++ b/Documentation/filesystems/fuse/index.rst @@ -12,3 +12,4 @@ FUSE (Filesystem in Userspace) Technical Documentation fuse-io fuse-io-uring fuse-passthrough + uapi/fuse-uapi-io-uring diff --git a/Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst b/Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst new file mode 100644 index 000000000000..8367be7ea29d --- /dev/null +++ b/Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst @@ -0,0 +1,126 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================================== +FUSE-over-io-uring uapi documentation +===================================== + +Commands +======== + +``enum fuse_uring_cmd``: + +``FUSE_IO_URING_CMD_ADD_QUEUE`` + Create a queue identified by ``fuse_uring_cmd_req.qid``. Queue-wide + options are passed in ``fuse_uring_cmd_req.flags``: + + ``FUSE_URING_ZERO_COPY`` + Enable zero-copy on this queue. Requires ``CAP_SYS_ADMIN`` and a buffer + pool, which is added separately via ``ADD_BUFPOOL`` before registering + entries (see `Zero-copy`_). + +``FUSE_IO_URING_CMD_ADD_BUFPOOL`` + Register the payload buffer pool for an existing queue. The server provides + a single contiguous region in ``fuse_uring_cmd_req.bufpool.uaddr`` / + ``.len``. This command must be issued after ``ADD_QUEUE`` and before + registering any payload-carrying entries on that queue. + ``fuse_uring_cmd_req.flags`` must be 0. Submitting this command with + ``IORING_URING_CMD_FIXED`` marks the pool as registered, which avoids per + i/o pinning/unpinning and mapping overhead (see `Buffer pools`_). + +``FUSE_IO_URING_CMD_REGISTER`` + Register a ring entry (a long-lived SQE that carries the request header + iovec). For a zero-copy queue, ``fuse_uring_cmd_req.ent_zero_copy_buf_index`` + indicates the reserved registered buffer table slot this entry uses for + zero-copy (see `Zero-copy`_). + +``FUSE_IO_URING_CMD_COMMIT_AND_FETCH`` + Commit the reply for a completed request and fetch the next one. The + request is identified by ``fuse_uring_cmd_req.commit_id`` (the value the + kernel reported in ``fuse_uring_ent_in_out.commit_id``). + +Structures +========== + +``struct fuse_uring_cmd_req`` (80-byte SQE command area): + +============================ ================================================== +Field Meaning +============================ ================================================== +``flags`` Command-specific flags (see each command). +``commit_id`` Request id, for ``COMMIT_AND_FETCH``. +``qid`` Queue index. +``bufpool.uaddr`` Pool base address, for ``ADD_BUFPOOL``. +``bufpool.len`` Pool length in bytes, for ``ADD_BUFPOOL``. +``bufpool.reserved`` Must be 0, for ``ADD_BUFPOOL``. +``ent_zero_copy_buf_index`` Per-entry zero-copy slot, for ``REGISTER``. +============================ ================================================== + +``struct fuse_uring_ent_in_out`` (reported by the kernel per request): + +============================ ================================================== +Field Meaning +============================ ================================================== +``flags`` ``FUSE_URING_ENT_ZERO_COPY`` if zero-copied. +``commit_id`` Id to echo back in ``COMMIT_AND_FETCH``. +``payload_sz`` Total payload size in bytes (see `Zero-copy`_). +``offset`` Payload buffer offset within the pool. +============================ ================================================== + +Buffer pools +============ +Setup: + +* Issue ``ADD_QUEUE`` for the qid. +* Issue ``ADD_BUFPOOL`` with ``bufpool.uaddr`` and ``bufpool.len`` pointing + at the region. +* Register entries with ``REGISTER``. + +For every request that has a payload, the kernel reports where the payload +lives in ``struct fuse_uring_ent_in_out`` (part of +``struct fuse_uring_req_header``): + +``offset`` + Byte offset, within the pool region, for this request's payload buffer. + The server adds this to the pool base address to locate the payload. + +``payload_sz`` + Number of payload bytes for this request. + +To use registered buffers, the server registers the pool region with io_uring +and submits ``ADD_BUFPOOL`` with ``IORING_URING_CMD_FIXED`` set in +``sqe->uring_cmd_flags`` and the index of the registered bufpool in +``sqe->buf_index``. Every SQE the server submits afterwards must follow the +same fixed-buffer protocol, carrying ``IORING_URING_CMD_FIXED`` and that same +``sqe->buf_index``. The same registered buffer can be reused for the server's +backing-store I/O as well (e.g. ``IORING_OP_READ_FIXED`` / +``IORING_OP_WRITE_FIXED``). + +Zero-copy +========= +Requirements: + +* The server must be privileged (``CAP_SYS_ADMIN``). +* A zero-copy queue: ``ADD_QUEUE`` with the ``FUSE_URING_ZERO_COPY`` flag set. +* A buffer pool: ``ADD_BUFPOOL``. +* For each entry, ``REGISTER`` with ``ent_zero_copy_buf_index`` set to the + index this entry uses in the server's io_uring registered-buffer table. + This is where the kernel registers the request's pages for the server to + access (it is separate from the payload pool). On a non-zero-copy queue this + field must be 0. + +Zero-copy is selected per open file. The server sets the open-file flag in +the ``FUSE_OPEN`` / ``FUSE_CREATE`` reply: + +``FOPEN_IO_URING_ZERO_COPY`` + Reads/writes on this open file should use zero-copy. + +For a request that is zero-copied, the kernel sets ``FUSE_URING_ENT_ZERO_COPY`` +in ``fuse_uring_ent_in_out.flags`` and places the request's pages at the +entry's ``ent_zero_copy_buf_index``. The server then issues +``IORING_OP_READ_FIXED`` / ``IORING_OP_WRITE_FIXED`` against that index to +transfer the data directly to/from the client's pages. + +For such a request, ``payload_sz`` includes the zero-copied page bytes +(transferred via the registered buffer at ``ent_zero_copy_buf_index``). Any +non-page-backed args (e.g. op headers) are still copied through the pool +payload buffer at ``offset``. diff --git a/MAINTAINERS b/MAINTAINERS index b141e6e7e92c..0338751dac07 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9774,7 +9774,7 @@ EXFAT FILE SYSTEM M: Namjae Jeon <linkinjeon@kernel.org> M: Sungjong Seo <sj1557.seo@samsung.com> R: Yuezhang Mo <yuezhang.mo@sony.com> -L: linux-fsdevel@vger.kernel.org +L: exfat@lists.linux.dev S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat.git F: fs/exfat/ @@ -14173,9 +14173,11 @@ M: Steve French <smfrench@gmail.com> M: Steve French <sfrench@samba.org> R: Sergey Senozhatsky <senozhatsky@chromium.org> R: Tom Talpey <tom@talpey.com> +R: ChenXiaoSong <chenxiaosong@chenxiaosong.com> +R: ChenXiaoSong <chenxiaosong@kylinos.cn> L: linux-cifs@vger.kernel.org S: Maintained -T: git https://git.samba.org/ksmbd.git ksmbd-for-next +T: git git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/smb.git ksmbd-for-next F: Documentation/filesystems/smb/ksmbd.rst F: fs/smb/common/ F: fs/smb/server/ @@ -19353,7 +19355,7 @@ F: drivers/ntb/hw/intel/ NTFS FILESYSTEM M: Namjae Jeon <linkinjeon@kernel.org> M: Hyunchul Lee <hyc.lee@gmail.com> -L: linux-fsdevel@vger.kernel.org +L: ntfs@lists.linux.dev S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/ntfs.git F: Documentation/filesystems/ntfs.rst diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 4d17ed264da1..6c5bec7da97c 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -1699,8 +1699,8 @@ ublk_auto_buf_register(const struct ublk_queue *ubq, struct request *req, { int ret; - ret = io_buffer_register_bvec(cmd, req, ublk_io_release, - io->buf.auto_reg.index, issue_flags); + ret = io_buffer_register_request(cmd, req, ublk_io_release, + io->buf.auto_reg.index, issue_flags); if (ret) { if (io->buf.auto_reg.flags & UBLK_AUTO_BUF_REG_FALLBACK) { ublk_auto_buf_reg_fallback(ubq, req->tag); @@ -1909,7 +1909,7 @@ static noinline void ublk_batch_dispatch_fail(struct ublk_queue *ubq, ublk_io_unlock(io); if (index != -1) - io_buffer_unregister_bvec(data->cmd, index, + io_buffer_unregister(data->cmd, index, data->issue_flags); } @@ -3192,8 +3192,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd, if (!req) return -EINVAL; - ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index, - issue_flags); + ret = io_buffer_register_request(cmd, req, ublk_io_release, index, + issue_flags); if (ret) { ublk_put_req_ref(io, req); return ret; @@ -3224,8 +3224,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd, if (!ublk_dev_support_zero_copy(ub) || !blk_rq_has_data(req)) return -EINVAL; - ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index, - issue_flags); + ret = io_buffer_register_request(cmd, req, ublk_io_release, index, + issue_flags); if (ret) return ret; @@ -3240,7 +3240,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd, if (!(ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY)) return -EINVAL; - return io_buffer_unregister_bvec(cmd, index, issue_flags); + return io_buffer_unregister(cmd, index, issue_flags); } static int ublk_check_fetch_buf(const struct ublk_device *ub, __u64 buf_addr) @@ -3384,7 +3384,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, goto out; /* - * io_buffer_unregister_bvec() doesn't access the ubq or io, + * io_buffer_unregister() doesn't access the ubq or io, * so no need to validate the q_id, tag, or task */ if (_IOC_NR(cmd_op) == UBLK_IO_UNREGISTER_IO_BUF) @@ -3456,7 +3456,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, req = ublk_fill_io_cmd(io, cmd); ublk_apply_io_buf(ub, io, cmd, addr, &auto_buf, &buf_idx); if (buf_idx != UBLK_INVALID_BUF_IDX) - io_buffer_unregister_bvec(cmd, buf_idx, issue_flags); + io_buffer_unregister(cmd, buf_idx, issue_flags); compl = ublk_need_complete_req(ub, io); if (req_op(req) == REQ_OP_ZONE_APPEND) @@ -3801,7 +3801,7 @@ static int ublk_batch_commit_io(struct ublk_queue *ubq, } if (buf_idx != UBLK_INVALID_BUF_IDX) - io_buffer_unregister_bvec(data->cmd, buf_idx, data->issue_flags); + io_buffer_unregister(data->cmd, buf_idx, data->issue_flags); if (req_op(req) == REQ_OP_ZONE_APPEND) req->__sector = ublk_batch_zone_lba(uc, elem); if (compl) diff --git a/drivers/mtd/ubi/attach.c b/drivers/mtd/ubi/attach.c index 0fa115cbf3ad..0ce7ff7400d3 100644 --- a/drivers/mtd/ubi/attach.c +++ b/drivers/mtd/ubi/attach.c @@ -771,7 +771,7 @@ void ubi_remove_av(struct ubi_attach_info *ai, struct ubi_ainf_volume *av) static int early_erase_peb(struct ubi_device *ubi, const struct ubi_attach_info *ai, int pnum, int ec) { - int err; + int err, torture = 0; struct ubi_ec_hdr *ec_hdr; if ((long long)ec >= UBI_MAX_ERASECOUNTER) { @@ -790,7 +790,7 @@ static int early_erase_peb(struct ubi_device *ubi, ec_hdr->ec = cpu_to_be64(ec); - err = ubi_io_sync_erase(ubi, pnum, 0); + err = ubi_io_sync_erase(ubi, pnum, &torture); if (err < 0) goto out_free; diff --git a/drivers/mtd/ubi/build.c b/drivers/mtd/ubi/build.c index 674ad87809df..910e6b0ff5e7 100644 --- a/drivers/mtd/ubi/build.c +++ b/drivers/mtd/ubi/build.c @@ -36,7 +36,7 @@ #define MTD_PARAM_LEN_MAX 64 /* Maximum number of comma-separated items in the 'mtd=' parameter */ -#define MTD_PARAM_MAX_COUNT 6 +#define MTD_PARAM_MAX_COUNT 7 /* Maximum value for the number of bad PEBs per 1024 PEBs */ #define MAX_MTD_UBI_BEB_LIMIT 768 @@ -56,6 +56,7 @@ * @max_beb_per1024: maximum expected number of bad PEBs per 1024 PEBs * @enable_fm: enable fastmap when value is non-zero * @need_resv_pool: reserve pool->max_size pebs when value is none-zero + * @wl_threshold: wear-leveling threshold, 0 means use CONFIG_MTD_UBI_WL_THRESHOLD */ struct mtd_dev_param { char name[MTD_PARAM_LEN_MAX]; @@ -64,6 +65,7 @@ struct mtd_dev_param { int max_beb_per1024; int enable_fm; int need_resv_pool; + int wl_threshold; }; /* Numbers of elements set in the @mtd_dev_param array */ @@ -832,6 +834,8 @@ static int autoresize(struct ubi_device *ubi, int vol_id) * @max_beb_per1024: maximum expected number of bad PEB per 1024 PEBs * @disable_fm: whether disable fastmap * @need_resv_pool: whether reserve pebs to fill fm_pool + * @wl_threshold: wear-leveling threshold for this UBI device; 0 means use + * %CONFIG_MTD_UBI_WL_THRESHOLD; accepted range is 2-65536 * * This function attaches MTD device @mtd_dev to UBI and assign @ubi_num number * to the newly created UBI device, unless @ubi_num is %UBI_DEV_NUM_AUTO, in @@ -848,7 +852,7 @@ static int autoresize(struct ubi_device *ubi, int vol_id) */ int ubi_attach_mtd_dev(struct mtd_info *mtd, int ubi_num, int vid_hdr_offset, int max_beb_per1024, bool disable_fm, - bool need_resv_pool) + bool need_resv_pool, int wl_threshold) { struct ubi_device *ubi; int i, err; @@ -859,6 +863,15 @@ int ubi_attach_mtd_dev(struct mtd_info *mtd, int ubi_num, if (!max_beb_per1024) max_beb_per1024 = CONFIG_MTD_UBI_BEB_LIMIT; + if (!wl_threshold) + wl_threshold = CONFIG_MTD_UBI_WL_THRESHOLD; + + if (wl_threshold < 2 || wl_threshold > 65536) { + pr_err("ubi: bad wear-leveling threshold %d\n", + wl_threshold); + return -EINVAL; + } + /* * Check if we already have the same MTD device attached. * @@ -944,6 +957,8 @@ int ubi_attach_mtd_dev(struct mtd_info *mtd, int ubi_num, ubi->ubi_num = ubi_num; ubi->vid_hdr_offset = vid_hdr_offset; ubi->autoresize_vol_id = -1; + ubi->wl_threshold = wl_threshold; + ubi->wl_free_max_diff = wl_threshold * 2; #ifdef CONFIG_MTD_UBI_FASTMAP ubi->fm_pool.used = ubi->fm_pool.size = 0; @@ -1044,7 +1059,7 @@ int ubi_attach_mtd_dev(struct mtd_info *mtd, int ubi_num, ubi->vol_count - UBI_INT_VOL_COUNT, UBI_INT_VOL_COUNT, ubi->vtbl_slots); ubi_msg(ubi, "max/mean erase counter: %d/%d, WL threshold: %d, image sequence number: %u", - ubi->max_ec, ubi->mean_ec, CONFIG_MTD_UBI_WL_THRESHOLD, + ubi->max_ec, ubi->mean_ec, ubi->wl_threshold, ubi->image_seq); ubi_msg(ubi, "available PEBs: %d, total reserved PEBs: %d, PEBs reserved for bad PEB handling: %d", ubi->avail_pebs, ubi->rsvd_pebs, ubi->beb_rsvd_pebs); @@ -1105,6 +1120,7 @@ int ubi_detach_mtd_dev(int ubi_num, int anyway) ubi->ref_count -= 1; if (ubi->ref_count) { if (!anyway) { + put_device(&ubi->dev); spin_unlock(&ubi_devices_lock); return -EBUSY; } @@ -1247,7 +1263,7 @@ static void ubi_notify_add(struct mtd_info *mtd) /* called while holding mtd_table_mutex */ mutex_lock_nested(&ubi_devices_mutex, SINGLE_DEPTH_NESTING); - err = ubi_attach_mtd_dev(mtd, UBI_DEV_NUM_AUTO, 0, 0, false, false); + err = ubi_attach_mtd_dev(mtd, UBI_DEV_NUM_AUTO, 0, 0, false, false, 0); mutex_unlock(&ubi_devices_mutex); if (err < 0) __put_mtd_device(mtd); @@ -1289,7 +1305,8 @@ static int __init ubi_init_attach(void) err = ubi_attach_mtd_dev(mtd, p->ubi_num, p->vid_hdr_offs, p->max_beb_per1024, p->enable_fm == 0, - p->need_resv_pool != 0); + p->need_resv_pool != 0, + p->wl_threshold); mutex_unlock(&ubi_devices_mutex); if (err < 0) { pr_err("UBI error: cannot attach mtd%d\n", @@ -1317,10 +1334,10 @@ static int __init ubi_init_attach(void) return 0; out_detach: - for (k = 0; k < i; k++) + for (k = 0; k < UBI_MAX_DEVICES; k++) if (ubi_devices[k]) { mutex_lock(&ubi_devices_mutex); - ubi_detach_mtd_dev(ubi_devices[k]->ubi_num, 1); + ubi_detach_mtd_dev(k, 1); mutex_unlock(&ubi_devices_mutex); } return err; @@ -1569,12 +1586,24 @@ static int ubi_mtd_param_parse(const char *val, const struct kernel_param *kp) } else p->need_resv_pool = 0; + token = tokens[6]; + if (token) { + int err = kstrtoint(token, 10, &p->wl_threshold); + + if (err) { + pr_err("UBI error: bad value for wl_threshold parameter: %s\n", + token); + return -EINVAL; + } + } else + p->wl_threshold = 0; + mtd_devs += 1; return 0; } module_param_call(mtd, ubi_mtd_param_parse, NULL, NULL, 0400); -MODULE_PARM_DESC(mtd, "MTD devices to attach. Parameter format: mtd=<name|num|path>[,<vid_hdr_offs>[,max_beb_per1024[,ubi_num]]].\n" +MODULE_PARM_DESC(mtd, "MTD devices to attach. Parameter format: mtd=<name|num|path>[,<vid_hdr_offs>[,max_beb_per1024[,ubi_num[,enable_fm[,need_resv_pool[,wl_threshold]]]]]].\n" "Multiple \"mtd\" parameters may be specified.\n" "MTD devices may be specified by their number, name, or path to the MTD character device node.\n" "Optional \"vid_hdr_offs\" parameter specifies UBI VID header position to be used by UBI. (default value if 0)\n" @@ -1587,9 +1616,10 @@ MODULE_PARM_DESC(mtd, "MTD devices to attach. Parameter format: mtd=<name|num|pa "Example 1: mtd=/dev/mtd0 - attach MTD device /dev/mtd0.\n" "Example 2: mtd=content,1984 mtd=4 - attach MTD device with name \"content\" using VID header offset 1984, and MTD device number 4 with default VID header offset.\n" "Example 3: mtd=/dev/mtd1,0,25 - attach MTD device /dev/mtd1 using default VID header offset and reserve 25*nand_size_in_blocks/1024 erase blocks for bad block handling.\n" + "\t(e.g. if the NAND *chipset* has 4096 PEB, 100 will be reserved for this UBI device).\n" "Example 4: mtd=/dev/mtd1,0,0,5 - attach MTD device /dev/mtd1 to UBI 5 and using default values for the other fields.\n" "example 5: mtd=1,0,0,5 mtd=2,0,0,6,1 - attach MTD device /dev/mtd1 to UBI 5 and disable fastmap; attach MTD device /dev/mtd2 to UBI 6 and enable fastmap.(only works when fastmap is enabled and fm_autoconvert=Y).\n" - "\t(e.g. if the NAND *chipset* has 4096 PEB, 100 will be reserved for this UBI device)."); + "Example 6: mtd=/dev/mtd0,0,0,0,0,0,256 mtd=/dev/mtd1,0,0,0,0,0,4096 - attach MTD device /dev/mtd0 with wear-leveling threshold 256, and MTD device /dev/mtd1 with threshold 4096.\n"); #ifdef CONFIG_MTD_UBI_FASTMAP module_param(fm_autoconvert, bool, 0644); MODULE_PARM_DESC(fm_autoconvert, "Set this parameter to enable fastmap automatically on images without a fastmap."); diff --git a/drivers/mtd/ubi/cdev.c b/drivers/mtd/ubi/cdev.c index fd39030dbf89..54acd8394d67 100644 --- a/drivers/mtd/ubi/cdev.c +++ b/drivers/mtd/ubi/cdev.c @@ -1112,7 +1112,7 @@ static long ctrl_cdev_ioctl(struct file *file, unsigned int cmd, mutex_lock(&ubi_devices_mutex); err = ubi_attach_mtd_dev(mtd, req.ubi_num, req.vid_hdr_offset, req.max_beb_per1024, !!req.disable_fm, - !!req.need_resv_pool); + !!req.need_resv_pool, req.wl_threshold); mutex_unlock(&ubi_devices_mutex); if (err < 0) put_mtd_device(mtd); diff --git a/drivers/mtd/ubi/fastmap-wl.c b/drivers/mtd/ubi/fastmap-wl.c index a0f750411f9d..e24bb6b18e90 100644 --- a/drivers/mtd/ubi/fastmap-wl.c +++ b/drivers/mtd/ubi/fastmap-wl.c @@ -238,7 +238,7 @@ void ubi_refill_pools_and_lock(struct ubi_device *ubi) if (left_free <= 0) break; - e = find_wl_entry(ubi, &ubi->free, WL_FREE_MAX_DIFF, + e = find_wl_entry(ubi, &ubi->free, ubi->wl_free_max_diff, !can_fill_pools(ubi, left_free)); self_check_in_wl_tree(ubi, e, &ubi->free); rb_erase(&e->u.rb, &ubi->free); @@ -392,18 +392,18 @@ static bool need_wear_leveling(struct ubi_device *ubi) if (!e) { if (!ubi->free.rb_node) return false; - e = find_wl_entry(ubi, &ubi->free, WL_FREE_MAX_DIFF, 0); + e = find_wl_entry(ubi, &ubi->free, ubi->wl_free_max_diff, 0); ec = e->ec; } else { ec = e->ec; if (ubi->free.rb_node) { - e = find_wl_entry(ubi, &ubi->free, WL_FREE_MAX_DIFF, 0); + e = find_wl_entry(ubi, &ubi->free, ubi->wl_free_max_diff, 0); ec = max(ec, e->ec); } } e = rb_entry(rb_first(&ubi->used), struct ubi_wl_entry, u.rb); - return ec - e->ec >= UBI_WL_THRESHOLD; + return ec - e->ec >= ubi->wl_threshold; } /* get_peb_for_wl - returns a PEB to be used internally by the WL sub-system. diff --git a/drivers/mtd/ubi/fastmap.c b/drivers/mtd/ubi/fastmap.c index 3bce1b4d8464..3f9fb7c181ff 100644 --- a/drivers/mtd/ubi/fastmap.c +++ b/drivers/mtd/ubi/fastmap.c @@ -1526,7 +1526,8 @@ int ubi_update_fastmap(struct ubi_device *ubi) if (!tmp_e) { if (old_fm && old_fm->e[i]) { - ret = ubi_sync_erase(ubi, old_fm->e[i], 0); + ret = ubi_sync_erase(ubi, old_fm->e[i], + &old_fm->to_be_tortured[i]); if (ret < 0) { ubi_err(ubi, "could not erase old fastmap PEB"); @@ -1578,7 +1579,8 @@ int ubi_update_fastmap(struct ubi_device *ubi) if (old_fm) { /* no fresh anchor PEB was found, reuse the old one */ if (!tmp_e) { - ret = ubi_sync_erase(ubi, old_fm->e[0], 0); + ret = ubi_sync_erase(ubi, old_fm->e[0], + &old_fm->to_be_tortured[0]); if (ret < 0) { ubi_err(ubi, "could not erase old anchor PEB"); diff --git a/drivers/mtd/ubi/io.c b/drivers/mtd/ubi/io.c index 915eb64cb001..d70d1313ad4e 100644 --- a/drivers/mtd/ubi/io.c +++ b/drivers/mtd/ubi/io.c @@ -519,8 +519,10 @@ error: /** * ubi_io_sync_erase - synchronously erase a physical eraseblock. * @ubi: UBI device description object - * @pnum: physical eraseblock number to erase - * @torture: if this physical eraseblock has to be tortured + * @pnum: the physical eraseblock number to erase + * @torture: if this physical eraseblock has to be tortured; cleared to zero + * once the torture test has completed successfully so that a retry + * of the erase does not torture the physical eraseblock again * * This function synchronously erases physical eraseblock @pnum. If @torture * flag is not zero, the physical eraseblock is checked by means of writing @@ -532,7 +534,7 @@ error: * codes in case of other errors. Note, %-EIO means that the physical * eraseblock is bad. */ -int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int torture) +int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int *torture) { int err, ret = 0; @@ -560,10 +562,11 @@ int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int torture) return err; } - if (torture) { + if (*torture) { ret = torture_peb(ubi, pnum); if (ret < 0) return ret; + *torture = 0; } err = do_sync_erase(ubi, pnum); diff --git a/drivers/mtd/ubi/ubi.h b/drivers/mtd/ubi/ubi.h index af466cd83ae0..8a9ac60ff2f4 100644 --- a/drivers/mtd/ubi/ubi.h +++ b/drivers/mtd/ubi/ubi.h @@ -516,6 +516,22 @@ struct ubi_debug_info { * @bgt_thread: background thread description object * @thread_enabled: if the background thread is enabled * @bgt_name: background thread name + * @wl_threshold: Maximum difference between two erase counters. If this + * threshold is exceeded, the WL sub-system starts moving + * data from used physical eraseblocks with low erase + * counter to free physical eraseblocks with high erase counter. + * @wl_free_max_diff: When a physical eraseblock is moved, the WL sub-system + * has to pick the target physical eraseblock to move to. + * The simplest way would be just to pick the one with the + * highest erase counter. But in certain workloads this + * could lead to an unlimited wear of one or few physical + * eraseblock. Indeed, imagine a situation when the picked + * physical eraseblock is constantly erased after the + * data is written to it. So, we have a constant which + * limits the highest erase counter of the free physical + * eraseblock to pick. Namely, the WL sub-system does not + * pick eraseblocks with erase counter greater than the + * lowest erase counter plus @wl_free_max_diff. * * @flash_size: underlying MTD device size (in bytes) * @peb_count: count of physical eraseblocks on the MTD device @@ -623,6 +639,8 @@ struct ubi_device { struct task_struct *bgt_thread; int thread_enabled; char bgt_name[sizeof(UBI_BGT_NAME_PATTERN)+2]; + int wl_threshold; + int wl_free_max_diff; /* I/O sub-system's stuff */ long long flash_size; @@ -901,7 +919,7 @@ int self_check_eba(struct ubi_device *ubi, struct ubi_attach_info *ai_fastmap, struct ubi_attach_info *ai_scan); /* wl.c */ -int ubi_sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e, int torture); +int ubi_sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e, int *torture); int ubi_wl_get_peb(struct ubi_device *ubi); int ubi_wl_put_peb(struct ubi_device *ubi, int vol_id, int lnum, int pnum, int torture); @@ -923,7 +941,7 @@ int ubi_io_read(const struct ubi_device *ubi, void *buf, int pnum, int offset, int len); int ubi_io_write(struct ubi_device *ubi, const void *buf, int pnum, int offset, int len); -int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int torture); +int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int *torture); int ubi_io_is_bad(const struct ubi_device *ubi, int pnum); int ubi_io_mark_bad(const struct ubi_device *ubi, int pnum); int ubi_io_read_ec_hdr(struct ubi_device *ubi, int pnum, @@ -938,7 +956,8 @@ int ubi_io_write_vid_hdr(struct ubi_device *ubi, int pnum, /* build.c */ int ubi_attach_mtd_dev(struct mtd_info *mtd, int ubi_num, int vid_hdr_offset, int max_beb_per1024, - bool disable_fm, bool need_resv_pool); + bool disable_fm, bool need_resv_pool, + int wl_threshold); int ubi_detach_mtd_dev(int ubi_num, int anyway); struct ubi_device *ubi_get_device(int ubi_num); void ubi_put_device(struct ubi_device *ubi); diff --git a/drivers/mtd/ubi/wl.c b/drivers/mtd/ubi/wl.c index e3705db8e570..5a0db928a90e 100644 --- a/drivers/mtd/ubi/wl.c +++ b/drivers/mtd/ubi/wl.c @@ -96,27 +96,6 @@ #define WL_RESERVED_PEBS 1 /* - * Maximum difference between two erase counters. If this threshold is - * exceeded, the WL sub-system starts moving data from used physical - * eraseblocks with low erase counter to free physical eraseblocks with high - * erase counter. - */ -#define UBI_WL_THRESHOLD CONFIG_MTD_UBI_WL_THRESHOLD - -/* - * When a physical eraseblock is moved, the WL sub-system has to pick the target - * physical eraseblock to move to. The simplest way would be just to pick the - * one with the highest erase counter. But in certain workloads this could lead - * to an unlimited wear of one or few physical eraseblock. Indeed, imagine a - * situation when the picked physical eraseblock is constantly erased after the - * data is written to it. So, we have a constant which limits the highest erase - * counter of the free physical eraseblock to pick. Namely, the WL sub-system - * does not pick eraseblocks with erase counter greater than the lowest erase - * counter plus %WL_FREE_MAX_DIFF. - */ -#define WL_FREE_MAX_DIFF (2*UBI_WL_THRESHOLD) - -/* * Maximum number of consecutive background thread failures which is enough to * switch to read-only mode. */ @@ -358,7 +337,7 @@ static struct ubi_wl_entry *find_wl_entry(struct ubi_device *ubi, * * This function looks for a wear leveling entry with medium erase counter, * but not greater or equivalent than the lowest erase counter plus - * %WL_FREE_MAX_DIFF/2. + * @ubi->wl_free_max_diff/2. */ static struct ubi_wl_entry *find_mean_wl_entry(struct ubi_device *ubi, struct rb_root *root) @@ -368,7 +347,7 @@ static struct ubi_wl_entry *find_mean_wl_entry(struct ubi_device *ubi, first = rb_entry(rb_first(root), struct ubi_wl_entry, u.rb); last = rb_entry(rb_last(root), struct ubi_wl_entry, u.rb); - if (last->ec - first->ec < WL_FREE_MAX_DIFF) { + if (last->ec - first->ec < ubi->wl_free_max_diff) { e = rb_entry(root->rb_node, struct ubi_wl_entry, u.rb); /* @@ -379,7 +358,7 @@ static struct ubi_wl_entry *find_mean_wl_entry(struct ubi_device *ubi, */ e = may_reserve_for_fm(ubi, e, root); } else - e = find_wl_entry(ubi, root, WL_FREE_MAX_DIFF/2, 0); + e = find_wl_entry(ubi, root, ubi->wl_free_max_diff/2, 0); return e; } @@ -443,12 +422,14 @@ static int prot_queue_del(struct ubi_device *ubi, int pnum) * ubi_sync_erase - synchronously erase a physical eraseblock. * @ubi: UBI device description object * @e: the physical eraseblock to erase - * @torture: if the physical eraseblock has to be tortured + * @torture: if the physical eraseblock has to be tortured; cleared to zero + * once the torture test has completed successfully so that a retry + * of the erase does not torture the physical eraseblock again * * This function returns zero in case of success and a negative error code in * case of failure. */ -int ubi_sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e, int torture) +int ubi_sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e, int *torture) { int err; struct ubi_ec_hdr *ec_hdr; @@ -706,7 +687,7 @@ static int wear_leveling_worker(struct ubi_device *ubi, struct ubi_work *wrk, #ifdef CONFIG_MTD_UBI_FASTMAP e1 = find_anchor_wl_entry(&ubi->used); if (e1 && ubi->fm_anchor && - (ubi->fm_anchor->ec - e1->ec >= UBI_WL_THRESHOLD)) { + (ubi->fm_anchor->ec - e1->ec >= ubi->wl_threshold)) { ubi->fm_do_produce_anchor = 1; /* * fm_anchor is no longer considered a good anchor. @@ -743,7 +724,7 @@ static int wear_leveling_worker(struct ubi_device *ubi, struct ubi_work *wrk, if (!e2) goto out_cancel; - if (!(e2->ec - e1->ec >= UBI_WL_THRESHOLD)) { + if (!(e2->ec - e1->ec >= ubi->wl_threshold)) { dbg_wl("no WL needed: min used EC %d, max free EC %d", e1->ec, e2->ec); @@ -1056,12 +1037,12 @@ static int ensure_wear_leveling(struct ubi_device *ubi, int nested) * We schedule wear-leveling only if the difference between the * lowest erase counter of used physical eraseblocks and a high * erase counter of free physical eraseblocks is greater than - * %UBI_WL_THRESHOLD. + * @ubi->wl_threshold. */ e1 = rb_entry(rb_first(&ubi->used), struct ubi_wl_entry, u.rb); - e2 = find_wl_entry(ubi, &ubi->free, WL_FREE_MAX_DIFF, 0); + e2 = find_wl_entry(ubi, &ubi->free, ubi->wl_free_max_diff, 0); - if (!(e2->ec - e1->ec >= UBI_WL_THRESHOLD)) + if (!(e2->ec - e1->ec >= ubi->wl_threshold)) goto out_unlock; #endif dbg_wl("schedule wear-leveling"); @@ -1113,7 +1094,7 @@ static int __erase_worker(struct ubi_device *ubi, struct ubi_work *wl_wrk) dbg_wl("erase PEB %d EC %d LEB %d:%d", pnum, e->ec, wl_wrk->vol_id, wl_wrk->lnum); - err = ubi_sync_erase(ubi, e, wl_wrk->torture); + err = ubi_sync_erase(ubi, e, &wl_wrk->torture); if (!err) { spin_lock(&ubi->wl_lock); @@ -1150,7 +1131,8 @@ static int __erase_worker(struct ubi_device *ubi, struct ubi_work *wl_wrk) int err1; /* Re-schedule the LEB for erasure */ - err1 = schedule_erase(ubi, e, vol_id, lnum, 0, true); + err1 = schedule_erase(ubi, e, vol_id, lnum, wl_wrk->torture, + true); if (err1) { spin_lock(&ubi->wl_lock); wl_entry_destroy(ubi, e); @@ -1757,7 +1739,7 @@ static void shutdown_work(struct ubi_device *ubi) static int erase_aeb(struct ubi_device *ubi, struct ubi_ainf_peb *aeb, bool sync) { struct ubi_wl_entry *e; - int err; + int err, torture = 0; e = kmem_cache_alloc(ubi_wl_entry_slab, GFP_KERNEL); if (!e) @@ -1768,7 +1750,7 @@ static int erase_aeb(struct ubi_device *ubi, struct ubi_ainf_peb *aeb, bool sync ubi->lookuptbl[e->pnum] = e; if (sync) { - err = ubi_sync_erase(ubi, e, false); + err = ubi_sync_erase(ubi, e, &torture); if (err) goto out_free; @@ -2090,7 +2072,7 @@ static struct ubi_wl_entry *get_peb_for_wl(struct ubi_device *ubi) { struct ubi_wl_entry *e; - e = find_wl_entry(ubi, &ubi->free, WL_FREE_MAX_DIFF, 0); + e = find_wl_entry(ubi, &ubi->free, ubi->wl_free_max_diff, 0); self_check_in_wl_tree(ubi, e, &ubi->free); ubi->free_count--; ubi_assert(ubi->free_count >= 0); diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig index 4b10d78ed99b..0b8d8905e38e 100644 --- a/fs/btrfs/Kconfig +++ b/fs/btrfs/Kconfig @@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 +# misc-next marker config BTRFS_FS tristate "Btrfs filesystem support" diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index bafc62cf5ebc..c641099d66e2 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -456,12 +456,12 @@ static int record_root_in_trans(struct btrfs_trans_handle *trans, * * When this is zero, they can trust root->last_trans and fly * through btrfs_record_root_in_trans without having to take the - * lock. smp_wmb() makes sure that all the writes above are - * done before we pop in the zero below + * lock. smp_wmb() makes sure readers that see the last_trans + * update also see IN_TRANS_SETUP set, and clear_bit_unlock() + * publishes the relocation setup before we clear the bit. */ ret = btrfs_init_reloc_root(trans, root); - smp_mb__before_atomic(); - clear_bit(BTRFS_ROOT_IN_TRANS_SETUP, &root->state); + clear_bit_unlock(BTRFS_ROOT_IN_TRANS_SETUP, &root->state); } return ret; } @@ -499,10 +499,12 @@ int btrfs_record_root_in_trans(struct btrfs_trans_handle *trans, * see record_root_in_trans for comments about IN_TRANS_SETUP usage * and barriers */ - smp_rmb(); - if (btrfs_get_root_last_trans(root) == trans->transid && - !test_bit(BTRFS_ROOT_IN_TRANS_SETUP, &root->state)) - return 0; + if (btrfs_get_root_last_trans(root) == trans->transid) { + /* Order the last_trans load before testing IN_TRANS_SETUP. */ + smp_rmb(); + if (!test_bit_acquire(BTRFS_ROOT_IN_TRANS_SETUP, &root->state)) + return 0; + } mutex_lock(&fs_info->reloc_mutex); ret = record_root_in_trans(trans, root, false); diff --git a/fs/ecryptfs/crypto.c b/fs/ecryptfs/crypto.c index 74b02b55e3f6..e67119b6029c 100644 --- a/fs/ecryptfs/crypto.c +++ b/fs/ecryptfs/crypto.c @@ -1197,7 +1197,7 @@ static int ecryptfs_read_headers_virt(char *page_virt, } else set_default_header_data(crypt_stat); rc = ecryptfs_parse_packet_set(crypt_stat, (page_virt + offset), - ecryptfs_dentry); + PAGE_SIZE - offset, ecryptfs_dentry); out: return rc; } diff --git a/fs/ecryptfs/ecryptfs_kernel.h b/fs/ecryptfs/ecryptfs_kernel.h index f4f56a92bd56..58165928ed1e 100644 --- a/fs/ecryptfs/ecryptfs_kernel.h +++ b/fs/ecryptfs/ecryptfs_kernel.h @@ -1,5 +1,5 @@ /* SPDX-License-Identifier: GPL-2.0-or-later */ -/** +/* * eCryptfs: Linux filesystem encryption layer * Kernel declarations. * @@ -204,7 +204,7 @@ struct ecryptfs_filename { char dentry_name[ECRYPTFS_ENCRYPTED_DENTRY_NAME_LEN + 1]; }; -/** +/* * This is the primary struct associated with each encrypted file. * * TODO: cache align/pack? @@ -255,7 +255,8 @@ struct ecryptfs_inode_info { }; /** - * ecryptfs_global_auth_tok - A key used to encrypt all new files under the mountpoint + * struct ecryptfs_global_auth_tok - A key used to encrypt all new files + * under the mountpoint * @flags: Status flags * @mount_crypt_stat_list: These auth_toks hang off the mount-wide * cryptographic context. Every time a new @@ -263,7 +264,6 @@ struct ecryptfs_inode_info { * the auth_toks on that list to the set of * auth_toks on the inode's crypt_stat * @global_auth_tok_key: The key from the user's keyring for the sig - * @global_auth_tok: The key contents * @sig: The key identifier * * ecryptfs_global_auth_tok structs refer to authentication token keys @@ -283,7 +283,7 @@ struct ecryptfs_global_auth_tok { }; /** - * ecryptfs_key_tfm - Persistent key tfm + * struct ecryptfs_key_tfm - Persistent key tfm * @key_tfm: crypto API handle to the key * @key_size: Key size in bytes * @key_tfm_mutex: Mutex to ensure only one operation in eCryptfs is @@ -306,7 +306,7 @@ struct ecryptfs_key_tfm { extern struct mutex key_tfm_list_mutex; -/** +/* * This struct is to enable a mount-wide passphrase/salt combo. This * is more or less a stopgap to provide similar functionality to other * crypto filesystems like EncFS or CFS until full policy support is @@ -580,7 +580,8 @@ int ecryptfs_generate_key_packet_set(char *dest_base, size_t *len, size_t max); int ecryptfs_parse_packet_set(struct ecryptfs_crypt_stat *crypt_stat, - unsigned char *src, struct dentry *ecryptfs_dentry); + unsigned char *src, size_t src_size, + struct dentry *ecryptfs_dentry); int ecryptfs_truncate(struct dentry *dentry, loff_t new_length); ssize_t ecryptfs_getxattr_lower(struct dentry *lower_dentry, struct inode *lower_inode, diff --git a/fs/ecryptfs/keystore.c b/fs/ecryptfs/keystore.c index ebebc9551f1f..51651314b7a6 100644 --- a/fs/ecryptfs/keystore.c +++ b/fs/ecryptfs/keystore.c @@ -894,6 +894,12 @@ ecryptfs_parse_tag_70_packet(char **filename, size_t *filename_size, "rc = [%d]\n", __func__, rc); goto out; } + if (s->parsed_tag_70_packet_size < (ECRYPTFS_SIG_SIZE + 2)) { + ecryptfs_printk(KERN_WARNING, "Invalid packet size [%zd]\n", + s->parsed_tag_70_packet_size); + rc = -EINVAL; + goto out; + } s->block_aligned_filename_size = (s->parsed_tag_70_packet_size - ECRYPTFS_SIG_SIZE - 1); if ((1 + s->packet_size_len + s->parsed_tag_70_packet_size) @@ -1384,10 +1390,20 @@ parse_tag_3_packet(struct ecryptfs_crypt_stat *crypt_stat, } (*new_auth_tok)->session_key.encrypted_key_size = (body_size - (ECRYPTFS_SALT_SIZE + 5)); + /* + * Although encrypted_key_size is copied into the + * encrypted_key[ECRYPTFS_MAX_ENCRYPTED_KEY_BYTES] buffer here, + * it later bounds operations on a smaller buffer: + * decrypt_passphrase_encrypted_session_key() sets decrypted_key_size = + * encrypted_key_size and decrypts into + * decrypted_key[ECRYPTFS_MAX_KEY_BYTES], then memcpy's into + * crypt_stat->key[ECRYPTFS_MAX_KEY_BYTES]. Limit to + * ECRYPTFS_MAX_KEY_BYTES to protect those smaller buffers. + */ if ((*new_auth_tok)->session_key.encrypted_key_size - > ECRYPTFS_MAX_ENCRYPTED_KEY_BYTES) { + > ECRYPTFS_MAX_KEY_BYTES) { printk(KERN_WARNING "Tag 3 packet contains key larger " - "than ECRYPTFS_MAX_ENCRYPTED_KEY_BYTES\n"); + "than ECRYPTFS_MAX_KEY_BYTES\n"); rc = -EINVAL; goto out_free; } @@ -1537,7 +1553,7 @@ parse_tag_11_packet(unsigned char *data, unsigned char *contents, } (*packet_size) += length_size; (*tag_11_contents_size) = (body_size - 14); - if (unlikely((*packet_size) + body_size + 1 > max_packet_size)) { + if (unlikely((*packet_size) + body_size > max_packet_size)) { printk(KERN_ERR "Packet size exceeds max\n"); rc = -EINVAL; goto out; @@ -1704,6 +1720,7 @@ out: * ecryptfs_parse_packet_set * @crypt_stat: The cryptographic context * @src: Virtual address of region of memory containing the packets + * @src_size: Size of the packet set buffer * @ecryptfs_dentry: The eCryptfs dentry associated with the packet set * * Get crypt_stat to have the file's session key if the requisite key @@ -1714,7 +1731,7 @@ out: * conditions. */ int ecryptfs_parse_packet_set(struct ecryptfs_crypt_stat *crypt_stat, - unsigned char *src, + unsigned char *src, size_t src_size, struct dentry *ecryptfs_dentry) { size_t i = 0; @@ -1736,7 +1753,11 @@ int ecryptfs_parse_packet_set(struct ecryptfs_crypt_stat *crypt_stat, * added the our &auth_tok_list */ next_packet_is_auth_tok_packet = 1; while (next_packet_is_auth_tok_packet) { - size_t max_packet_size = ((PAGE_SIZE - 8) - i); + size_t max_packet_size; + + if (i >= src_size) + break; + max_packet_size = src_size - i; switch (src[i]) { case ECRYPTFS_TAG_3_PACKET_TYPE: @@ -1751,12 +1772,16 @@ int ecryptfs_parse_packet_set(struct ecryptfs_crypt_stat *crypt_stat, goto out_wipe_list; } i += packet_size; + if (i > src_size) { + rc = -EIO; + goto out_wipe_list; + } rc = parse_tag_11_packet((unsigned char *)&src[i], sig_tmp_space, ECRYPTFS_SIG_SIZE, &tag_11_contents_size, &tag_11_packet_size, - max_packet_size); + src_size - i); if (rc) { ecryptfs_printk(KERN_ERR, "No valid " "(ecryptfs-specific) literal " @@ -1768,6 +1793,10 @@ int ecryptfs_parse_packet_set(struct ecryptfs_crypt_stat *crypt_stat, goto out_wipe_list; } i += tag_11_packet_size; + if (i > src_size) { + rc = -EIO; + goto out_wipe_list; + } if (ECRYPTFS_SIG_SIZE != tag_11_contents_size) { ecryptfs_printk(KERN_ERR, "Expected " "signature of size [%d]; " @@ -1793,6 +1822,10 @@ int ecryptfs_parse_packet_set(struct ecryptfs_crypt_stat *crypt_stat, goto out_wipe_list; } i += packet_size; + if (i > src_size) { + rc = -EIO; + goto out_wipe_list; + } crypt_stat->flags |= ECRYPTFS_ENCRYPTED; break; case ECRYPTFS_TAG_11_PACKET_TYPE: diff --git a/fs/ecryptfs/messaging.c b/fs/ecryptfs/messaging.c index 03c60f0850ca..73c7b8215e66 100644 --- a/fs/ecryptfs/messaging.c +++ b/fs/ecryptfs/messaging.c @@ -166,6 +166,7 @@ int ecryptfs_exorcise_daemon(struct ecryptfs_daemon *daemon) mutex_unlock(&daemon->mux); goto out; } + mutex_lock(&ecryptfs_msg_ctx_lists_mux); list_for_each_entry_safe(msg_ctx, msg_ctx_tmp, &daemon->msg_ctx_out_queue, daemon_out_list) { list_del(&msg_ctx->daemon_out_list); @@ -174,6 +175,7 @@ int ecryptfs_exorcise_daemon(struct ecryptfs_daemon *daemon) "the out queue of a dying daemon\n", __func__); ecryptfs_msg_ctx_alloc_to_free(msg_ctx); } + mutex_unlock(&ecryptfs_msg_ctx_lists_mux); hlist_del(&daemon->euid_chain); mutex_unlock(&daemon->mux); kfree_sensitive(daemon); @@ -284,9 +286,16 @@ ecryptfs_send_message_locked(char *data, int data_len, u8 msg_type, mutex_unlock(&ecryptfs_msg_ctx_lists_mux); rc = ecryptfs_send_miscdev(data, data_len, *msg_ctx, msg_type, 0, daemon); - if (rc) + if (rc) { printk(KERN_ERR "%s: Error attempting to send message to " "userspace daemon; rc = [%d]\n", __func__, rc); + mutex_lock(&ecryptfs_msg_ctx_lists_mux); + mutex_lock(&(*msg_ctx)->mux); + ecryptfs_msg_ctx_alloc_to_free(*msg_ctx); + mutex_unlock(&(*msg_ctx)->mux); + mutex_unlock(&ecryptfs_msg_ctx_lists_mux); + *msg_ctx = NULL; + } out: return rc; } diff --git a/fs/ecryptfs/miscdev.c b/fs/ecryptfs/miscdev.c index 5a7d08149922..68804399a5df 100644 --- a/fs/ecryptfs/miscdev.c +++ b/fs/ecryptfs/miscdev.c @@ -360,7 +360,7 @@ ecryptfs_miscdev_write(struct file *file, const char __user *buf, u32 seq; size_t packet_size, packet_size_length; char *data; - unsigned char packet_size_peek[ECRYPTFS_MAX_PKT_LEN_SIZE]; + unsigned char packet_size_peek[ECRYPTFS_MAX_PKT_LEN_SIZE] = { }; ssize_t rc; if (count == 0) { @@ -376,7 +376,8 @@ ecryptfs_miscdev_write(struct file *file, const char __user *buf, } if (copy_from_user(packet_size_peek, &buf[PKT_LEN_OFFSET], - sizeof(packet_size_peek))) { + min_t(size_t, count - PKT_LEN_OFFSET, + sizeof(packet_size_peek)))) { printk(KERN_WARNING "%s: Error while inspecting packet size\n", __func__); return -EFAULT; diff --git a/fs/ecryptfs/mmap.c b/fs/ecryptfs/mmap.c index 2c2b12fedeae..07a6c8448100 100644 --- a/fs/ecryptfs/mmap.c +++ b/fs/ecryptfs/mmap.c @@ -355,24 +355,17 @@ out: */ static int ecryptfs_write_inode_size_to_header(struct inode *ecryptfs_inode) { - char *file_size_virt; + __be64 file_size; int rc; - file_size_virt = kmalloc(sizeof(u64), GFP_KERNEL); - if (!file_size_virt) { - rc = -ENOMEM; - goto out; - } - put_unaligned_be64(i_size_read(ecryptfs_inode), file_size_virt); - rc = ecryptfs_write_lower(ecryptfs_inode, file_size_virt, 0, - sizeof(u64)); - kfree(file_size_virt); + file_size = cpu_to_be64(i_size_read(ecryptfs_inode)); + rc = ecryptfs_write_lower(ecryptfs_inode, (char *)&file_size, 0, + sizeof(file_size)); if (rc < 0) printk(KERN_ERR "%s: Error writing file size to header; " "rc = [%d]\n", __func__, rc); else rc = 0; -out: return rc; } @@ -510,21 +503,8 @@ static sector_t ecryptfs_bmap(struct address_space *mapping, sector_t block) return block; } -#include <linux/buffer_head.h> - const struct address_space_operations ecryptfs_aops = { - /* - * XXX: This is pretty broken for multiple reasons: ecryptfs does not - * actually use buffer_heads, and ecryptfs will crash without - * CONFIG_BLOCK. But it matches the behavior before the default for - * address_space_operations without the ->dirty_folio method was - * cleaned up, so this is the best we can do without maintainer - * feedback. - */ -#ifdef CONFIG_BLOCK - .dirty_folio = block_dirty_folio, - .invalidate_folio = block_invalidate_folio, -#endif + .dirty_folio = filemap_dirty_folio, .writepages = ecryptfs_writepages, .read_folio = ecryptfs_read_folio, .write_begin = ecryptfs_write_begin, diff --git a/fs/ecryptfs/super.c b/fs/ecryptfs/super.c index 3bc21d677564..686b2b4a9cb5 100644 --- a/fs/ecryptfs/super.c +++ b/fs/ecryptfs/super.c @@ -150,6 +150,13 @@ static int ecryptfs_show_options(struct seq_file *m, struct dentry *root) if (mount_crypt_stat->global_default_cipher_key_size) seq_printf(m, ",ecryptfs_key_bytes=%zd", mount_crypt_stat->global_default_cipher_key_size); + if (mount_crypt_stat->flags & ECRYPTFS_GLOBAL_ENCRYPT_FILENAMES) { + seq_printf(m, ",ecryptfs_fn_cipher=%s", + mount_crypt_stat->global_default_fn_cipher_name); + if (mount_crypt_stat->global_default_fn_cipher_key_bytes) + seq_printf(m, ",ecryptfs_fn_key_bytes=%zd", + mount_crypt_stat->global_default_fn_cipher_key_bytes); + } if (mount_crypt_stat->flags & ECRYPTFS_PLAINTEXT_PASSTHROUGH_ENABLED) seq_printf(m, ",ecryptfs_passthrough"); if (mount_crypt_stat->flags & ECRYPTFS_XATTR_METADATA_ENABLED) diff --git a/fs/erofs/Kconfig b/fs/erofs/Kconfig index 2dfc313588d2..37495e95a78d 100644 --- a/fs/erofs/Kconfig +++ b/fs/erofs/Kconfig @@ -132,11 +132,13 @@ config EROFS_FS_ZIP_LZMA Say N if you want to disable LZMA compression support. config EROFS_FS_ZIP_LZMA_DEFAULT_MAX_STREAMS - int "EROFS LZMA default maximum decompression streams" + int "EROFS LZMA default maximum decompression streams" if SMP depends on EROFS_FS_ZIP_LZMA range 1 NR_CPUS if SMP range 1 1 if !SMP - default 16 + default 16 if SMP && NR_CPUS >= 16 + default NR_CPUS if SMP + default 1 help By default EROFS allocates one LZMA decompression stream per CPU. Each stream can hold a dictionary of up to 8 MiB taken from the @@ -144,7 +146,7 @@ config EROFS_FS_ZIP_LZMA_DEFAULT_MAX_STREAMS of memory. This caps the default; the lzma_streams module parameter still overrides it. - If unsure, keep the default of 16. + If unsure, keep the suggested default (16 or lower). config EROFS_FS_ZIP_DEFLATE bool "EROFS DEFLATE compressed data support" diff --git a/fs/erofs/data.c b/fs/erofs/data.c index 20962f8b60a8..0885b1f2fc92 100644 --- a/fs/erofs/data.c +++ b/fs/erofs/data.c @@ -497,7 +497,7 @@ static int erofs_file_mmap_prepare(struct vm_area_desc *desc) #define erofs_file_mmap_prepare generic_file_readonly_mmap_prepare #endif -static loff_t erofs_file_llseek(struct file *file, loff_t offset, int whence) +loff_t erofs_file_llseek(struct file *file, loff_t offset, int whence) { struct inode *inode = file->f_mapping->host; const struct iomap_ops *ops = &erofs_iomap_ops; diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index 9b7370f0f3df..65974e57aebf 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -415,6 +415,7 @@ void *erofs_read_metabuf(struct erofs_buf *buf, struct super_block *sb, int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *dev); int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, u64 start, u64 len); +loff_t erofs_file_llseek(struct file *file, loff_t offset, int whence); int erofs_map_blocks(struct inode *inode, struct erofs_map_blocks *map); void erofs_onlinefolio_init(struct folio *folio); void erofs_onlinefolio_split(struct folio *folio); diff --git a/fs/erofs/ishare.c b/fs/erofs/ishare.c index fa7d4112dec5..372e9db8fd8c 100644 --- a/fs/erofs/ishare.c +++ b/fs/erofs/ishare.c @@ -148,6 +148,13 @@ static int erofs_ishare_mmap(struct file *file, struct vm_area_struct *vma) return generic_file_readonly_mmap(file, vma); } +static ssize_t erofs_ishare_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + return filemap_splice_read(in->private_data, ppos, pipe, len, flags); +} + static int erofs_ishare_fadvise(struct file *file, loff_t offset, loff_t len, int advice) { @@ -156,12 +163,12 @@ static int erofs_ishare_fadvise(struct file *file, loff_t offset, const struct file_operations erofs_ishare_fops = { .open = erofs_ishare_file_open, - .llseek = generic_file_llseek, + .llseek = erofs_file_llseek, .read_iter = erofs_ishare_file_read_iter, .mmap = erofs_ishare_mmap, .release = erofs_ishare_file_release, .get_unmapped_area = thp_get_unmapped_area, - .splice_read = filemap_splice_read, + .splice_read = erofs_ishare_splice_read, .fadvise = erofs_ishare_fadvise, }; diff --git a/fs/exfat/balloc.c b/fs/exfat/balloc.c index e66ebf899778..c0ddd522c1e1 100644 --- a/fs/exfat/balloc.c +++ b/fs/exfat/balloc.c @@ -340,14 +340,27 @@ int exfat_trim_fs(struct inode *inode, struct fstrim_range *range) mutex_lock(&sbi->bitmap_lock); trim_begin = trim_end = exfat_find_free_bitmap(sb, clu_start); - if (trim_begin == EXFAT_EOF_CLUSTER) + /* + * exfat_find_free_bitmap() may wrap around to the beginning of + * the bitmap. Reject a cluster outside the requested range. + */ + if (trim_begin == EXFAT_EOF_CLUSTER || + trim_begin < clu_start || trim_begin > clu_end) goto unlock; - next_free_clu = exfat_find_free_bitmap(sb, trim_end + 1); - if (next_free_clu == EXFAT_EOF_CLUSTER) - goto unlock; + for (;;) { + if (trim_end >= clu_end) + break; + + next_free_clu = exfat_find_free_bitmap(sb, trim_end + 1); + /* + * Stop if the search wrapped around or moved beyond the requested + * FITRIM range. + */ + if (next_free_clu == EXFAT_EOF_CLUSTER || + next_free_clu <= trim_end || next_free_clu > clu_end) + break; - do { if (next_free_clu == trim_end + 1) { /* extend trim range for continuous free cluster */ trim_end++; @@ -368,17 +381,11 @@ int exfat_trim_fs(struct inode *inode, struct fstrim_range *range) trim_begin = trim_end = next_free_clu; } - if (next_free_clu >= clu_end) - break; - if (fatal_signal_pending(current)) { err = -ERESTARTSYS; goto unlock; } - - next_free_clu = exfat_find_free_bitmap(sb, next_free_clu + 1); - } while (next_free_clu != EXFAT_EOF_CLUSTER && - next_free_clu > trim_end); + } /* try to trim remainder */ count = trim_end - trim_begin + 1; diff --git a/fs/exfat/dir.c b/fs/exfat/dir.c index 4fd4ec52b8c0..fe73b1380c5d 100644 --- a/fs/exfat/dir.c +++ b/fs/exfat/dir.c @@ -87,7 +87,7 @@ static int exfat_readdir(struct inode *inode, loff_t *cpos, struct exfat_dir_ent exfat_bytes_to_cluster(sbi, i_size_read(inode)), ei->flags); dentries_per_clu = sbi->dentries_per_clu; - max_dentries = min(MAX_EXFAT_DENTRIES, + max_dentries = (unsigned int)min_t(u64, MAX_EXFAT_DENTRIES, exfat_cluster_to_dentries(sbi, sbi->num_clusters)); clu_offset = exfat_dentries_to_cluster(sbi, dentry); @@ -299,7 +299,13 @@ int exfat_alloc_new_dir(struct inode *inode, struct exfat_chain *clu) if (ret) return ret; - return exfat_zeroed_cluster(inode, clu->dir); + ret = exfat_zeroed_cluster(inode, clu->dir); + if (ret) { + exfat_free_cluster(inode, clu); + return ret; + } + + return 0; } int exfat_calc_num_entries(struct exfat_uni_name *p_uniname) diff --git a/fs/exfat/exfat_fs.h b/fs/exfat/exfat_fs.h index 9be50949ce34..a9131fe03302 100644 --- a/fs/exfat/exfat_fs.h +++ b/fs/exfat/exfat_fs.h @@ -294,12 +294,10 @@ struct exfat_inode_info { /* on-disk position of directory entry or 0 */ loff_t i_pos; loff_t valid_size; - /* page-aligned size that has been zeroed out for mmap */ + /* block-aligned size zeroed in the page cache (>= valid_size) */ loff_t zeroed_size; /* hash by i_location */ struct hlist_node i_hash_fat; - /* protect bmap against truncate */ - struct rw_semaphore truncate_lock; struct inode vfs_inode; /* File creation time */ struct timespec64 i_crtime; @@ -487,10 +485,10 @@ static inline u32 exfat_dentries_to_bytes(u32 dentry) /* * helpers for cluster size to dentry size conversion. */ -static inline u32 exfat_cluster_to_dentries(struct exfat_sb_info *sbi, +static inline u64 exfat_cluster_to_dentries(struct exfat_sb_info *sbi, u32 nr_clusters) { - return nr_clusters << (sbi->cluster_size_bits - DENTRY_SIZE_BITS); + return (u64)nr_clusters << (sbi->cluster_size_bits - DENTRY_SIZE_BITS); } static inline u32 exfat_dentries_to_cluster(struct exfat_sb_info *sbi, diff --git a/fs/exfat/file.c b/fs/exfat/file.c index 5fc13378d35f..a2a9ee1a2004 100644 --- a/fs/exfat/file.c +++ b/fs/exfat/file.c @@ -16,6 +16,7 @@ #include <linux/falloc.h> #include <linux/fileattr.h> #include <linux/iomap.h> +#include <linux/pagemap.h> #include "exfat_raw.h" #include "exfat_fs.h" @@ -412,7 +413,6 @@ int exfat_setattr(struct mnt_idmap *idmap, struct dentry *dentry, * about to be freed. */ inode_dio_wait(inode); - down_write(&EXFAT_I(inode)->truncate_lock); truncate_setsize(inode, attr->ia_size); /* @@ -420,7 +420,6 @@ int exfat_setattr(struct mnt_idmap *idmap, struct dentry *dentry, * is already written by it, so mark_inode_dirty() is unneeded. */ exfat_truncate(inode); - up_write(&EXFAT_I(inode)->truncate_lock); } else mark_inode_dirty(inode); @@ -565,7 +564,7 @@ static int exfat_ioctl_get_volume_label(struct super_block *sb, unsigned long ar if (ret < 0) return ret; - ret = exfat_utf16_to_nls(sb, &uniname, label, uniname.name_len); + ret = exfat_utf16_to_nls(sb, &uniname, label, sizeof(label)); if (ret < 0) return ret; @@ -654,6 +653,104 @@ int exfat_file_fsync(struct file *filp, loff_t start, loff_t end, int datasync) return blkdev_issue_flush(inode->i_sb->s_bdev); } +/* + * exfat_zero_new_range - zero [start, end) without overwriting uptodate blocks + * + * Uptodate blocks may contain data written through a shared mapping beyond + * valid_size. + */ +static int exfat_zero_new_range(struct inode *inode, loff_t start, loff_t end) +{ + struct address_space *mapping = inode->i_mapping; + unsigned int blocksize = i_blocksize(inode); + loff_t pos = start; + int err; + + while (pos < end) { + loff_t next = min_t(loff_t, + round_down(pos, PAGE_SIZE) + PAGE_SIZE, end); + struct folio *folio; + loff_t bpos; + + folio = filemap_get_folio(mapping, pos >> PAGE_SHIFT); + if (IS_ERR(folio)) { + err = iomap_zero_range(inode, pos, next - pos, NULL, + &exfat_iomap_ops, NULL, NULL); + if (err < 0) + return err; + pos = next; + continue; + } + + if (folio_test_uptodate(folio)) { + folio_lock(folio); + if (folio->mapping == mapping) + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); + pos = next; + continue; + } + + /* + * Zero not-uptodate block runs. iomap_zero_range() requires an + * unlocked folio, so recheck ->mapping after each call. + */ + folio_lock(folio); + bpos = pos; + while (bpos < next) { + loff_t rstart, rend; + + if (folio->mapping != mapping) { + folio_unlock(folio); + err = iomap_zero_range(inode, bpos, next - bpos, + NULL, &exfat_iomap_ops, NULL, NULL); + if (err < 0) { + folio_put(folio); + return err; + } + folio_lock(folio); + break; + } + + if (iomap_is_partially_uptodate(folio, + offset_in_folio(folio, bpos), blocksize)) { + bpos += blocksize; + continue; + } + + rstart = bpos; + rend = min_t(loff_t, bpos + blocksize, next); + while (rend < next && + !iomap_is_partially_uptodate(folio, + offset_in_folio(folio, rend), blocksize)) + rend = min_t(loff_t, rend + blocksize, next); + + folio_unlock(folio); + err = iomap_zero_range(inode, rstart, rend - rstart, + NULL, &exfat_iomap_ops, NULL, NULL); + if (err < 0) { + folio_put(folio); + return err; + } + folio_lock(folio); + bpos = rend; + } + + /* + * Dirty only a fully uptodate folio. Dirtying a partial folio could + * write uninitialised cache contents over valid on-disk blocks. + */ + if (folio->mapping == mapping && folio_test_uptodate(folio)) + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); + pos = next; + } + + return 0; +} + static int exfat_extend_valid_size(struct inode *inode, loff_t new_valid_size) { struct exfat_inode_info *ei = EXFAT_I(inode); @@ -661,18 +758,41 @@ static int exfat_extend_valid_size(struct inode *inode, loff_t new_valid_size) int ret = 0; if (old_valid_size < new_valid_size) { + /* Do not re-zero blocks already covered by zeroed_size. */ + loff_t gap_start = max(old_valid_size, ei->zeroed_size); + if (i_size_read(inode) < new_valid_size) { - i_size_write(inode, new_valid_size); - mark_inode_dirty(inode); + /* + * Allocate clusters before increasing i_size. The gap + * may already be zeroed, so the subsequent zeroing + * can be skipped. + */ + ret = exfat_cont_expand(inode, new_valid_size); + if (ret) + return ret; } - ret = iomap_zero_range(inode, old_valid_size, - new_valid_size - old_valid_size, NULL, - &exfat_write_iomap_ops, NULL, NULL); + /* + * Revoke writable PTEs while zeroing the gap. A racing mmap + * store re-faults through exfat_page_mkwrite() after valid_size + * is updated. + */ + filemap_invalidate_lock(inode->i_mapping); + if (gap_start < new_valid_size) + unmap_mapping_range(inode->i_mapping, gap_start, + new_valid_size - gap_start, 0); + ret = exfat_zero_new_range(inode, gap_start, new_valid_size); + filemap_invalidate_unlock(inode->i_mapping); if (ret) { truncate_setsize(inode, old_valid_size); exfat_truncate(inode); + return ret; } + + ei->valid_size = new_valid_size; + if (ei->zeroed_size < round_up(new_valid_size, i_blocksize(inode))) + ei->zeroed_size = round_up(new_valid_size, i_blocksize(inode)); + mark_inode_dirty(inode); } return ret; @@ -825,39 +945,39 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); struct exfat_inode_info *ei = EXFAT_I(inode); vm_fault_t ret; - loff_t new_valid_size, mmap_valid_size; + loff_t new_valid_size, mmap_valid_size, fault_page_start; if (!inode_trylock(inode)) return VM_FAULT_RETRY; mmap_valid_size = ((loff_t)vmf->pgoff + 1) << PAGE_SHIFT; + fault_page_start = ((loff_t)vmf->pgoff) << PAGE_SHIFT; new_valid_size = min(mmap_valid_size, i_size_read(inode)); if (ei->valid_size < new_valid_size) { - if (ei->zeroed_size < mmap_valid_size) { + if (ei->zeroed_size < fault_page_start) { int err; /* - * Only zero the range that hasn't been zeroed yet for - * this mmap write path. zeroed_size tracks the largest - * page-aligned offset that has already been zeroed. - * - * This prevents unnecessarily zeroing out the entire - * tail page on every page fault when userspace writes - * data byte-by-byte through mmap (after a small - * fallocate). It fixes data corruption in the tail page - * while preserving the existing valid_size semantics. + * Zero only the gap below the faulting page. The read + * fault populated its folio and iomap_page_mkwrite() + * will dirty it. */ - err = iomap_zero_range(inode, ei->zeroed_size, - mmap_valid_size - ei->zeroed_size, NULL, - &exfat_iomap_ops, NULL, NULL); + err = exfat_zero_new_range(inode, ei->zeroed_size, + fault_page_start); if (err < 0) { inode_unlock(inode); return vmf_fs_error(err); } - ei->zeroed_size = mmap_valid_size; } + /* + * Track zeroed_size by block, not page, because writeback stops + * at i_size recording blocks wholly beyond it could skip a + * later required zeroing. + */ + if (ei->zeroed_size < round_up(new_valid_size, i_blocksize(inode))) + ei->zeroed_size = round_up(new_valid_size, i_blocksize(inode)); ei->valid_size = new_valid_size; mark_inode_dirty(inode); } @@ -866,7 +986,7 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf) file_update_time(vmf->vma->vm_file); filemap_invalidate_lock_shared(inode->i_mapping); - ret = iomap_page_mkwrite(vmf, &exfat_write_iomap_ops, NULL); + ret = iomap_page_mkwrite(vmf, &exfat_iomap_ops, NULL); filemap_invalidate_unlock_shared(inode->i_mapping); sb_end_pagefault(inode->i_sb); inode_unlock(inode); @@ -876,7 +996,6 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf) static const struct vm_operations_struct exfat_file_vm_ops = { .fault = filemap_fault, - .map_pages = filemap_map_pages, .page_mkwrite = exfat_page_mkwrite, }; @@ -887,21 +1006,6 @@ static int exfat_file_mmap_prepare(struct vm_area_desc *desc) if (unlikely(exfat_forced_shutdown(file_inode(desc->file)->i_sb))) return -EIO; - if (vma_desc_test_all(desc, VMA_SHARED_BIT, VMA_MAYWRITE_BIT)) { - struct inode *inode = file_inode(file); - loff_t from, to; - int err; - - from = ((loff_t)desc->pgoff << PAGE_SHIFT); - to = min_t(loff_t, i_size_read(inode), - from + vma_desc_size(desc)); - if (EXFAT_I(inode)->valid_size < to) { - err = exfat_extend_valid_size(inode, to); - if (err) - return err; - } - } - file_accessed(file); desc->vm_ops = &exfat_file_vm_ops; return 0; diff --git a/fs/exfat/inode.c b/fs/exfat/inode.c index 89826aea5e1e..ccd13630187e 100644 --- a/fs/exfat/inode.c +++ b/fs/exfat/inode.c @@ -291,10 +291,9 @@ static sector_t exfat_aop_bmap(struct address_space *mapping, sector_t block) { sector_t blocknr; - /* exfat_get_cluster() assumes the requested blocknr isn't truncated. */ - down_read(&EXFAT_I(mapping->host)->truncate_lock); + inode_lock_shared(mapping->host); blocknr = iomap_bmap(mapping, block, &exfat_iomap_ops); - up_read(&EXFAT_I(mapping->host)->truncate_lock); + inode_unlock_shared(mapping->host); return blocknr; } diff --git a/fs/exfat/iomap.c b/fs/exfat/iomap.c index 4b9207bc4ab0..8911aa84a730 100644 --- a/fs/exfat/iomap.c +++ b/fs/exfat/iomap.c @@ -177,11 +177,18 @@ static int exfat_write_iomap_end(struct inode *inode, loff_t pos, loff_t length, if (ei->valid_size < end) { ei->valid_size = end; - if (ei->zeroed_size < end) - ei->zeroed_size = end; dirtied = true; } + /* + * IOMAP_F_ZERO_TAIL zeroes the remainder of the last block. Track that + * block as zeroed so later valid_size extensions do not zero it again. + */ + if (iomap->flags & IOMAP_F_ZERO_TAIL) + end = round_up(end, i_blocksize(inode)); + if (ei->zeroed_size < end) + ei->zeroed_size = end; + if (dirtied || iomap->flags & IOMAP_F_SIZE_CHANGED) mark_inode_dirty(inode); diff --git a/fs/exfat/namei.c b/fs/exfat/namei.c index cd9c9eca58f8..a4dc83b5949c 100644 --- a/fs/exfat/namei.c +++ b/fs/exfat/namei.c @@ -471,6 +471,7 @@ static int exfat_add_entry(struct inode *inode, const char *path, struct exfat_entry_set_cache es; int clu_size = 0; unsigned int start_clu = EXFAT_FREE_CLUSTER; + bool dir_allocated = false; ret = exfat_resolve_path(inode, path, &uniname); if (ret) @@ -497,6 +498,7 @@ static int exfat_add_entry(struct inode *inode, const char *path, } start_clu = clu.dir; clu_size = sbi->cluster_size; + dir_allocated = true; } /* update the directory entry */ @@ -507,8 +509,21 @@ static int exfat_add_entry(struct inode *inode, const char *path, exfat_init_ext_entry(&es, num_entries, &uniname, NULL, 0); ret = exfat_put_dentry_set(&es, IS_DIRSYNC(inode)); - if (ret) + if (ret) { + int cleanup_ret; + + cleanup_ret = exfat_get_dentry_set(&es, sb, &info->dir, + dentry, ES_ALL_ENTRIES); + if (!cleanup_ret) { + exfat_remove_entries(inode, &es, ES_IDX_FILE, false); + cleanup_ret = exfat_put_dentry_set(&es, + IS_DIRSYNC(inode)); + } + + if (!cleanup_ret && dir_allocated) + exfat_free_cluster(inode, &clu); goto out; + } info->entry = dentry; info->flags = ALLOC_NO_FAT_CHAIN; @@ -1116,11 +1131,6 @@ static int exfat_move_file(struct inode *parent_inode, exfat_init_ext_entry(&new_es, num_new_entries, p_uniname, &mov_es, num_extra_entries); - exfat_remove_entries(parent_inode, &mov_es, ES_IDX_FILE, false); - - ei->dir = newdir; - ei->entry = newentry; - ret = exfat_put_dentry_set(&new_es, IS_DIRSYNC(parent_inode)); if (ret) { /* Best-effort delete to avoid duplicate entries */ @@ -1134,6 +1144,11 @@ static int exfat_move_file(struct inode *parent_inode, goto put_mov_es; } + exfat_remove_entries(parent_inode, &mov_es, ES_IDX_FILE, false); + + ei->dir = newdir; + ei->entry = newentry; + return exfat_put_dentry_set(&mov_es, IS_DIRSYNC(parent_inode)); put_mov_es: diff --git a/fs/exfat/super.c b/fs/exfat/super.c index 388db271c6bf..a9ea36ba2693 100644 --- a/fs/exfat/super.c +++ b/fs/exfat/super.c @@ -195,7 +195,6 @@ static struct inode *exfat_alloc_inode(struct super_block *sb) if (!ei) return NULL; - init_rwsem(&ei->truncate_lock); return &ei->vfs_inode; } diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index 064f5b537423..eb2f955b1e2a 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -766,10 +766,25 @@ static void __remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type) spin_unlock(&im->ino_lock); } -void f2fs_add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type) +static void f2fs_wait_for_inode_record(struct f2fs_sb_info *sbi, int mode) +{ + if (mode != APPEND_INO && mode != UPDATE_INO) + return; + + /* Let's wait for some pending updates for APPEND_INO and UPDATE_INO. */ + flush_workqueue(sbi->evict_wq); +} + +static void __f2fs_add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, + unsigned int devidx, int type) { /* add new dirty ino entry into list */ - __add_ino_entry(sbi, ino, 0, type); + __add_ino_entry(sbi, ino, devidx, type); +} + +void f2fs_add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type) +{ + __f2fs_add_ino_entry(sbi, ino, 0, type); } void f2fs_remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type) @@ -798,6 +813,8 @@ void f2fs_release_ino_entry(struct f2fs_sb_info *sbi, bool all) for (i = all ? ORPHAN_INO : APPEND_INO; i < MAX_INO_ENTRY; i++) { struct inode_management *im = &sbi->im[i]; + f2fs_wait_for_inode_record(sbi, i); + spin_lock(&im->ino_lock); list_for_each_entry_safe(e, tmp, &im->ino_list, list) { list_del(&e->list); @@ -812,7 +829,7 @@ void f2fs_release_ino_entry(struct f2fs_sb_info *sbi, bool all) void f2fs_set_dirty_device(struct f2fs_sb_info *sbi, nid_t ino, unsigned int devidx, int type) { - __add_ino_entry(sbi, ino, devidx, type); + __f2fs_add_ino_entry(sbi, ino, devidx, type); } bool f2fs_is_dirty_device(struct f2fs_sb_info *sbi, nid_t ino, @@ -864,14 +881,14 @@ void f2fs_release_orphan_inode(struct f2fs_sb_info *sbi) void f2fs_add_orphan_inode(struct inode *inode) { /* add new orphan ino entry into list */ - __add_ino_entry(F2FS_I_SB(inode), inode->i_ino, 0, ORPHAN_INO); + f2fs_add_ino_entry(F2FS_I_SB(inode), inode->i_ino, ORPHAN_INO); f2fs_update_inode_page(inode); } void f2fs_remove_orphan_inode(struct f2fs_sb_info *sbi, nid_t ino) { /* remove orphan entry from orphan list */ - __remove_ino_entry(sbi, ino, ORPHAN_INO); + f2fs_remove_ino_entry(sbi, ino, ORPHAN_INO); } static int recover_orphan_inode(struct f2fs_sb_info *sbi, nid_t ino) diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index dad5c2bd5888..21f396ebe22c 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -384,11 +384,30 @@ static void f2fs_write_end_bio(struct bio *bio) bio_put(bio); } +static void f2fs_write_end_io_work(struct work_struct *work) +{ + struct bio *bio = &container_of(work, struct f2fs_bio, work)->bio; + + f2fs_write_end_bio(bio); +} + static void f2fs_write_end_io(struct bio *bio) { + struct f2fs_sb_info *sbi; + iostat_update_and_unbind_ctx(bio); - f2fs_write_end_bio(bio); + sbi = bio->bi_private; + + if (in_atomic() && bio->bi_iter.bi_size > sbi->max_atc_write_bio_size) { + struct work_struct *w; + + w = &container_of(bio, struct f2fs_bio, bio)->work; + INIT_WORK(w, f2fs_write_end_io_work); + queue_work(sbi->wq, w); + } else { + f2fs_write_end_bio(bio); + } } #ifdef CONFIG_BLK_DEV_ZONED @@ -1216,7 +1235,7 @@ int f2fs_reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count) if (unlikely(is_inode_flag_set(dn->inode, FI_NO_ALLOC))) return -EPERM; - err = inc_valid_block_count(sbi, dn->inode, &count, true); + err = inc_valid_block_count(sbi, dn->inode, &count, true, false); if (unlikely(err)) return err; @@ -1288,10 +1307,11 @@ retry: if (folio_test_large(folio)) { pgoff_t folio_index = mapping_align_index(mapping, index); + unsigned long nr_pages = folio_nr_pages(folio); f2fs_folio_put(folio, true); invalidate_inode_pages2_range(mapping, folio_index, - folio_index + folio_nr_pages(folio) - 1); + folio_index + nr_pages - 1); f2fs_schedule_timeout(DEFAULT_SCHEDULE_TIMEOUT); goto retry; } @@ -1488,7 +1508,7 @@ static int __allocate_data_block(struct dnode_of_data *dn, int seg_type) dn->data_blkaddr = f2fs_data_blkaddr(dn); if (dn->data_blkaddr == NULL_ADDR) { - err = inc_valid_block_count(sbi, dn->inode, &count, true); + err = inc_valid_block_count(sbi, dn->inode, &count, true, false); if (unlikely(err)) return err; } @@ -1497,8 +1517,11 @@ static int __allocate_data_block(struct dnode_of_data *dn, int seg_type) old_blkaddr = dn->data_blkaddr; err = f2fs_allocate_data_block(sbi, NULL, old_blkaddr, &dn->data_blkaddr, &sum, seg_type, NULL); - if (err) + if (err) { + if (old_blkaddr == NULL_ADDR) + dec_valid_block_count(sbi, dn->inode, count); return err; + } if (GET_SEGNO(sbi, old_blkaddr) != NULL_SEGNO) f2fs_invalidate_internal_cache(sbi, old_blkaddr, 1); @@ -3900,7 +3923,7 @@ repeat: * Will wait that below with our IO control. */ folio = f2fs_filemap_get_folio(mapping, index, - FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_NOFS, + FGP_LOCK | FGP_WRITE | FGP_CREAT, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) { err = PTR_ERR(folio); @@ -4460,13 +4483,24 @@ int f2fs_init_wq(struct f2fs_sb_info *sbi) { sbi->wq = alloc_workqueue("f2fs_wq", WQ_UNBOUND | WQ_HIGHPRI, num_online_cpus()); - return sbi->wq ? 0 : -ENOMEM; + if (!sbi->wq) + return -ENOMEM; + + sbi->evict_wq = alloc_workqueue("f2fs_evict_wq", + WQ_UNBOUND | WQ_HIGHPRI, num_online_cpus()); + if (!sbi->evict_wq) { + destroy_workqueue(sbi->wq); + return -ENOMEM; + } + return 0; } void f2fs_destroy_wq(struct f2fs_sb_info *sbi) { if (sbi->wq) destroy_workqueue(sbi->wq); + if (sbi->evict_wq) + destroy_workqueue(sbi->evict_wq); } int __init f2fs_init_bio_entry_cache(void) diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index af88db8fdb71..ff379aff4472 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -352,10 +352,6 @@ static void update_mem_info(struct f2fs_sb_info *sbi) get_cache: si->cache_mem = 0; - /* build gc */ - if (sbi->gc_thread) - si->cache_mem += sizeof(struct f2fs_gc_kthread); - /* build merge flush thread */ if (SM_I(sbi)->fcc_info) si->cache_mem += sizeof(struct flush_cmd_control); diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c index a9563f7fcd88..fd0e2cd31a81 100644 --- a/fs/f2fs/dir.c +++ b/fs/f2fs/dir.c @@ -320,6 +320,7 @@ start_find_bucket: de = find_in_block(dir, dentry_folio, fname, &max_slots, use_hash); if (IS_ERR(de)) { + f2fs_folio_put(dentry_folio, false); *res_folio = ERR_CAST(de); de = NULL; break; @@ -460,7 +461,7 @@ void f2fs_set_link(struct inode *dir, struct f2fs_dir_entry *de, folio_mark_dirty(folio); inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); - f2fs_mark_inode_dirty_sync(dir, false); + f2fs_mark_inode_dirty_sync(dir, true); f2fs_folio_put(folio, true); } @@ -615,7 +616,7 @@ void f2fs_update_parent_metadata(struct inode *dir, struct inode *inode, clear_inode_flag(inode, FI_NEW_INODE); } inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); - f2fs_mark_inode_dirty_sync(dir, false); + f2fs_mark_inode_dirty_sync(dir, true); if (F2FS_I(dir)->i_current_depth != current_depth) f2fs_i_depth_write(dir, current_depth); @@ -927,7 +928,7 @@ void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct folio *folio, f2fs_folio_put(folio, true); inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); - f2fs_mark_inode_dirty_sync(dir, false); + f2fs_mark_inode_dirty_sync(dir, true); if (inode) f2fs_drop_nlink(dir, inode); diff --git a/fs/f2fs/extent_cache.c b/fs/f2fs/extent_cache.c index 61f6b9714366..37cf9fa8d537 100644 --- a/fs/f2fs/extent_cache.c +++ b/fs/f2fs/extent_cache.c @@ -17,6 +17,7 @@ #include "f2fs.h" #include "node.h" +#include "segment.h" #include <trace/events/f2fs.h> bool sanity_check_extent_cache(struct inode *inode, struct folio *ifolio) @@ -62,6 +63,14 @@ bool sanity_check_extent_cache(struct inode *inode, struct folio *ifolio) __func__, inode->i_ino, ei.blk, ei.fofs, ei.len); return false; } + + if ((GET_SEGOFF_FROM_SEG0(sbi, ei.blk) % BLKS_PER_SEC(sbi)) || + (ei.len % BLKS_PER_SEC(sbi))) { + f2fs_warn(sbi, "%s: device alias inode (ino=%llx)'s extent info [%u, %u, %u] is not aligned to section size %u", + __func__, inode->i_ino, ei.blk, ei.fofs, ei.len, + BLKS_PER_SEC(sbi)); + return false; + } return true; } diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 8376bbe58ee3..d5778c7e50ca 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -1403,6 +1403,8 @@ struct f2fs_dev_info { unsigned int total_segments; block_t start_blk; block_t end_blk; + bool has_alias; + bool is_reserving; #ifdef CONFIG_BLK_DEV_ZONED unsigned int nr_blkz; /* Total number of zones */ unsigned long *blkz_seq; /* Bitmap indicating sequential zones */ @@ -1747,6 +1749,33 @@ struct decompress_io_ctx { #define MAX_COMPRESS_LOG_SIZE 8 #define MAX_COMPRESS_WINDOW_SIZE(log_size) ((PAGE_SIZE) << (log_size)) +struct f2fs_gc_kthread { + struct task_struct *f2fs_gc_task; + wait_queue_head_t gc_wait_queue_head; + + /* for gc sleep time */ + unsigned int urgent_sleep_time; + unsigned int min_sleep_time; + unsigned int max_sleep_time; + unsigned int no_gc_sleep_time; + + /* for changing gc mode */ + bool gc_wake; + + /* for GC_MERGE mount option */ + wait_queue_head_t fggc_wq; /* + * caller of f2fs_balance_fs() + * will wait on this wait queue. + */ + + /* for gc control for zoned devices */ + unsigned int no_zoned_gc_percent; + unsigned int boost_zoned_gc_percent; + unsigned int valid_thresh_ratio; + unsigned int boost_gc_multiple; + unsigned int boost_gc_greedy; +}; + struct f2fs_sb_info { struct super_block *sb; /* pointer to VFS super block */ struct proc_dir_entry *s_proc; /* proc entry */ @@ -1772,6 +1801,8 @@ struct f2fs_sb_info { struct f2fs_sm_info *sm_info; /* segment manager */ /* for bio operations */ + /* Largest write bio size completed in atomic context (atc). */ + u32 max_atc_write_bio_size; struct f2fs_bio_info *write_io[NR_PAGE_TYPE]; /* for write bios */ /* keep migration IO order for LFS mode */ struct f2fs_rwsem io_order_lock; @@ -1854,6 +1885,7 @@ struct f2fs_sb_info { block_t last_valid_block_count; /* for recovery */ block_t reserved_blocks; /* configurable reserved blocks */ block_t current_reserved_blocks; /* current reserved blocks */ + block_t alias_reserved_blocks; /* reserved blocks for device alias */ /* Additional tracking for no checkpoint mode */ block_t unusable_block_count; /* # of blocks saved by last cp */ @@ -1882,7 +1914,7 @@ struct f2fs_sb_info { * semaphore for GC, avoid * race between GC and GC or CP */ - struct f2fs_gc_kthread *gc_thread; /* GC thread */ + struct f2fs_gc_kthread gc_thread; /* GC thread */ struct atgc_management am; /* atgc management */ unsigned int cur_victim_sec; /* current victim section num */ unsigned int gc_mode; /* current GC state */ @@ -1980,6 +2012,8 @@ struct f2fs_sb_info { struct workqueue_struct *wq; /* bio completion workqueue */ + struct workqueue_struct *evict_wq; /* inode eviction workqueue */ + /* * If we are in irq context, let's update error information into * on-disk superblock in the work. @@ -2556,7 +2590,8 @@ static inline unsigned int get_available_block_count(struct f2fs_sb_info *sbi, block_t avail_user_block_count; avail_user_block_count = sbi->user_block_count - - sbi->current_reserved_blocks; + sbi->current_reserved_blocks - + sbi->alias_reserved_blocks; if (test_opt(sbi, RESERVE_ROOT) && !__allow_reserved_root(sbi, inode, cap)) avail_user_block_count -= F2FS_OPTION(sbi).root_reserved_blocks; @@ -2573,7 +2608,8 @@ static inline unsigned int get_available_block_count(struct f2fs_sb_info *sbi, static inline void f2fs_i_blocks_write(struct inode *, block_t, bool, bool); static inline int inc_valid_block_count(struct f2fs_sb_info *sbi, - struct inode *inode, blkcnt_t *count, bool partial) + struct inode *inode, blkcnt_t *count, + bool partial, bool alias_reserved) { long long diff = 0, release = 0; block_t avail_user_block_count; @@ -2596,10 +2632,16 @@ static inline int inc_valid_block_count(struct f2fs_sb_info *sbi, spin_lock(&sbi->stat_lock); + if (alias_reserved) + sbi->alias_reserved_blocks -= *count; + avail_user_block_count = get_available_block_count(sbi, inode, true); diff = (long long)sbi->total_valid_block_count + *count - avail_user_block_count; if (unlikely(diff > 0)) { + if (alias_reserved) + sbi->alias_reserved_blocks += *count; + if (!partial) { spin_unlock(&sbi->stat_lock); release = *count; @@ -3831,6 +3873,8 @@ int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc); void f2fs_remove_donate_inode(struct inode *inode); void f2fs_evict_inode(struct inode *inode); void f2fs_handle_failed_inode(struct inode *inode, struct f2fs_lock_context *lc); +int f2fs_init_evict_inode_work(void); +void f2fs_destroy_evict_inode_work(void); /* * namei.c @@ -4007,6 +4051,8 @@ int f2fs_flush_device_cache(struct f2fs_sb_info *sbi); void f2fs_destroy_flush_cmd_control(struct f2fs_sb_info *sbi, bool free); void f2fs_invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr, unsigned int len); +void f2fs_reserve_device_alias(struct f2fs_sb_info *sbi, block_t addr, + unsigned int len); bool f2fs_is_checkpointed_data(struct f2fs_sb_info *sbi, block_t blkaddr); int f2fs_start_discard_thread(struct f2fs_sb_info *sbi); void f2fs_drop_discard_cmd(struct f2fs_sb_info *sbi); @@ -4095,10 +4141,26 @@ static inline struct inode *fio_inode(struct f2fs_io_info *fio) #define MIN_FRAGMENT_SIZE 1 #define MAX_FRAGMENT_SIZE 512 -static inline bool f2fs_need_rand_seg(struct f2fs_sb_info *sbi) +static inline bool f2fs_need_rand_blk(struct f2fs_sb_info *sbi, + enum log_type type) +{ + if (type == CURSEG_COLD_DATA_PINNED) + return false; + return F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK; +} + +static inline bool f2fs_need_rand_seg(struct f2fs_sb_info *sbi, + enum log_type type) +{ + if (type == CURSEG_COLD_DATA_PINNED) + return false; + return F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_SEG; +} + +static inline bool f2fs_need_rand_seg_blk(struct f2fs_sb_info *sbi, + enum log_type type) { - return F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_SEG || - F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK; + return f2fs_need_rand_blk(sbi, type) || f2fs_need_rand_seg(sbi, type); } /* @@ -4226,7 +4288,9 @@ int f2fs_gc(struct f2fs_sb_info *sbi, struct f2fs_gc_control *gc_control); void f2fs_build_gc_manager(struct f2fs_sb_info *sbi); int f2fs_gc_range(struct f2fs_sb_info *sbi, unsigned int start_seg, unsigned int end_seg, - bool dry_run, unsigned int dry_run_sections); + bool dry_run, unsigned int dry_run_sections, bool lock); +void f2fs_reset_gc_victim_resource(struct f2fs_sb_info *sbi, + unsigned int start, unsigned int end); int f2fs_resize_fs(struct file *filp, __u64 block_count); int __init f2fs_create_garbage_collection_cache(void); void f2fs_destroy_garbage_collection_cache(void); diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index 089759366cdc..fa770fade8dc 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -36,23 +36,69 @@ #include <trace/events/f2fs.h> #include <uapi/linux/f2fs.h> -static void f2fs_zero_post_eof_page(struct inode *inode, - loff_t new_size, bool lock) +static int fill_zero(struct inode *inode, pgoff_t index, + loff_t start, loff_t len); + +static int do_zero_post_eof_page(struct inode *inode, loff_t new_size) +{ + loff_t old_size = i_size_read(inode); + unsigned int offset, len; + pgoff_t index; + int err; + + offset = old_size & (PAGE_SIZE - 1); + + if (!offset) + return 0; + + len = min_t(loff_t, PAGE_SIZE - offset, new_size - old_size); + index = old_size >> PAGE_SHIFT; + + if (f2fs_has_inline_data(inode)) { + /* data post eof should be always zero */ + if (new_size <= MAX_INLINE_DATA(inode)) + return 0; + err = f2fs_convert_inline_inode(inode); + if (err) + return err; + } + + err = fill_zero(inode, index, offset, len); + if (err) + return err; + return filemap_write_and_wait_range(inode->i_mapping, + old_size, old_size + len - 1); +} + +static int f2fs_zero_post_eof_page(struct inode *inode, + loff_t new_size, bool lock, bool writeback) { loff_t old_size = i_size_read(inode); + bool strict = + F2FS_OPTION(F2FS_I_SB(inode)).fsync_mode == FSYNC_MODE_STRICT; if (old_size >= new_size) - return; + return 0; - if (mapping_empty(inode->i_mapping)) - return; + if (!strict && mapping_empty(inode->i_mapping)) + return 0; if (lock) filemap_invalidate_lock(inode->i_mapping); /* zero or drop pages only in range of [old_size, new_size] */ - truncate_inode_pages_range(inode->i_mapping, old_size, new_size); + truncate_inode_pages_range(inode->i_mapping, old_size, new_size - 1); if (lock) filemap_invalidate_unlock(inode->i_mapping); + + if (!writeback || !strict) + return 0; + /* + * In fsync_mode=strict, when we expand an unaligned EOF size, we + * should zero post EOF data and writeback the data immediately, + * so that it can avoid exposing stale data after metadata flush + * and POR. + */ + return do_zero_post_eof_page(inode, new_size); } static vm_fault_t f2fs_filemap_fault(struct vm_fault *vmf) @@ -132,7 +178,10 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf) f2fs_bug_on(sbi, f2fs_has_inline_data(inode)); - f2fs_zero_post_eof_page(inode, (folio->index + 1) << PAGE_SHIFT, true); + err = f2fs_zero_post_eof_page(inode, + (folio->index + 1) << PAGE_SHIFT, true, false); + if (err) + goto out_pagefault; file_update_time(vmf->vma->vm_file); filemap_invalidate_lock_shared(inode->i_mapping); @@ -189,7 +238,7 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf) out_sem: filemap_invalidate_unlock_shared(inode->i_mapping); - +out_pagefault: sb_end_pagefault(inode->i_sb); out: ret = vmf_fs_error(err); @@ -813,13 +862,19 @@ int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock) if (IS_DEVICE_ALIASING(inode)) { struct extent_tree *et = F2FS_I(inode)->extent_tree[EX_READ]; - struct extent_info ei = et->largest; + struct extent_info ei; + + read_lock(&et->lock); + ei = et->largest; + read_unlock(&et->lock); f2fs_invalidate_blocks(sbi, ei.blk, ei.len); dec_valid_block_count(sbi, inode, ei.len); f2fs_update_time(sbi, REQ_TIME); + f2fs_drop_extent_tree(inode); + f2fs_folio_put(ifolio, true); goto out; } @@ -1096,24 +1151,31 @@ int f2fs_setattr(struct mnt_idmap *idmap, struct dentry *dentry, if ((attr->ia_valid & ATTR_SIZE)) { if (mapping_large_folio_support(inode->i_mapping)) return -EOPNOTSUPP; - if (!f2fs_is_compress_backend_ready(inode) || - IS_DEVICE_ALIASING(inode)) + if (IS_DEVICE_ALIASING(inode)) + return -EPERM; + if (!f2fs_is_compress_backend_ready(inode)) return -EOPNOTSUPP; if (is_inode_flag_set(inode, FI_COMPRESS_RELEASED) && !IS_ALIGNED(attr->ia_size, F2FS_BLK_TO_BYTES(fi->i_cluster_size))) return -EINVAL; - /* - * To prevent scattered pin block generation, we don't allow - * smaller/equal size unaligned truncation for pinned file. - * We only support overwrite IO to pinned file, so don't - * care about larger size truncation. - */ - if (f2fs_is_pinned_file(inode) && - attr->ia_size <= i_size_read(inode) && - !IS_ALIGNED(attr->ia_size, - F2FS_BLK_TO_BYTES(CAP_BLKS_PER_SEC(sbi)))) - return -EINVAL; + + if (f2fs_is_pinned_file(inode)) { + /* + * It may break section-aligned fallocate recovery + * mechanism, so do not allow larger size truncation. + */ + if (attr->ia_size > i_size_read(inode)) + return -EINVAL; + /* + * To prevent scattered pin block generation, we don't + * allow smaller/equal size unaligned truncation for + * pinned file. + */ + else if (!IS_ALIGNED(attr->ia_size, + F2FS_BLK_TO_BYTES(CAP_BLKS_PER_SEC(sbi)))) + return -EINVAL; + } } if (is_quota_modification(idmap, inode, attr)) { @@ -1165,8 +1227,12 @@ int f2fs_setattr(struct mnt_idmap *idmap, struct dentry *dentry, f2fs_down_write(&fi->i_gc_rwsem[WRITE]); filemap_invalidate_lock(inode->i_mapping); - if (attr->ia_size > old_size) - f2fs_zero_post_eof_page(inode, attr->ia_size, false); + if (attr->ia_size > old_size) { + err = f2fs_zero_post_eof_page(inode, + attr->ia_size, false, true); + if (err) + goto err_out; + } truncate_setsize(inode, attr->ia_size); if (attr->ia_size <= old_size) @@ -1175,6 +1241,7 @@ int f2fs_setattr(struct mnt_idmap *idmap, struct dentry *dentry, * do not trim all blocks after i_size if target size is * larger than i_size. */ +err_out: filemap_invalidate_unlock(inode->i_mapping); f2fs_up_write(&fi->i_gc_rwsem[WRITE]); if (err) @@ -1286,7 +1353,9 @@ static int f2fs_punch_hole(struct inode *inode, loff_t offset, loff_t len) if (ret) return ret; - f2fs_zero_post_eof_page(inode, offset + len, true); + ret = f2fs_zero_post_eof_page(inode, offset + len, true, false); + if (ret) + return ret; pg_start = ((unsigned long long) offset) >> PAGE_SHIFT; pg_end = ((unsigned long long) offset + len) >> PAGE_SHIFT; @@ -1573,7 +1642,9 @@ static int f2fs_do_collapse(struct inode *inode, loff_t offset, loff_t len) f2fs_down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]); filemap_invalidate_lock(inode->i_mapping); - f2fs_zero_post_eof_page(inode, offset + len, false); + ret = f2fs_zero_post_eof_page(inode, offset + len, false, false); + if (ret) + goto out_unlock; f2fs_lock_op(sbi, &lc); f2fs_drop_extent_tree(inode); @@ -1581,6 +1652,7 @@ static int f2fs_do_collapse(struct inode *inode, loff_t offset, loff_t len) ret = __exchange_data_block(inode, inode, end, start, nrpages - end, true); f2fs_unlock_op(sbi, &lc); +out_unlock: filemap_invalidate_unlock(inode->i_mapping); f2fs_up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]); return ret; @@ -1613,11 +1685,14 @@ static int f2fs_collapse_range(struct inode *inode, loff_t offset, loff_t len) /* write out all moved pages, if possible */ filemap_invalidate_lock(inode->i_mapping); - filemap_write_and_wait_range(inode->i_mapping, offset, LLONG_MAX); + ret = filemap_write_and_wait_range(inode->i_mapping, offset, LLONG_MAX); + if (ret) + goto out_unlock; truncate_pagecache(inode, offset); new_size = i_size_read(inode) - len; ret = f2fs_truncate_blocks(inode, new_size, true); +out_unlock: filemap_invalidate_unlock(inode->i_mapping); if (!ret) f2fs_i_size_write(inode, new_size); @@ -1699,7 +1774,9 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len, if (ret) return ret; - f2fs_zero_post_eof_page(inode, offset + len, true); + ret = f2fs_zero_post_eof_page(inode, offset + len, true, false); + if (ret) + return ret; pg_start = ((unsigned long long) offset) >> PAGE_SHIFT; pg_end = ((unsigned long long) offset + len) >> PAGE_SHIFT; @@ -1834,7 +1911,9 @@ static int f2fs_insert_range(struct inode *inode, loff_t offset, loff_t len) f2fs_down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]); filemap_invalidate_lock(mapping); - f2fs_zero_post_eof_page(inode, offset + len, false); + ret = f2fs_zero_post_eof_page(inode, offset + len, false, false); + if (ret) + goto out_unlock; truncate_pagecache(inode, offset); while (!ret && idx > pg_start) { @@ -1852,6 +1931,7 @@ static int f2fs_insert_range(struct inode *inode, loff_t offset, loff_t len) idx + delta, nr, false); f2fs_unlock_op(sbi, &lc); } +out_unlock: filemap_invalidate_unlock(mapping); f2fs_up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]); if (ret) @@ -1894,7 +1974,9 @@ static int f2fs_expand_inode_data(struct inode *inode, loff_t offset, if (err) return err; - f2fs_zero_post_eof_page(inode, offset + len, true); + err = f2fs_zero_post_eof_page(inode, offset + len, true, true); + if (err) + return err; f2fs_balance_fs(sbi, true); @@ -2126,6 +2208,9 @@ static int f2fs_setflags_common(struct inode *inode, u32 iflags, u32 mask) if (IS_NOQUOTA(inode)) return -EPERM; + if (IS_DEVICE_ALIASING(inode)) + return -EPERM; + if ((iflags ^ masked_flags) & F2FS_CASEFOLD_FL) { if (!f2fs_sb_has_casefold(F2FS_I_SB(inode))) return -EOPNOTSUPP; @@ -2674,6 +2759,17 @@ static int f2fs_ioc_get_encryption_policy(struct file *filp, unsigned long arg) return fscrypt_ioctl_get_policy(filp, (void __user *)arg); } +static int f2fs_ioc_get_dev_alias_status(struct file *filp, unsigned long arg) +{ + struct inode *inode = file_inode(filp); + + if (!IS_DEVICE_ALIASING(inode)) + return -EINVAL; + + return put_user(F2FS_HAS_BLOCKS(inode) ? F2FS_DEV_ALIAS_STATUS_RESERVED : + F2FS_DEV_ALIAS_STATUS_RELEASED, (u32 __user *)arg); +} + static int f2fs_ioc_get_encryption_pwsalt(struct file *filp, unsigned long arg) { struct inode *inode = file_inode(filp); @@ -3079,6 +3175,9 @@ static int f2fs_ioc_defragment(struct file *filp, unsigned long arg) if (f2fs_readonly(sbi->sb)) return -EROFS; + if (IS_DEVICE_ALIASING(inode)) + return -EOPNOTSUPP; + if (copy_from_user(&range, (struct f2fs_defragment __user *)arg, sizeof(range))) return -EFAULT; @@ -3117,8 +3216,9 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in, struct inode *dst = file_inode(file_out); struct f2fs_sb_info *sbi = F2FS_I_SB(src); struct f2fs_lock_context lc; - size_t olen = len, dst_max_i_size = 0; - size_t dst_osize; + size_t olen = len; + loff_t dst_max_i_size = 0; + loff_t dst_osize, dst_end; int ret; if (file_in->f_path.mnt != file_out->f_path.mnt || @@ -3131,7 +3231,8 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in, if (!S_ISREG(src->i_mode) || !S_ISREG(dst->i_mode)) return -EINVAL; - if (IS_ENCRYPTED(src) || IS_ENCRYPTED(dst)) + if (IS_ENCRYPTED(src) || IS_ENCRYPTED(dst) || + IS_DEVICE_ALIASING(src) || IS_DEVICE_ALIASING(dst)) return -EOPNOTSUPP; if (pos_out < 0 || pos_in < 0) @@ -3140,8 +3241,6 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in, if (src == dst) { if (pos_in == pos_out) return 0; - if (pos_out > pos_in && pos_out < pos_in + len) - return -EINVAL; } inode_lock(src); @@ -3167,6 +3266,8 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in, goto out_unlock; if (len == 0) olen = len = src->i_size - pos_in; + if (src == dst && pos_out > pos_in && pos_out < pos_in + len) + goto out_unlock; if (pos_in + len == src->i_size) len = ALIGN(src->i_size, F2FS_BLKSIZE) - pos_in; if (len == 0) { @@ -3175,8 +3276,15 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in, } dst_osize = dst->i_size; - if (pos_out + olen > dst->i_size) - dst_max_i_size = pos_out + olen; + if (olen > LLONG_MAX - pos_out) + goto out_unlock; + dst_end = pos_out + olen; + if (dst_end > dst->i_size) { + ret = inode_newsize_ok(dst, dst_end); + if (ret) + goto out_unlock; + dst_max_i_size = dst_end; + } /* verify the end result is block aligned */ if (!IS_ALIGNED(pos_in, F2FS_BLKSIZE) || @@ -3612,6 +3720,241 @@ static int f2fs_ioc_get_dev_alias_file(struct file *filp, unsigned long arg) (u32 __user *)arg); } +static bool f2fs_get_dev_alias_extent(struct f2fs_sb_info *sbi, + struct dentry *dentry, + struct extent_info *ei) +{ + int i; + + for (i = 1; i < sbi->s_ndevs; i++) { + char *name = strrchr(FDEV(i).path, '/'); + + name = name ? name + 1 : FDEV(i).path; + if (strcmp(name, dentry->d_name.name)) + continue; + + ei->blk = FDEV(i).start_blk; + ei->len = FDEV(i).total_segments << sbi->log_blocks_per_seg; + ei->fofs = 0; + return true; + } + return false; +} + +static int f2fs_ioc_reserve_dev_alias(struct file *filp) +{ + struct inode *inode = file_inode(filp); + struct f2fs_sb_info *sbi = F2FS_I_SB(inode); + struct extent_tree *et = F2FS_I(inode)->extent_tree[EX_READ]; + struct extent_info ei; + struct cp_control cpc = { CP_SYNC, 0, 0, 0 }; + struct f2fs_lock_context lc, glc; + blkcnt_t count; + unsigned int start, end; + int type, err; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + + if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) + return -EINVAL; + + err = mnt_want_write_file(filp); + if (err) + return err; + + inode_lock(inode); + + if (!IS_DEVICE_ALIASING(inode)) { + err = -EINVAL; + goto out_inode_unlock; + } + + if (F2FS_HAS_BLOCKS(inode)) { + err = 0; + goto out_inode_unlock; + } + + if (!f2fs_get_dev_alias_extent(sbi, filp->f_path.dentry, &ei)) { + f2fs_warn(sbi, "device alias file (%s, ino=%llu) has no matching device", + filp->f_path.dentry->d_name.name, + (unsigned long long)inode->i_ino); + set_sbi_flag(sbi, SBI_NEED_FSCK); + f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE); + err = -EFSCORRUPTED; + goto out_inode_unlock; + } + + spin_lock(&sbi->stat_lock); + if (sbi->total_valid_block_count + ei.len > + get_available_block_count(sbi, inode, true)) { + spin_unlock(&sbi->stat_lock); + err = -ENOSPC; + goto out_inode_unlock; + } + sbi->alias_reserved_blocks += ei.len; + spin_unlock(&sbi->stat_lock); + + spin_lock(&FREE_I(sbi)->segmap_lock); + FDEV(f2fs_target_device_index(sbi, ei.blk)).is_reserving = true; + spin_unlock(&FREE_I(sbi)->segmap_lock); + + start = GET_SEGNO(sbi, ei.blk); + end = GET_SEGNO(sbi, ei.blk + ei.len - 1); + + /* Acquire gc_lock for victim reset, curseg resize, and range GC */ + f2fs_down_write_trace(&sbi->gc_lock, &glc); + + /* Reset the victim information to prevent GC from targeting the range */ + f2fs_reset_gc_victim_resource(sbi, start, end); + + /* Move out cursegs from the target range */ + for (type = CURSEG_HOT_DATA; type < NR_CURSEG_PERSIST_TYPE; type++) { + err = f2fs_allocate_segment_for_resize(sbi, type, start, end); + if (err) + goto out_gc_unlock; + } + + f2fs_lock_op(sbi, &lc); + + if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { + err = -EINVAL; + f2fs_unlock_op(sbi, &lc); + goto out_gc_unlock; + } + + /* do GC to move out valid blocks in the range all at once! */ + err = f2fs_gc_range(sbi, start, end, false, 0, false); + if (err) { + f2fs_unlock_op(sbi, &lc); + goto out_gc_unlock; + } + + count = ei.len; + err = inc_valid_block_count(sbi, inode, &count, false, true); + if (err) { + f2fs_unlock_op(sbi, &lc); + goto out_gc_unlock; + } + + write_lock(&et->lock); + et->largest = ei; + write_unlock(&et->lock); + clear_inode_flag(inode, FI_NO_EXTENT); + + f2fs_reserve_device_alias(sbi, ei.blk, ei.len); + + i_size_write(inode, (loff_t)ei.len << sbi->log_blocksize); + f2fs_update_inode_page(inode); + + spin_lock(&FREE_I(sbi)->segmap_lock); + FDEV(f2fs_target_device_index(sbi, ei.blk)).is_reserving = false; + spin_unlock(&FREE_I(sbi)->segmap_lock); + + f2fs_unlock_op(sbi, &lc); + f2fs_up_write_trace(&sbi->gc_lock, &glc); + + inode_unlock(inode); + mnt_drop_write_file(filp); + + return f2fs_write_checkpoint(sbi, &cpc); + +out_gc_unlock: + spin_lock(&sbi->stat_lock); + sbi->alias_reserved_blocks -= ei.len; + spin_unlock(&sbi->stat_lock); + + spin_lock(&FREE_I(sbi)->segmap_lock); + FDEV(f2fs_target_device_index(sbi, ei.blk)).is_reserving = false; + spin_unlock(&FREE_I(sbi)->segmap_lock); + f2fs_up_write_trace(&sbi->gc_lock, &glc); + +out_inode_unlock: + inode_unlock(inode); + mnt_drop_write_file(filp); + return err; +} + +static int f2fs_ioc_release_dev_alias(struct file *filp) +{ + struct inode *inode = file_inode(filp); + struct f2fs_sb_info *sbi = F2FS_I_SB(inode); + struct extent_tree *et = F2FS_I(inode)->extent_tree[EX_READ]; + struct extent_info ei = {0, }; + struct cp_control cpc = { CP_SYNC, 0, 0, 0 }; + struct f2fs_lock_context lc, glc; + int err; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + + if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) + return -EINVAL; + + err = mnt_want_write_file(filp); + if (err) + return err; + + inode_lock(inode); + + if (!IS_DEVICE_ALIASING(inode)) { + err = -EINVAL; + goto out_inode_unlock; + } + + if (!F2FS_HAS_BLOCKS(inode)) { + err = 0; + goto out_inode_unlock; + } + + err = filemap_write_and_wait(inode->i_mapping); + if (err) + goto out_inode_unlock; + + read_lock(&et->lock); + ei = et->largest; + read_unlock(&et->lock); + + f2fs_down_write_trace(&sbi->gc_lock, &glc); + f2fs_lock_op(sbi, &lc); + + if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) { + err = -EINVAL; + f2fs_unlock_op(sbi, &lc); + f2fs_up_write_trace(&sbi->gc_lock, &glc); + goto out_inode_unlock; + } + + filemap_invalidate_lock(inode->i_mapping); + truncate_setsize(inode, 0); + + err = f2fs_truncate_blocks(inode, 0, false); + if (err) + i_size_write(inode, (loff_t)ei.len << sbi->log_blocksize); + filemap_invalidate_unlock(inode->i_mapping); + + if (err) { + f2fs_unlock_op(sbi, &lc); + f2fs_up_write_trace(&sbi->gc_lock, &glc); + goto out_inode_unlock; + } + + f2fs_update_inode_page(inode); + + f2fs_unlock_op(sbi, &lc); + f2fs_up_write_trace(&sbi->gc_lock, &glc); + + inode_unlock(inode); + mnt_drop_write_file(filp); + + return f2fs_write_checkpoint(sbi, &cpc); + +out_inode_unlock: + inode_unlock(inode); + mnt_drop_write_file(filp); + return err; +} + static int f2fs_ioc_io_prio(struct file *filp, unsigned long arg) { struct inode *inode = file_inode(filp); @@ -4037,7 +4380,7 @@ static int reserve_compress_blocks(struct dnode_of_data *dn, pgoff_t count, } ret = inc_valid_block_count(sbi, dn->inode, - &to_reserved, false); + &to_reserved, false, false); if (unlikely(ret)) return ret; @@ -4469,7 +4812,7 @@ static int redirty_blocks(struct inode *inode, pgoff_t page_idx, int len) page_idx = folio_next_index(folio); } while (page_len < len); - do { + while (redirty_idx < page_idx) { folio = filemap_lock_folio(mapping, redirty_idx); /* It will never fail, when folio has pinned above */ @@ -4482,7 +4825,7 @@ static int redirty_blocks(struct inode *inode, pgoff_t page_idx, int len) redirty_idx = folio_next_index(folio); folio_unlock(folio); folio_put_refs(folio, 2); - } while (redirty_idx < page_idx); + } return ret; } @@ -4738,8 +5081,14 @@ static long __f2fs_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) return f2fs_ioc_compress_file(filp); case F2FS_IOC_GET_DEV_ALIAS_FILE: return f2fs_ioc_get_dev_alias_file(filp, arg); + case F2FS_IOC_GET_DEV_ALIAS_STATUS: + return f2fs_ioc_get_dev_alias_status(filp, arg); case F2FS_IOC_IO_PRIO: return f2fs_ioc_io_prio(filp, arg); + case F2FS_IOC_RESERVE_DEV_ALIAS: + return f2fs_ioc_reserve_dev_alias(filp); + case F2FS_IOC_RELEASE_DEV_ALIAS: + return f2fs_ioc_release_dev_alias(filp); default: return -ENOTTY; } @@ -4998,8 +5347,10 @@ static ssize_t f2fs_write_checks(struct kiocb *iocb, struct iov_iter *from) if (err) return err; - f2fs_zero_post_eof_page(inode, - iocb->ki_pos + iov_iter_count(from), true); + err = f2fs_zero_post_eof_page(inode, + iocb->ki_pos + iov_iter_count(from), true, true); + if (err) + return err; return count; } @@ -5526,7 +5877,10 @@ long f2fs_compat_ioctl(struct file *file, unsigned int cmd, unsigned long arg) case F2FS_IOC_DECOMPRESS_FILE: case F2FS_IOC_COMPRESS_FILE: case F2FS_IOC_GET_DEV_ALIAS_FILE: + case F2FS_IOC_GET_DEV_ALIAS_STATUS: case F2FS_IOC_IO_PRIO: + case F2FS_IOC_RESERVE_DEV_ALIAS: + case F2FS_IOC_RELEASE_DEV_ALIAS: break; default: return -ENOIOCTLCMD; diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index ffaa7ba76a1b..c4da2f31805b 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -31,9 +31,9 @@ static unsigned int count_bits(const unsigned long *addr, static int gc_thread_func(void *data) { struct f2fs_sb_info *sbi = data; - struct f2fs_gc_kthread *gc_th = sbi->gc_thread; - wait_queue_head_t *wq = &sbi->gc_thread->gc_wait_queue_head; - wait_queue_head_t *fggc_wq = &sbi->gc_thread->fggc_wq; + struct f2fs_gc_kthread *gc_th = &sbi->gc_thread; + wait_queue_head_t *wq = &sbi->gc_thread.gc_wait_queue_head; + wait_queue_head_t *fggc_wq = &sbi->gc_thread.fggc_wq; unsigned int wait_ms; struct f2fs_gc_control gc_control = { .victim_segno = NULL_SEGNO, @@ -193,13 +193,9 @@ next: int f2fs_start_gc_thread(struct f2fs_sb_info *sbi) { - struct f2fs_gc_kthread *gc_th; + struct f2fs_gc_kthread *gc_th = &sbi->gc_thread; dev_t dev = sbi->sb->s_bdev->bd_dev; - gc_th = f2fs_kmalloc(sbi, sizeof(struct f2fs_gc_kthread), GFP_KERNEL); - if (!gc_th) - return -ENOMEM; - gc_th->urgent_sleep_time = DEF_GC_THREAD_URGENT_SLEEP_TIME; gc_th->valid_thresh_ratio = DEF_GC_THREAD_VALID_THRESH_RATIO; gc_th->boost_gc_multiple = BOOST_GC_MULTIPLE; @@ -221,16 +217,14 @@ int f2fs_start_gc_thread(struct f2fs_sb_info *sbi) gc_th->gc_wake = false; - sbi->gc_thread = gc_th; - init_waitqueue_head(&sbi->gc_thread->gc_wait_queue_head); - init_waitqueue_head(&sbi->gc_thread->fggc_wq); - sbi->gc_thread->f2fs_gc_task = kthread_run(gc_thread_func, sbi, + init_waitqueue_head(&gc_th->gc_wait_queue_head); + init_waitqueue_head(&gc_th->fggc_wq); + gc_th->f2fs_gc_task = kthread_run(gc_thread_func, sbi, "f2fs_gc-%u:%u", MAJOR(dev), MINOR(dev)); if (IS_ERR(gc_th->f2fs_gc_task)) { int err = PTR_ERR(gc_th->f2fs_gc_task); - kfree(gc_th); - sbi->gc_thread = NULL; + gc_th->f2fs_gc_task = NULL; return err; } @@ -241,14 +235,14 @@ int f2fs_start_gc_thread(struct f2fs_sb_info *sbi) void f2fs_stop_gc_thread(struct f2fs_sb_info *sbi) { - struct f2fs_gc_kthread *gc_th = sbi->gc_thread; + struct f2fs_gc_kthread *gc_th = &sbi->gc_thread; - if (!gc_th) + if (!gc_th->f2fs_gc_task) return; + kthread_stop(gc_th->f2fs_gc_task); + gc_th->f2fs_gc_task = NULL; wake_up_all(&gc_th->fggc_wq); - kfree(gc_th); - sbi->gc_thread = NULL; } static int select_gc_type(struct f2fs_sb_info *sbi, int gc_type) @@ -316,7 +310,7 @@ static void select_policy(struct f2fs_sb_info *sbi, int gc_type, p->max_search = sbi->max_victim_search; /* let's select beginning hot/small space first. */ - if (f2fs_need_rand_seg(sbi)) { + if (f2fs_need_rand_seg_blk(sbi, type)) { p->offset = get_random_u32_below(MAIN_SECS(sbi) * SEGS_PER_SEC(sbi)); SIT_I(sbi)->last_victim[p->gc_mode] = p->offset; @@ -796,7 +790,7 @@ int f2fs_get_victim(struct f2fs_sb_info *sbi, unsigned int *result, if (one_time) { p.one_time_gc = one_time; if (has_enough_free_secs(sbi, 0, NR_PERSISTENT_LOG)) - valid_thresh_ratio = sbi->gc_thread->valid_thresh_ratio; + valid_thresh_ratio = sbi->gc_thread.valid_thresh_ratio; } retry: @@ -1807,9 +1801,9 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi, if (f2fs_sb_has_blkzoned(sbi) && !has_enough_free_blocks(sbi, - sbi->gc_thread->boost_zoned_gc_percent)) + sbi->gc_thread.boost_zoned_gc_percent)) window_granularity *= - sbi->gc_thread->boost_gc_multiple; + sbi->gc_thread.boost_gc_multiple; end_segno = start_segno + window_granularity; } @@ -2156,8 +2150,9 @@ void f2fs_build_gc_manager(struct f2fs_sb_info *sbi) int f2fs_gc_range(struct f2fs_sb_info *sbi, unsigned int start_seg, unsigned int end_seg, - bool dry_run, unsigned int dry_run_sections) + bool dry_run, unsigned int dry_run_sections, bool lock) { + struct f2fs_lock_context lc; unsigned int segno; unsigned int gc_secs = dry_run_sections; @@ -2170,56 +2165,79 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi, .ilist = LIST_HEAD_INIT(gc_list.ilist), .iroot = RADIX_TREE_INIT(gc_list.iroot, GFP_NOFS), }; + int err = 0; + + if (lock) + f2fs_down_write_trace(&sbi->gc_lock, &lc); /* * avoid migrating empty section, as it can be allocated by * log in parallel. */ if (!get_valid_blocks(sbi, segno, true)) - continue; + goto next; if (is_cursec(sbi, GET_SEC_FROM_SEG(sbi, segno))) - continue; + goto next; do_garbage_collect(sbi, segno, &gc_list, FG_GC, true, false); put_gc_inode(&gc_list); - if (!dry_run && get_valid_blocks(sbi, segno, true)) - return -EAGAIN; + if (!dry_run && get_valid_blocks(sbi, segno, true)) { + err = -EAGAIN; + goto next; + } if (dry_run && dry_run_sections && - !get_valid_blocks(sbi, segno, true) && --gc_secs == 0) - break; + !get_valid_blocks(sbi, segno, true)) { + --gc_secs; + goto next; + } if (fatal_signal_pending(current)) - return -ERESTARTSYS; + err = -ERESTARTSYS; +next: + if (lock) + f2fs_up_write_trace(&sbi->gc_lock, &lc); + if (err) + return err; + if (dry_run && dry_run_sections && !gc_secs) + return 0; } return 0; } +void f2fs_reset_gc_victim_resource(struct f2fs_sb_info *sbi, + unsigned int start, unsigned int end) +{ + int i; + + mutex_lock(&DIRTY_I(sbi)->seglist_lock); + for (i = 0; i < MAX_GC_POLICY; i++) + if (SIT_I(sbi)->last_victim[i] >= start && + SIT_I(sbi)->last_victim[i] <= end) + SIT_I(sbi)->last_victim[i] = 0; + + for (i = BG_GC; i <= FG_GC; i++) + if (sbi->next_victim_seg[i] >= start && + sbi->next_victim_seg[i] <= end) + sbi->next_victim_seg[i] = NULL_SEGNO; + mutex_unlock(&DIRTY_I(sbi)->seglist_lock); +} + static int free_segment_range(struct f2fs_sb_info *sbi, unsigned int secs, bool dry_run) { unsigned int next_inuse, start, end; struct cp_control cpc = { CP_RESIZE, 0, 0, 0 }; - int gc_mode, gc_type; int err = 0; int type; - /* Force block allocation for GC */ MAIN_SECS(sbi) -= secs; start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi); end = MAIN_SEGS(sbi) - 1; - mutex_lock(&DIRTY_I(sbi)->seglist_lock); - for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++) - if (SIT_I(sbi)->last_victim[gc_mode] >= start) - SIT_I(sbi)->last_victim[gc_mode] = 0; - - for (gc_type = BG_GC; gc_type <= FG_GC; gc_type++) - if (sbi->next_victim_seg[gc_type] >= start) - sbi->next_victim_seg[gc_type] = NULL_SEGNO; - mutex_unlock(&DIRTY_I(sbi)->seglist_lock); + f2fs_reset_gc_victim_resource(sbi, start, end); /* Move out cursegs from the target range */ for (type = CURSEG_HOT_DATA; type < NR_CURSEG_PERSIST_TYPE; type++) { @@ -2229,7 +2247,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi, } /* do GC to move out valid blocks in the range */ - err = f2fs_gc_range(sbi, start, end, dry_run, 0); + err = f2fs_gc_range(sbi, start, end, dry_run, 0, false); if (err || dry_run) goto out; diff --git a/fs/f2fs/gc.h b/fs/f2fs/gc.h index 6c4d4567571e..b015742fb455 100644 --- a/fs/f2fs/gc.h +++ b/fs/f2fs/gc.h @@ -45,32 +45,7 @@ #define NR_GC_CHECKPOINT_SECS (3) /* data/node/dentry sections */ -struct f2fs_gc_kthread { - struct task_struct *f2fs_gc_task; - wait_queue_head_t gc_wait_queue_head; - - /* for gc sleep time */ - unsigned int urgent_sleep_time; - unsigned int min_sleep_time; - unsigned int max_sleep_time; - unsigned int no_gc_sleep_time; - - /* for changing gc mode */ - bool gc_wake; - - /* for GC_MERGE mount option */ - wait_queue_head_t fggc_wq; /* - * caller of f2fs_balance_fs() - * will wait on this wait queue. - */ - /* for gc control for zoned devices */ - unsigned int no_zoned_gc_percent; - unsigned int boost_zoned_gc_percent; - unsigned int valid_thresh_ratio; - unsigned int boost_gc_multiple; - unsigned int boost_gc_greedy; -}; struct gc_inode_list { struct list_head ilist; @@ -197,6 +172,6 @@ static inline bool need_to_boost_gc(struct f2fs_sb_info *sbi) { if (f2fs_sb_has_blkzoned(sbi)) return !has_enough_free_blocks(sbi, - sbi->gc_thread->boost_zoned_gc_percent); + sbi->gc_thread.boost_zoned_gc_percent); return has_enough_invalid_blocks(sbi); } diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c index e2f7bedf1552..aec06fb4fd76 100644 --- a/fs/f2fs/inline.c +++ b/fs/f2fs/inline.c @@ -732,7 +732,7 @@ void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, f2fs_folio_put(folio, true); inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); - f2fs_mark_inode_dirty_sync(dir, false); + f2fs_mark_inode_dirty_sync(dir, true); if (inode) f2fs_drop_nlink(dir, inode); diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c index c95e0b126da4..bac1e360d966 100644 --- a/fs/f2fs/inode.c +++ b/fs/f2fs/inode.c @@ -9,6 +9,7 @@ #include <linux/f2fs_fs.h> #include <linux/writeback.h> #include <linux/sched/mm.h> +#include <linux/swap.h> #include <linux/lz4.h> #include <linux/zstd.h> #include <linux/fserror.h> @@ -24,6 +25,18 @@ extern const struct address_space_operations f2fs_compress_aops; #endif +#define NUM_PREALLOC_EVICT_INODE_WORK 8 + +static struct kmem_cache *evict_inode_work_cache; +static mempool_t *evict_inode_work_pool; + +struct evict_inode_work { + struct work_struct work; + struct f2fs_sb_info *sbi; + nid_t ino; + unsigned int add_ino_entry_bits; +}; + void f2fs_mark_inode_dirty_sync(struct inode *inode, bool sync) { if (is_inode_flag_set(inode, FI_NEW_INODE)) @@ -637,6 +650,9 @@ make_now: inode->i_fop = &f2fs_dir_operations; inode->i_mapping->a_ops = &f2fs_dblock_aops; mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); + + /* Let's prepare APPEND/UPDATE_INO before future access. */ + flush_workqueue(sbi->evict_wq); } else if (S_ISLNK(inode->i_mode)) { if (file_is_encrypt(inode)) inode->i_op = &f2fs_encrypted_symlink_inode_operations; @@ -854,16 +870,32 @@ void f2fs_remove_donate_inode(struct inode *inode) spin_unlock(&sbi->inode_lock[DONATE_INODE]); } +static void f2fs_record_inode_state(struct f2fs_sb_info *sbi, nid_t ino, + unsigned int bits) +{ + if (bits & BIT(APPEND_INO)) + f2fs_add_ino_entry(sbi, ino, APPEND_INO); + if (bits & BIT(UPDATE_INO)) + f2fs_add_ino_entry(sbi, ino, UPDATE_INO); +} + +static void f2fs_evict_inode_work(struct work_struct *work) +{ + struct evict_inode_work *ew = + container_of(work, struct evict_inode_work, work); + + f2fs_record_inode_state(ew->sbi, ew->ino, ew->add_ino_entry_bits); + + mempool_free(ew, evict_inode_work_pool); +} + /* - * Called at the last iput() if i_nlink is zero + * Return true, if we shouldn't go through post_evict_inode. */ -void f2fs_evict_inode(struct inode *inode) +static bool f2fs_pre_evict_inode(struct inode *inode) { struct f2fs_sb_info *sbi = F2FS_I_SB(inode); struct f2fs_inode_info *fi = F2FS_I(inode); - nid_t xnid = fi->i_xattr_nid; - int err = 0; - bool freeze_protected = false; f2fs_abort_atomic_write(inode, true); @@ -883,13 +915,13 @@ void f2fs_evict_inode(struct inode *inode) truncate_inode_pages_final(&inode->i_data); if ((inode->i_nlink || is_bad_inode(inode)) && - test_opt(sbi, COMPRESS_CACHE) && f2fs_compressed_file(inode)) + test_opt(sbi, COMPRESS_CACHE) && f2fs_compressed_file(inode)) f2fs_invalidate_compress_pages(sbi, inode->i_ino); if (inode->i_ino == F2FS_NODE_INO(sbi) || - inode->i_ino == F2FS_META_INO(sbi) || - inode->i_ino == F2FS_COMPRESS_INO(sbi)) - goto out_clear; + inode->i_ino == F2FS_META_INO(sbi) || + inode->i_ino == F2FS_COMPRESS_INO(sbi)) + return true; f2fs_bug_on(sbi, get_dirty_pages(inode)); f2fs_remove_dirty_inode(inode); @@ -898,14 +930,18 @@ void f2fs_evict_inode(struct inode *inode) if (!IS_DEVICE_ALIASING(inode)) f2fs_destroy_extent_tree(inode); - if (inode->i_nlink || is_bad_inode(inode)) - goto no_delete; + return false; +} - err = f2fs_dquot_initialize(inode); - if (err) { - err = 0; +static void f2fs_delete_inode(struct inode *inode) +{ + struct f2fs_sb_info *sbi = F2FS_I_SB(inode); + bool freeze_protected = false; + struct f2fs_lock_context lc; + int err = 0; + + if (f2fs_dquot_initialize(inode)) set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR); - } f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO); f2fs_remove_ino_entry(sbi, inode->i_ino, UPDATE_INO); @@ -924,30 +960,30 @@ retry: if (time_to_inject(sbi, FAULT_EVICT_INODE)) err = -EIO; - if (!err) { - struct f2fs_lock_context lc; - - f2fs_lock_op(sbi, &lc); - err = f2fs_remove_inode_page(inode); - f2fs_unlock_op(sbi, &lc); - if (err == -ENOENT) { - err = 0; - - /* - * in fuzzed image, another node may has the same - * block address as inode's, if it was truncated - * previously, truncation of inode node will fail. - */ - if (is_inode_flag_set(inode, FI_DIRTY_INODE)) { - f2fs_warn(F2FS_I_SB(inode), - "f2fs_evict_inode: inconsistent node id, ino:%llu", - inode->i_ino); - f2fs_inode_synced(inode); - set_sbi_flag(sbi, SBI_NEED_FSCK); - } + if (err) + goto error_check; + + f2fs_lock_op(sbi, &lc); + err = f2fs_remove_inode_page(inode); + f2fs_unlock_op(sbi, &lc); + + if (err == -ENOENT) { + err = 0; + + /* + * in fuzzed image, another node may has the same + * block address as inode's, if it was truncated + * previously, truncation of inode node will fail. + */ + if (is_inode_flag_set(inode, FI_DIRTY_INODE)) { + f2fs_warn(F2FS_I_SB(inode), + "f2fs_evict_inode: inconsistent node id, ino:%llu", + inode->i_ino); + f2fs_inode_synced(inode); + set_sbi_flag(sbi, SBI_NEED_FSCK); } } - +error_check: /* give more chances, if ENOMEM case */ if (err == -ENOMEM) { err = 0; @@ -957,27 +993,38 @@ retry: if (IS_DEVICE_ALIASING(inode)) f2fs_destroy_extent_tree(inode); - if (err) { - f2fs_update_inode_page(inode); - if (dquot_initialize_needed(inode)) - set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR); + if (!err) + goto unfreeze_out; - /* - * If both f2fs_truncate() and f2fs_update_inode_page() failed - * due to fuzzed corrupted inode, call f2fs_inode_synced() to - * avoid triggering later f2fs_bug_on(). - */ - if (is_inode_flag_set(inode, FI_DIRTY_INODE)) { - f2fs_warn(sbi, - "f2fs_evict_inode: inode is dirty, ino:%llu", - inode->i_ino); - f2fs_inode_synced(inode); - set_sbi_flag(sbi, SBI_NEED_FSCK); - } + f2fs_update_inode_page(inode); + + if (dquot_initialize_needed(inode)) + set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR); + + /* + * If both f2fs_truncate() and f2fs_update_inode_page() failed + * due to fuzzed corrupted inode, call f2fs_inode_synced() to + * avoid triggering later f2fs_bug_on(). + */ + if (is_inode_flag_set(inode, FI_DIRTY_INODE)) { + f2fs_warn(sbi, + "f2fs_evict_inode: inode is dirty, ino:%llu", + inode->i_ino); + f2fs_inode_synced(inode); + set_sbi_flag(sbi, SBI_NEED_FSCK); } +unfreeze_out: if (freeze_protected) sb_end_intwrite(inode->i_sb); -no_delete: +} + +static void f2fs_post_evict_inode(struct inode *inode) +{ + struct f2fs_sb_info *sbi = F2FS_I_SB(inode); + struct f2fs_inode_info *fi = F2FS_I(inode); + nid_t xnid = fi->i_xattr_nid; + unsigned int record_bits = 0; + dquot_drop(inode); stat_dec_inline_xattr(inode); @@ -1003,12 +1050,32 @@ no_delete: inode->i_ino); if (xnid) invalidate_mapping_pages(NODE_MAPPING(sbi), xnid, xnid); - if (inode->i_nlink) { - if (is_inode_flag_set(inode, FI_APPEND_WRITE)) - f2fs_add_ino_entry(sbi, inode->i_ino, APPEND_INO); - if (is_inode_flag_set(inode, FI_UPDATE_WRITE)) - f2fs_add_ino_entry(sbi, inode->i_ino, UPDATE_INO); + + if (!inode->i_nlink) + goto skip_record; + + if (is_inode_flag_set(inode, FI_APPEND_WRITE)) + record_bits = BIT(APPEND_INO); + if (is_inode_flag_set(inode, FI_UPDATE_WRITE)) + record_bits = BIT(UPDATE_INO); + + if (!record_bits) + goto skip_record; + + /* Let's do this in workqueue out of the direct reclaim path. */ + if (current_is_kswapd()) { + f2fs_record_inode_state(sbi, inode->i_ino, record_bits); + } else { + struct evict_inode_work *ew = + mempool_alloc(evict_inode_work_pool, GFP_NOFS); + + ew->sbi = sbi; + ew->ino = inode->i_ino; + ew->add_ino_entry_bits = record_bits; + INIT_WORK(&ew->work, f2fs_evict_inode_work); + queue_work(sbi->evict_wq, &ew->work); } +skip_record: if (is_inode_flag_set(inode, FI_FREE_NID)) { f2fs_alloc_nid_failed(sbi, inode->i_ino); clear_inode_flag(inode, FI_FREE_NID); @@ -1019,7 +1086,22 @@ no_delete: * In that case, f2fs_check_nid_range() is enough to give a clue. */ } -out_clear: +} + +/* + * Called at the last iput() if i_nlink is zero + */ +void f2fs_evict_inode(struct inode *inode) +{ + if (f2fs_pre_evict_inode(inode)) + goto clear_out; + + if (!inode->i_nlink && !is_bad_inode(inode)) + f2fs_delete_inode(inode); + + f2fs_post_evict_inode(inode); + +clear_out: fscrypt_put_encryption_info(inode); clear_inode(inode); } @@ -1079,3 +1161,29 @@ out: /* iput will drop the inode object */ iput(inode); } + +int __init f2fs_init_evict_inode_work(void) +{ + evict_inode_work_cache = + kmem_cache_create("f2fs_evict_inode_work", + sizeof(struct evict_inode_work), 0, 0, NULL); + if (!evict_inode_work_cache) + goto fail; + evict_inode_work_pool = + mempool_create_slab_pool(NUM_PREALLOC_EVICT_INODE_WORK, + evict_inode_work_cache); + if (!evict_inode_work_pool) + goto fail_free_cache; + return 0; + +fail_free_cache: + kmem_cache_destroy(evict_inode_work_cache); +fail: + return -ENOMEM; +} + +void f2fs_destroy_evict_inode_work(void) +{ + mempool_destroy(evict_inode_work_pool); + kmem_cache_destroy(evict_inode_work_cache); +} diff --git a/fs/f2fs/iostat.c b/fs/f2fs/iostat.c index ae265e3e9b2c..12d4e18a6a50 100644 --- a/fs/f2fs/iostat.c +++ b/fs/f2fs/iostat.c @@ -332,6 +332,12 @@ void f2fs_destroy_iostat_processing(void) int f2fs_init_iostat(struct f2fs_sb_info *sbi) { + /* + * The f2fs_iostat tracepoint emits a fixed number of read folio order + * buckets; make sure every order fits so none is silently dropped. + */ + BUILD_BUG_ON(NR_PAGE_ORDERS > F2FS_IOSTAT_RD_FOLIO_ORDERS); + /* init iostat info */ spin_lock_init(&sbi->iostat_lock); spin_lock_init(&sbi->iostat_lat_lock); diff --git a/fs/f2fs/namei.c b/fs/f2fs/namei.c index 5ae647a352aa..87fa5a598c41 100644 --- a/fs/f2fs/namei.c +++ b/fs/f2fs/namei.c @@ -425,6 +425,9 @@ static int f2fs_link(struct dentry *old_dentry, struct inode *dir, if (!f2fs_is_checkpoint_ready(sbi)) return -ENOSPC; + if (IS_DEVICE_ALIASING(inode)) + return -EPERM; + err = fscrypt_prepare_link(old_dentry, dir, dentry); if (err) return err; @@ -568,6 +571,9 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry) trace_f2fs_unlink_enter(dir, dentry); + if (IS_DEVICE_ALIASING(inode)) + return -EPERM; + if (unlikely(f2fs_cp_error(sbi))) { err = -EIO; goto out; @@ -946,6 +952,9 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, bool old_is_dir = S_ISDIR(old_inode->i_mode); int err; + if (IS_DEVICE_ALIASING(old_inode)) + return -EPERM; + if (unlikely(f2fs_cp_error(sbi))) return -EIO; if (!f2fs_is_checkpoint_ready(sbi)) @@ -1016,6 +1025,8 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, } if (new_inode) { + if (IS_DEVICE_ALIASING(new_inode)) + return -EPERM; err = -ENOTEMPTY; if (old_is_dir && !f2fs_empty_dir(new_inode)) @@ -1076,7 +1087,7 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir, f2fs_up_write(&F2FS_I(old_inode)->i_sem); inode_set_ctime_current(old_inode); - f2fs_mark_inode_dirty_sync(old_inode, false); + f2fs_mark_inode_dirty_sync(old_inode, true); f2fs_delete_entry(old_entry, old_folio, old_dir, NULL); old_folio = NULL; @@ -1143,6 +1154,9 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, int old_nlink = 0, new_nlink = 0; int err; + if (IS_DEVICE_ALIASING(old_inode) || IS_DEVICE_ALIASING(new_inode)) + return -EPERM; + if (unlikely(f2fs_cp_error(sbi))) return -EIO; if (!f2fs_is_checkpoint_ready(sbi)) @@ -1246,7 +1260,7 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, f2fs_i_links_write(old_dir, old_nlink > 0); f2fs_up_write(&F2FS_I(old_dir)->i_sem); } - f2fs_mark_inode_dirty_sync(old_dir, false); + f2fs_mark_inode_dirty_sync(old_dir, true); /* update directory entry info of new dir inode */ f2fs_set_link(new_dir, new_entry, new_folio, old_inode); @@ -1265,7 +1279,7 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry, f2fs_i_links_write(new_dir, new_nlink > 0); f2fs_up_write(&F2FS_I(new_dir)->i_sem); } - f2fs_mark_inode_dirty_sync(new_dir, false); + f2fs_mark_inode_dirty_sync(new_dir, true); if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_STRICT) { f2fs_add_ino_entry(sbi, old_dir->i_ino, TRANS_DIR_INO); diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 38917e4a7319..46bea52e35c3 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -660,6 +660,7 @@ sanity_check: __builtin_return_address(0), ni->ino, ni->nid, ni->blk_addr, ni->version, ni->flag); f2fs_handle_error(sbi, ERROR_INCONSISTENT_NAT); + return -EFSCORRUPTED; } /* cache nat entry */ @@ -1617,7 +1618,7 @@ page_hit: if (!err) return folio; out_err: - folio_clear_uptodate(folio); + clear_node_folio_dirty(folio); out_put_err: /* ENOENT comes from read_node_folio which is not an error. */ if (err != -ENOENT) @@ -1789,7 +1790,7 @@ static bool __write_node_folio(struct folio *folio, bool atomic, bool do_fsync, /* get old block addr of this node page */ nid = nid_of_node(folio); - if (f2fs_sanity_check_node_footer(sbi, folio, nid, + if (f2fs_sanity_check_node_footer(sbi, folio, folio->index, NODE_TYPE_REGULAR, false)) { fserror_report_metadata(sbi->sb, -EFSCORRUPTED, GFP_NOFS); f2fs_stop_checkpoint(sbi, false, STOP_CP_REASON_CORRUPTED_NID); diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c index 89af8407b667..aaa5227739c8 100644 --- a/fs/f2fs/recovery.c +++ b/fs/f2fs/recovery.c @@ -116,18 +116,26 @@ static void del_fsync_inode(struct fsync_inode_entry *entry, int drop) } static int init_recovered_filename(const struct inode *dir, + struct inode *inode, struct f2fs_inode *raw_inode, struct f2fs_filename *fname, struct qstr *usr_fname) { + struct f2fs_sb_info *sbi = F2FS_I_SB(inode); int err; memset(fname, 0, sizeof(*fname)); fname->disk_name.len = le32_to_cpu(raw_inode->i_namelen); fname->disk_name.name = raw_inode->i_name; - if (WARN_ON(fname->disk_name.len > F2FS_NAME_LEN)) - return -ENAMETOOLONG; + if (unlikely(!fname->disk_name.len || + fname->disk_name.len > F2FS_NAME_LEN)) { + f2fs_err(sbi, "invalid recovered filename length %u for ino %llu", + fname->disk_name.len, inode->i_ino); + set_sbi_flag(sbi, SBI_NEED_FSCK); + f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE); + return -EFSCORRUPTED; + } if (!IS_ENCRYPTED(dir)) { usr_fname->name = fname->disk_name.name; @@ -158,6 +166,22 @@ static int init_recovered_filename(const struct inode *dir, return 0; } +static const char *recover_printable_name(struct inode *inode, + struct f2fs_inode *raw, + int *name_len) +{ + static const char encrypted_name[] = "<encrypted>"; + + if (file_enc_name(inode)) { + *name_len = sizeof(encrypted_name) - 1; + return encrypted_name; + } + + *name_len = min_t(unsigned int, le32_to_cpu(raw->i_namelen), + F2FS_NAME_LEN); + return raw->i_name; +} + static int recover_dentry(struct inode *inode, struct folio *ifolio, struct list_head *dir_list) { @@ -170,7 +194,8 @@ static int recover_dentry(struct inode *inode, struct folio *ifolio, struct inode *dir, *einode; struct fsync_inode_entry *entry; int err = 0; - char *name; + const char *name; + int name_len; entry = get_fsync_inode(dir_list, pino); if (!entry) { @@ -184,7 +209,7 @@ static int recover_dentry(struct inode *inode, struct folio *ifolio, } dir = entry->inode; - err = init_recovered_filename(dir, raw_inode, &fname, &usr_fname); + err = init_recovered_filename(dir, inode, raw_inode, &fname, &usr_fname); if (err) goto out; retry: @@ -229,12 +254,9 @@ retry: out_put: f2fs_folio_put(folio, false); out: - if (file_enc_name(inode)) - name = "<encrypted>"; - else - name = raw_inode->i_name; - f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %s, dir = %llu, err = %d", - __func__, ino_of_node(ifolio), name, + name = recover_printable_name(inode, raw_inode, &name_len); + f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %llu, err = %d", + __func__, ino_of_node(ifolio), name_len, name, IS_ERR(dir) ? 0 : dir->i_ino, err); return err; } @@ -282,7 +304,8 @@ static int recover_inode(struct inode *inode, struct folio *folio) { struct f2fs_inode *raw = F2FS_INODE(folio); struct f2fs_inode_info *fi = F2FS_I(inode); - char *name; + const char *name; + int name_len; int err; inode->i_mode = le16_to_cpu(raw->i_mode); @@ -331,13 +354,11 @@ static int recover_inode(struct inode *inode, struct folio *folio) f2fs_mark_inode_dirty_sync(inode, true); - if (file_enc_name(inode)) - name = "<encrypted>"; - else - name = F2FS_INODE(folio)->i_name; + name = recover_printable_name(inode, raw, &name_len); - f2fs_notice(F2FS_I_SB(inode), "recover_inode: ino = %x, name = %s, inline = %x", - ino_of_node(folio), name, raw->i_inline); + f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x", + __func__, ino_of_node(folio), name_len, name, + raw->i_inline); return 0; } diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index 642668a1847c..bfbd99349a0c 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -261,7 +261,7 @@ retry: } else { blkcnt_t count = 1; - err = inc_valid_block_count(sbi, inode, &count, true); + err = inc_valid_block_count(sbi, inode, &count, true, false); if (err) { f2fs_put_dnode(&dn); return err; @@ -452,15 +452,14 @@ void f2fs_balance_fs(struct f2fs_sb_info *sbi, bool need) f2fs_submit_merged_write(sbi, DATA); f2fs_submit_all_merged_ipu_writes(sbi); - if (test_opt(sbi, GC_MERGE) && sbi->gc_thread && - sbi->gc_thread->f2fs_gc_task) { + if (test_opt(sbi, GC_MERGE) && sbi->gc_thread.f2fs_gc_task) { DEFINE_WAIT(wait); - prepare_to_wait(&sbi->gc_thread->fggc_wq, &wait, + prepare_to_wait(&sbi->gc_thread.fggc_wq, &wait, TASK_UNINTERRUPTIBLE); - wake_up(&sbi->gc_thread->gc_wait_queue_head); + wake_up(&sbi->gc_thread.gc_wait_queue_head); io_schedule(); - finish_wait(&sbi->gc_thread->fggc_wq, &wait); + finish_wait(&sbi->gc_thread.fggc_wq, &wait); } else { struct f2fs_gc_control gc_control = { .victim_segno = NULL_SEGNO, @@ -566,21 +565,59 @@ static int __submit_flush_wait(struct f2fs_sb_info *sbi, return ret; } +static void f2fs_flush_end_io(struct bio *bio) +{ + complete(bio->bi_private); +} + +struct f2fs_flush_bio { + struct bio bio; + struct completion wait; +}; + static int submit_flush_wait(struct f2fs_sb_info *sbi, nid_t ino) { + struct f2fs_flush_bio *flush_bio; + unsigned long devices = 0; int ret = 0; int i; if (!f2fs_is_multi_device(sbi)) return __submit_flush_wait(sbi, sbi->sb->s_bdev); + flush_bio = kmalloc(array_size(sbi->s_ndevs, sizeof(*flush_bio)), + GFP_NOFS | __GFP_NOFAIL); + for (i = 0; i < sbi->s_ndevs; i++) { if (!f2fs_is_dirty_device(sbi, ino, i, FLUSH_INO)) continue; - ret = __submit_flush_wait(sbi, FDEV(i).bdev); - if (ret) - break; + + bio_init(&flush_bio[i].bio, FDEV(i).bdev, NULL, 0, + REQ_OP_WRITE | REQ_SYNC | REQ_PREFLUSH); + init_completion(&flush_bio[i].wait); + flush_bio[i].bio.bi_private = &flush_bio[i].wait; + flush_bio[i].bio.bi_end_io = f2fs_flush_end_io; + devices |= BIT(i); + submit_bio(&flush_bio[i].bio); } + + for (i = 0; i < sbi->s_ndevs; i++) { + int err; + + if (!(devices & BIT(i))) + continue; + + wait_for_completion(&flush_bio[i].wait); + err = blk_status_to_errno(flush_bio[i].bio.bi_status); + trace_f2fs_issue_flush(FDEV(i).bdev, test_opt(sbi, NOBARRIER), + test_opt(sbi, FLUSH_MERGE), err); + if (!err) + f2fs_update_iostat(sbi, NULL, FS_FLUSH_IO, 0); + else if (!ret) + ret = err; + bio_uninit(&flush_bio[i].bio); + } + kfree(flush_bio); return ret; } @@ -2502,35 +2539,42 @@ static int update_sit_entry_for_alloc(struct f2fs_sb_info *sbi, struct seg_entry unsigned int segno, block_t blkaddr, unsigned int offset, int del) { bool exist; + int del_count = del; + int i; - exist = f2fs_test_and_set_bit(offset, se->cur_valid_map); - if (unlikely(exist)) { - f2fs_err(sbi, "Bitmap was wrongly set, blk:%u", blkaddr); - f2fs_bug_on(sbi, 1); - se->valid_blocks--; - del = 0; - } + f2fs_bug_on(sbi, GET_SEGNO(sbi, blkaddr) != GET_SEGNO(sbi, blkaddr + del_count - 1)); - if (f2fs_block_unit_discard(sbi) && - !f2fs_test_and_set_bit(offset, se->discard_map)) - sbi->discard_blks--; + for (i = 0; i < del_count; i++) { + exist = f2fs_test_and_set_bit(offset + i, se->cur_valid_map); + if (unlikely(exist)) { + f2fs_err(sbi, "Bitmap was wrongly set, blk:%u", blkaddr + i); + f2fs_bug_on(sbi, 1); + se->valid_blocks--; + del -= 1; + continue; + } - /* - * SSR should never reuse block which is checkpointed - * or newly invalidated. - */ - if (!is_sbi_flag_set(sbi, SBI_CP_DISABLED)) { - if (!f2fs_test_and_set_bit(offset, se->ckpt_valid_map)) { - se->ckpt_valid_blocks++; - if (__is_large_section(sbi)) - get_sec_entry(sbi, segno)->ckpt_valid_blocks++; + if (f2fs_block_unit_discard(sbi) && + !f2fs_test_and_set_bit(offset + i, se->discard_map)) + sbi->discard_blks--; + + /* + * SSR should never reuse block which is checkpointed + * or newly invalidated. + */ + if (!is_sbi_flag_set(sbi, SBI_CP_DISABLED)) { + if (!f2fs_test_and_set_bit(offset + i, se->ckpt_valid_map)) { + se->ckpt_valid_blocks++; + if (__is_large_section(sbi)) + get_sec_entry(sbi, segno)->ckpt_valid_blocks++; + } } - } - if (!f2fs_test_bit(offset, se->ckpt_valid_map)) { - se->ckpt_valid_blocks += del; - if (__is_large_section(sbi)) - get_sec_entry(sbi, segno)->ckpt_valid_blocks += del; + if (!f2fs_test_bit(offset + i, se->ckpt_valid_map)) { + se->ckpt_valid_blocks += 1; + if (__is_large_section(sbi)) + get_sec_entry(sbi, segno)->ckpt_valid_blocks += 1; + } } if (__is_large_section(sbi)) @@ -2585,9 +2629,14 @@ void f2fs_invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr, unsigned int segno = GET_SEGNO(sbi, addr); struct sit_info *sit_i = SIT_I(sbi); block_t addr_start = addr, addr_end = addr + len - 1; - unsigned int seg_num = GET_SEGNO(sbi, addr_end) - segno + 1; + unsigned int seg_num; unsigned int i = 1, max_blocks = sbi->blocks_per_seg, cnt; + if (len == 0) + return; + + seg_num = GET_SEGNO(sbi, addr_end) - segno + 1; + f2fs_bug_on(sbi, addr == NULL_ADDR); if (addr == NEW_ADDR || addr == COMPRESS_ADDR) return; @@ -2620,6 +2669,52 @@ void f2fs_invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr, up_write(&sit_i->sentry_lock); } +void f2fs_reserve_device_alias(struct f2fs_sb_info *sbi, block_t addr, + unsigned int len) +{ + unsigned int segno = GET_SEGNO(sbi, addr); + struct sit_info *sit_i = SIT_I(sbi); + block_t addr_start = addr, addr_end = addr + len - 1; + unsigned int seg_num; + unsigned int i = 1, max_blocks = sbi->blocks_per_seg, cnt; + + if (len == 0) + return; + + seg_num = GET_SEGNO(sbi, addr_end) - segno + 1; + + down_write(&sit_i->sentry_lock); + + if (seg_num == 1) + cnt = len; + else + cnt = max_blocks - GET_BLKOFF_FROM_SEG0(sbi, addr); + + do { + update_segment_mtime(sbi, addr_start, 0); + update_sit_entry(sbi, addr_start, cnt); + __set_test_and_inuse(sbi, segno); + + /* Remove the segment from PRE (prefree) to prevent checkpoint from freeing it! */ + mutex_lock(&DIRTY_I(sbi)->seglist_lock); + if (test_and_clear_bit(segno, DIRTY_I(sbi)->dirty_segmap[PRE])) + DIRTY_I(sbi)->nr_dirty[PRE]--; + mutex_unlock(&DIRTY_I(sbi)->seglist_lock); + + /* add it into dirty seglist */ + locate_dirty_segment(sbi, segno); + + /* update @addr_start and @cnt and @segno */ + addr_start = START_BLOCK(sbi, ++segno); + if (++i == seg_num) + cnt = GET_BLKOFF_FROM_SEG0(sbi, addr_end) + 1; + else + cnt = max_blocks; + } while (i <= seg_num); + + up_write(&sit_i->sentry_lock); +} + bool f2fs_is_checkpointed_data(struct f2fs_sb_info *sbi, block_t blkaddr) { struct sit_info *sit_i = SIT_I(sbi); @@ -2758,8 +2853,13 @@ static int is_next_segment_free(struct f2fs_sb_info *sbi, unsigned int segno = curseg->segno + 1; struct free_segmap_info *free_i = FREE_I(sbi); - if (segno < MAIN_SEGS(sbi) && segno % SEGS_PER_SEC(sbi)) + if (segno < MAIN_SEGS(sbi) && segno % SEGS_PER_SEC(sbi)) { + int devi = f2fs_target_device_index(sbi, START_BLOCK(sbi, segno)); + + if (f2fs_dev_is_reserving(sbi, devi)) + return 0; return !test_bit(segno, free_i->free_segmap); + } return 0; } @@ -2778,7 +2878,8 @@ static int get_new_segment(struct f2fs_sb_info *sbi, unsigned int alloc_policy = sbi->allocate_section_policy; unsigned int alloc_hint = sbi->allocate_section_hint; bool init = true; - int i; + bool looped = false; + int i, devi; int ret = 0; spin_lock(&free_i->segmap_lock); @@ -2791,8 +2892,13 @@ static int get_new_segment(struct f2fs_sb_info *sbi, if (!new_sec && ((*newseg + 1) % SEGS_PER_SEC(sbi))) { segno = find_next_zero_bit(free_i->free_segmap, GET_SEG_FROM_SEC(sbi, hint + 1), *newseg + 1); - if (segno < GET_SEG_FROM_SEC(sbi, hint + 1)) + if (segno < GET_SEG_FROM_SEC(sbi, hint + 1)) { + devi = f2fs_target_device_index(sbi, START_BLOCK(sbi, segno)); + + if (f2fs_dev_is_alloc_blocked(sbi, devi, pinning)) + goto find_other_zone; goto got_it; + } } #ifdef CONFIG_BLK_DEV_ZONED @@ -2828,33 +2934,42 @@ static int get_new_segment(struct f2fs_sb_info *sbi, find_other_zone: secno = find_next_zero_bit(free_i->free_secmap, MAIN_SECS(sbi), hint); -#ifdef CONFIG_BLK_DEV_ZONED - if (secno >= MAIN_SECS(sbi) && f2fs_sb_has_blkzoned(sbi)) { - /* Write only to sequential zones */ - if (sbi->blkzone_alloc_policy == BLKZONE_ALLOC_ONLY_SEQ) { - hint = GET_SEC_FROM_SEG(sbi, sbi->first_seq_zone_segno); - secno = find_next_zero_bit(free_i->free_secmap, MAIN_SECS(sbi), hint); - } else - secno = find_first_zero_bit(free_i->free_secmap, - MAIN_SECS(sbi)); - if (secno >= MAIN_SECS(sbi)) { - ret = -ENOSPC; - f2fs_bug_on(sbi, 1); - goto out_unlock; - } - } -#endif - if (secno >= MAIN_SECS(sbi)) { - secno = find_first_zero_bit(free_i->free_secmap, - MAIN_SECS(sbi)); - if (secno >= MAIN_SECS(sbi)) { + if (looped) { ret = -ENOSPC; f2fs_bug_on(sbi, !pinning); goto out_unlock; } + hint = 0; +#ifdef CONFIG_BLK_DEV_ZONED + /* Write only to sequential zones */ + if (f2fs_sb_has_blkzoned(sbi) && + sbi->blkzone_alloc_policy == BLKZONE_ALLOC_ONLY_SEQ) + hint = GET_SEC_FROM_SEG(sbi, sbi->first_seq_zone_segno); +#endif + looped = true; + goto find_other_zone; } + segno = GET_SEG_FROM_SEC(sbi, secno); + + devi = f2fs_target_device_index(sbi, START_BLOCK(sbi, segno)); + + if (f2fs_dev_is_alloc_blocked(sbi, devi, pinning)) { + while (devi < sbi->s_ndevs && + f2fs_dev_is_alloc_blocked(sbi, devi, pinning)) { + unsigned int end_segno = GET_SEGNO(sbi, FDEV(devi).end_blk); + + hint = GET_SEC_FROM_SEG(sbi, end_segno) + 1; + devi++; + } + goto find_other_zone; + } + + if (sec_usage_check(sbi, secno)) { + hint = secno + 1; + goto find_other_zone; + } zoneno = GET_ZONE_FROM_SEC(sbi, secno); /* give up on finding another zone */ @@ -2937,7 +3052,7 @@ static unsigned int __get_next_segno(struct f2fs_sb_info *sbi, int type) sanity_check_seg_type(sbi, seg_type); if (__is_large_section(sbi)) { - if (f2fs_need_rand_seg(sbi)) { + if (f2fs_need_rand_seg_blk(sbi, type)) { unsigned int hint = GET_SEC_FROM_SEG(sbi, curseg->segno); if (GET_SEC_FROM_SEG(sbi, curseg->segno + 1) != hint) @@ -2946,7 +3061,7 @@ static unsigned int __get_next_segno(struct f2fs_sb_info *sbi, int type) GET_SEG_FROM_SEC(sbi, hint + 1) - 1); } return curseg->segno; - } else if (f2fs_need_rand_seg(sbi)) { + } else if (f2fs_need_rand_seg_blk(sbi, type)) { return get_random_u32_below(MAIN_SECS(sbi) * SEGS_PER_SEC(sbi)); } @@ -3002,7 +3117,7 @@ static int new_curseg(struct f2fs_sb_info *sbi, int type, bool new_sec) curseg->next_segno = segno; reset_curseg(sbi, type, 1); curseg->alloc_type = LFS; - if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK) + if (f2fs_need_rand_blk(sbi, type)) curseg->fragment_remained_chunk = get_random_u32_inclusive(1, sbi->max_fragment_chunk); return 0; @@ -3358,14 +3473,15 @@ retry: f2fs_unlock_op(sbi, &lc); if (f2fs_sb_has_blkzoned(sbi) && err == -EAGAIN && gc_required) { - f2fs_down_write_trace(&sbi->gc_lock, &lc); err = f2fs_gc_range(sbi, 0, sbi->first_seq_zone_segno - 1, - true, ZONED_PIN_SEC_REQUIRED_COUNT); - f2fs_up_write_trace(&sbi->gc_lock, &lc); - - gc_required = false; - if (!err) + true, ZONED_PIN_SEC_REQUIRED_COUNT, true); + if (err) + return err; + err = f2fs_sync_fs(sbi->sb, 1); + if (!err) { + gc_required = false; goto retry; + } } return err; @@ -3820,7 +3936,7 @@ int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio, curseg->next_blkoff = f2fs_find_next_ssr_block(sbi, curseg); } else { curseg->next_blkoff++; - if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK) + if (f2fs_need_rand_blk(sbi, type)) f2fs_randomize_chunk(sbi, curseg); } if (curseg->next_blkoff >= f2fs_usable_blks_in_seg(sbi, curseg->segno)) diff --git a/fs/f2fs/segment.h b/fs/f2fs/segment.h index b0c06b3580b4..db1079169a23 100644 --- a/fs/f2fs/segment.h +++ b/fs/f2fs/segment.h @@ -796,20 +796,6 @@ F2FS_IPU_POLICY(F2FS_IPU_ASYNC); F2FS_IPU_POLICY(F2FS_IPU_NOCACHE); F2FS_IPU_POLICY(F2FS_IPU_HONOR_OPU_WRITE); -static inline unsigned int curseg_segno(struct f2fs_sb_info *sbi, - int type) -{ - struct curseg_info *curseg = CURSEG_I(sbi, type); - return curseg->segno; -} - -static inline unsigned char curseg_alloc_type(struct f2fs_sb_info *sbi, - int type) -{ - struct curseg_info *curseg = CURSEG_I(sbi, type); - return curseg->alloc_type; -} - static inline bool valid_main_segno(struct f2fs_sb_info *sbi, unsigned int segno) { @@ -954,10 +940,32 @@ static inline block_t sum_blk_addr(struct f2fs_sb_info *sbi, int base, int type) - (base + 1) + type; } +static inline bool f2fs_dev_is_reserving(struct f2fs_sb_info *sbi, int devi) +{ + if (!f2fs_sb_has_device_alias(sbi)) + return false; + return FDEV(devi).is_reserving; +} + +static inline bool f2fs_dev_is_alloc_blocked(struct f2fs_sb_info *sbi, + int devi, bool pinning) +{ + if (!f2fs_sb_has_device_alias(sbi)) + return false; + return (pinning && FDEV(devi).has_alias) || FDEV(devi).is_reserving; +} + static inline bool sec_usage_check(struct f2fs_sb_info *sbi, unsigned int secno) { if (is_cursec(sbi, secno) || (sbi->cur_victim_sec == secno)) return true; + if (f2fs_sb_has_device_alias(sbi)) { + block_t start_blk = START_BLOCK(sbi, GET_SEG_FROM_SEC(sbi, secno)); + int devi = f2fs_target_device_index(sbi, start_blk); + + if (f2fs_dev_is_reserving(sbi, devi)) + return true; + } return false; } diff --git a/fs/f2fs/shrinker.c b/fs/f2fs/shrinker.c index b88babcf6ab4..4f6bf5926de4 100644 --- a/fs/f2fs/shrinker.c +++ b/fs/f2fs/shrinker.c @@ -109,7 +109,8 @@ unsigned long f2fs_shrink_scan(struct shrinker *shrink, freed += f2fs_shrink_age_extent_tree(sbi, nr >> 2); /* shrink read extent cache entries */ - freed += f2fs_shrink_read_extent_tree(sbi, nr >> 2); + if (freed < nr) + freed += f2fs_shrink_read_extent_tree(sbi, nr >> 2); /* shrink clean nat cache entries */ if (freed < nr) diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index be22c31015ef..7e0a6c93ce39 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2058,7 +2058,8 @@ static void f2fs_put_super(struct super_block *sb) if (!get_pages(sbi, i)) continue; f2fs_err(sbi, "detect filesystem reference count leak during " - "umount, type: %d, count: %lld", i, get_pages(sbi, i)); + "umount, type: %d, count: %lld, err: %d, cp_err: %d", + i, get_pages(sbi, i), err, f2fs_cp_error(sbi)); f2fs_bug_on(sbi, 1); } @@ -2456,9 +2457,9 @@ static int f2fs_show_options(struct seq_file *seq, struct dentry *root) seq_puts(seq, "adaptive"); else if (F2FS_OPTION(sbi).fs_mode == FS_MODE_LFS) seq_puts(seq, "lfs"); - else if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_SEG) + else if (f2fs_need_rand_seg(sbi, NO_CHECK_TYPE)) seq_puts(seq, "fragment:segment"); - else if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK) + else if (f2fs_need_rand_blk(sbi, NO_CHECK_TYPE)) seq_puts(seq, "fragment:block"); seq_printf(seq, ",active_logs=%u", F2FS_OPTION(sbi).active_logs); if (test_opt(sbi, RESERVE_ROOT) || test_opt(sbi, RESERVE_NODE)) @@ -2943,11 +2944,11 @@ static int __f2fs_remount(struct fs_context *fc, struct super_block *sb) if ((flags & SB_RDONLY) || (F2FS_OPTION(sbi).bggc_mode == BGGC_MODE_OFF && !test_opt(sbi, GC_MERGE))) { - if (sbi->gc_thread) { + if (sbi->gc_thread.f2fs_gc_task) { f2fs_stop_gc_thread(sbi); need_restart_gc = true; } - } else if (!sbi->gc_thread) { + } else if (!sbi->gc_thread.f2fs_gc_task) { err = f2fs_start_gc_thread(sbi); if (err) goto restore_opts; @@ -3168,7 +3169,7 @@ static ssize_t f2fs_quota_read(struct super_block *sb, int type, char *data, repeat: folio = mapping_read_folio_gfp(mapping, off >> PAGE_SHIFT, - GFP_NOFS); + GFP_KERNEL); if (IS_ERR(folio)) { if (PTR_ERR(folio) == -ENOMEM) { memalloc_retry_wait(GFP_NOFS); @@ -5004,6 +5005,39 @@ static void f2fs_tuning_parameters(struct f2fs_sb_info *sbi) sbi->readdir_ra = true; } +static void f2fs_restore_device_alias(struct f2fs_sb_info *sbi) +{ + struct inode *root = d_inode(sbi->sb->s_root); + struct f2fs_dir_entry *de; + struct folio *folio; + int i; + + if (!f2fs_sb_has_device_alias(sbi)) + return; + + for (i = 1; i < sbi->s_ndevs; i++) { + char *name = strrchr(FDEV(i).path, '/'); + struct inode *inode; + struct qstr qstr; + + name = name ? name + 1 : FDEV(i).path; + qstr.name = name; + qstr.len = strlen(name); + + de = f2fs_find_entry(root, &qstr, &folio); + if (!de) + continue; + + inode = f2fs_iget(sbi->sb, le32_to_cpu(de->ino)); + if (!IS_ERR(inode)) { + if (IS_DEVICE_ALIASING(inode)) + FDEV(i).has_alias = true; + iput(inode); + } + f2fs_folio_put(folio, 0); + } +} + static int f2fs_fill_super(struct super_block *sb, struct fs_context *fc) { struct f2fs_fs_context *ctx = fc->fs_private; @@ -5068,6 +5102,7 @@ try_onemore: sb->s_fs_info = sbi; sbi->raw_super = raw_super; + sbi->max_atc_write_bio_size = UINT_MAX; INIT_WORK(&sbi->s_error_work, f2fs_record_error_work); memcpy(sbi->errors, raw_super->s_errors, MAX_F2FS_ERRORS); @@ -5207,6 +5242,7 @@ try_onemore: sbi->last_valid_block_count = sbi->total_valid_block_count; sbi->reserved_blocks = 0; sbi->current_reserved_blocks = 0; + sbi->alias_reserved_blocks = 0; limit_reserve_root(sbi); adjust_unusable_cap_perc(sbi); @@ -5434,6 +5470,8 @@ reset_checkpoint: f2fs_update_time(sbi, REQ_TIME); clear_sbi_flag(sbi, SBI_CP_DISABLED_QUICK); + f2fs_restore_device_alias(sbi); + sbi->umount_lock_holder = NULL; return 0; @@ -5701,10 +5739,16 @@ static int __init init_f2fs_fs(void) err = f2fs_init_xattr_cache(); if (err) goto free_casefold_cache; - err = register_filesystem(&f2fs_fs_type); + err = f2fs_init_evict_inode_work(); if (err) goto free_xattr_cache; + err = register_filesystem(&f2fs_fs_type); + if (err) + goto free_evict_inode_cache; return 0; + +free_evict_inode_cache: + f2fs_destroy_evict_inode_work(); free_xattr_cache: f2fs_destroy_xattr_cache(); free_casefold_cache: @@ -5747,6 +5791,7 @@ fail: static void __exit exit_f2fs_fs(void) { unregister_filesystem(&f2fs_fs_type); + f2fs_destroy_evict_inode_work(); f2fs_destroy_xattr_cache(); f2fs_destroy_casefold_cache(); f2fs_destroy_compress_cache(); diff --git a/fs/f2fs/sysfs.c b/fs/f2fs/sysfs.c index 665687244c93..3201e2185fea 100644 --- a/fs/f2fs/sysfs.c +++ b/fs/f2fs/sysfs.c @@ -75,7 +75,7 @@ static ssize_t f2fs_sbi_show(struct f2fs_attr *a, static unsigned char *__struct_ptr(struct f2fs_sb_info *sbi, int struct_type) { if (struct_type == GC_THREAD) - return (unsigned char *)sbi->gc_thread; + return (unsigned char *)&sbi->gc_thread; else if (struct_type == SM_INFO) return (unsigned char *)SM_I(sbi); else if (struct_type == DCC_INFO) @@ -557,7 +557,7 @@ out: return -EINVAL; cprc->ckpt_thread_ioprio = IOPRIO_PRIO_VALUE(class, level); - if (test_opt(sbi, MERGE_CHECKPOINT)) { + if (cprc->f2fs_issue_ckpt) { ret = set_task_ioprio(cprc->f2fs_issue_ckpt, cprc->ckpt_thread_ioprio); if (ret) @@ -664,20 +664,20 @@ out: sbi->gc_mode = GC_NORMAL; } else if (t == 1) { sbi->gc_mode = GC_URGENT_HIGH; - if (sbi->gc_thread) { - sbi->gc_thread->gc_wake = true; + if (sbi->gc_thread.f2fs_gc_task) { + sbi->gc_thread.gc_wake = true; wake_up_interruptible_all( - &sbi->gc_thread->gc_wait_queue_head); + &sbi->gc_thread.gc_wait_queue_head); wake_up_discard_thread(sbi, true); } } else if (t == 2) { sbi->gc_mode = GC_URGENT_LOW; } else if (t == 3) { sbi->gc_mode = GC_URGENT_MID; - if (sbi->gc_thread) { - sbi->gc_thread->gc_wake = true; + if (sbi->gc_thread.f2fs_gc_task) { + sbi->gc_thread.gc_wake = true; wake_up_interruptible_all( - &sbi->gc_thread->gc_wait_queue_head); + &sbi->gc_thread.gc_wait_queue_head); } } else { return -EINVAL; @@ -934,14 +934,14 @@ out: if (!strcmp(a->attr.name, "gc_boost_gc_multiple")) { if (t < 1 || t > SEGS_PER_SEC(sbi)) return -EINVAL; - sbi->gc_thread->boost_gc_multiple = (unsigned int)t; + sbi->gc_thread.boost_gc_multiple = (unsigned int)t; return count; } if (!strcmp(a->attr.name, "gc_boost_gc_greedy")) { if (t > GC_GREEDY) return -EINVAL; - sbi->gc_thread->boost_gc_greedy = (unsigned int)t; + sbi->gc_thread.boost_gc_greedy = (unsigned int)t; return count; } @@ -989,8 +989,8 @@ out: if (sbi->cprc_info.f2fs_issue_ckpt) set_user_nice(sbi->cprc_info.f2fs_issue_ckpt, PRIO_TO_NICE(sbi->critical_task_priority)); - if (sbi->gc_thread && sbi->gc_thread->f2fs_gc_task) - set_user_nice(sbi->gc_thread->f2fs_gc_task, + if (sbi->gc_thread.f2fs_gc_task) + set_user_nice(sbi->gc_thread.f2fs_gc_task, PRIO_TO_NICE(sbi->critical_task_priority)); return count; } @@ -1007,13 +1007,15 @@ static ssize_t f2fs_sbi_store(struct f2fs_attr *a, ssize_t ret; bool gc_entry = (!strcmp(a->attr.name, "gc_urgent") || a->struct_type == GC_THREAD); + bool thread_entry = !strcmp(a->attr.name, "ckpt_thread_ioprio") || + !strcmp(a->attr.name, "critical_task_priority"); - if (gc_entry) { + if (gc_entry || thread_entry) { if (!down_read_trylock(&sbi->sb->s_umount)) return -EAGAIN; } ret = __sbi_store(a, sbi, buf, count); - if (gc_entry) + if (gc_entry || thread_entry) up_read(&sbi->sb->s_umount); return ret; @@ -1266,6 +1268,7 @@ F2FS_SBI_RW_ATTR(gc_idle_interval, interval_time[GC_TIME]); F2FS_SBI_RW_ATTR(umount_discard_timeout, interval_time[UMOUNT_DISCARD_TIMEOUT]); F2FS_SBI_RW_ATTR(gc_pin_file_thresh, gc_pin_file_threshold); F2FS_SBI_RW_ATTR(gc_reclaimed_segments, gc_reclaimed_segs); +F2FS_SBI_RW_ATTR(max_atc_write_bio_size, max_atc_write_bio_size); F2FS_SBI_GENERAL_RW_ATTR(max_victim_search); F2FS_SBI_GENERAL_RW_ATTR(migration_granularity); F2FS_SBI_GENERAL_RW_ATTR(migration_window_granularity); @@ -1509,6 +1512,7 @@ static struct attribute *f2fs_attrs[] = { ATTR_LIST(seq_file_ra_mul), ATTR_LIST(gc_segment_mode), ATTR_LIST(gc_reclaimed_segments), + ATTR_LIST(max_atc_write_bio_size), ATTR_LIST(max_fragment_chunk), ATTR_LIST(max_fragment_hole), ATTR_LIST(current_atomic_write), diff --git a/fs/f2fs/xattr.c b/fs/f2fs/xattr.c index ed33e5110f2a..6728d1488cad 100644 --- a/fs/f2fs/xattr.c +++ b/fs/f2fs/xattr.c @@ -119,7 +119,7 @@ static int f2fs_xattr_advise_set(const struct xattr_handler *handler, unsigned char old_advise = F2FS_I(inode)->i_advise; unsigned char new_advise; - if (!inode_owner_or_capable(&nop_mnt_idmap, inode)) + if (!inode_owner_or_capable(idmap, inode)) return -EPERM; if (value == NULL) return -EINVAL; diff --git a/fs/fuse/args.h b/fs/fuse/args.h index ecfe51a192af..5173264a1261 100644 --- a/fs/fuse/args.h +++ b/fs/fuse/args.h @@ -42,6 +42,8 @@ struct fuse_args { bool is_pinned:1; bool invalidate_vmap:1; bool abort_on_kill:1; + /* server requested io-uring zero-copy for this op */ + bool zero_copy:1; struct fuse_in_arg in_args[4]; struct fuse_arg out_args[2]; void (*end)(struct fuse_args *args, int error); diff --git a/fs/fuse/cuse.c b/fs/fuse/cuse.c index 3c15b5ba16d7..4079cf8e5974 100644 --- a/fs/fuse/cuse.c +++ b/fs/fuse/cuse.c @@ -530,7 +530,8 @@ static int cuse_channel_open(struct inode *inode, struct file *file) INIT_LIST_HEAD(&cc->list); - cc->fc.chan->initialized = 1; + /* Pairs with smp_load_acquire() readers of fch->initialized */ + smp_store_release(&cc->fc.chan->initialized, 1); rc = cuse_send_init(cc); if (rc) { fuse_dev_put(fud); @@ -653,6 +654,11 @@ static void __exit cuse_exit(void) { misc_deregister(&cuse_miscdev); class_destroy(cuse_class); + /* + * Wait for pending call_rcu() callbacks that call back into + * this module via fc->release (cuse_fc_release). + */ + rcu_barrier(); } module_init(cuse_init); diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 5763a7cd3b37..4fec31fc0b84 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -75,17 +75,23 @@ void fuse_chan_set_initialized(struct fuse_chan *fch, struct fuse_chan_param *pa fch->minor = param->minor; fch->max_write = param->max_write; fch->max_pages = param->max_pages; + + if (param->io_uring_enabled) + fuse_uring_conn_init(fch); } - /* Make sure stores before this are seen on another CPU */ - smp_wmb(); - fch->initialized = 1; + /* Pairs with smp_load_acquire() readers of fch->initialized */ + smp_store_release(&fch->initialized, 1); wake_up_all(&fch->blocked_waitq); } static bool fuse_block_alloc(struct fuse_chan *fch, bool for_background) { - return !fch->initialized || (for_background && fch->blocked) || + /* Pairs with smp_store_release() in fuse_chan_set_initialized() */ + if (!smp_load_acquire(&fch->initialized)) + return true; + + return (for_background && fch->blocked) || (fch->io_uring && fch->connected && !fuse_uring_ready(fch)); } @@ -120,9 +126,6 @@ static struct fuse_req *fuse_get_req(struct fuse_chan *fch, bool for_background) goto out; } - /* Matches smp_wmb() in fuse_chan_set_initialized() */ - smp_rmb(); - err = -ENOTCONN; if (!fch->connected) goto out; @@ -210,10 +213,13 @@ EXPORT_SYMBOL_GPL(fuse_req_hash); /* * A new request is available, wake fiq->waitq */ -static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq) +static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq, bool sync) __releases(fiq->lock) { - wake_up(&fiq->waitq); + if (sync) + wake_up_sync(&fiq->waitq); + else + wake_up(&fiq->waitq); kill_fasync(&fiq->fasync, SIGIO, POLL_IN); spin_unlock(&fiq->lock); } @@ -230,7 +236,7 @@ void fuse_dev_queue_forget(struct fuse_iqueue *fiq, if (fiq->connected) { fiq->forget_list_tail->next = forget; fiq->forget_list_tail = forget; - fuse_dev_wake_and_unlock(fiq); + fuse_dev_wake_and_unlock(fiq, false); } else { kfree(forget); spin_unlock(&fiq->lock); @@ -240,7 +246,8 @@ void fuse_dev_queue_forget(struct fuse_iqueue *fiq, void fuse_dev_queue_interrupt(struct fuse_iqueue *fiq, struct fuse_req *req) { spin_lock(&fiq->lock); - if (list_empty(&req->intr_entry)) { + /* Repeat FR_SENT test after obtaining the lock to prevent race with fuse_resend() */ + if (list_empty(&req->intr_entry) && test_bit(FR_SENT, &req->flags)) { list_add_tail(&req->intr_entry, &fiq->interrupts); /* * Pairs with smp_mb() implied by test_and_set_bit() @@ -251,7 +258,7 @@ void fuse_dev_queue_interrupt(struct fuse_iqueue *fiq, struct fuse_req *req) list_del_init(&req->intr_entry); spin_unlock(&fiq->lock); } else { - fuse_dev_wake_and_unlock(fiq); + fuse_dev_wake_and_unlock(fiq, false); } } else { spin_unlock(&fiq->lock); @@ -281,11 +288,13 @@ EXPORT_SYMBOL_GPL(fuse_request_assign_unique); static void fuse_dev_queue_req(struct fuse_iqueue *fiq, struct fuse_req *req) { + bool sync = test_and_clear_bit(FR_SYNC_WAKEUP, &req->flags); + spin_lock(&fiq->lock); if (fiq->connected) { fuse_request_assign_unique_locked(fiq, req); list_add_tail(&req->list, &fiq->pending); - fuse_dev_wake_and_unlock(fiq); + fuse_dev_wake_and_unlock(fiq, sync); } else { spin_unlock(&fiq->lock); req->out.h.error = -ENOTCONN; @@ -397,7 +406,8 @@ void fuse_chan_max_background_set(struct fuse_chan *fch, unsigned int val) fch->max_background = val; fch->blocked = fch->num_background >= fch->max_background; if (!fch->blocked) - wake_up(&fch->blocked_waitq); + wake_up_nr(&fch->blocked_waitq, + fch->max_background - fch->num_background); spin_unlock(&fch->bg_lock); } @@ -411,11 +421,6 @@ void fuse_chan_set_fc(struct fuse_chan *fch, struct fuse_conn *fc) fch->conn = fc; } -void fuse_chan_io_uring_enable(struct fuse_chan *fch) -{ - fch->io_uring = 1; -} - void fuse_pqueue_init(struct fuse_pqueue *fpq) { spin_lock_init(&fpq->lock); @@ -725,7 +730,7 @@ static void request_wait_answer(struct fuse_req *req) if (req->args->abort_on_kill) { fuse_chan_abort(fch, false); - return; + goto wait_for_finish; } if (test_bit(FR_URING, &req->flags)) @@ -736,6 +741,7 @@ static void request_wait_answer(struct fuse_req *req) return; } +wait_for_finish: /* * Either request is already in userspace, or it was forced. * Wait it out. @@ -752,6 +758,11 @@ static void __fuse_request_send(struct fuse_req *req) /* acquire extra reference, since request is still needed after fuse_request_end() */ __fuse_get_request(req); + /* + * This is a synchronous request: the caller will block waiting for + * the answer. Hint the scheduler via wake_up_sync(). + */ + set_bit(FR_SYNC_WAKEUP, &req->flags); fuse_send_one(fiq, req); request_wait_answer(req); @@ -1249,11 +1260,25 @@ int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop, if (folio) { size = folio_size(folio); - if (zeroing && count < size) - folio_zero_range(folio, 0, size); + if (zeroing && count < size) { + /* + * When the copy is skipped the folio already holds the + * payload, so only the bytes outside [offset, offset + + * count) may be zeroed. + * + * Otherwise, the whole folio is cleared first so that a + * failed copy leaves zeros rather than stale folio + * contents. + */ + if (cs->skip_folio_copy) + folio_zero_segments(folio, 0, offset, + offset + count, size); + else + folio_zero_range(folio, 0, size); + } } - while (count) { + while (!cs->skip_folio_copy && count) { if (cs->write && cs->pipebufs && folio) { /* * Can't control lifetime of pipe buffers, so always @@ -1346,6 +1371,10 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs, for (i = 0; !err && i < numargs; i++) { struct fuse_arg *arg = &args[i]; if (i == numargs - 1 && argpages) + /* + * if cs->skip_folio_copy is set, this just does any + * needed zeroing. No copying is involved. + */ err = fuse_copy_folios(cs, arg->size, zeroing); else err = fuse_copy_one(cs, arg->value, arg->size); @@ -1760,7 +1789,7 @@ out: void fuse_chan_resend(struct fuse_chan *fch) { struct fuse_dev *fud; - struct fuse_req *req, *next; + struct fuse_req *req; struct fuse_iqueue *fiq = &fch->iq; LIST_HEAD(to_queue); unsigned int i; @@ -1775,24 +1804,20 @@ void fuse_chan_resend(struct fuse_chan *fch) struct fuse_pqueue *fpq = &fud->pq; spin_lock(&fpq->lock); - for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) - list_splice_tail_init(&fpq->processing[i], &to_queue); + for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) { + struct list_head *this_queue = &fpq->processing[i]; + + list_for_each_entry(req, this_queue, list) + clear_bit(FR_SENT, &req->flags); + list_splice_tail_init(this_queue, &to_queue); + } spin_unlock(&fpq->lock); } spin_unlock(&fch->lock); - list_for_each_entry_safe(req, next, &to_queue, list) { - set_bit(FR_PENDING, &req->flags); - clear_bit(FR_SENT, &req->flags); - /* mark the request as resend request */ - req->in.h.unique |= FUSE_UNIQUE_RESEND; - } - spin_lock(&fiq->lock); if (!fiq->connected) { spin_unlock(&fiq->lock); - list_for_each_entry(req, &to_queue, list) - clear_bit(FR_PENDING, &req->flags); fuse_dev_end_requests(&to_queue); return; } @@ -1801,12 +1826,16 @@ void fuse_chan_resend(struct fuse_chan *fch) * intr_entry on fiq->interrupts after the request is re-queued. */ list_for_each_entry(req, &to_queue, list) { + set_bit(FR_PENDING, &req->flags); + /* mark the request as resend request */ + req->in.h.unique |= FUSE_UNIQUE_RESEND; + if (test_bit(FR_INTERRUPTED, &req->flags)) list_del_init(&req->intr_entry); } /* iq and pq requests are both oldest to newest */ list_splice(&to_queue, &fiq->pending); - fuse_dev_wake_and_unlock(fiq); + fuse_dev_wake_and_unlock(fiq, false); } /* Look up request on processing list by unique ID */ @@ -1888,7 +1917,8 @@ static ssize_t fuse_dev_do_write(struct fuse_dev *fud, * initialized and connected state */ err = -EINVAL; - if (!fch->initialized || !fch->connected) + /* Pairs with smp_store_release() in fuse_chan_set_initialized() */ + if (!smp_load_acquire(&fch->initialized) || !fch->connected) goto copy_finish; /* Don't try to move folios (yet) */ diff --git a/fs/fuse/dev.h b/fs/fuse/dev.h index aed69fd14c41..8d25378c0918 100644 --- a/fs/fuse/dev.h +++ b/fs/fuse/dev.h @@ -22,6 +22,7 @@ struct fuse_chan_param { unsigned int minor; unsigned int max_write; unsigned int max_pages; + bool io_uring_enabled; }; struct fuse_chan *fuse_chan_new(void); @@ -34,7 +35,6 @@ void fuse_chan_max_background_set(struct fuse_chan *fch, unsigned int val); unsigned int fuse_chan_num_waiting(struct fuse_chan *fch); void fuse_chan_set_fc(struct fuse_chan *fch, struct fuse_conn *fc); void fuse_chan_set_initialized(struct fuse_chan *fch, struct fuse_chan_param *param); -void fuse_chan_io_uring_enable(struct fuse_chan *fch); ssize_t fuse_chan_send(struct fuse_chan *fch, struct fuse_args *args); int fuse_chan_send_bg(struct fuse_chan *fch, struct fuse_args *args, gfp_t gfp_flags); int fuse_chan_send_notify_reply(struct fuse_chan *fch, struct fuse_args *args, u64 unique); diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c index 77c8cec43d9c..e22a48c9a678 100644 --- a/fs/fuse/dev_uring.c +++ b/fs/fuse/dev_uring.c @@ -9,6 +9,7 @@ #include "dev_uring_i.h" #include "fuse_trace.h" +#include <linux/bitmap.h> #include <linux/fs.h> #include <linux/io_uring/cmd.h> @@ -21,6 +22,8 @@ MODULE_PARM_DESC(enable_uring, #define FUSE_URING_IOV_HEADERS 0 #define FUSE_URING_IOV_PAYLOAD 1 +#define FUSE_URING_ADD_QUEUE_FLAGS (FUSE_URING_ZERO_COPY) + bool fuse_uring_enabled(void) { return enable_uring; @@ -30,6 +33,11 @@ struct fuse_uring_pdu { struct fuse_ring_ent *ent; }; +struct fuse_zero_copy_bvs { + unsigned int nr_bvs; + struct bio_vec bvs[]; +}; + static const struct fuse_iqueue_ops fuse_io_uring_ops; enum fuse_uring_header_type { @@ -41,6 +49,32 @@ enum fuse_uring_header_type { FUSE_URING_HEADER_RING_ENT, }; +static inline bool bufpool_enabled(struct fuse_ring_queue *queue) +{ + return queue->payload_mode == FUSE_PAYLOAD_BUFPOOL; +} + +static inline bool bufpool_registered(struct fuse_ring_queue *queue) +{ + return queue->bufpool && queue->bufpool->registered; +} + +/* + * For a registered bufpool, every sqe that drives a payload import (REGISTER, + * COMMIT_AND_FETCH) must carry the registered buffer index of the pool. + * This also must be called from the command's issue handler, where cmd->sqe is + * still valid + */ +static inline bool fuse_uring_cmd_index_ok(struct io_uring_cmd *cmd, + struct fuse_ring_queue *queue) +{ + if (!bufpool_registered(queue)) + return true; + + return (cmd->flags & IORING_URING_CMD_FIXED) && + READ_ONCE(cmd->sqe->buf_index) == queue->bufpool->registered_index; +} + static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd, struct fuse_ring_ent *ring_ent) { @@ -86,8 +120,36 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue) } } +static bool can_zero_copy_req(struct fuse_ring_ent *ent, struct fuse_req *req) +{ + struct fuse_args *args = req->args; + + if (!ent->queue->zero_copy || !args->zero_copy) + return false; + + if (args->opcode != FUSE_READ && args->opcode != FUSE_WRITE) + return false; + + return args->in_pages || args->out_pages; +} + +static void zero_copy_unregister(struct io_uring_cmd *cmd, + struct fuse_ring_ent *ent, + unsigned int issue_flags) +{ + if (ent->zero_copied) { + int err = io_buffer_unregister(cmd, ent->zero_copy_index, + issue_flags); + + if (err) + pr_warn_ratelimited("qid=%d zero-copy unregister failed: %d\n", + ent->queue->qid, err); + ent->zero_copied = false; + } +} + static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req, - int error) + int error, unsigned int issue_flags) { struct fuse_ring_queue *queue = ent->queue; struct fuse_ring *ring = queue->ring; @@ -107,6 +169,8 @@ static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req, spin_unlock(&queue->lock); + zero_copy_unregister(ent->cmd, ent, issue_flags); + if (error) req->out.h.error = error; @@ -204,7 +268,7 @@ void fuse_uring_destruct(struct fuse_chan *fch) return; for (qid = 0; qid < ring->nr_queues; qid++) { - struct fuse_ring_queue *queue = ring->queues[qid]; + struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]); struct fuse_ring_ent *ent, *next; if (!queue) @@ -222,8 +286,9 @@ void fuse_uring_destruct(struct fuse_chan *fch) } kfree(queue->fpq.processing); + kfree(queue->bufpool); kfree(queue); - ring->queues[qid] = NULL; + WRITE_ONCE(ring->queues[qid], NULL); } kfree(ring->queues); @@ -238,7 +303,6 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch) { struct fuse_ring *ring; size_t nr_queues = num_possible_cpus(); - struct fuse_ring *res = NULL; size_t max_payload_size; ring = kzalloc_obj(*ring, GFP_KERNEL_ACCOUNT); @@ -258,12 +322,6 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch) spin_unlock(&fch->lock); goto out_err; } - if (fch->ring) { - /* race, another thread created the ring in the meantime */ - spin_unlock(&fch->lock); - res = fch->ring; - goto out_err; - } init_waitqueue_head(&ring->stop_waitq); @@ -278,11 +336,18 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch) out_err: kfree(ring->queues); kfree(ring); - return res; + return NULL; +} + +void fuse_uring_conn_init(struct fuse_chan *fch) +{ + if (fuse_uring_create(fch)) + fch->io_uring = 1; } static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring, - int qid) + int qid, bool zero_copy, + bool fail_if_exists) { struct fuse_chan *fch = ring->chan; struct fuse_ring_queue *queue; @@ -290,16 +355,17 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring, queue = kzalloc_obj(*queue, GFP_KERNEL_ACCOUNT); if (!queue) - return NULL; + return ERR_PTR(-ENOMEM); pq = fuse_pqueue_alloc(); if (!pq) { kfree(queue); - return NULL; + return ERR_PTR(-ENOMEM); } queue->qid = qid; queue->ring = ring; spin_lock_init(&queue->lock); + queue->zero_copy = zero_copy; INIT_LIST_HEAD(&queue->ent_avail_queue); INIT_LIST_HEAD(&queue->ent_commit_queue); @@ -316,14 +382,17 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring, if (ring->queues[qid]) { spin_unlock(&fch->lock); kfree(queue->fpq.processing); + kfree(queue->bufpool); kfree(queue); - return ring->queues[qid]; + return fail_if_exists ? ERR_PTR(-EEXIST) : ring->queues[qid]; } /* - * write_once and lock as the caller mostly doesn't take the lock at all + * fch->lock serializes concurrent creators for this qid. + * smp_store_release() are for the lockless readers who must see a + * fully initialized queue after &ring->queues[qid] is set */ - WRITE_ONCE(ring->queues[qid], queue); + smp_store_release(&ring->queues[qid], queue); spin_unlock(&fch->lock); return queue; @@ -434,7 +503,7 @@ static void fuse_uring_log_ent_state(struct fuse_ring *ring) struct fuse_ring_ent *ent; for (qid = 0; qid < ring->nr_queues; qid++) { - struct fuse_ring_queue *queue = ring->queues[qid]; + struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]); if (!queue) continue; @@ -643,30 +712,57 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent, return 0; } +static int fuse_uring_import_payload(struct fuse_ring_ent *ent, int dir, + struct iov_iter *iter, + unsigned int issue_flags) +{ + void __user *base = ent->payload.iov_base; + size_t len = ent->payload.iov_len; + int err = 0; + + if (!base) { + memset(iter, 0, sizeof(*iter)); + return 0; + } + + if (bufpool_registered(ent->queue)) + err = io_uring_cmd_import_fixed((u64)(uintptr_t)base, len, dir, + iter, ent->cmd, issue_flags); + else + err = import_ubuf(dir, base, len, iter); + + if (err) + pr_info_ratelimited("fuse: Import of user buffer failed\n"); + + return err; +} + static int setup_fuse_copy_state(struct fuse_copy_state *cs, - struct fuse_ring *ring, struct fuse_req *req, + struct fuse_req *req, struct fuse_ring_ent *ent, int dir, - struct iov_iter *iter) + struct iov_iter *iter, + unsigned int issue_flags) { int err; - err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter); - if (err) { - pr_info_ratelimited("fuse: Import of user buffer failed\n"); + err = fuse_uring_import_payload(ent, dir, iter, issue_flags); + if (err) return err; - } fuse_copy_init(cs, dir == ITER_DEST, iter); + if (ent->zero_copied) + cs->skip_folio_copy = true; + cs->is_uring = true; cs->req = req; return 0; } -static int fuse_uring_copy_from_ring(struct fuse_ring *ring, - struct fuse_req *req, - struct fuse_ring_ent *ent) +static int fuse_uring_copy_from_ring(struct fuse_req *req, + struct fuse_ring_ent *ent, + unsigned int issue_flags) { struct fuse_copy_state cs; struct fuse_args *args = req->args; @@ -679,7 +775,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring, if (err) return err; - err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_SOURCE, &iter); + err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter, + issue_flags); if (err) return err; @@ -688,11 +785,68 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring, return err; } +static void fuse_zero_copy_release(void *priv) +{ + struct fuse_zero_copy_bvs *zc_bvs = priv; + unsigned int i; + + for (i = 0; i < zc_bvs->nr_bvs; i++) + folio_put(page_folio(zc_bvs->bvs[i].bv_page)); + + kvfree(zc_bvs); +} + +static int fuse_uring_set_up_zero_copy(struct fuse_ring_ent *ent, + struct fuse_req *req, + unsigned int issue_flags) +{ + struct fuse_args_pages *ap; + int err, i, ddir = 0; + struct fuse_zero_copy_bvs *zc_bvs; + struct bio_vec *bvs; + + /* out_pages indicates a read, in_pages indicates a write */ + if (req->args->out_pages) + ddir |= IO_BUF_DEST; + if (req->args->in_pages) + ddir |= IO_BUF_SOURCE; + + ap = container_of(req->args, typeof(*ap), args); + + zc_bvs = kvmalloc_flex(*zc_bvs, bvs, ap->num_folios, + GFP_KERNEL_ACCOUNT); + if (!zc_bvs) + return -ENOMEM; + + zc_bvs->nr_bvs = ap->num_folios; + bvs = zc_bvs->bvs; + for (i = 0; i < ap->num_folios; i++) { + bvs[i].bv_page = folio_page(ap->folios[i], 0); + bvs[i].bv_offset = ap->descs[i].offset; + bvs[i].bv_len = ap->descs[i].length; + folio_get(ap->folios[i]); + } + + err = io_buffer_register_bvec(ent->cmd, bvs, ap->num_folios, + fuse_zero_copy_release, zc_bvs, + ddir, ent->zero_copy_index, + issue_flags); + if (err) { + fuse_zero_copy_release(zc_bvs); + return err; + } + + ent->zero_copied = true; + + return 0; +} + /* * Copy data from the req to the ring buffer */ -static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, - struct fuse_ring_ent *ent) +static int fuse_uring_args_to_ring(struct fuse_req *req, + struct fuse_ring_ent *ent, + unsigned int issue_flags) { struct fuse_copy_state cs; struct fuse_args *args = req->args; @@ -705,7 +859,15 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_DEST, &iter); + if (can_zero_copy_req(ent, req)) { + ent_in_out.flags |= FUSE_URING_ENT_ZERO_COPY; + err = fuse_uring_set_up_zero_copy(ent, req, issue_flags); + if (err) + return err; + } + + err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter, + issue_flags); if (err) return err; @@ -735,15 +897,32 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; + /* + * on a zero-copied write the pages are registered for the server to + * read via a fixed-buffer op rather than copied into the payload + * buffer, so copied_sz does not account for it. The server still needs + * the total inbound size to know how many bytes to read from the + * registered buffer, so add the page arg (always the last in-arg) back + * in + */ + if (cs.skip_folio_copy && args->in_pages) + ent_in_out.payload_sz += + args->in_args[args->in_numargs - 1].size; + + if (bufpool_enabled(ent->queue) && ent->payload.iov_base) + ent_in_out.offset = + (uintptr_t)ent->payload.iov_base - ent->queue->bufpool->base_uaddr; + return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT, &ent_in_out, sizeof(ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, - struct fuse_req *req) + struct fuse_req *req, + unsigned int issue_flags) { struct fuse_ring_queue *queue = ent->queue; - struct fuse_ring *ring = queue->ring; + struct fuse_in_header in_header; int err; err = -EIO; @@ -758,23 +937,124 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, return err; /* copy the request */ - err = fuse_uring_args_to_ring(ring, req, ent); + err = fuse_uring_args_to_ring(req, ent, issue_flags); if (unlikely(err)) { pr_info_ratelimited("Copy to ring failed: %d\n", err); return err; } /* copy fuse_in_header */ - return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->in.h, - sizeof(req->in.h)); + in_header = req->in.h; + return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &in_header, + sizeof(in_header)); +} + +static bool fuse_uring_req_has_copyable_payload(struct fuse_ring_ent *ent, + struct fuse_req *req) +{ + struct fuse_args *args = req->args; + + if (!can_zero_copy_req(ent, req)) + return args->in_numargs > 1 || args->out_numargs; + + /* + * the asymmetry between in_numargs > 2 and out_numargs > 1 is because + * the per-op header is extracted before fuse_copy_args() for inargs but + * not for outargs + */ + if ((args->in_numargs > 1) && (!args->in_pages || args->in_numargs > 2)) + return true; + if (args->out_numargs && (!args->out_pages || args->out_numargs > 1)) + return true; + + return false; +} + +static int fuse_uring_select_buffer(struct fuse_ring_ent *ent) +{ + struct fuse_ring_queue *queue = ent->queue; + struct fuse_bufpool *pool = queue->bufpool; + unsigned int id; + + lockdep_assert_held(&queue->lock); + + id = find_first_bit(pool->free_map, pool->nr_bufs); + if (id >= pool->nr_bufs) + return -ENOBUFS; + + WARN_ON_ONCE(ent->payload.iov_base); + __clear_bit(id, pool->free_map); + + ent->buf_id = id; + ent->payload.iov_base = + (void __user *)(pool->base_uaddr + id * pool->buf_size); + ent->payload.iov_len = pool->buf_size; + + return 0; +} + +static void fuse_uring_recycle_buffer(struct fuse_ring_ent *ent) +{ + struct iovec *ent_payload = &ent->payload; + struct fuse_ring_queue *queue = ent->queue; + struct fuse_bufpool *pool; + + lockdep_assert_held(&queue->lock); + + if (!bufpool_enabled(queue) || !ent_payload->iov_base) + return; + + pool = queue->bufpool; + + /* a buffer should never be recycled twice */ + WARN_ON_ONCE(test_bit(ent->buf_id, pool->free_map)); + __set_bit(ent->buf_id, pool->free_map); + + memset(ent_payload, 0, sizeof(*ent_payload)); + ent->buf_id = 0; +} + +static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent, + struct fuse_req *req) +{ + bool buffer_selected; + bool has_payload; + + if (!bufpool_enabled(ent->queue)) + return 0; + + buffer_selected = !!ent->payload.iov_base; + has_payload = fuse_uring_req_has_copyable_payload(ent, req); + + if (has_payload && !buffer_selected) + return fuse_uring_select_buffer(ent); + + if (!has_payload && buffer_selected) + fuse_uring_recycle_buffer(ent); + + return 0; +} + +static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent, + struct fuse_req *req) +{ + if (!bufpool_enabled(ent->queue)) + return 0; + + /* no payload to copy, can skip selecting a buffer */ + if (!fuse_uring_req_has_copyable_payload(ent, req)) + return 0; + + return fuse_uring_select_buffer(ent); } static int fuse_uring_prepare_send(struct fuse_ring_ent *ent, - struct fuse_req *req) + struct fuse_req *req, + unsigned int issue_flags) { int err; - err = fuse_uring_copy_to_ring(ent, req); + err = fuse_uring_copy_to_ring(ent, req, issue_flags); if (!err) { set_bit(FR_SENT, &req->flags); trace_fuse_request_sent(req); @@ -788,7 +1068,7 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent, ent->state = FRRS_INVALID; spin_unlock(&ent->queue->lock); - fuse_uring_req_end(ent, req, err); + fuse_uring_req_end(ent, req, err, issue_flags); } return err; @@ -856,9 +1136,12 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent) /* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); - if (req) - fuse_uring_add_req_to_ring_ent(ent, req); + if (!req || fuse_uring_next_req_update_buffer(ent, req)) { + fuse_uring_recycle_buffer(ent); + return NULL; + } + fuse_uring_add_req_to_ring_ent(ent, req); return req; } @@ -870,12 +1153,13 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent) static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, unsigned int issue_flags) { - struct fuse_ring *ring = ent->queue->ring; + struct fuse_out_header out_header; ssize_t err = -EFAULT; - if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h, - sizeof(req->out.h))) + if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &out_header, + sizeof(out_header))) goto out; + req->out.h = out_header; err = fuse_uring_out_header_has_err(&req->out.h, req); if (err) { @@ -883,9 +1167,9 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, goto out; } - err = fuse_uring_copy_from_ring(ring, req, ent); + err = fuse_uring_copy_from_ring(req, ent, issue_flags); out: - fuse_uring_req_end(ent, req, err); + fuse_uring_req_end(ent, req, err, issue_flags); } /* @@ -895,7 +1179,8 @@ out: * Else, there is no next fuse request and this returns false. */ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent, - struct fuse_ring_queue *queue) + struct fuse_ring_queue *queue, + unsigned int issue_flags) { int err; struct fuse_req *req; @@ -907,7 +1192,7 @@ retry: spin_unlock(&queue->lock); if (req) { - err = fuse_uring_prepare_send(ent, req); + err = fuse_uring_prepare_send(ent, req, issue_flags); if (err) goto retry; } @@ -967,7 +1252,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, if (qid >= ring->nr_queues) return -EINVAL; - queue = ring->queues[qid]; + queue = READ_ONCE(ring->queues[qid]); if (!queue) return err; fpq = &queue->fpq; @@ -981,6 +1266,11 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, return err; } + if (!fuse_uring_cmd_index_ok(cmd, queue)) { + spin_unlock(&queue->lock); + return -EINVAL; + } + /* Find a request based on the unique ID of the fuse request * This should get revised, as it needs a hash calculation and list * search. And full struct fuse_pqueue is needed (memory overhead). @@ -1002,8 +1292,14 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, if (err != 0) { pr_info_ratelimited("qid=%d commit_id %llu state %d", queue->qid, commit_id, ent->state); + fuse_uring_recycle_buffer(ent); spin_unlock(&queue->lock); - fuse_uring_req_end(ent, req, err); + /* + * Unregister any zero copyable pages since ent->cmd is null + * when it hits fuse_uring_req_end() in this path + */ + zero_copy_unregister(cmd, ent, issue_flags); + fuse_uring_req_end(ent, req, err, issue_flags); return err; } @@ -1019,8 +1315,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, * fuse requests would otherwise not get processed - committing * and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). + * + * If there is no next request or if all buffers are busy (if using a + * bufpool), the cmd is not returned to userspace. The entry is left + * available and the cmd only returns to userspace when there's a + * next request and an available buffer. */ - if (fuse_uring_get_next_fuse_req(ent, queue)) + if (fuse_uring_get_next_fuse_req(ent, queue, issue_flags)) fuse_uring_send(ent, cmd, 0, issue_flags); return 0; } @@ -1035,7 +1336,7 @@ static bool is_ring_ready(struct fuse_ring *ring, int current_qid) if (current_qid == qid) continue; - queue = ring->queues[qid]; + queue = READ_ONCE(ring->queues[qid]); if (!queue) { ready = false; break; @@ -1122,10 +1423,14 @@ static struct fuse_ring_ent * fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, struct fuse_ring_queue *queue) { + const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req); struct fuse_ring *ring = queue->ring; struct fuse_ring_ent *ent; struct iovec iov[FUSE_URING_IOV_SEGS]; struct iovec *headers, *payload; + unsigned int zero_copy_index; + int err; err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov); @@ -1135,6 +1440,10 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, return ERR_PTR(err); } + zero_copy_index = READ_ONCE(cmd_req->ent_zero_copy_buf_index); + if (zero_copy_index && !queue->zero_copy) + return ERR_PTR(-EINVAL); + err = -EINVAL; headers = &iov[FUSE_URING_IOV_HEADERS]; if (headers->iov_len < sizeof(struct fuse_uring_req_header)) { @@ -1143,11 +1452,29 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, } payload = &iov[FUSE_URING_IOV_PAYLOAD]; - if (payload->iov_len < ring->max_payload_sz) { - pr_info_ratelimited("Invalid req payload len %zu\n", - payload->iov_len); - return ERR_PTR(err); + + spin_lock(&queue->lock); + if (bufpool_enabled(queue)) { + if (payload->iov_base || payload->iov_len || + !fuse_uring_cmd_index_ok(cmd, queue)) { + spin_unlock(&queue->lock); + return ERR_PTR(err); + } + } else { + if (payload->iov_len < ring->max_payload_sz) { + spin_unlock(&queue->lock); + pr_info_ratelimited("Invalid req payload len %zu\n", + payload->iov_len); + return ERR_PTR(err); + } + if (queue->zero_copy) { + spin_unlock(&queue->lock); + pr_info_ratelimited("Can only use zero copy with bufpools\n"); + return ERR_PTR(err); + } + queue->payload_mode = FUSE_PAYLOAD_PER_ENT; } + spin_unlock(&queue->lock); err = -ENOMEM; ent = kzalloc_obj(*ent, GFP_KERNEL_ACCOUNT); @@ -1158,7 +1485,9 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, ent->queue = queue; ent->headers = headers->iov_base; - ent->payload = payload->iov_base; + if (queue->payload_mode == FUSE_PAYLOAD_PER_ENT) + ent->payload = *payload; + ent->zero_copy_index = zero_copy_index; atomic_inc(&ring->queue_refs); return ent; @@ -1176,26 +1505,21 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, struct fuse_ring *ring = smp_load_acquire(&fch->ring); struct fuse_ring_queue *queue; struct fuse_ring_ent *ent; - int err; unsigned int qid = READ_ONCE(cmd_req->qid); - err = -ENOMEM; - if (!ring) { - ring = fuse_uring_create(fch); - if (!ring) - return err; - } + if (!ring) + return -EINVAL; if (qid >= ring->nr_queues) { pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid); return -EINVAL; } - queue = ring->queues[qid]; + queue = READ_ONCE(ring->queues[qid]); if (!queue) { - queue = fuse_uring_create_queue(ring, qid); - if (!queue) - return err; + queue = fuse_uring_create_queue(ring, qid, false, false); + if (IS_ERR(queue)) + return PTR_ERR(queue); } /* @@ -1210,6 +1534,110 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, return fuse_uring_do_register(ent, cmd, issue_flags); } +static int fuse_uring_add_queue(struct io_uring_cmd *cmd, struct fuse_chan *fch) +{ + const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req); + struct fuse_ring *ring = smp_load_acquire(&fch->ring); + unsigned int qid = READ_ONCE(cmd_req->qid); + uint64_t flags = READ_ONCE(cmd_req->flags); + struct fuse_ring_queue *queue; + bool zero_copy = flags & FUSE_URING_ZERO_COPY; + + if (!ring) + return -EINVAL; + + if (qid >= ring->nr_queues) { + pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid); + return -EINVAL; + } + + if (flags & ~FUSE_URING_ADD_QUEUE_FLAGS) + return -EINVAL; + + if (zero_copy && !capable(CAP_SYS_ADMIN)) + return -EPERM; + + queue = fuse_uring_create_queue(ring, qid, zero_copy, true); + if (IS_ERR(queue)) + return PTR_ERR(queue); + + return 0; +} + +static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd, + struct fuse_chan *fch) +{ + const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req); + unsigned int qid = READ_ONCE(cmd_req->qid); + uint64_t flags = READ_ONCE(cmd_req->flags); + /* paired with the smp_store_release() in fuse_uring_create */ + struct fuse_ring *ring = smp_load_acquire(&fch->ring); + struct fuse_ring_queue *queue; + struct fuse_bufpool *pool; + uintptr_t pool_uaddr; + unsigned int pool_len, nr_bufs; + size_t pool_size, buf_size; + bool registered = cmd->flags & IORING_URING_CMD_FIXED; + + if (!ring || qid >= ring->nr_queues || flags) + return -EINVAL; + + /* reserved for future use, must be zero */ + if (READ_ONCE(cmd_req->bufpool.reserved)) + return -EINVAL; + + /* Pairs with smp_store_release() in fuse_uring_create_queue() */ + queue = smp_load_acquire(&ring->queues[qid]); + if (!queue) + return -EINVAL; + + pool_uaddr = READ_ONCE(cmd_req->bufpool.uaddr); + pool_len = READ_ONCE(cmd_req->bufpool.len); + + /* each buffer holds the max payload size */ + buf_size = queue->ring->max_payload_sz; + + nr_bufs = pool_len / buf_size; + if (!nr_bufs) + return -EINVAL; + + pool_size = struct_size(pool, free_map, BITS_TO_LONGS(nr_bufs)); + pool = kzalloc(pool_size, GFP_KERNEL_ACCOUNT); + if (!pool) + return -ENOMEM; + + pool->base_uaddr = pool_uaddr; + pool->buf_size = buf_size; + pool->nr_bufs = nr_bufs; + /* all buffers are free */ + bitmap_set(pool->free_map, 0, nr_bufs); + + /* + * A registered bufpool is reached through an io_uring fixed buffer, so + * the pool is registered iff this command was submitted with + * IORING_URING_CMD_FIXED. The registered buffer index is taken from + * sqe->buf_index. + */ + if (registered) { + pool->registered = true; + pool->registered_index = READ_ONCE(cmd->sqe->buf_index); + } + + spin_lock(&queue->lock); + if (queue->payload_mode != FUSE_PAYLOAD_UNSET) { + spin_unlock(&queue->lock); + kfree(pool); + return -EINVAL; + } + queue->bufpool = pool; + queue->payload_mode = FUSE_PAYLOAD_BUFPOOL; + spin_unlock(&queue->lock); + + return 0; +} + /* * Entry function from io_uring to handle the given passthrough command * (op code IORING_OP_URING_CMD) @@ -1251,8 +1679,10 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) /* * fuse_uring_register() needs the ring to be initialized, * we need to know the max payload size + * + * Pairs with smp_store_release() in fuse_chan_set_initialized() */ - if (!fch->initialized) + if (!smp_load_acquire(&fch->initialized)) return -EAGAIN; switch (cmd_op) { @@ -1274,6 +1704,18 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) return err; } break; + case FUSE_IO_URING_CMD_ADD_QUEUE: + err = fuse_uring_add_queue(cmd, fch); + if (err) + pr_info_once("FUSE_IO_URING_CMD_ADD_QUEUE failed err=%d\n", + err); + return err; + case FUSE_IO_URING_CMD_ADD_BUFPOOL: + err = fuse_uring_add_bufpool(cmd, fch); + if (err) + pr_info_once("FUSE_IO_URING_ADD_BUFPOOL failed err=%d\n", + err); + return err; default: return -EINVAL; } @@ -1295,9 +1737,10 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw) int err; if (!tw.cancel) { - err = fuse_uring_prepare_send(ent, ent->fuse_req); + err = fuse_uring_prepare_send(ent, ent->fuse_req, issue_flags); if (err) { - if (!fuse_uring_get_next_fuse_req(ent, queue)) + if (!fuse_uring_get_next_fuse_req(ent, queue, + issue_flags)) return; err = 0; } @@ -1307,11 +1750,12 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw) spin_lock(&queue->lock); list_del_init(&ent->list); + fuse_uring_recycle_buffer(ent); spin_unlock(&queue->lock); io_uring_cmd_done(cmd, err, issue_flags); - fuse_uring_req_end(ent, ent->fuse_req, err); + fuse_uring_req_end(ent, ent->fuse_req, err, issue_flags); kfree(ent); if (atomic_dec_and_test(&queue->ring->queue_refs)) wake_up_all(&queue->ring->stop_waitq); @@ -1330,7 +1774,7 @@ static struct fuse_ring_queue *fuse_uring_task_to_queue(struct fuse_ring *ring) ring->nr_queues)) qid = 0; - queue = ring->queues[qid]; + queue = READ_ONCE(ring->queues[qid]); WARN_ONCE(!queue, "Missing queue for qid %d\n", qid); return queue; @@ -1368,15 +1812,16 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req) req->ring_queue = queue; ent = list_first_entry_or_null(&queue->ent_avail_queue, struct fuse_ring_ent, list); - if (ent) - fuse_uring_add_req_to_ring_ent(ent, req); - else - list_add_tail(&req->list, &queue->fuse_req_queue); - spin_unlock(&queue->lock); - if (ent) - fuse_uring_dispatch_ent(ent); + if (!ent || fuse_uring_prep_buffer(ent, req)) { + list_add_tail(&req->list, &queue->fuse_req_queue); + spin_unlock(&queue->lock); + return; + } + fuse_uring_add_req_to_ring_ent(ent, req); + spin_unlock(&queue->lock); + fuse_uring_dispatch_ent(ent); return; err_unlock: @@ -1424,10 +1869,9 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req) */ req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req, list); - if (ent && req) { + if (ent && req && !fuse_uring_prep_buffer(ent, req)) { fuse_uring_add_req_to_ring_ent(ent, req); spin_unlock(&queue->lock); - fuse_uring_dispatch_ent(ent); } else { spin_unlock(&queue->lock); diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h index 55f8d04e4b0b..263d0f8b9714 100644 --- a/fs/fuse/dev_uring_i.h +++ b/fs/fuse/dev_uring_i.h @@ -7,6 +7,8 @@ #ifndef _FS_FUSE_DEV_URING_I_H #define _FS_FUSE_DEV_URING_I_H +#include <linux/uio.h> + #include "fuse_dev_i.h" #ifdef CONFIG_FUSE_IO_URING @@ -36,11 +38,50 @@ enum fuse_ring_req_state { FRRS_RELEASED, }; +/* how a queue's payload buffers are provided */ +enum fuse_queue_payload_mode { + /* not yet committed (a bufpool may still be added) */ + FUSE_PAYLOAD_UNSET = 0, + /* each entry registers its own payload buffer */ + FUSE_PAYLOAD_PER_ENT, + /* each entry's payload buffer is assigned from a bufpool */ + FUSE_PAYLOAD_BUFPOOL, +}; + +struct fuse_bufpool { + bool registered; + + /* + * io_uring registered buffer table index for this pool, bound at + * ADD_BUFPOOL time. Only valid if the bufpool is registered + */ + u16 registered_index; + + /* starting uaddr of the bufpool */ + uintptr_t base_uaddr; + + /* size of each buffer in the pool */ + size_t buf_size; + + /* total number of buffers in the pool */ + unsigned int nr_bufs; + + /* bitmap tracking which buffers are free */ + unsigned long free_map[]; +}; + /** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { /* userspace buffer */ struct fuse_uring_req_header __user *headers; - void __user *payload; + struct iovec payload; + + /* buffer id in the pool, if bufpools are used. ignored otherwise */ + unsigned int buf_id; + + /* true if the request's pages are being zero-copied */ + bool zero_copied; + unsigned int zero_copy_index; /* the ring queue that owns the request */ struct fuse_ring_queue *queue; @@ -99,6 +140,14 @@ struct fuse_ring_queue { unsigned int active_background; bool stopped; + + /* how this queue's payload buffers are provided */ + enum fuse_queue_payload_mode payload_mode; + + /* only allocated when payload_mode == FUSE_PAYLOAD_BUFPOOL */ + struct fuse_bufpool *bufpool; + + bool zero_copy; }; /* @@ -135,6 +184,7 @@ struct fuse_ring { bool ready; }; +void fuse_uring_conn_init(struct fuse_chan *fch); void fuse_uring_stop_queues(struct fuse_ring *ring); void fuse_uring_abort_end_requests(struct fuse_ring *ring); int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags); @@ -174,6 +224,10 @@ static inline bool fuse_uring_ready(struct fuse_chan *fch) #else /* CONFIG_FUSE_IO_URING */ +static inline void fuse_uring_conn_init(struct fuse_chan *fch) +{ +} + static inline void fuse_uring_abort(struct fuse_chan *fch) { } diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c index d4e0029810c0..24514d1309e1 100644 --- a/fs/fuse/dir.c +++ b/fs/fuse/dir.c @@ -34,7 +34,7 @@ struct dentry_bucket { #define FUSE_HASH_BITS 5 #define FUSE_HASH_SIZE (1 << FUSE_HASH_BITS) static struct dentry_bucket dentry_hash[FUSE_HASH_SIZE]; -struct delayed_work dentry_tree_work; +static struct delayed_work dentry_tree_work; /* Minimum invalidation work queue frequency */ #define FUSE_DENTRY_INVAL_FREQ_MIN 5 @@ -96,6 +96,7 @@ static void fuse_advise_use_readdirplus(struct inode *dir) struct fuse_dentry { u64 time; + u64 epoch; union { struct rcu_head rcu; struct rb_node node; @@ -236,6 +237,13 @@ void fuse_dentry_tree_cleanup(void) WARN_ON_ONCE(!RB_EMPTY_ROOT(&dentry_hash[i].tree)); } +void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch) +{ + struct fuse_dentry *fd = dentry->d_fsdata; + + fd->epoch = epoch; +} + static inline void __fuse_dentry_settime(struct dentry *dentry, u64 time) { ((struct fuse_dentry *) dentry->d_fsdata)->time = time; @@ -387,10 +395,11 @@ static int fuse_dentry_revalidate(struct inode *dir, const struct qstr *name, struct fuse_mount *fm; struct fuse_conn *fc; struct fuse_inode *fi; + struct fuse_dentry *fd = entry->d_fsdata; int ret; fc = get_fuse_conn_super(dir->i_sb); - if (entry->d_time < atomic_read(&fc->epoch)) + if (fd->epoch < atomic_read(&fc->epoch)) goto invalid; inode = d_inode_rcu(entry); @@ -480,10 +489,10 @@ static int fuse_dentry_init(struct dentry *dentry) RB_CLEAR_NODE(&fd->node); dentry->d_fsdata = fd; /* - * Initialising d_time (epoch) to '0' ensures the dentry is invalid + * Initialising epoch to '0' ensures the dentry is invalid * if compared to fc->epoch, which is initialized to '1'. */ - dentry->d_time = 0; + fuse_dentry_set_epoch(dentry, 0); return 0; } @@ -641,7 +650,7 @@ static struct dentry *fuse_lookup(struct inode *dir, struct dentry *entry, goto out_err; entry = newent ? newent : entry; - entry->d_time = epoch; + fuse_dentry_set_epoch(entry, epoch); if (outarg_valid) fuse_change_entry_timeout(entry, &outarg); else @@ -898,7 +907,7 @@ static int fuse_create_open(struct mnt_idmap *idmap, struct inode *dir, } kfree(forget); d_instantiate(entry, inode); - entry->d_time = epoch; + fuse_dentry_set_epoch(entry, epoch); fuse_change_entry_timeout(entry, &outentry); fuse_dir_changed(dir); err = generic_file_open(inode, file); @@ -1028,10 +1037,10 @@ static struct dentry *create_new_entry(struct mnt_idmap *idmap, struct fuse_moun return d; if (d) { - d->d_time = epoch; + fuse_dentry_set_epoch(d, epoch); fuse_change_entry_timeout(d, &outarg); } else { - entry->d_time = epoch; + fuse_dentry_set_epoch(entry, epoch); fuse_change_entry_timeout(entry, &outarg); } fuse_dir_changed(dir); @@ -2169,10 +2178,8 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, filemap_invalidate_lock(mapping); fault_blocked = true; err = fuse_dax_break_layouts(inode, 0, -1); - if (err) { - filemap_invalidate_unlock(mapping); - return err; - } + if (err) + goto unlock; } if (attr->ia_valid & ATTR_OPEN) { @@ -2199,7 +2206,7 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, ATTR_TIMES_SET)) { err = write_inode_now(inode, true); if (err) - return err; + goto unlock; fuse_set_nowrite(inode); fuse_release_nowrite(inode); @@ -2307,6 +2314,7 @@ error: clear_bit(FUSE_I_SIZE_UNSTABLE, &fi->state); +unlock: if (fault_blocked) filemap_invalidate_unlock(mapping); return err; diff --git a/fs/fuse/file.c b/fs/fuse/file.c index f2c081f09791..865382a9e9dc 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -272,7 +272,7 @@ static int fuse_open(struct inode *inode, struct file *file) filemap_invalidate_lock(inode->i_mapping); err = fuse_dax_break_layouts(inode, 0, -1); if (err) - goto out_inode_unlock; + goto out_unlock; } if (is_wb_truncate || dax_truncate) @@ -296,9 +296,9 @@ static int fuse_open(struct inode *inode, struct file *file) else if (!(ff->open_flags & FOPEN_KEEP_CACHE)) invalidate_inode_pages2(inode->i_mapping); } +out_unlock: if (dax_truncate) filemap_invalidate_unlock(inode->i_mapping); -out_inode_unlock: if (is_wb_truncate || dax_truncate) inode_unlock(inode); @@ -605,6 +605,7 @@ void fuse_read_args_fill(struct fuse_io_args *ia, struct file *file, loff_t pos, args->out_argvar = true; args->out_numargs = 1; args->out_args[0].size = count; + args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY; } static void fuse_release_user_pages(struct fuse_args_pages *ap, ssize_t nres, @@ -1153,6 +1154,7 @@ static void fuse_write_args_fill(struct fuse_io_args *ia, struct fuse_file *ff, args->out_numargs = 1; args->out_args[0].size = sizeof(ia->write.out); args->out_args[0].value = &ia->write.out; + args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY; } static unsigned int fuse_write_flags(struct kiocb *iocb) diff --git a/fs/fuse/fuse_dev_i.h b/fs/fuse/fuse_dev_i.h index 668c8391d61c..4b412a76225f 100644 --- a/fs/fuse/fuse_dev_i.h +++ b/fs/fuse/fuse_dev_i.h @@ -38,6 +38,8 @@ struct fuse_iqueue; * @FR_PRIVATE: request is on private list * @FR_ASYNC: request is asynchronous * @FR_URING: request is handled through fuse-io-uring + * @FR_SYNC_WAKEUP: use synchronous wakeup when queueing this request to + * give the scheduler a hint about the waker task */ enum fuse_req_flag { FR_ISREPLY, @@ -53,6 +55,7 @@ enum fuse_req_flag { FR_PRIVATE, FR_ASYNC, FR_URING, + FR_SYNC_WAKEUP, }; /** @@ -325,6 +328,8 @@ struct fuse_copy_state { bool write:1; bool move_folios:1; bool is_uring:1; + /* set when the payload is zero-copied. folios are filled in place */ + bool skip_folio_copy:1; struct { unsigned int copied_sz; /* copied size into the user buffer */ } ring; diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index 85f738c53122..c8d4c5f3af7e 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -1054,6 +1054,8 @@ u64 fuse_time_to_jiffies(u64 sec, u32 nsec); void fuse_change_entry_timeout(struct dentry *entry, struct fuse_entry_out *o); +void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch); + /* * Initialize fuse_conn */ diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index d975073c6029..1c6ee01c6796 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -791,6 +791,9 @@ static int fuse_opt_fd(struct fs_context *fsc, struct file *file) { struct fuse_fs_context *ctx = fsc->fs_private; + if (ctx->fud) + return invalfc(fsc, "Multiple fd specified"); + if (file->f_op != &fuse_dev_operations) return invalfc(fsc, "fd is not a fuse device"); /* @@ -1272,6 +1275,7 @@ static void process_init_reply(struct fuse_args *args, int error) struct fuse_mount *fm = ia->fm; struct fuse_conn *fc = fm->fc; struct fuse_init_out *arg = &ia->out; + bool io_uring_enabled = false; bool ok = true; if (error || arg->major != FUSE_KERNEL_VERSION) @@ -1402,7 +1406,7 @@ static void process_init_reply(struct fuse_args *args, int error) ok = false; } if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled()) - fuse_chan_io_uring_enable(fc->chan); + io_uring_enabled = true; if (flags & FUSE_REQUEST_TIMEOUT) timeout = arg->request_timeout; @@ -1416,6 +1420,7 @@ static void process_init_reply(struct fuse_args *args, int error) fm->sb->s_bdi->ra_pages = min(fm->sb->s_bdi->ra_pages, ra_pages); + fm->sb->s_bdi->io_pages = fc->max_pages; fc->minor = arg->minor; fc->max_write = arg->minor < 5 ? 4096 : arg->max_write; fc->max_write = max_t(unsigned, 4096, fc->max_write); @@ -1432,6 +1437,7 @@ static void process_init_reply(struct fuse_args *args, int error) .minor = fc->minor, .max_write = fc->max_write, .max_pages = fc->max_pages, + .io_uring_enabled = io_uring_enabled, }; fuse_chan_set_initialized(fc->chan, &cp); } @@ -1479,7 +1485,7 @@ static struct fuse_init_args *fuse_new_init(struct fuse_mount *fm) * the reply - server is either sending IORING_OP_URING_CMD or not. */ if (fuse_uring_enabled()) - flags |= FUSE_OVER_IO_URING; + flags |= FUSE_OVER_IO_URING | FUSE_HAS_IO_URING_BUFPOOL; ia->in.flags = flags; ia->in.flags2 = flags >> 32; @@ -1639,6 +1645,8 @@ static int fuse_fill_super_submount(struct super_block *sb, fuse_fill_attr_from_inode(&root_attr, parent_fi); root = fuse_iget(sb, parent_fi->nodeid, 0, &root_attr, 0, 0, fuse_get_evict_ctr(fm->fc)); + if (!root) + return -ENOMEM; /* * This inode is just a duplicate, so it is not looked up and * its nlookup should not be incremented. fuse_iget() does diff --git a/fs/fuse/readdir.c b/fs/fuse/readdir.c index 0e1321491747..5ca87151d70d 100644 --- a/fs/fuse/readdir.c +++ b/fs/fuse/readdir.c @@ -260,7 +260,7 @@ retry: } if (fc->readdirplus_auto) set_bit(FUSE_I_INIT_RDPLUS, &get_fuse_inode(inode)->state); - dentry->d_time = epoch; + fuse_dentry_set_epoch(dentry, epoch); fuse_change_entry_timeout(dentry, o); dput(dentry); diff --git a/fs/fuse/req_timeout.c b/fs/fuse/req_timeout.c index 6cc6fc491343..95a1acd7bc08 100644 --- a/fs/fuse/req_timeout.c +++ b/fs/fuse/req_timeout.c @@ -128,18 +128,12 @@ static void set_request_timeout(struct fuse_chan *fch, unsigned int timeout) void fuse_init_server_timeout(struct fuse_chan *fch, unsigned int timeout) { - if (!timeout && !fuse_max_req_timeout && !fuse_default_req_timeout) - return; - if (!timeout) timeout = fuse_default_req_timeout; - if (fuse_max_req_timeout) { - if (timeout) - timeout = min(fuse_max_req_timeout, timeout); - else - timeout = fuse_max_req_timeout; - } + timeout = min_not_zero(timeout, fuse_max_req_timeout); + if (!timeout) + return; timeout = max(FUSE_TIMEOUT_TIMER_FREQ, timeout); diff --git a/fs/lockd/clntproc.c b/fs/lockd/clntproc.c index f06faf577cea..f8018bfe9c64 100644 --- a/fs/lockd/clntproc.c +++ b/fs/lockd/clntproc.c @@ -487,9 +487,12 @@ static const struct file_lock_operations nlmclnt_lock_ops = { static void nlmclnt_locks_init_private(struct file_lock *fl, struct nlm_host *host) { fl->fl_u.nfs_fl.state = 0; + fl->fl_ops = NULL; fl->fl_u.nfs_fl.owner = nlmclnt_find_lockowner(host, fl->c.flc_owner); INIT_LIST_HEAD(&fl->fl_u.nfs_fl.list); + if (!fl->fl_u.nfs_fl.owner) + return; fl->fl_ops = &nlmclnt_lock_ops; } diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index 368d20daf67b..c926b7e43827 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -86,15 +86,17 @@ bl_free_device(struct pnfs_block_dev *dev) { bl_unregister_dev(dev); - if (dev->nr_children) { + if (dev->children) { int i; for (i = 0; i < dev->nr_children; i++) bl_free_device(&dev->children[i]); kfree(dev->children); - } else { - if (dev->bdev_file) - fput(dev->bdev_file); + dev->children = NULL; + dev->nr_children = 0; + } else if (dev->bdev_file) { + fput(dev->bdev_file); + dev->bdev_file = NULL; } } @@ -444,6 +446,7 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, out_blkdev_put: fput(d->bdev_file); + d->bdev_file = NULL; return error; } @@ -479,8 +482,11 @@ bl_parse_concat(struct nfs_server *server, struct pnfs_block_dev *d, for (i = 0; i < v->concat.volumes_count; i++) { ret = bl_parse_deviceid(server, &d->children[i], volumes, v->concat.volumes[i], gfp_mask); - if (ret) + if (ret) { + bl_free_device(&d->children[i]); + bl_free_device(d); return ret; + } d->nr_children++; d->children[i].start += len; @@ -508,8 +514,11 @@ bl_parse_stripe(struct nfs_server *server, struct pnfs_block_dev *d, for (i = 0; i < v->stripe.volumes_count; i++) { ret = bl_parse_deviceid(server, &d->children[i], volumes, v->stripe.volumes[i], gfp_mask); - if (ret) + if (ret) { + bl_free_device(&d->children[i]); + bl_free_device(d); return ret; + } d->nr_children++; len += d->children[i].len; diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c index 10f2354ba304..3fb10c8e4271 100644 --- a/fs/nfs/callback_proc.c +++ b/fs/nfs/callback_proc.c @@ -291,7 +291,8 @@ static u32 initiate_file_draining(struct nfs_client *clp, pnfs_set_layout_stateid(lo, &args->cbl_stateid, NULL, true); switch (pnfs_mark_matching_lsegs_return(lo, &free_me_list, &args->cbl_range, - be32_to_cpu(args->cbl_stateid.seqid))) { + be32_to_cpu(args->cbl_stateid.seqid), + args->cbl_layoutchanged)) { case 0: case -EBUSY: /* There are layout segments that need to be returned */ @@ -317,7 +318,7 @@ out: nfs_iput_and_deactive(ino); out_noput: trace_nfs4_cb_layoutrecall_file(clp, &args->cbl_fh, ino, - &args->cbl_stateid, -rv); + &args->cbl_stateid, args->cbl_layoutchanged, -rv); return rv; } diff --git a/fs/nfs/client.c b/fs/nfs/client.c index 4dcb91ab3039..60386330aeec 100644 --- a/fs/nfs/client.c +++ b/fs/nfs/client.c @@ -215,9 +215,21 @@ static void nfs_cb_idr_remove_locked(struct nfs_client *clp) { struct nfs_net *nn = net_generic(clp->cl_net, nfs_net_id); - if (clp->cl_cb_ident) + if (clp->cl_cb_ident) { idr_remove(&nn->cb_ident_idr, clp->cl_cb_ident); + clp->cl_cb_ident = 0; + } +} + +void nfs_cb_idr_remove(struct nfs_client *clp) +{ + struct nfs_net *nn = net_generic(clp->cl_net, nfs_net_id); + + spin_lock(&nn->nfs_client_lock); + nfs_cb_idr_remove_locked(clp); + spin_unlock(&nn->nfs_client_lock); } +EXPORT_SYMBOL_GPL(nfs_cb_idr_remove); static void pnfs_init_server(struct nfs_server *server) { diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c index 9546d2195c25..ab3e441464a6 100644 --- a/fs/nfs/delegation.c +++ b/fs/nfs/delegation.c @@ -447,11 +447,14 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred, struct nfs_inode *nfsi = NFS_I(inode); struct nfs_delegation *delegation, *old_delegation; struct nfs_delegation *freeme = NULL; + bool orphaned = false; int status = 0; delegation = kmalloc_obj(*delegation, GFP_KERNEL_ACCOUNT); - if (delegation == NULL) + if (delegation == NULL) { + nfs4_proc_delegreturn(inode, cred, stateid, NULL, 0); return -ENOMEM; + } nfs4_stateid_copy(&delegation->stateid, stateid); refcount_set(&delegation->refcount, 1); delegation->type = type; @@ -500,11 +503,15 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred, goto out; } if (test_and_set_bit(NFS_DELEGATION_RETURNING, - &old_delegation->flags)) + &old_delegation->flags)) { + orphaned = true; goto out; + } } - if (!nfs_detach_delegations_locked(nfsi, old_delegation, clp)) + if (!nfs_detach_delegations_locked(nfsi, old_delegation, clp)) { + orphaned = true; goto out; + } freeme = old_delegation; add_new: /* @@ -539,8 +546,11 @@ add_new: nfs_update_delegated_mtime(inode); out: spin_unlock(&clp->cl_lock); - if (delegation != NULL) + if (delegation != NULL) { + if (orphaned) + nfs_do_return_delegation(inode, delegation, 0); __nfs_free_delegation(delegation); + } if (freeme != NULL) { nfs_do_return_delegation(inode, freeme, 0); nfs_mark_delegation_revoked(server, freeme); @@ -594,7 +604,7 @@ delay: spin_lock(&server->delegations_lock); if (list_empty(&delegation->entry)) refcount_inc(&delegation->refcount); - list_move_tail(&delegation->entry, &server->delegations_return); + list_move_tail(&delegation->entry, &server->delegations_delayed); spin_unlock(&server->delegations_lock); set_bit(NFS4CLNT_DELEGRETURN_DELAYED, &server->nfs_client->cl_state); abort: diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c index 36f2e8588922..49394123bd09 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -2208,6 +2208,10 @@ int nfs_atomic_open(struct inode *dir, struct dentry *dentry, goto out; } file->f_mode |= FMODE_CAN_ODIRECT; + if (test_bit(NFS_CONTEXT_O_DIRECT, &ctx->flags)) { + file->f_flags |= O_DIRECT; + open_flags |= O_DIRECT; + } err = nfs_finish_open(ctx, ctx->dentry, file, open_flags); trace_nfs_atomic_open_exit(dir, ctx, open_flags, err); @@ -2319,6 +2323,13 @@ int nfs_atomic_open_v23(struct inode *dir, struct dentry *dentry, if (open_flags & O_CREAT) { error = nfs_do_create(dir, dentry, mode, open_flags); if (!error) { + /* With UNCHECKED mode, a server may return NFS3_OK for + * a pre-existing non-regular file (e.g. a symlink). + * Let the VFS handle it; calling finish_open() would + * hit no_open() and return -ENXIO. + */ + if (!d_is_reg(dentry)) + return finish_no_open(file, NULL); file->f_mode |= FMODE_CREATED; return finish_open(file, dentry, NULL); } else if (error != -EEXIST || open_flags & O_EXCL) @@ -2669,6 +2680,12 @@ int nfs_symlink(struct mnt_idmap *idmap, struct inode *dir, return error; } + if (unlikely(!d_is_symlink(dentry))) { + d_drop(dentry); + folio_put(folio); + return 0; + } + nfs_set_verifier(dentry, nfs_save_change_attribute(dir)); /* diff --git a/fs/nfs/filelayout/filelayoutdev.c b/fs/nfs/filelayout/filelayoutdev.c index 7226989ee4d5..d06d303fdcc3 100644 --- a/fs/nfs/filelayout/filelayoutdev.c +++ b/fs/nfs/filelayout/filelayoutdev.c @@ -280,7 +280,7 @@ nfs4_fl_prepare_ds(struct pnfs_layout_segment *lseg, u32 ds_idx) status = nfs4_pnfs_ds_connect(s, ds, devid, dataserver_timeo, dataserver_retrans, 4, - s->nfs_client->cl_minorversion); + s->nfs_client->cl_minorversion, true); if (status) { nfs4_mark_deviceid_unavailable(devid); ret = NULL; diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c index c4aa995026f6..7fe8b91fa47c 100644 --- a/fs/nfs/flexfilelayout/flexfilelayout.c +++ b/fs/nfs/flexfilelayout/flexfilelayout.c @@ -1322,7 +1322,8 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task, struct pnfs_layout_hdr *lo = lseg->pls_layout; struct inode *inode = lo->plh_inode; struct nfs4_deviceid_node *devid = FF_LAYOUT_DEVID_NODE(lseg, idx, dss_id); - struct nfs4_slot_table *tbl = &clp->cl_session->fc_slot_table; + struct nfs4_slot_table *tbl = nfs4_has_session(clp) ? + &clp->cl_session->fc_slot_table : clp->cl_slot_tbl; switch (op_status) { case NFS4_OK: @@ -1543,6 +1544,17 @@ static void ff_layout_io_track_ds_error(struct pnfs_layout_segment *lseg, case -EACCES: *op_status = status = NFS4ERR_ACCESS; break; + case -ECANCELED: + /* + * In-flight I/O we cancelled to return a recalled or + * revoked layout. Report it as a failure to reach the + * device (NFS4ERR_NXIO), like the transport errors + * above, so the server can reconcile the affected mirror + * instance. We aborted the I/O ourselves rather than + * observe the device fail, so don't condemn it below. + */ + *op_status = status = NFS4ERR_NXIO; + break; default: return; } @@ -1553,6 +1565,15 @@ static void ff_layout_io_track_ds_error(struct pnfs_layout_segment *lseg, mirror, dss_id, offset, length, status, opnum, nfs_io_gfp_mask()); + /* + * I/O we cancelled ourselves to return a recalled or revoked layout + * is reported above so the server can reconcile the mirror, but we + * have no evidence the device is at fault: don't mark it unreachable + * or force a return. + */ + if (error == -ECANCELED) + goto out; + switch (status) { case NFS4ERR_DELAY: case NFS4ERR_GRACE: @@ -1572,6 +1593,7 @@ static void ff_layout_io_track_ds_error(struct pnfs_layout_segment *lseg, lseg); } +out: dprintk("%s: err %d op %d status %u\n", __func__, err, opnum, status); } @@ -2462,7 +2484,7 @@ static void ff_layout_cancel_io(struct pnfs_layout_segment *lseg) clnt = ds_clp->cl_rpcclient; if (!clnt) continue; - if (!rpc_cancel_tasks(clnt, -EAGAIN, + if (!rpc_cancel_tasks(clnt, -ECANCELED, ff_layout_match_io, lseg)) continue; rpc_clnt_disconnect(clnt); diff --git a/fs/nfs/flexfilelayout/flexfilelayoutdev.c b/fs/nfs/flexfilelayout/flexfilelayoutdev.c index 1109462a9699..8be5c730e101 100644 --- a/fs/nfs/flexfilelayout/flexfilelayoutdev.c +++ b/fs/nfs/flexfilelayout/flexfilelayoutdev.c @@ -399,7 +399,8 @@ nfs4_ff_layout_prepare_ds(struct pnfs_layout_segment *lseg, status = nfs4_pnfs_ds_connect(s, ds, &mirror->dss[dss_id].mirror_ds->id_node, dataserver_timeo, dataserver_retrans, mirror->dss[dss_id].mirror_ds->ds_versions[0].version, - mirror->dss[dss_id].mirror_ds->ds_versions[0].minor_version); + mirror->dss[dss_id].mirror_ds->ds_versions[0].minor_version, + mirror->dss[dss_id].mirror_ds->ds_versions[0].tightly_coupled); /* connect success, check rsize/wsize limit */ if (!status) { diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c index 5bcd4027d203..3022454f7698 100644 --- a/fs/nfs/inode.c +++ b/fs/nfs/inode.c @@ -507,6 +507,7 @@ nfs_fhget(struct super_block *sb, struct nfs_fh *fh, struct nfs_fattr *fattr) inode->i_blocks = 0; nfsi->write_io = 0; nfsi->read_io = 0; + nfsi->uncacheable_file_data = false; nfsi->read_cache_jiffies = fattr->time_start; nfsi->attr_gencount = fattr->gencount; @@ -561,6 +562,12 @@ nfs_fhget(struct super_block *sb, struct nfs_fh *fh, struct nfs_fattr *fattr) } else if (fattr_supported & NFS_ATTR_FATTR_SPACE_USED && fattr->size != 0) nfs_set_cache_invalid(inode, NFS_INO_INVALID_BLOCKS); + if (fattr->valid & NFS_ATTR_FATTR_UNCACHEABLE_FILE_DATA) + nfsi->uncacheable_file_data = + fattr->aux_flags & NFS_AUX_UNCACHEABLE_FILE_DATA; + else if (S_ISREG(inode->i_mode) && + (fattr_supported & NFS_ATTR_FATTR_UNCACHEABLE_FILE_DATA)) + nfs_set_cache_invalid(inode, NFS_INO_INVALID_UNCACHEABLE_FILE_DATA); nfs_setsecurity(inode, fattr); @@ -1531,9 +1538,7 @@ int nfs_clear_invalid_mapping(struct address_space *mapping) ret = nfs_invalidate_mapping(inode, mapping); trace_nfs_invalidate_mapping_exit(inode, ret); - clear_bit_unlock(NFS_INO_INVALIDATING, bitlock); - smp_mb__after_atomic(); - wake_up_bit(bitlock, NFS_INO_INVALIDATING); + clear_and_wake_up_bit(NFS_INO_INVALIDATING, bitlock); out: return ret; } @@ -1975,7 +1980,8 @@ static int nfs_inode_finish_partial_attr_update(const struct nfs_fattr *fattr, NFS_INO_INVALID_ATIME | NFS_INO_INVALID_CTIME | NFS_INO_INVALID_MTIME | NFS_INO_INVALID_SIZE | NFS_INO_INVALID_BLOCKS | NFS_INO_INVALID_OTHER | - NFS_INO_INVALID_NLINK | NFS_INO_INVALID_BTIME; + NFS_INO_INVALID_NLINK | NFS_INO_INVALID_BTIME | + NFS_INO_INVALID_UNCACHEABLE_FILE_DATA; unsigned long cache_validity = NFS_I(inode)->cache_validity; enum nfs4_change_attr_type ctype = NFS_SERVER(inode)->change_attr_type; @@ -2297,7 +2303,8 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) nfsi->cache_validity &= ~(NFS_INO_INVALID_ATTR | NFS_INO_INVALID_ATIME | NFS_INO_REVAL_FORCED - | NFS_INO_INVALID_BLOCKS); + | NFS_INO_INVALID_BLOCKS + | NFS_INO_INVALID_UNCACHEABLE_FILE_DATA); /* Do atomic weak cache consistency updates */ nfs_wcc_update_inode(inode, fattr); @@ -2337,7 +2344,8 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) | NFS_INO_INVALID_NLINK | NFS_INO_INVALID_MODE | NFS_INO_INVALID_OTHER - | NFS_INO_INVALID_BTIME; + | NFS_INO_INVALID_BTIME + | NFS_INO_INVALID_UNCACHEABLE_FILE_DATA; if (S_ISDIR(inode->i_mode)) nfs_force_lookup_revalidate(inode); attr_changed = true; @@ -2461,6 +2469,14 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) nfsi->cache_validity |= save_cache_validity & NFS_INO_INVALID_BLOCKS; + if (fattr->valid & NFS_ATTR_FATTR_UNCACHEABLE_FILE_DATA) + nfsi->uncacheable_file_data = + fattr->aux_flags & NFS_AUX_UNCACHEABLE_FILE_DATA; + else if (S_ISREG(inode->i_mode) && + (fattr_supported & NFS_ATTR_FATTR_UNCACHEABLE_FILE_DATA)) + nfsi->cache_validity |= + save_cache_validity & NFS_INO_INVALID_UNCACHEABLE_FILE_DATA; + /* Update attrtimeo value if we're out of the unstable period */ if (attr_changed) { nfs_inc_stats(inode, NFSIOS_ATTRINVALIDATE); diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h index 7f96a258af76..abc81f5ae578 100644 --- a/fs/nfs/internal.h +++ b/fs/nfs/internal.h @@ -225,6 +225,7 @@ void nfs_server_copy_userdata(struct nfs_server *, struct nfs_server *); extern void nfs_put_client(struct nfs_client *); extern void nfs_free_client(struct nfs_client *); +void nfs_cb_idr_remove(struct nfs_client *clp); extern struct nfs_client *nfs4_find_client_ident(struct net *, int); extern struct nfs_client * nfs4_find_client_sessionid(struct net *, const struct sockaddr *, @@ -250,7 +251,8 @@ extern struct nfs_client *nfs4_set_ds_client(struct nfs_server *mds_srv, int ds_addrlen, int ds_proto, unsigned int ds_timeo, unsigned int ds_retrans, - u32 minor_version); + u32 minor_version, + bool tightly_coupled); extern struct rpc_clnt *nfs4_find_or_create_ds_client(struct nfs_client *, struct inode *); extern void nfs4_session_limit_rwsize(struct nfs_server *server); @@ -479,7 +481,7 @@ extern int nfs_local_doio(struct nfs_client *, const struct rpc_call_ops *); extern int nfs_local_commit(struct nfsd_file *, struct nfs_commit_data *, - const struct rpc_call_ops *, int); + const struct rpc_call_ops *); extern bool nfs_server_is_local(const struct nfs_client *clp); #else /* CONFIG_NFS_LOCALIO */ @@ -501,7 +503,7 @@ static inline int nfs_local_doio(struct nfs_client *clp, } static inline int nfs_local_commit(struct nfsd_file *localio, struct nfs_commit_data *data, - const struct rpc_call_ops *call_ops, int how) + const struct rpc_call_ops *call_ops) { return -EINVAL; } diff --git a/fs/nfs/localio.c b/fs/nfs/localio.c index e55c5977fcc3..f42b6112a613 100644 --- a/fs/nfs/localio.c +++ b/fs/nfs/localio.c @@ -52,7 +52,6 @@ struct nfs_local_fsync_ctx { struct nfsd_file *localio; struct nfs_commit_data *data; struct work_struct work; - struct completion *done; }; static bool localio_enabled __read_mostly = true; @@ -699,6 +698,29 @@ static void nfs_local_call_read(struct work_struct *work) } } +/* + * Decide whether LOCALIO must defer submission to the dedicated + * !WQ_MEM_RECLAIM nfslocaliod_workqueue rather than issue the IO inline. + * + * LOCALIO issues IO directly into a stacked local filesystem (e.g. XFS), + * which may in turn flush its own !WQ_MEM_RECLAIM workqueue. Doing so from a + * memory-reclaim context -- either a WQ_MEM_RECLAIM worker (most importantly + * writeback's wb_workfn running on bdi_wq) or an explicit reclaim task + * (PF_MEMALLOC) -- would trip check_flush_dependency() and risks a + * forward-progress deadlock; see commit b9f5dd57f4a5 ("nfs/localio: use + * dedicated workqueues for filesystem read and write"). In that case defer + * to nfslocaliod_workqueue. + * + * Otherwise (ordinary application/task context, e.g. O_DIRECT or fsync-driven + * submission) issue the IO inline: this preserves the NFS client's inherent + * application-context parallelism and avoids the per-IO workqueue hop. + */ +static inline bool nfs_local_defer_io(void) +{ + return (current->flags & PF_MEMALLOC) || + current_is_workqueue_mem_reclaim(); +} + static void nfs_local_do_read(struct nfs_local_kiocb *iocb, const struct rpc_call_ops *call_ops) { @@ -711,7 +733,10 @@ static void nfs_local_do_read(struct nfs_local_kiocb *iocb, hdr->res.eof = false; INIT_WORK(&iocb->work, nfs_local_call_read); - queue_work(nfslocaliod_workqueue, &iocb->work); + if (nfs_local_defer_io()) + queue_work(nfslocaliod_workqueue, &iocb->work); + else + nfs_local_call_read(&iocb->work); } static void @@ -929,7 +954,10 @@ static void nfs_local_do_write(struct nfs_local_kiocb *iocb, nfs_set_local_verifier(hdr->inode, hdr->res.verf, hdr->args.stable); INIT_WORK(&iocb->work, nfs_local_call_write); - queue_work(nfslocaliod_workqueue, &iocb->work); + if (nfs_local_defer_io()) + queue_work(nfslocaliod_workqueue, &iocb->work); + else + nfs_local_call_write(&iocb->work); } static struct nfs_local_kiocb * @@ -1071,8 +1099,6 @@ nfs_local_fsync_work(struct work_struct *work) status = nfs_local_run_commit(nfs_to->nfsd_file_file(ctx->localio), ctx->data); nfs_local_commit_done(ctx->data, status); - if (ctx->done != NULL) - complete(ctx->done); nfs_local_fsync_ctx_free(ctx); current->flags = old_flags; @@ -1088,14 +1114,13 @@ nfs_local_fsync_ctx_alloc(struct nfs_commit_data *data, ctx->localio = localio; ctx->data = data; INIT_WORK(&ctx->work, nfs_local_fsync_work); - ctx->done = NULL; } return ctx; } int nfs_local_commit(struct nfsd_file *localio, struct nfs_commit_data *data, - const struct rpc_call_ops *call_ops, int how) + const struct rpc_call_ops *call_ops) { struct nfs_local_fsync_ctx *ctx; @@ -1108,13 +1133,18 @@ int nfs_local_commit(struct nfsd_file *localio, nfs_local_init_commit(data, call_ops); - if (how & FLUSH_SYNC) { - DECLARE_COMPLETION_ONSTACK(done); - ctx->done = &done; - queue_work(nfslocaliod_workqueue, &ctx->work); - wait_for_completion(&done); - } else + /* + * Run the commit (fsync) inline when not in a memory-reclaim context, + * rather than bouncing through nfslocaliod_workqueue; see + * nfs_local_defer_io(). Completion (nfs_commit_release_pages -> + * nfs_commit_end) then runs synchronously, which higher layers cope + * with: __nfs_commit_inode() dispatches async and waits via + * wait_on_commit(). + */ + if (nfs_local_defer_io()) queue_work(nfslocaliod_workqueue, &ctx->work); + else + nfs_local_fsync_work(&ctx->work); return 0; } diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c index 71c271a1700a..b661f446ea49 100644 --- a/fs/nfs/nfs4client.c +++ b/fs/nfs/nfs4client.c @@ -261,6 +261,7 @@ struct nfs_client *nfs4_alloc_client(const struct nfs_client_initdata *cl_init) return clp; error: + nfs_cb_idr_remove(clp); nfs_free_client(clp); return ERR_PTR(err); } @@ -791,7 +792,7 @@ static int nfs4_set_client(struct nfs_server *server, struct nfs_client *nfs4_set_ds_client(struct nfs_server *mds_srv, const struct sockaddr_storage *ds_addr, int ds_addrlen, int ds_proto, unsigned int ds_timeo, unsigned int ds_retrans, - u32 minor_version) + u32 minor_version, bool tightly_coupled) { struct rpc_timeout ds_timeout; struct nfs_client *mds_clp = mds_srv->nfs_client; @@ -838,7 +839,8 @@ struct nfs_client *nfs4_set_ds_client(struct nfs_server *mds_srv, if (test_bit(NFS_CS_NETUNREACH_FATAL, &mds_clp->cl_flags)) __set_bit(NFS_CS_NETUNREACH_FATAL, &cl_init.init_flags); - __set_bit(NFS_CS_PNFS, &cl_init.init_flags); + if (tightly_coupled) + __set_bit(NFS_CS_PNFS, &cl_init.init_flags); cl_init.max_connect = NFS_MAX_TRANSPORTS; /* * Set an authflavor equual to the MDS value. Use the MDS nfs_client @@ -915,20 +917,22 @@ static int nfs4_server_common_setup(struct nfs_server *server, return error; /* data servers support only a subset of NFSv4.1 */ - if (is_ds_only_client(server->nfs_client)) - return -EPROTONOSUPPORT; + if (is_ds_only_client(server->nfs_client)) { + error = -EPROTONOSUPPORT; + goto out_free_delegation_hash; + } /* We must ensure the session is initialised first */ error = nfs4_init_session(server->nfs_client); if (error < 0) - return error; + goto out_free_delegation_hash; nfs_server_set_init_caps(server); /* Probe the root fh to retrieve its FSID and filehandle */ error = nfs4_get_rootfh(server, mntfh, auth_probe); if (error < 0) - return error; + goto out_free_delegation_hash; dprintk("Server FSID: %llx:%llx\n", (unsigned long long) server->fsid.major, @@ -937,7 +941,7 @@ static int nfs4_server_common_setup(struct nfs_server *server, error = nfs_probe_server(server, mntfh); if (error < 0) - return error; + goto out_free_delegation_hash; nfs4_session_limit_rwsize(server); nfs4_session_limit_xasize(server); @@ -949,6 +953,11 @@ static int nfs4_server_common_setup(struct nfs_server *server, server->mount_time = jiffies; server->destroy = nfs4_destroy_server; return 0; + +out_free_delegation_hash: + kfree(server->delegation_hash_table); + server->delegation_hash_table = NULL; + return error; } /* diff --git a/fs/nfs/nfs4file.c b/fs/nfs/nfs4file.c index be40e126c539..6401f6363f75 100644 --- a/fs/nfs/nfs4file.c +++ b/fs/nfs/nfs4file.c @@ -91,6 +91,8 @@ nfs4_file_open(struct inode *inode, struct file *filp) nfs_fscache_open_file(inode, filp); err = 0; filp->f_mode |= FMODE_CAN_ODIRECT; + if (test_bit(NFS_CONTEXT_O_DIRECT, &ctx->flags)) + filp->f_flags |= O_DIRECT; out_put_ctx: put_nfs_open_context(ctx); diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 5709c6fea85b..04b1987115d5 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -225,8 +225,9 @@ const u32 nfs4_fattr_bitmap[3] = { | FATTR4_WORD1_TIME_METADATA | FATTR4_WORD1_TIME_MODIFY | FATTR4_WORD1_MOUNTED_ON_FILEID, + FATTR4_WORD2_UNCACHEABLE_FILE_DATA #ifdef CONFIG_NFS_V4_SECURITY_LABEL - FATTR4_WORD2_SECURITY_LABEL + | FATTR4_WORD2_SECURITY_LABEL #endif }; @@ -250,6 +251,7 @@ static const u32 nfs4_pnfs_open_bitmap[3] = { #ifdef CONFIG_NFS_V4_SECURITY_LABEL | FATTR4_WORD2_SECURITY_LABEL #endif + | FATTR4_WORD2_UNCACHEABLE_FILE_DATA }; static const u32 nfs4_open_noattr_bitmap[3] = { @@ -306,6 +308,15 @@ static void nfs4_bitmap_copy_adjust(__u32 *dst, const __u32 *src, unsigned long cache_validity; memcpy(dst, src, NFS4_BITMASK_SZ*sizeof(*dst)); + /* + * The uncacheable_file_data attribute applies only to regular files + * (NF4REG); a server must reject a query of it on any other object + * type with NFS4ERR_INVAL. Never request it unless the target is + * known to be a regular file (callers with an unknown object type, + * e.g. LOOKUP, pass a NULL inode). + */ + if (!inode || !S_ISREG(inode->i_mode)) + dst[2] &= ~FATTR4_WORD2_UNCACHEABLE_FILE_DATA; if (!inode || !nfs_have_read_or_write_delegation(inode)) return; @@ -327,6 +338,9 @@ static void nfs4_bitmap_copy_adjust(__u32 *dst, const __u32 *src, if (!(cache_validity & NFS_INO_INVALID_BTIME)) dst[1] &= ~FATTR4_WORD1_TIME_CREATE; + if (!(cache_validity & NFS_INO_INVALID_UNCACHEABLE_FILE_DATA)) + dst[2] &= ~FATTR4_WORD2_UNCACHEABLE_FILE_DATA; + if (nfs_have_delegated_mtime(inode)) { if (!(cache_validity & NFS_INO_INVALID_ATIME)) dst[1] &= ~(FATTR4_WORD1_TIME_ACCESS|FATTR4_WORD1_TIME_ACCESS_SET); @@ -3839,6 +3853,16 @@ nfs4_atomic_open(struct inode *dir, struct nfs_open_context *ctx, if (IS_ERR(state)) return ERR_CAST(state); + + /* + * Use O_DIRECT if file was marked as Uncacheable, see: + * https://datatracker.ietf.org/doc/draft-ietf-nfsv4-uncacheable-files/ + */ + if (!(open_flags & O_DIRECT) && NFS_I(state->inode)->uncacheable_file_data) { + if (!(open_flags & O_APPEND)) + set_bit(NFS_CONTEXT_O_DIRECT, &ctx->flags); + } + return state->inode; } @@ -3857,7 +3881,7 @@ static void nfs4_close_context(struct nfs_open_context *ctx, int is_sync) #define FATTR4_WORD1_NFS40_MASK (2*FATTR4_WORD1_MOUNTED_ON_FILEID - 1UL) #define FATTR4_WORD2_NFS41_MASK (2*FATTR4_WORD2_SUPPATTR_EXCLCREAT - 1UL) -#define FATTR4_WORD2_NFS42_MASK (2*FATTR4_WORD2_OPEN_ARGUMENTS - 1UL) +#define FATTR4_WORD2_NFS42_MASK (2*FATTR4_WORD2_UNCACHEABLE_FILE_DATA - 1UL) #define FATTR4_WORD2_NFS42_TIME_DELEG_MASK \ (FATTR4_WORD2_TIME_DELEG_MODIFY|FATTR4_WORD2_TIME_DELEG_ACCESS) @@ -3981,6 +4005,8 @@ static int _nfs4_server_capabilities(struct nfs_server *server, struct nfs_fh *f memcpy(server->attr_bitmask_nl, res.attr_bitmask, sizeof(server->attr_bitmask)); server->attr_bitmask_nl[2] &= ~FATTR4_WORD2_SECURITY_LABEL; + if (!(res.attr_bitmask[2] & FATTR4_WORD2_UNCACHEABLE_FILE_DATA)) + server->fattr_valid &= ~NFS_ATTR_FATTR_UNCACHEABLE_FILE_DATA; if (res.open_caps.oa_share_access_want[0] & NFS4_SHARE_WANT_OPEN_XOR_DELEGATION) @@ -4591,6 +4617,7 @@ static int _nfs4_proc_lookup(struct rpc_clnt *clnt, struct inode *dir, .rpc_resp = &res, }; unsigned short task_flags = 0; + __u32 bitmask[NFS4_BITMASK_SZ]; if (nfs_server_capable(dir, NFS_CAP_MOVEABLE)) task_flags = RPC_TASK_MOVEABLE; @@ -4599,7 +4626,13 @@ static int _nfs4_proc_lookup(struct rpc_clnt *clnt, struct inode *dir, if (nfs_lookup_is_soft_revalidate(dentry)) task_flags |= RPC_TASK_TIMEOUT; - args.bitmask = nfs4_bitmask(server, fattr->label); + /* + * The looked-up object's type is unknown here, so gate out the + * regular-file-only uncacheable_file_data attribute (NULL inode). + */ + nfs4_bitmap_copy_adjust(bitmask, nfs4_bitmask(server, fattr->label), + NULL, 0); + args.bitmask = bitmask; nfs_fattr_init(fattr); @@ -4713,13 +4746,20 @@ static int _nfs4_proc_lookupp(struct inode *inode, .rpc_resp = &res, }; unsigned short task_flags = 0; + __u32 bitmask[NFS4_BITMASK_SZ]; if (server->flags & NFS_MOUNT_SOFTREVAL) task_flags |= RPC_TASK_TIMEOUT; if (server->caps & NFS_CAP_MOVEABLE) task_flags |= RPC_TASK_MOVEABLE; - args.bitmask = nfs4_bitmask(server, fattr->label); + /* + * The looked-up object's type is unknown here, so gate out the + * regular-file-only uncacheable_file_data attribute (NULL inode). + */ + nfs4_bitmap_copy_adjust(bitmask, nfs4_bitmask(server, fattr->label), + NULL, 0); + args.bitmask = bitmask; nfs_fattr_init(fattr); nfs4_init_sequence(server->nfs_client, &args.seq_args, &res.seq_res, 0, 0); @@ -5134,6 +5174,7 @@ struct nfs4_createdata { struct nfs4_create_res res; struct nfs_fh fh; struct nfs_fattr fattr; + u32 bitmask[NFS4_BITMASK_SZ]; }; static struct nfs4_createdata *nfs4_alloc_createdata(struct inode *dir, @@ -5157,7 +5198,14 @@ static struct nfs4_createdata *nfs4_alloc_createdata(struct inode *dir, data->arg.name = name; data->arg.attrs = sattr; data->arg.ftype = ftype; - data->arg.bitmask = nfs4_bitmask(server, data->fattr.label); + /* + * CREATE only makes non-regular objects, so gate out the + * regular-file-only uncacheable_file_data attribute (NULL inode). + */ + nfs4_bitmap_copy_adjust(data->bitmask, + nfs4_bitmask(server, data->fattr.label), + NULL, 0); + data->arg.bitmask = data->bitmask; data->arg.umask = current_umask(); data->res.server = server; data->res.fh = &data->fh; @@ -5809,6 +5857,13 @@ void nfs4_bitmask_set(__u32 bitmask[], const __u32 src[], bitmask[1] |= FATTR4_WORD1_SPACE_USED; if (cache_validity & NFS_INO_INVALID_BTIME) bitmask[1] |= FATTR4_WORD1_TIME_CREATE; + /* + * uncacheable_file_data (attr 87) applies only to regular files; a + * directory can reach here via DELEGRETURN of a directory delegation. + */ + if ((cache_validity & NFS_INO_INVALID_UNCACHEABLE_FILE_DATA) && + S_ISREG(inode->i_mode)) + bitmask[2] |= FATTR4_WORD2_UNCACHEABLE_FILE_DATA; if (cache_validity & NFS_INO_INVALID_SIZE) bitmask[0] |= FATTR4_WORD0_SIZE; @@ -7762,6 +7817,7 @@ static int nfs4_add_lease(struct file *file, int arg, struct file_lease **lease, { struct inode *inode = file_inode(file); fmode_t type = arg == F_RDLCK ? FMODE_READ : FMODE_WRITE; + fl_owner_t owner = (*lease)->c.flc_owner; int ret; /* No delegation, no lease */ @@ -7771,7 +7827,8 @@ static int nfs4_add_lease(struct file *file, int arg, struct file_lease **lease, if (ret || nfs4_have_delegation(inode, type, 0)) return ret; /* We raced with a delegation return */ - nfs4_delete_lease(file, priv); + dprintk("%s: raced with a delegation return\n", __func__); + nfs4_delete_lease(file, &owner); return -EAGAIN; } diff --git a/fs/nfs/nfs4session.c b/fs/nfs/nfs4session.c index 5c128957a0a4..175390e5b93f 100644 --- a/fs/nfs/nfs4session.c +++ b/fs/nfs/nfs4session.c @@ -626,27 +626,34 @@ int nfs4_init_session(struct nfs_client *clp) return nfs41_check_session_ready(clp); } -int nfs4_init_ds_session(struct nfs_client *clp, unsigned long lease_time) +int nfs4_init_ds_session(struct nfs_client *clp, unsigned long lease_time, + bool tightly_coupled) { struct nfs4_session *session = clp->cl_session; int ret; spin_lock(&clp->cl_lock); - if (test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) { - /* - * Do not set NFS_CS_CHECK_LEASE_TIME instead set the - * DS lease to be equal to the MDS lease. - */ + /* + * Do not set NFS_CS_CHECK_LEASE_TIME instead set the + * DS lease to be equal to the MDS lease. + * + * A v4.0 DS has no session, so seed the lease every time. + */ + if (!session || + test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) { clp->cl_lease_time = lease_time; clp->cl_last_renewal = jiffies; } spin_unlock(&clp->cl_lock); + if (!session) + return 0; + ret = nfs41_check_session_ready(clp); if (ret) return ret; /* Test for the DS role */ - if (!is_ds_client(clp)) + if (tightly_coupled && !is_ds_client(clp)) return -ENODEV; return 0; } diff --git a/fs/nfs/nfs4session.h b/fs/nfs/nfs4session.h index d2569f599977..ee2f4baf16a1 100644 --- a/fs/nfs/nfs4session.h +++ b/fs/nfs/nfs4session.h @@ -122,7 +122,8 @@ extern int nfs4_setup_session_slot_tables(struct nfs4_session *ses); extern struct nfs4_session *nfs4_alloc_session(struct nfs_client *clp); extern void nfs4_destroy_session(struct nfs4_session *session); extern int nfs4_init_session(struct nfs_client *clp); -extern int nfs4_init_ds_session(struct nfs_client *, unsigned long); +extern int nfs4_init_ds_session(struct nfs_client *clp, unsigned long lease_time, + bool tightly_coupled); /* * Determine if sessions are in use. diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c index 305a772e5497..a5dec0473e22 100644 --- a/fs/nfs/nfs4state.c +++ b/fs/nfs/nfs4state.c @@ -351,6 +351,26 @@ const struct cred *nfs4_get_clid_cred(struct nfs_client *clp) return cred; } +static bool +nfs4_get_state_owner_active_locked(struct nfs4_state_owner *sp) +{ + struct nfs_server *server = sp->so_server; + + /* + * A counted state owner may dereference so_server until the final + * nfs4_put_state_owner(). Pin the superblock when reviving an idle + * owner so umount cannot free the server underneath it. + */ + if (atomic_read(&sp->so_count) == 0) { + if (!nfs_sb_active(server->super)) + return false; + if (!list_empty(&sp->so_lru)) + list_del_init(&sp->so_lru); + } + atomic_inc(&sp->so_count); + return true; +} + static struct nfs4_state_owner * nfs4_find_state_owner_locked(struct nfs_server *server, const struct cred *cred) { @@ -369,9 +389,8 @@ nfs4_find_state_owner_locked(struct nfs_server *server, const struct cred *cred) else if (cmp > 0) p = &parent->rb_right; else { - if (!list_empty(&sp->so_lru)) - list_del_init(&sp->so_lru); - atomic_inc(&sp->so_count); + if (!nfs4_get_state_owner_active_locked(sp)) + return NULL; return sp; } } @@ -397,9 +416,8 @@ nfs4_insert_state_owner_locked(struct nfs4_state_owner *new) else if (cmp > 0) p = &parent->rb_right; else { - if (!list_empty(&sp->so_lru)) - list_del_init(&sp->so_lru); - atomic_inc(&sp->so_count); + if (!nfs4_get_state_owner_active_locked(sp)) + return NULL; return sp; } } @@ -449,6 +467,10 @@ nfs4_alloc_state_owner(struct nfs_server *server, sp = kzalloc_obj(*sp, gfp_flags); if (!sp) return NULL; + if (!nfs_sb_active(server->super)) { + kfree(sp); + return NULL; + } sp->so_seqid.owner_id = atomic64_inc_return(&server->owner_ctr); sp->so_server = server; sp->so_cred = get_cred(cred); @@ -534,8 +556,10 @@ struct nfs4_state_owner *nfs4_get_state_owner(struct nfs_server *server, spin_lock(&clp->cl_lock); sp = nfs4_insert_state_owner_locked(new); spin_unlock(&clp->cl_lock); - if (sp != new) + if (sp != new) { nfs4_free_state_owner(new); + nfs_sb_deactive(server->super); + } out: nfs4_gc_state_owners(server); return sp; @@ -557,6 +581,7 @@ void nfs4_put_state_owner(struct nfs4_state_owner *sp) { struct nfs_server *server = sp->so_server; struct nfs_client *clp = server->nfs_client; + struct super_block *sb = server->super; if (!atomic_dec_and_lock(&sp->so_count, &clp->cl_lock)) return; @@ -564,6 +589,7 @@ void nfs4_put_state_owner(struct nfs4_state_owner *sp) sp->so_expires = jiffies; list_add_tail(&sp->so_lru, &server->state_owners_lru); spin_unlock(&clp->cl_lock); + nfs_sb_deactive(sb); } /** diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h index 1ed677810d9d..b5c89eeef2bc 100644 --- a/fs/nfs/nfs4trace.h +++ b/fs/nfs/nfs4trace.h @@ -33,7 +33,8 @@ { NFS_ATTR_FATTR_CHANGE, "CHANGE" }, \ { NFS_ATTR_FATTR_OWNER_NAME, "OWNER_NAME" }, \ { NFS_ATTR_FATTR_GROUP_NAME, "GROUP_NAME" }, \ - { NFS_ATTR_FATTR_BTIME, "BTIME" }) + { NFS_ATTR_FATTR_BTIME, "BTIME" }, \ + { NFS_ATTR_FATTR_UNCACHEABLE_FILE_DATA, "UNCACHEABLE_FILE_DATA" }) DECLARE_EVENT_CLASS(nfs4_clientid_event, TP_PROTO( @@ -1515,7 +1516,60 @@ DECLARE_EVENT_CLASS(nfs4_inode_stateid_callback_event, ), \ TP_ARGS(clp, fhandle, inode, stateid, error)) DEFINE_NFS4_INODE_STATEID_CALLBACK_EVENT(nfs4_cb_recall); -DEFINE_NFS4_INODE_STATEID_CALLBACK_EVENT(nfs4_cb_layoutrecall_file); + +TRACE_EVENT(nfs4_cb_layoutrecall_file, + TP_PROTO( + const struct nfs_client *clp, + const struct nfs_fh *fhandle, + const struct inode *inode, + const nfs4_stateid *stateid, + unsigned int changed, + int error + ), + + TP_ARGS(clp, fhandle, inode, stateid, changed, error), + + TP_STRUCT__entry( + __field(unsigned long, error) + __field(dev_t, dev) + __field(u32, fhandle) + __field(u64, fileid) + __string(dstaddr, clp ? clp->cl_hostname : "unknown") + __field(int, stateid_seq) + __field(u32, stateid_hash) + __field(unsigned int, changed) + ), + + TP_fast_assign( + __entry->error = error < 0 ? -error : 0; + __entry->fhandle = nfs_fhandle_hash(fhandle); + if (!IS_ERR_OR_NULL(inode)) { + __entry->fileid = inode->i_ino; + __entry->dev = inode->i_sb->s_dev; + } else { + __entry->fileid = 0; + __entry->dev = 0; + } + __assign_str(dstaddr); + __entry->stateid_seq = + be32_to_cpu(stateid->seqid); + __entry->stateid_hash = + nfs_stateid_hash(stateid); + __entry->changed = changed; + ), + + TP_printk( + "error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x " + "stateid=%d:0x%08x dstaddr=%s clora_changed=%u", + -__entry->error, + show_nfs4_status(__entry->error), + MAJOR(__entry->dev), MINOR(__entry->dev), + (unsigned long long)__entry->fileid, + __entry->fhandle, + __entry->stateid_seq, __entry->stateid_hash, + __get_str(dstaddr), __entry->changed + ) +); #define show_stateid_type(type) \ __print_symbolic(type, \ diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c index c23c2eee1b5c..fc049ce4ba8a 100644 --- a/fs/nfs/nfs4xdr.c +++ b/fs/nfs/nfs4xdr.c @@ -120,7 +120,8 @@ static int decode_layoutget(struct xdr_stream *xdr, struct rpc_rqst *req, 3*nfstime4_maxsz + \ nfs4_owner_maxsz + \ nfs4_group_maxsz + nfs4_label_maxsz + \ - decode_mdsthreshold_maxsz)) + decode_mdsthreshold_maxsz + \ + 1)) /* uncacheable_file_data */ #define nfs4_fattr_maxsz (nfs4_fattr_bitmap_maxsz + \ nfs4_fattr_value_maxsz) #define decode_getattr_maxsz (op_decode_hdr_maxsz + nfs4_fattr_maxsz) @@ -4380,6 +4381,30 @@ static int decode_attr_open_arguments(struct xdr_stream *xdr, uint32_t *bitmap, return 0; } +static int decode_attr_uncacheable_file_data(struct xdr_stream *xdr, uint32_t *bitmap, + uint32_t *res, uint64_t *flags) +{ + int status = 0; + __be32 *p; + + if (unlikely(bitmap[2] & (FATTR4_WORD2_UNCACHEABLE_FILE_DATA - 1U))) + return -EIO; + if (likely(bitmap[2] & FATTR4_WORD2_UNCACHEABLE_FILE_DATA)) { + p = xdr_inline_decode(xdr, 4); + if (unlikely(!p)) + return -EIO; + if (be32_to_cpup(p)) + *res |= NFS_AUX_UNCACHEABLE_FILE_DATA; + else + *res &= ~NFS_AUX_UNCACHEABLE_FILE_DATA; + bitmap[2] &= ~FATTR4_WORD2_UNCACHEABLE_FILE_DATA; + *flags |= NFS_ATTR_FATTR_UNCACHEABLE_FILE_DATA; + } + dprintk("%s: uncacheable_file_data: =%s\n", __func__, + (*res & NFS_AUX_UNCACHEABLE_FILE_DATA) == 0 ? "false" : "true"); + return status; +} + static int verify_attr_len(struct xdr_stream *xdr, unsigned int savep, uint32_t attrlen) { unsigned int attrwords = XDR_QUADLEN(attrlen); @@ -4725,6 +4750,8 @@ static int decode_getfattr_attrs(struct xdr_stream *xdr, uint32_t *bitmap, uint32_t type; int32_t err; + fattr->aux_flags = 0; + status = decode_attr_type(xdr, bitmap, &type); if (status < 0) goto xdr_error; @@ -4843,6 +4870,12 @@ static int decode_getfattr_attrs(struct xdr_stream *xdr, uint32_t *bitmap, goto xdr_error; fattr->valid |= status; + status = decode_attr_uncacheable_file_data(xdr, bitmap, &fattr->aux_flags, + &fattr->valid); + if (status < 0) + goto xdr_error; + + status = 0; xdr_error: dprintk("%s: xdr returned %d\n", __func__, -status); return status; diff --git a/fs/nfs/nfstrace.h b/fs/nfs/nfstrace.h index 4ada21f4eebd..b15c1732c869 100644 --- a/fs/nfs/nfstrace.h +++ b/fs/nfs/nfstrace.h @@ -33,7 +33,8 @@ { NFS_INO_INVALID_XATTR, "INVALID_XATTR" }, \ { NFS_INO_INVALID_NLINK, "INVALID_NLINK" }, \ { NFS_INO_INVALID_MODE, "INVALID_MODE" }, \ - { NFS_INO_INVALID_BTIME, "INVALID_BTIME" }) + { NFS_INO_INVALID_BTIME, "INVALID_BTIME" }, \ + { NFS_INO_INVALID_UNCACHEABLE_FILE_DATA, "INVALID_UNCACHEABLE_FILE_DATA" }) #define nfs_show_nfsi_flags(v) \ __print_flags(v, "|", \ diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c index 7715e2bd5871..9a08fd076e0a 100644 --- a/fs/nfs/pnfs.c +++ b/fs/nfs/pnfs.c @@ -432,7 +432,8 @@ bool nfs4_layout_refresh_old_stateid(nfs4_stateid *dst, goto out; } /* Try to update the seqid to the most recent */ - err = pnfs_mark_matching_lsegs_return(lo, &head, &range, 0); + err = pnfs_mark_matching_lsegs_return(lo, &head, &range, 0, + true); if (err != -EBUSY) { dst->seqid = lo->plh_stateid.seqid; *dst_range = range; @@ -486,7 +487,7 @@ static int pnfs_mark_layout_stateid_return(struct pnfs_layout_hdr *lo, .length = NFS4_MAX_UINT64, }; - return pnfs_mark_matching_lsegs_return(lo, lseg_list, &range, seq); + return pnfs_mark_matching_lsegs_return(lo, lseg_list, &range, seq, true); } static int @@ -524,7 +525,7 @@ pnfs_layout_io_set_failed(struct pnfs_layout_hdr *lo, u32 iomode) spin_lock(&inode->i_lock); pnfs_layout_set_fail_bit(lo, pnfs_iomode_to_fail_bit(iomode)); - pnfs_mark_matching_lsegs_return(lo, &head, &range, 0); + pnfs_mark_matching_lsegs_return(lo, &head, &range, 0, true); spin_unlock(&inode->i_lock); pnfs_free_lseg_list(&head); dprintk("%s Setting layout IOMODE_%s fail bit\n", __func__, @@ -1461,7 +1462,7 @@ _pnfs_return_layout(struct inode *ino) } valid_layout = pnfs_layout_is_valid(lo); pnfs_clear_layoutcommit(ino, &tmp_list); - pnfs_mark_matching_lsegs_return(lo, &tmp_list, &range, 0); + pnfs_mark_matching_lsegs_return(lo, &tmp_list, &range, 0, true); /* Don't send a LAYOUTRETURN if list was initially empty */ @@ -2100,15 +2101,6 @@ static bool pnfs_is_first_layoutget(struct pnfs_layout_hdr *lo) return test_bit(NFS_LAYOUT_FIRST_LAYOUTGET, &lo->plh_flags); } -static void pnfs_clear_first_layoutget(struct pnfs_layout_hdr *lo) -{ - unsigned long *bitlock = &lo->plh_flags; - - clear_bit_unlock(NFS_LAYOUT_FIRST_LAYOUTGET, bitlock); - smp_mb__after_atomic(); - wake_up_bit(bitlock, NFS_LAYOUT_FIRST_LAYOUTGET); -} - static void _add_to_server_list(struct pnfs_layout_hdr *lo, struct nfs_server *server) { @@ -2284,7 +2276,8 @@ lookup_again: iomode, lo, lseg, PNFS_UPDATE_LAYOUT_INVALID_OPEN); nfs4_schedule_stateid_recovery(server, ctx->state); - pnfs_clear_first_layoutget(lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, + &lo->plh_flags); pnfs_put_layout_hdr(lo); goto lookup_again; } @@ -2353,7 +2346,8 @@ lookup_again: if (!exception.retry) goto out_put_layout_hdr; if (first) - pnfs_clear_first_layoutget(lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, + &lo->plh_flags); trace_pnfs_update_layout(ino, pos, count, iomode, lo, lseg, PNFS_UPDATE_LAYOUT_RETRY); pnfs_put_layout_hdr(lo); @@ -2365,7 +2359,7 @@ lookup_again: out_put_layout_hdr: if (first) - pnfs_clear_first_layoutget(lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, &lo->plh_flags); trace_pnfs_update_layout(ino, pos, count, iomode, lo, lseg, PNFS_UPDATE_LAYOUT_EXIT); pnfs_put_layout_hdr(lo); @@ -2457,7 +2451,7 @@ static void _lgopen_prepare_attached(struct nfs4_opendata *data, lgp = pnfs_alloc_init_layoutget_args(ino, ctx, ¤t_stateid, &rng, nfs_io_gfp_mask()); if (!lgp) { - pnfs_clear_first_layoutget(lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, &lo->plh_flags); nfs_layoutget_end(lo); pnfs_put_layout_hdr(lo); return; @@ -2561,7 +2555,8 @@ void nfs4_lgopen_release(struct nfs4_layoutget *lgp) { if (lgp != NULL) { if (lgp->lo) { - pnfs_clear_first_layoutget(lgp->lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, + &lgp->lo->plh_flags); nfs_layoutget_end(lgp->lo); } pnfs_layoutget_free(lgp); @@ -2621,7 +2616,7 @@ pnfs_layout_process(struct nfs4_layoutget *lgp) .iomode = IOMODE_ANY, .length = NFS4_MAX_UINT64, }; - pnfs_mark_matching_lsegs_return(lo, &free_me, &range, 0); + pnfs_mark_matching_lsegs_return(lo, &free_me, &range, 0, true); goto out_forget; } else { /* We have a completely new layout */ @@ -2652,6 +2647,7 @@ out_forget: * @tmp_list: list header to be used with pnfs_free_lseg_list() * @return_range: describe layout segment ranges to be returned * @seq: stateid seqid to match + * @cancel_io: signal io be cancelled * * This function is mainly intended for use by layoutrecall. It attempts * to free the layout segment immediately, or else to mark it for return @@ -2666,7 +2662,7 @@ int pnfs_mark_matching_lsegs_return(struct pnfs_layout_hdr *lo, struct list_head *tmp_list, const struct pnfs_layout_range *return_range, - u32 seq) + u32 seq, bool cancel_io) { struct pnfs_layout_segment *lseg, *next; struct nfs_server *server = NFS_SERVER(lo->plh_inode); @@ -2692,7 +2688,8 @@ pnfs_mark_matching_lsegs_return(struct pnfs_layout_hdr *lo, continue; remaining++; set_bit(NFS_LSEG_LAYOUTRETURN, &lseg->pls_flags); - pnfs_lseg_cancel_io(server, lseg); + if (cancel_io) + pnfs_lseg_cancel_io(server, lseg); } if (remaining) { @@ -2727,7 +2724,8 @@ pnfs_mark_layout_for_return(struct inode *inode, * segments at hand when sending layoutreturn. See pnfs_put_lseg() * for how it works. */ - if (pnfs_mark_matching_lsegs_return(lo, &lo->plh_return_segs, range, 0) != -EBUSY) { + if (pnfs_mark_matching_lsegs_return(lo, &lo->plh_return_segs, range, 0, + true) != -EBUSY) { const struct cred *cred; nfs4_stateid stateid; enum pnfs_iomode iomode; @@ -2842,7 +2840,7 @@ restart: pnfs_get_layout_hdr(lo); pnfs_set_plh_return_info(lo, range->iomode, 0); if (pnfs_mark_matching_lsegs_return(lo, &lo->plh_return_segs, - range, 0) != 0 || + range, 0, true) != 0 || !pnfs_prepare_layoutreturn(lo, &stateid, &cred, &iomode)) { spin_unlock(&inode->i_lock); rcu_read_unlock(); @@ -3273,15 +3271,6 @@ pnfs_generic_pg_readpages(struct nfs_pageio_descriptor *desc) } EXPORT_SYMBOL_GPL(pnfs_generic_pg_readpages); -static void pnfs_clear_layoutcommitting(struct inode *inode) -{ - unsigned long *bitlock = &NFS_I(inode)->flags; - - clear_bit_unlock(NFS_INO_LAYOUTCOMMITTING, bitlock); - smp_mb__after_atomic(); - wake_up_bit(bitlock, NFS_INO_LAYOUTCOMMITTING); -} - /* * There can be multiple RW segments. */ @@ -3306,7 +3295,7 @@ static void pnfs_list_write_lseg_done(struct inode *inode, struct list_head *lis pnfs_put_lseg(lseg); } - pnfs_clear_layoutcommitting(inode); + clear_and_wake_up_bit(NFS_INO_LAYOUTCOMMITTING, &NFS_I(inode)->flags); } void pnfs_set_lo_fail(struct pnfs_layout_segment *lseg) @@ -3446,7 +3435,7 @@ out_unlock: spin_unlock(&inode->i_lock); kfree(data); clear_layoutcommitting: - pnfs_clear_layoutcommitting(inode); + clear_and_wake_up_bit(NFS_INO_LAYOUTCOMMITTING, &NFS_I(inode)->flags); goto out; } EXPORT_SYMBOL_GPL(pnfs_layoutcommit_inode); diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h index eb39859c216c..bab81f769636 100644 --- a/fs/nfs/pnfs.h +++ b/fs/nfs/pnfs.h @@ -300,7 +300,7 @@ int pnfs_mark_matching_lsegs_invalid(struct pnfs_layout_hdr *lo, int pnfs_mark_matching_lsegs_return(struct pnfs_layout_hdr *lo, struct list_head *tmp_list, const struct pnfs_layout_range *recall_range, - u32 seq); + u32 seq, bool cancel_io); int pnfs_mark_layout_stateid_invalid(struct pnfs_layout_hdr *lo, struct list_head *lseg_list); bool pnfs_roc(struct inode *ino, struct nfs4_layoutreturn_args *args, @@ -421,7 +421,8 @@ struct nfs4_pnfs_ds *nfs4_pnfs_ds_add(const struct net *net, void nfs4_pnfs_v3_ds_connect_unload(void); int nfs4_pnfs_ds_connect(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds, struct nfs4_deviceid_node *devid, unsigned int timeo, - unsigned int retrans, u32 version, u32 minor_version); + unsigned int retrans, u32 version, u32 minor_version, + bool tightly_coupled); struct nfs4_pnfs_ds_addr *nfs4_decode_mp_ds_addr(struct net *net, struct xdr_stream *xdr, gfp_t gfp_flags); diff --git a/fs/nfs/pnfs_nfs.c b/fs/nfs/pnfs_nfs.c index 648c95b78eea..b539e1a44d26 100644 --- a/fs/nfs/pnfs_nfs.c +++ b/fs/nfs/pnfs_nfs.c @@ -881,7 +881,8 @@ static int _nfs4_pnfs_v4_ds_connect(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds, unsigned int timeo, unsigned int retrans, - u32 minor_version) + u32 minor_version, + bool tightly_coupled) { struct nfs_client *clp = ERR_PTR(-EIO); struct nfs_client *mds_clp = mds_srv->nfs_client; @@ -971,12 +972,14 @@ static int _nfs4_pnfs_v4_ds_connect(struct nfs_server *mds_srv, clp = nfs4_set_ds_client(mds_srv, &da->da_addr, da->da_addrlen, ds_proto, - timeo, retrans, minor_version); + timeo, retrans, minor_version, + tightly_coupled); if (IS_ERR(clp)) continue; status = nfs4_init_ds_session(clp, - mds_srv->nfs_client->cl_lease_time); + mds_srv->nfs_client->cl_lease_time, + tightly_coupled); if (status) { nfs_put_client(clp); clp = ERR_PTR(-EIO); @@ -1004,7 +1007,8 @@ out: */ int nfs4_pnfs_ds_connect(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds, struct nfs4_deviceid_node *devid, unsigned int timeo, - unsigned int retrans, u32 version, u32 minor_version) + unsigned int retrans, u32 version, u32 minor_version, + bool tightly_coupled) { int err; @@ -1027,7 +1031,7 @@ int nfs4_pnfs_ds_connect(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds, break; case 4: err = _nfs4_pnfs_v4_ds_connect(mds_srv, ds, timeo, retrans, - minor_version); + minor_version, tightly_coupled); break; default: dprintk("%s: unsupported DS version %d\n", __func__, version); diff --git a/fs/nfs/write.c b/fs/nfs/write.c index d2b03ceaeb4f..623e7ef1f73d 100644 --- a/fs/nfs/write.c +++ b/fs/nfs/write.c @@ -739,17 +739,18 @@ static void nfs_inode_remove_request(struct nfs_page *req) nfs_page_group_lock(req); if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) { struct folio *folio = nfs_page_to_folio(req->wb_head); - struct address_space *mapping = folio->mapping; - spin_lock(&mapping->i_private_lock); if (likely(folio)) { + struct address_space *mapping = folio->mapping; + + spin_lock(&mapping->i_private_lock); folio->private = NULL; folio_clear_private(folio); clear_bit(PG_MAPPED, &req->wb_head->wb_flags); - } - spin_unlock(&mapping->i_private_lock); + spin_unlock(&mapping->i_private_lock); - folio_end_dropbehind(folio); + folio_end_dropbehind(folio); + } } nfs_page_group_unlock(req); @@ -1664,7 +1665,7 @@ int nfs_initiate_commit(struct rpc_clnt *clnt, struct nfs_commit_data *data, dprintk("NFS: initiated commit call\n"); if (localio) - return nfs_local_commit(localio, data, call_ops, how); + return nfs_local_commit(localio, data, call_ops); task = rpc_run_task(&task_setup_data); if (IS_ERR(task)) diff --git a/fs/nilfs2/Kconfig b/fs/nilfs2/Kconfig index 7dae168e346e..0a5ace60e6ab 100644 --- a/fs/nilfs2/Kconfig +++ b/fs/nilfs2/Kconfig @@ -3,7 +3,7 @@ config NILFS2_FS tristate "NILFS2 file system support" select BUFFER_HEAD select CRC32 - select LEGACY_DIRECT_IO + select FS_IOMAP help NILFS2 is a log-structured file system (LFS) supporting continuous snapshotting. In addition to versioning capability of the entire diff --git a/fs/nilfs2/Makefile b/fs/nilfs2/Makefile index 43b60b8a4d07..516e6b85a03c 100644 --- a/fs/nilfs2/Makefile +++ b/fs/nilfs2/Makefile @@ -3,4 +3,4 @@ obj-$(CONFIG_NILFS2_FS) += nilfs2.o nilfs2-y := inode.o file.o dir.o super.o namei.o page.o mdt.o \ btnode.o bmap.o btree.o direct.o dat.o recovery.o \ the_nilfs.o segbuf.o segment.o cpfile.o sufile.o \ - ifile.o alloc.o gcinode.o ioctl.o sysfs.o + ifile.o alloc.o gcinode.o ioctl.o sysfs.o iomap.o diff --git a/fs/nilfs2/file.c b/fs/nilfs2/file.c index f93b68c4877c..ad2e87c049c9 100644 --- a/fs/nilfs2/file.c +++ b/fs/nilfs2/file.c @@ -10,9 +10,12 @@ #include <linux/fs.h> #include <linux/filelock.h> #include <linux/mm.h> +#include <linux/uio.h> +#include <linux/iomap.h> #include <linux/writeback.h> #include "nilfs.h" #include "segment.h" +#include "iomap.h" int nilfs_sync_file(struct file *file, loff_t start, loff_t end, int datasync) { @@ -133,20 +136,51 @@ static int nilfs_file_mmap_prepare(struct vm_area_desc *desc) return 0; } +static int nilfs_file_open(struct inode *inode, struct file *file) +{ + file->f_mode |= FMODE_CAN_ODIRECT; + return generic_file_open(inode, file); +} + +static ssize_t nilfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to) +{ + if (iocb->ki_flags & IOCB_DIRECT) { + return iomap_dio_rw(iocb, to, &nilfs_iomap_ops, + NULL, 0, NULL, 0); + } else + return generic_file_read_iter(iocb, to); +} + +static ssize_t nilfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) +{ + /* + * NILFS2 cannot perform true direct I/O writes: new blocks are + * delay-allocated and are only given a real disk address when + * the segment constructor writes them out as part of a log, + * which works directly on buffer_head lists rather than + * through iomap. Fall back to the ordinary buffered write path + * for O_DIRECT writes. + */ + if (iocb->ki_flags & IOCB_DIRECT) + iocb->ki_flags &= ~IOCB_DIRECT; + + return generic_file_write_iter(iocb, from); +} + /* * We have mostly NULL's here: the current defaults are ok for * the nilfs filesystem. */ const struct file_operations nilfs_file_operations = { .llseek = generic_file_llseek, - .read_iter = generic_file_read_iter, - .write_iter = generic_file_write_iter, + .read_iter = nilfs_file_read_iter, + .write_iter = nilfs_file_write_iter, .unlocked_ioctl = nilfs_ioctl, #ifdef CONFIG_COMPAT .compat_ioctl = nilfs_compat_ioctl, #endif /* CONFIG_COMPAT */ .mmap_prepare = nilfs_file_mmap_prepare, - .open = generic_file_open, + .open = nilfs_file_open, /* .release = nilfs_release_file, */ .fsync = nilfs_sync_file, .splice_read = filemap_splice_read, diff --git a/fs/nilfs2/inode.c b/fs/nilfs2/inode.c index 34e6096069ad..cbe9658ae065 100644 --- a/fs/nilfs2/inode.c +++ b/fs/nilfs2/inode.c @@ -257,18 +257,6 @@ static int nilfs_write_end(const struct kiocb *iocb, return err ? : copied; } -static ssize_t -nilfs_direct_IO(struct kiocb *iocb, struct iov_iter *iter) -{ - struct inode *inode = file_inode(iocb->ki_filp); - - if (iov_iter_rw(iter) == WRITE) - return 0; - - /* Needs synchronization with the cleaner */ - return blockdev_direct_IO(iocb, inode, iter, nilfs_get_block); -} - const struct address_space_operations nilfs_aops = { .read_folio = nilfs_read_folio, .writepages = nilfs_writepages, @@ -277,7 +265,6 @@ const struct address_space_operations nilfs_aops = { .write_begin = nilfs_write_begin, .write_end = nilfs_write_end, .invalidate_folio = block_invalidate_folio, - .direct_IO = nilfs_direct_IO, .migrate_folio = buffer_migrate_folio_norefs, .is_partially_uptodate = block_is_partially_uptodate, }; diff --git a/fs/nilfs2/iomap.c b/fs/nilfs2/iomap.c new file mode 100644 index 000000000000..3ae3bf6ed368 --- /dev/null +++ b/fs/nilfs2/iomap.c @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * NILFS iomap support implementation. + * + * Written by Viacheslav Dubeyko. + */ + +#include <linux/iomap.h> +#include <linux/pagemap.h> +#include "nilfs.h" +#include "mdt.h" +#include "iomap.h" + +static int nilfs_iomap_begin(struct inode *inode, loff_t offset, + loff_t length, unsigned int flags, + struct iomap *iomap, struct iomap *srcmap) +{ + struct the_nilfs *nilfs = inode->i_sb->s_fs_info; + struct nilfs_inode_info *ii = NILFS_I(inode); + sector_t blkoff = offset >> inode->i_blkbits; + unsigned int maxblocks; + __u64 blknum = 0; + int ret; + + /* Completely beyond EOF. Treat as hole */ + if (i_size_read(inode) <= offset) { + iomap->type = IOMAP_HOLE; + iomap->addr = IOMAP_NULL_ADDR; + iomap->offset = offset; + iomap->length = length; + return 0; + } + + /* Clamp length if the requested range goes beyond i_size */ + if (offset + length > i_size_read(inode)) { + loff_t i_size = i_size_read(inode); + unsigned int blocksize = i_blocksize(inode); + + length = round_up(i_size, blocksize) - offset; + } + + maxblocks = min_t(loff_t, length >> inode->i_blkbits, INT_MAX); + if (maxblocks == 0) + maxblocks = 1; + + down_read(&NILFS_MDT(nilfs->ns_dat)->mi_sem); + ret = nilfs_bmap_lookup_contig(ii->i_bmap, blkoff, &blknum, maxblocks); + up_read(&NILFS_MDT(nilfs->ns_dat)->mi_sem); + + if (ret == -ENOENT) { + iomap->type = IOMAP_HOLE; + iomap->addr = IOMAP_NULL_ADDR; + iomap->offset = offset; + iomap->length = min_t(loff_t, length, i_blocksize(inode)); + return 0; + } else if (ret < 0) + return ret; + + iomap->bdev = inode->i_sb->s_bdev; + iomap->offset = offset; + iomap->length = min_t(loff_t, length, (loff_t)ret << inode->i_blkbits); + iomap->addr = (loff_t)blknum << inode->i_blkbits; + iomap->type = IOMAP_MAPPED; + iomap->flags = IOMAP_F_MERGED; + + return 0; +} + +const struct iomap_ops nilfs_iomap_ops = { + .iomap_begin = nilfs_iomap_begin, +}; diff --git a/fs/nilfs2/iomap.h b/fs/nilfs2/iomap.h new file mode 100644 index 000000000000..adef3e22346d --- /dev/null +++ b/fs/nilfs2/iomap.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0+ */ +/* + * NILFS iomap support declarations. + * + * Written by Viacheslav Dubeyko. + */ + +#ifndef _NILFS_IOMAP_H +#define _NILFS_IOMAP_H + +extern const struct iomap_ops nilfs_iomap_ops; + +#endif /* _NILFS_IOMAP_H */ diff --git a/fs/ntfs/Kconfig b/fs/ntfs/Kconfig index 6a6acde9ba91..f88f4138f921 100644 --- a/fs/ntfs/Kconfig +++ b/fs/ntfs/Kconfig @@ -10,6 +10,17 @@ config NTFS_FS To compile this as a module, choose M here: the module will be called ntfs. +config NTFS_FS_WOF_COMPRESSION + bool "NTFS Windows system compression support" + depends on NTFS_FS + help + Enable read support for Windows system-compressed files. + Windows more aggressively compresses system files such as binaries + and DLLs using system compression. It is optimized for files that + are rarely modified. + Enabling this option builds the decompression support into the NTFS + module. + config NTFS_DEBUG bool "NTFS debugging support" depends on NTFS_FS diff --git a/fs/ntfs/Makefile b/fs/ntfs/Makefile index e120c2e69862..ee8987e496a8 100644 --- a/fs/ntfs/Makefile +++ b/fs/ntfs/Makefile @@ -7,4 +7,7 @@ ntfs-y := aops.o attrib.o collate.o dir.o file.o index.o inode.o \ upcase.o bitmap.o lcnalloc.o logfile.o reparse.o compress.o \ iomap.o debug.o sysctl.o object_id.o bdev-io.o +ntfs-$(CONFIG_NTFS_FS_WOF_COMPRESSION) += wof.o \ + lib/decompress_common.o lib/lzx_decompress.o lib/xpress_decompress.o + ccflags-$(CONFIG_NTFS_DEBUG) += -DDEBUG diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c index 173de4cbee0f..517d9a1563e7 100644 --- a/fs/ntfs/aops.c +++ b/fs/ntfs/aops.c @@ -90,6 +90,14 @@ static int ntfs_read_folio(struct file *file, struct folio *folio) folio_unlock(folio); return -EOPNOTSUPP; } + if (NInoWofCompressed(ni)) { +#ifdef CONFIG_NTFS_FS_WOF_COMPRESSION + return ntfs_read_wof_compressed_block(folio); +#else + folio_unlock(folio); + return -EOPNOTSUPP; +#endif + } /* Compressed data streams are handled in compress.c. */ if (NInoNonResident(ni) && NInoCompressed(ni)) return ntfs_read_compressed_block(folio); @@ -136,11 +144,12 @@ static sector_t ntfs_bmap(struct address_space *mapping, sector_t block) ntfs_debug("Entering for mft_no 0x%llx, logical block 0x%llx.", ni->mft_no, (unsigned long long)block); if (ni->type != AT_DATA || !NInoNonResident(ni) || NInoEncrypted(ni) || - NInoMstProtected(ni)) { + NInoWofCompressed(ni) || NInoMstProtected(ni)) { ntfs_error(vol->sb, "BMAP does not make sense for %s attributes, returning 0.", (ni->type != AT_DATA) ? "non-data" : (!NInoNonResident(ni) ? "resident" : - "encrypted")); + (NInoWofCompressed(ni) ? "WOF-compressed" : + "encrypted"))); return 0; } /* None of these can happen. */ @@ -234,7 +243,8 @@ static void ntfs_readahead(struct readahead_control *rac) * Resident files are not cached in the page cache, * and readahead is not implemented for compressed files. */ - if (!NInoNonResident(ni) || NInoCompressed(ni)) + if (!NInoNonResident(ni) || NInoCompressed(ni) || + NInoWofCompressed(ni)) return; iomap_readahead(&ntfs_read_iomap_ops, &ctx, NULL); } @@ -286,6 +296,9 @@ static int ntfs_writepages(struct address_space *mapping, static int ntfs_swap_activate(struct swap_info_struct *sis, struct file *swap_file, sector_t *span) { + if (NInoWofCompressed(NTFS_I(file_inode(swap_file)))) + return -EOPNOTSUPP; + return iomap_swapfile_activate(sis, swap_file, span, &ntfs_read_iomap_ops); } diff --git a/fs/ntfs/attrib.c b/fs/ntfs/attrib.c index 58f32aac5f61..60264833bb63 100644 --- a/fs/ntfs/attrib.c +++ b/fs/ntfs/attrib.c @@ -693,15 +693,42 @@ static bool ntfs_non_resident_attr_value_is_valid(const struct attr_record *a) u32 attr_len; u32 min_len; u16 mp_offset; + u16 name_offset; + u32 name_end; attr_len = le32_to_cpu(a->length); min_len = offsetof(struct attr_record, data.non_resident.initialized_size) + sizeof(a->data.non_resident.initialized_size); + + /* Sparse and compressed attributes have the extra compressed_size field */ + if (a->flags & (ATTR_IS_SPARSE | ATTR_COMPRESSION_MASK)) + min_len += sizeof(a->data.non_resident.compressed_size); + if (attr_len < min_len) return false; mp_offset = le16_to_cpu(a->data.non_resident.mapping_pairs_offset); - return mp_offset >= min_len && mp_offset <= attr_len; + if (mp_offset < min_len || mp_offset > attr_len) + return false; + + if (a->name_length) { + name_offset = le16_to_cpu(a->name_offset); + + if (name_offset < min_len || name_offset >= attr_len) + return false; + + name_end = name_offset + a->name_length * sizeof(__le16); + if (name_end > attr_len || name_end > mp_offset) + return false; + } + + /* Ensure there's room for the compressed_size field if needed. */ + if (!(a->flags & (ATTR_IS_SPARSE | ATTR_COMPRESSION_MASK)) && + attr_len - mp_offset < + sizeof(a->data.non_resident.compressed_size)) + return false; + + return true; } static bool ntfs_attr_value_is_valid(struct ntfs_volume *vol, @@ -2110,7 +2137,7 @@ int ntfs_attr_make_non_resident(struct ntfs_inode *ni, const u32 data_size) ni->runlist.count = 0; write_lock_irqsave(&ni->size_lock, flags); ni->allocated_size = new_size; - if (NInoSparse(ni) || NInoCompressed(ni)) { + if ((NInoSparse(ni) && !NInoWofCompressed(ni)) || NInoCompressed(ni)) { ni->itype.compressed.size = ni->allocated_size; if (a->data.non_resident.compression_unit) { ni->itype.compressed.block_size = 1U << @@ -3531,8 +3558,13 @@ int ntfs_attr_record_move_away(struct ntfs_attr_search_ctx *ctx, int extra) unmap_mft_record(ni); err = ntfs_attr_record_move_to(ctx, ni); - if (err) + if (err) { ntfs_error(sb, "Couldn't move attribute to MFT record"); + if (ntfs_mft_record_free(base_ni->vol, ni)) + ntfs_error(sb, "Couldn't free empty MFT record"); + else + ntfs_inode_close(ni); + } return err; } @@ -5264,6 +5296,7 @@ int ntfs_attr_remove(struct ntfs_inode *ni, const __le32 type, __le16 *name, * On success a buffer is allocated with the content of the attribute * and which needs to be freed when it's not needed anymore. If the * @data_size parameter is non-NULL then the data size is set there. + * On error, an ERR_PTR() containing the negative error code is returned. */ void *ntfs_attr_readall(struct ntfs_inode *ni, const __le32 type, __le16 *name, u32 name_len, s64 *data_size) @@ -5278,6 +5311,7 @@ void *ntfs_attr_readall(struct ntfs_inode *ni, const __le32 type, bmp_vi = ntfs_attr_iget(VFS_I(ni), type, name, name_len); if (IS_ERR(bmp_vi)) { + ret = ERR_PTR(PTR_ERR(bmp_vi)); ntfs_debug("ntfs_attr_iget failed"); goto err_exit; } @@ -5287,17 +5321,21 @@ void *ntfs_attr_readall(struct ntfs_inode *ni, const __le32 type, (bmp_ni->type != AT_BITMAP || bmp_ni->data_size > ((ni->vol->nr_clusters + 7) >> 3))) { ntfs_error(sb, "Invalid attribute data size"); + ret = ERR_PTR(-EIO); goto out; } data = kvmalloc(bmp_ni->data_size, GFP_NOFS); - if (!data) + if (!data) { + ret = ERR_PTR(-ENOMEM); goto out; + } size = ntfs_inode_attr_pread(VFS_I(bmp_ni), 0, bmp_ni->data_size, (u8 *)data); if (size != bmp_ni->data_size) { ntfs_error(sb, "ntfs_attr_pread failed"); + ret = size < 0 ? ERR_PTR((int)size) : ERR_PTR(-EIO); kvfree(data); goto out; } diff --git a/fs/ntfs/bdev-io.c b/fs/ntfs/bdev-io.c index 27d7c2767a33..86db4d9298ed 100644 --- a/fs/ntfs/bdev-io.c +++ b/fs/ntfs/bdev-io.c @@ -33,7 +33,7 @@ int ntfs_bdev_read(struct block_device *bdev, char *data, loff_t start, size_t s unsigned int done = 0, added; int error; struct bio *bio; - enum req_op op; + blk_opf_t op; sector_t sector = start >> SECTOR_SHIFT; if (start & (SECTOR_SIZE - 1)) @@ -66,7 +66,7 @@ int ntfs_bdev_read(struct block_device *bdev, char *data, loff_t start, size_t s error = submit_bio_wait(bio); bio_put(bio); - if (op == REQ_OP_READ) + if ((op & REQ_OP_MASK) == REQ_OP_READ) invalidate_kernel_vmap_range(data, size); return error; } diff --git a/fs/ntfs/compress.c b/fs/ntfs/compress.c index 76bd806b41ed..2225630b19d7 100644 --- a/fs/ntfs/compress.c +++ b/fs/ntfs/compress.c @@ -21,6 +21,7 @@ #include <linux/slab.h> #include "attrib.h" +#include "ntfs_codec.h" #include "inode.h" #include "debug.h" #include "ntfs.h" @@ -97,26 +98,6 @@ void free_compression_buffers(void) } /* - * zero_partial_compressed_page - zero out of bounds compressed page region - * @page: page to zero - * @initialized_size: initialized size of the attribute - */ -static void zero_partial_compressed_page(struct page *page, - const s64 initialized_size) -{ - u8 *kp = page_address(page); - unsigned int kp_ofs; - - ntfs_debug("Zeroing page region outside initialized size."); - if (((s64)page->__folio_index << PAGE_SHIFT) >= initialized_size) { - clear_page(kp); - return; - } - kp_ofs = initialized_size & ~PAGE_MASK; - memset(kp + kp_ofs, 0, PAGE_SIZE - kp_ofs); -} - -/* * handle_bounds_compressed_page - test for&handle out of bounds compressed page * @page: page to check and handle * @i_size: file size @@ -125,9 +106,21 @@ static void zero_partial_compressed_page(struct page *page, static inline void handle_bounds_compressed_page(struct page *page, const loff_t i_size, const s64 initialized_size) { - if ((page->__folio_index >= (initialized_size >> PAGE_SHIFT)) && - (initialized_size < i_size)) - zero_partial_compressed_page(page, initialized_size); + loff_t pos = page_offset(page); + + if ((pos + PAGE_SIZE > initialized_size) && + (initialized_size < i_size)) { + size_t offset; + + ntfs_debug("Zeroing page region outside initialized size."); + if (pos >= initialized_size) + offset = 0; + else + offset = offset_in_page(initialized_size); + zero_user_segment(page, offset, PAGE_SIZE); + } else { + flush_dcache_page(page); + } } /* @@ -185,6 +178,7 @@ static int ntfs_decompress(struct page *dest_pages[], int completed_pages[], /* Variables for uncompressed data / destination. */ struct page *dp; /* Current destination page being worked on. */ + u8 *dp_kaddr; /* Local kmap for the current destination page. */ u8 *dp_addr; /* Current pointer into dp. */ u8 *dp_sb_start; /* Start of current sub-block in dp. */ u8 *dp_sb_end; /* End of current sb in dp (dp_sb_start + NTFS_SB_SIZE). */ @@ -199,6 +193,7 @@ static int ntfs_decompress(struct page *dest_pages[], int completed_pages[], /* Default error code. */ int err = -EOVERFLOW; + dp_kaddr = NULL; ntfs_debug("Entering, cb_size = 0x%x.", cb_size); do_next_sb: ntfs_debug("Beginning sub-block at offset = 0x%zx in the cb.", @@ -231,8 +226,6 @@ return_error: */ handle_bounds_compressed_page(dp, i_size, initialized_size); - flush_dcache_page(dp); - kunmap_local(page_address(dp)); SetPageUptodate(dp); unlock_page(dp); if (di == xpage) @@ -278,7 +271,8 @@ return_error: } /* We have a valid destination page. Setup the destination pointers. */ - dp_addr = (u8 *)page_address(dp) + do_sb_start; + dp_kaddr = kmap_local_page(dp); + dp_addr = dp_kaddr + do_sb_start; /* Now, we are ready to process the current sub-block (sb). */ if (!(le16_to_cpup((__le16 *)cb) & NTFS_SB_IS_COMPRESSED)) { @@ -299,6 +293,8 @@ return_error: /* Advance destination position to next sub-block. */ *dest_ofs += NTFS_SB_SIZE; *dest_ofs &= ~PAGE_MASK; + kunmap_local(dp_kaddr); + dp_kaddr = NULL; if (!(*dest_ofs)) { finalize_page: /* @@ -333,6 +329,8 @@ do_next_tag: } /* We have finished the current sub-block. */ *dest_ofs &= ~PAGE_MASK; + kunmap_local(dp_kaddr); + dp_kaddr = NULL; if (!(*dest_ofs)) goto finalize_page; goto do_next_sb; @@ -352,7 +350,7 @@ do_next_tag: u8 *dp_back_addr; /* Check if we are done / still in range. */ - if (cb >= cb_sb_end || dp_addr > dp_sb_end) + if (cb >= cb_sb_end || dp_addr >= dp_sb_end) break; /* Determine token type and parse appropriately.*/ @@ -438,6 +436,8 @@ do_next_tag: goto do_next_tag; return_overflow: + if (dp_kaddr) + kunmap_local(dp_kaddr); ntfs_error(NULL, "Failed. Returning -EOVERFLOW."); goto return_error; } @@ -465,14 +465,14 @@ int ntfs_read_compressed_block(struct folio *folio) struct page *page = &folio->page; loff_t i_size; s64 initialized_size; - struct address_space *mapping = page->mapping; + struct address_space *mapping = folio->mapping; struct ntfs_inode *ni = NTFS_I(mapping->host); struct ntfs_volume *vol = ni->vol; struct super_block *sb = vol->sb; struct runlist_element *rl; unsigned long flags; u8 *cb, *cb_pos, *cb_end; - unsigned long offset, index = page->__folio_index; + unsigned long offset, index = folio->index; u32 cb_size = ni->itype.compressed.block_size; u64 cb_size_mask = cb_size - 1UL; s64 vcn; @@ -566,7 +566,6 @@ int ntfs_read_compressed_block(struct folio *folio) * least wasting our time. */ if (!PageDirty(page) && (!PageUptodate(page))) { - kmap_local_page(page); continue; } unlock_page(page); @@ -652,8 +651,7 @@ lock_retry_remap: } lock_page(lpage); - memcpy(cb_pos, page_address(lpage) + page_ofs, - vol->cluster_size); + memcpy_from_page(cb_pos, lpage, page_ofs, vol->cluster_size); unlock_page(lpage); put_page(lpage); cb_pos += vol->cluster_size; @@ -692,14 +690,7 @@ lock_retry_remap: for (; cur_page < cb_max_page; cur_page++) { page = pages[cur_page]; if (page) { - if (likely(!cur_ofs)) - clear_page(page_address(page)); - else - memset(page_address(page) + cur_ofs, 0, - PAGE_SIZE - - cur_ofs); - flush_dcache_page(page); - kunmap_local(page_address(page)); + memzero_page(page, cur_ofs, PAGE_SIZE - cur_ofs); SetPageUptodate(page); unlock_page(page); if (cur_page == xpage) @@ -717,8 +708,7 @@ lock_retry_remap: if (cb_max_ofs && cb_pos < cb_end) { page = pages[cur_page]; if (page) - memset(page_address(page) + cur_ofs, 0, - cb_max_ofs - cur_ofs); + memzero_page(page, cur_ofs, cb_max_ofs - cur_ofs); /* * No need to update cb_pos at this stage: * cb_pos += cb_max_ofs - cur_ofs; @@ -739,7 +729,7 @@ lock_retry_remap: for (; cur_page < cb_max_page; cur_page++) { page = pages[cur_page]; if (page) - memcpy(page_address(page) + cur_ofs, cb_pos, + memcpy_to_page(page, cur_ofs, cb_pos, PAGE_SIZE - cur_ofs); cb_pos += PAGE_SIZE - cur_ofs; cur_ofs = 0; @@ -750,7 +740,7 @@ lock_retry_remap: if (cb_max_ofs && cb_pos < cb_end) { page = pages[cur_page]; if (page) - memcpy(page_address(page) + cur_ofs, cb_pos, + memcpy_to_page(page, cur_ofs, cb_pos, cb_max_ofs - cur_ofs); cb_pos += cb_max_ofs - cur_ofs; cur_ofs = cb_max_ofs; @@ -767,8 +757,6 @@ lock_retry_remap: */ handle_bounds_compressed_page(page, i_size, initialized_size); - flush_dcache_page(page); - kunmap_local(page_address(page)); SetPageUptodate(page); unlock_page(page); if (cur2_page == xpage) @@ -787,7 +775,7 @@ lock_retry_remap: unsigned int prev_cur_page = cur_page; ntfs_debug("Found compressed compression block."); - err = ntfs_decompress(pages, completed_pages, &cur_page, + err = ntfs_lznt1_codec_ops.decompress_pages(pages, completed_pages, &cur_page, &cur_ofs, cb_max_page, cb_max_ofs, xpage, &xpage_done, cb_pos, cb_size - (cb_pos - cb), i_size, initialized_size); @@ -804,7 +792,6 @@ lock_retry_remap: page = pages[prev_cur_page]; if (page) { flush_dcache_page(page); - kunmap_local(page_address(page)); unlock_page(page); if (prev_cur_page != xpage) put_page(page); @@ -822,14 +809,15 @@ lock_retry_remap: for (cur_page = 0; cur_page < max_page; cur_page++) { page = pages[cur_page]; if (page) { + folio = page_folio(page); + ntfs_error(vol->sb, "Still have pages left! Terminating them with extreme prejudice. Inode 0x%llx, page index 0x%lx.", - ni->mft_no, page->__folio_index); - flush_dcache_page(page); - kunmap_local(page_address(page)); - unlock_page(page); + ni->mft_no, folio->index); + flush_dcache_folio(folio); + folio_unlock(folio); if (cur_page != xpage) - put_page(page); + folio_put(folio); pages[cur_page] = NULL; } } @@ -864,7 +852,6 @@ err_out: page = pages[i]; if (page) { flush_dcache_page(page); - kunmap_local(page_address(page)); unlock_page(page); if (i != xpage) put_page(page); @@ -908,6 +895,12 @@ struct compress_context { s16 prev[NTFS_SB_SIZE]; }; +struct ntfs_compress_workspace { + struct page **pages; + char *outbuf; + unsigned int nr_pages; +}; + /* * Hash the next 3-byte sequence in the input buffer */ @@ -1084,12 +1077,11 @@ static void ntfs_skip_position(struct compress_context *pctx, const int i) * * Returns the size of the compressed block, including the * header (minimal size is 2, maximum size is 4098) - * 0 if an error has been met. + * A negative error code if an error has been met. */ -static unsigned int ntfs_compress_block(const char *inbuf, const int bufsize, - char *outbuf) +static int ntfs_compress_block(struct compress_context *pctx, + const char *inbuf, const int bufsize, char *outbuf) { - struct compress_context *pctx; int i; /* current position */ int j; /* end of best match from current position */ int k; /* end of best match from next position */ @@ -1104,10 +1096,6 @@ static unsigned int ntfs_compress_block(const char *inbuf, const int bufsize, int tag; /* current value of tag */ int ntag; /* count of bits still undefined in tag */ - pctx = kvzalloc(sizeof(struct compress_context), GFP_NOFS); - if (!pctx) - return -ENOMEM; - /* * All hash chains start as empty. The special value '-1' indicates the * end of each hash chain. @@ -1263,22 +1251,76 @@ static unsigned int ntfs_compress_block(const char *inbuf, const int bufsize, xout = NTFS_SB_SIZE + 2; } - /* - * Free the compression context and return the total number of bytes - * written to 'outbuf'. - */ - kvfree(pctx); return xout; } +static int ntfs_compress_workspace_init(struct ntfs_inode *ni, + struct ntfs_compress_workspace *ws) +{ + unsigned int size, i; + + size = ni->itype.compressed.block_size + 2 * + (ni->itype.compressed.block_size / NTFS_SB_SIZE) + 2; + ws->nr_pages = DIV_ROUND_UP(size, PAGE_SIZE); + ws->pages = kcalloc(ws->nr_pages, sizeof(*ws->pages), GFP_NOFS); + if (!ws->pages) + return -ENOMEM; + + for (i = 0; i < ws->nr_pages; i++) { + ws->pages[i] = alloc_page(GFP_NOFS); + if (!ws->pages[i]) + goto free_pages; + } + + ws->outbuf = vmap(ws->pages, ws->nr_pages, VM_MAP, PAGE_KERNEL); + if (!ws->outbuf) + goto free_pages; + return 0; + +free_pages: + while (i) + put_page(ws->pages[--i]); + kfree(ws->pages); + return -ENOMEM; +} + +static void ntfs_compress_workspace_free(struct ntfs_compress_workspace *ws) +{ + unsigned int i; + + vunmap(ws->outbuf); + for (i = 0; i < ws->nr_pages; i++) + put_page(ws->pages[i]); + kfree(ws->pages); +} + +static void ntfs_copy_cb(struct page **pages, int pages_per_cb, + unsigned int page_offset, + struct ntfs_compress_workspace *ws, unsigned int bytes) +{ + unsigned int copied = 0, i; + + for (i = 0; i < pages_per_cb && copied < bytes; i++) { + unsigned int offset = i ? 0 : page_offset; + unsigned int len = min(bytes - copied, PAGE_SIZE - offset); + void *addr = kmap_local_page(pages[i]); + + memcpy(ws->outbuf + copied, addr + offset, len); + kunmap_local(addr); + copied += len; + } +} + static int ntfs_write_cb(struct ntfs_inode *ni, loff_t pos, struct page **pages, - int pages_per_cb) + int pages_per_cb, unsigned int page_offset, + struct compress_context *ctx, struct ntfs_compress_workspace *ws) { struct ntfs_volume *vol = ni->vol; - char *outbuf = NULL, *pbuf, *inbuf; - u32 compsz, p, insz = pages_per_cb << PAGE_SHIFT; + char *outbuf = ws->outbuf, *pbuf; + u32 compsz, p, insz = ni->itype.compressed.block_size; s32 rounded, bio_size; - unsigned int sz, bsz; + int sz; + unsigned int bsz; bool fail = false, allzeroes; /* a single compressed zero */ static char onezero[] = {0x01, 0xb0, 0x00, 0x00}; @@ -1286,54 +1328,36 @@ static int ntfs_write_cb(struct ntfs_inode *ni, loff_t pos, struct page **pages, static char twozeroes[] = {0x02, 0xb0, 0x00, 0x00, 0x00}; /* more compressed zeroes, to be followed by some count */ static char morezeroes[] = {0x03, 0xb0, 0x02, 0x00}; - struct page **pages_disk = NULL, *pg; - s64 bio_lcn; + s64 bio_lcn, bio_pos; struct runlist_element *rlc, *rl; int i, err; - int pages_count = (round_up(ni->itype.compressed.block_size + 2 * - (ni->itype.compressed.block_size / NTFS_SB_SIZE) + 2, PAGE_SIZE)) / PAGE_SIZE; + u32 cb_clusters = ni->itype.compressed.block_clusters; size_t new_rl_count; struct bio *bio = NULL; - loff_t new_length; + loff_t cb_pos, new_length; s64 new_vcn; - inbuf = vmap(pages, pages_per_cb, VM_MAP, PAGE_KERNEL_RO); - if (!inbuf) - return -ENOMEM; - - /* may need 2 extra bytes per block and 2 more bytes */ - pages_disk = kcalloc(pages_count, sizeof(struct page *), GFP_NOFS); - if (!pages_disk) { - vunmap(inbuf); - return -ENOMEM; - } - - for (i = 0; i < pages_count; i++) { - pg = alloc_page(GFP_KERNEL); - if (!pg) { - err = -ENOMEM; - goto out; - } - pages_disk[i] = pg; - lock_page(pg); - kmap_local_page(pg); - } - - outbuf = vmap(pages_disk, pages_count, VM_MAP, PAGE_KERNEL); - if (!outbuf) { - err = -ENOMEM; - goto out; - } - compsz = 0; allzeroes = true; for (p = 0; (p < insz) && !fail; p += NTFS_SB_SIZE) { + unsigned int input_offset = page_offset + p; + unsigned int page_idx = input_offset >> PAGE_SHIFT; + const char *input; + void *addr; + if ((p + NTFS_SB_SIZE) < insz) bsz = NTFS_SB_SIZE; else bsz = insz - p; pbuf = &outbuf[compsz]; - sz = ntfs_compress_block(&inbuf[p], bsz, pbuf); + addr = kmap_local_page(pages[page_idx]); + input = addr + offset_in_page(input_offset); + sz = ntfs_lznt1_codec_ops.compress_subblock(ctx, input, bsz, pbuf); + kunmap_local(addr); + if (sz < 0) { + err = sz; + goto out; + } /* fail if all the clusters (or more) are needed */ if (!sz || ((compsz + sz + vol->cluster_size + 2) > ni->itype.compressed.block_size)) @@ -1360,28 +1384,25 @@ static int ntfs_write_cb(struct ntfs_inode *ni, loff_t pos, struct page **pages, } } + cb_pos = pos & ~((loff_t)ni->itype.compressed.block_size - 1); + new_vcn = ntfs_bytes_to_cluster(vol, cb_pos); + if (!fail && !allzeroes) { outbuf[compsz++] = 0; outbuf[compsz++] = 0; rounded = ((compsz - 1) | (vol->cluster_size - 1)) + 1; memset(&outbuf[compsz], 0, rounded - compsz); bio_size = rounded; - pages = pages_disk; } else if (allzeroes) { - err = 0; + err = ntfs_non_resident_attr_punch_hole(ni, new_vcn, cb_clusters); goto out; } else { + ntfs_copy_cb(pages, pages_per_cb, page_offset, ws, insz); bio_size = insz; } - new_vcn = ntfs_bytes_to_cluster(vol, - pos & ~((loff_t)ni->itype.compressed.block_size - 1)); new_length = ntfs_bytes_to_cluster(vol, round_up(bio_size, vol->cluster_size)); - err = ntfs_non_resident_attr_punch_hole(ni, new_vcn, ni->itype.compressed.block_clusters); - if (err < 0) - goto out; - rlc = ntfs_cluster_alloc(vol, new_vcn, new_length, -1, DATA_ZONE, false, true, true); if (IS_ERR(rlc)) { @@ -1390,74 +1411,56 @@ static int ntfs_write_cb(struct ntfs_inode *ni, loff_t pos, struct page **pages, } bio_lcn = rlc->lcn; + bio_pos = ntfs_cluster_to_bytes(vol, bio_lcn); + bio = bio_alloc(vol->sb->s_bdev, DIV_ROUND_UP(bio_size, PAGE_SIZE), + REQ_OP_WRITE, GFP_NOIO); + bio->bi_iter.bi_sector = ntfs_bytes_to_sector(vol, bio_pos); + + for (i = 0; bio_size; i++) { + unsigned int len = min_t(unsigned int, bio_size, PAGE_SIZE); + + if (bio_add_page(bio, ws->pages[i], len, 0) != len) { + err = -EIO; + bio_put(bio); + goto free_rlc; + } + bio_size -= len; + } + + err = submit_bio_wait(bio); + bio_put(bio); + if (err) + goto free_rlc; + + /* Do not discard the old compression block until the new one is safe. */ + err = ntfs_non_resident_attr_punch_hole(ni, new_vcn, cb_clusters); + if (err) + goto free_rlc; + down_write(&ni->runlist.lock); rl = ntfs_runlists_merge(&ni->runlist, rlc, 0, &new_rl_count); if (IS_ERR(rl)) { up_write(&ni->runlist.lock); ntfs_error(vol->sb, "Failed to merge runlists"); err = PTR_ERR(rl); - if (ntfs_cluster_free_from_rl(vol, rlc)) - ntfs_error(vol->sb, "Failed to free hot clusters."); - kvfree(rlc); - goto out; + goto free_rlc; } ni->runlist.count = new_rl_count; ni->runlist.rl = rl; + rlc = NULL; err = ntfs_attr_update_mapping_pairs(ni, 0); up_write(&ni->runlist.lock); - if (err) { + if (err) err = -EIO; - goto out; - } - - i = 0; - while (bio_size > 0) { - int page_size; - - if (bio_size >= PAGE_SIZE) { - page_size = PAGE_SIZE; - bio_size -= PAGE_SIZE; - } else { - page_size = bio_size; - bio_size = 0; - } - -setup_bio: - if (!bio) { - bio = bio_alloc(vol->sb->s_bdev, 1, REQ_OP_WRITE, - GFP_NOIO); - bio->bi_iter.bi_sector = - ntfs_bytes_to_sector(vol, - ntfs_cluster_to_bytes(vol, bio_lcn + i)); - } - - if (!bio_add_page(bio, pages[i], page_size, 0)) { - err = submit_bio_wait(bio); - bio_put(bio); - if (err) - goto out; - bio = NULL; - goto setup_bio; - } - i++; - } + goto out; - err = submit_bio_wait(bio); - bio_put(bio); +free_rlc: + if (ntfs_cluster_free_from_rl(vol, rlc)) + ntfs_error(vol->sb, "Failed to free hot clusters."); + kvfree(rlc); out: - vunmap(outbuf); - for (i = 0; i < pages_count; i++) { - pg = pages_disk[i]; - if (pg) { - kunmap_local(page_address(pg)); - unlock_page(pg); - put_page(pg); - } - } - kfree(pages_disk); - vunmap(inbuf); NInoSetFileNameDirty(ni); mark_mft_record_dirty(ni); @@ -1467,31 +1470,39 @@ out: int ntfs_compress_write(struct ntfs_inode *ni, loff_t pos, size_t count, struct iov_iter *from) { + struct ntfs_compress_workspace ws = {}; + struct compress_context *ctx; struct folio *folio; struct page **pages = NULL, *page; - int pages_per_cb = ni->itype.compressed.block_size >> PAGE_SHIFT; + int pages_per_cb; int cb_size = ni->itype.compressed.block_size, cb_off, err = 0; int i, ip; size_t written = 0; struct address_space *mapping = VFS_I(ni)->i_mapping; - if (NInoCompressed(ni) && pos + count > ni->allocated_size) { - int err; - loff_t end = pos + count; - - err = ntfs_attr_expand(ni, end, - round_up(end, ni->itype.compressed.block_size)); - if (err) - return err; - } + pages_per_cb = DIV_ROUND_UP(offset_in_page(pos & ~(cb_size - 1)) + + cb_size, PAGE_SIZE); pages = kmalloc_array(pages_per_cb, sizeof(struct page *), GFP_NOFS); if (!pages) return -ENOMEM; + ctx = kvzalloc_obj(*ctx, GFP_NOFS); + if (!ctx) { + kfree(pages); + return -ENOMEM; + } + err = ntfs_compress_workspace_init(ni, &ws); + if (err) { + kvfree(ctx); + kfree(pages); + return err; + } while (count) { pgoff_t index; size_t copied, bytes; + unsigned int page_offset; + bool full_cb; int off; off = pos & (cb_size - 1); @@ -1500,7 +1511,11 @@ int ntfs_compress_write(struct ntfs_inode *ni, loff_t pos, size_t count, bytes = count; cb_off = pos & ~(cb_size - 1); + page_offset = offset_in_page(cb_off); + pages_per_cb = DIV_ROUND_UP(page_offset + cb_size, PAGE_SIZE); index = cb_off >> PAGE_SHIFT; + full_cb = !off && bytes == cb_size && !page_offset && + !(cb_size & (PAGE_SIZE - 1)); if (unlikely(fault_in_iov_iter_readable(from, bytes))) { err = -EFAULT; @@ -1508,7 +1523,10 @@ int ntfs_compress_write(struct ntfs_inode *ni, loff_t pos, size_t count, } for (i = 0; i < pages_per_cb; i++) { - folio = read_mapping_folio(mapping, index + i, NULL); + if (full_cb) + folio = filemap_grab_folio(mapping, index + i); + else + folio = read_mapping_folio(mapping, index + i, NULL); if (IS_ERR(folio)) { for (ip = 0; ip < i; ip++) { folio_unlock(page_folio(pages[ip])); @@ -1518,7 +1536,8 @@ int ntfs_compress_write(struct ntfs_inode *ni, loff_t pos, size_t count, goto out; } - folio_lock(folio); + if (!full_cb) + folio_lock(folio); pages[i] = folio_page(folio, 0); } @@ -1548,13 +1567,26 @@ int ntfs_compress_write(struct ntfs_inode *ni, loff_t pos, size_t count, } } - err = ntfs_write_cb(ni, pos, pages, pages_per_cb); + if (!copied) { + err = -EFAULT; + goto release_pages; + } + + err = ntfs_write_cb(ni, pos, pages, pages_per_cb, page_offset, ctx, &ws); + if (!err && pos + copied > ni->initialized_size) { + mutex_lock(&ni->mrec_lock); + err = ntfs_attr_set_initialized_size(ni, pos + copied); + mutex_unlock(&ni->mrec_lock); + } +release_pages: for (i = 0; i < pages_per_cb; i++) { folio = page_folio(pages[i]); - if (i < ip) { + if (!err) { folio_clear_dirty(folio); folio_mark_uptodate(folio); + } else { + folio_clear_uptodate(folio); } folio_unlock(folio); folio_put(folio); @@ -1570,9 +1602,18 @@ int ntfs_compress_write(struct ntfs_inode *ni, loff_t pos, size_t count, } out: + ntfs_compress_workspace_free(&ws); + kvfree(ctx); kfree(pages); if (err < 0) written = err; return written; } + +const struct ntfs_codec_ops ntfs_lznt1_codec_ops = { + .id = NTFS_CODEC_LZNT1, + .name = "lznt1", + .decompress_pages = ntfs_decompress, + .compress_subblock = ntfs_compress_block, +}; diff --git a/fs/ntfs/dir.c b/fs/ntfs/dir.c index 6fa9ae3377cb..2d594cbb4ebe 100644 --- a/fs/ntfs/dir.c +++ b/fs/ntfs/dir.c @@ -966,13 +966,14 @@ filldir: */ private = file->private_data; kfree(private->key); - private->key = kmalloc(le16_to_cpu(next->key_length), GFP_KERNEL); + private->key = kmemdup(&next->key.file_name, + le16_to_cpu(next->key_length), + GFP_KERNEL); if (!private->key) { err = -ENOMEM; goto out; } - memcpy(private->key, &next->key.file_name, le16_to_cpu(next->key_length)); private->key_length = next->key_length; break; } diff --git a/fs/ntfs/ea.c b/fs/ntfs/ea.c index 0cd192752b7c..cdd306933d73 100644 --- a/fs/ntfs/ea.c +++ b/fs/ntfs/ea.c @@ -122,17 +122,19 @@ static int ntfs_get_ea(struct inode *inode, const char *name, size_t name_len, p_ea_info = ntfs_attr_readall(ni, AT_EA_INFORMATION, NULL, 0, &ea_info_size); - if (!p_ea_info || ea_info_size != sizeof(struct ea_information)) { + if (IS_ERR(p_ea_info)) + return PTR_ERR(p_ea_info); + if (ea_info_size != sizeof(struct ea_information)) { kvfree(p_ea_info); - return -ENODATA; + return -EIO; } ea_info_qlen = le32_to_cpu(p_ea_info->ea_query_length); kvfree(p_ea_info); ea_buf = ntfs_attr_readall(ni, AT_EA, NULL, 0, &all_ea_size); - if (!ea_buf) - return -ENODATA; + if (IS_ERR(ea_buf)) + return PTR_ERR(ea_buf); if (ea_info_qlen > all_ea_size) { err = -EIO; @@ -196,6 +198,9 @@ static int ntfs_set_ea(struct inode *inode, const char *name, size_t name_len, struct ea_attr *p_ea; u32 ea_info_qsize = 0; char *ea_buf = NULL; + char *new_ea_buf; + char *old_ea_buf = NULL; + struct ea_information old_ea_info; size_t new_ea_size = ALIGN(struct_size(p_ea, ea_name, 1 + name_len + val_size), 4); s64 ea_off, ea_info_size, all_ea_size, ea_size; @@ -205,10 +210,22 @@ static int ntfs_set_ea(struct inode *inode, const char *name, size_t name_len, if (ntfs_attr_exist(ni, AT_EA_INFORMATION, AT_UNNAMED, 0)) { p_ea_info = ntfs_attr_readall(ni, AT_EA_INFORMATION, NULL, 0, &ea_info_size); - if (!p_ea_info || ea_info_size != sizeof(struct ea_information)) + if (IS_ERR(p_ea_info)) { + err = PTR_ERR(p_ea_info); + p_ea_info = NULL; goto out; + } + if (ea_info_size != sizeof(struct ea_information)) { + err = -EIO; + goto out; + } ea_buf = ntfs_attr_readall(ni, AT_EA, NULL, 0, &all_ea_size); + if (IS_ERR(ea_buf)) { + err = PTR_ERR(ea_buf); + ea_buf = NULL; + goto out; + } if (!ea_buf) { ea_info_qsize = 0; kvfree(p_ea_info); @@ -249,6 +266,22 @@ create_ea_info: err = -EEXIST; goto out; } + if ((flags & XATTR_REPLACE) && !val_size) { + old_ea_info = *p_ea_info; + old_ea_buf = kvmemdup(ea_buf, all_ea_size, GFP_NOFS); + if (!old_ea_buf) { + err = -ENOMEM; + goto out; + } + } + + /* Check the final $EA size before removing the old entry. */ + if (val_size && + ntfs_attr_size_bounds_check(ni->vol, AT_EA, + ea_info_qsize - ea_size + new_ea_size)) { + err = -EFBIG; + goto out; + } p_ea = (struct ea_attr *)(ea_buf + ea_off); @@ -267,17 +300,39 @@ create_ea_info: ea_info_qsize -= ea_size; p_ea_info->ea_query_length = cpu_to_le32(ea_info_qsize); - err = ntfs_write_ea(ni, AT_EA_INFORMATION, (char *)p_ea_info, 0, - sizeof(struct ea_information), false); - if (err) - goto out; + if ((flags & XATTR_REPLACE) && !val_size && !ea_info_qsize) { + err = ntfs_attr_remove(ni, AT_EA, AT_UNNAMED, 0); + if (err) + goto out; - err = ntfs_write_ea(ni, AT_EA, ea_buf, 0, ea_info_qsize, true); - if (err) + err = ntfs_attr_remove(ni, AT_EA_INFORMATION, AT_UNNAMED, 0); + if (err) { + /* Restore the original $EA if $EA_INFORMATION removal failed. */ + ntfs_attr_add(ni, AT_EA, AT_UNNAMED, 0, old_ea_buf, + all_ea_size); + ea_info_qsize = le32_to_cpu(old_ea_info.ea_query_length); + } goto out; + } if ((flags & XATTR_REPLACE) && !val_size) { - /* Remove xattr. */ + err = ntfs_write_ea(ni, AT_EA, ea_buf, 0, ea_info_qsize, + true); + if (err) { + ntfs_write_ea(ni, AT_EA, old_ea_buf, 0, + all_ea_size, false); + goto out; + } + + err = ntfs_write_ea(ni, AT_EA_INFORMATION, (char *)p_ea_info, + 0, sizeof(struct ea_information), false); + if (err) { + ntfs_write_ea(ni, AT_EA, old_ea_buf, 0, + all_ea_size, false); + ntfs_write_ea(ni, AT_EA_INFORMATION, + (char *)&old_ea_info, 0, + sizeof(old_ea_info), false); + } goto out; } } else { @@ -285,22 +340,30 @@ create_ea_info: err = -ENODATA; goto out; } - } - kvfree(ea_buf); + if (ntfs_attr_size_bounds_check(ni->vol, AT_EA, + ea_info_qsize + new_ea_size)) { + err = -EFBIG; + goto out; + } + } alloc_new_ea: - ea_buf = kzalloc(new_ea_size, GFP_NOFS); - if (!ea_buf) { + new_ea_buf = kvzalloc(ea_info_qsize + new_ea_size, GFP_NOFS); + if (!new_ea_buf) { err = -ENOMEM; goto out; } + if (ea_info_qsize) + memcpy(new_ea_buf, ea_buf, ea_info_qsize); + kvfree(ea_buf); + ea_buf = new_ea_buf; + p_ea = (struct ea_attr *)(ea_buf + ea_info_qsize); /* * EA and REPARSE_POINT compatibility not checked any more, * required by Windows 10, but having both may lead to * problems with earlier versions. */ - p_ea = (struct ea_attr *)ea_buf; memcpy(p_ea->ea_name, name, name_len); p_ea->ea_name_length = name_len; p_ea->ea_name[name_len] = 0; @@ -312,8 +375,7 @@ alloc_new_ea: p_ea_info->ea_length = cpu_to_le16(ea_packed); p_ea_info->ea_query_length = cpu_to_le32(ea_info_qsize + new_ea_size); - if (ea_packed > 0xffff || - ntfs_attr_size_bounds_check(ni->vol, AT_EA, new_ea_size)) { + if (ea_packed > 0xffff) { err = -EFBIG; goto out; } @@ -322,13 +384,13 @@ alloc_new_ea: * no EA or EA_INFORMATION : add them */ if (!ntfs_attr_exist(ni, AT_EA, AT_UNNAMED, 0)) { - err = ntfs_attr_add(ni, AT_EA, AT_UNNAMED, 0, (char *)p_ea, - new_ea_size); + err = ntfs_attr_add(ni, AT_EA, AT_UNNAMED, 0, ea_buf, + ea_info_qsize + new_ea_size); if (err) goto out; } else { - err = ntfs_write_ea(ni, AT_EA, (char *)p_ea, ea_info_qsize, - new_ea_size, false); + err = ntfs_write_ea(ni, AT_EA, ea_buf, 0, + ea_info_qsize + new_ea_size, true); if (err) goto out; } @@ -348,6 +410,7 @@ out: NInoClearHasEA(ni); kvfree(ea_buf); + kvfree(old_ea_buf); kvfree(p_ea_info); return err; @@ -357,37 +420,35 @@ out: * Check for the presence of an EA "$LXDEV" (used by WSL) * and return its value as a device address */ -int ntfs_ea_get_wsl_inode(struct inode *inode, dev_t *rdevp, unsigned int flags) +int ntfs_ea_get_wsl_inode(struct inode *inode, dev_t *rdevp, unsigned int flags, + bool *has_lxmod) { int err; __le32 v; + *has_lxmod = false; + if (!(flags & NTFS_VOL_UID)) { /* Load uid to lxuid EA */ err = ntfs_get_ea(inode, "$LXUID", sizeof("$LXUID") - 1, &v, sizeof(v)); - if (err < 0) - return err; - if (err != sizeof(v)) - return -EIO; - i_uid_write(inode, le32_to_cpu(v)); + if (err == sizeof(v)) + i_uid_write(inode, le32_to_cpu(v)); } if (!(flags & NTFS_VOL_GID)) { /* Load gid to lxgid EA */ err = ntfs_get_ea(inode, "$LXGID", sizeof("$LXGID") - 1, &v, sizeof(v)); - if (err < 0) - return err; - if (err != sizeof(v)) - return -EIO; - i_gid_write(inode, le32_to_cpu(v)); + if (err == sizeof(v)) + i_gid_write(inode, le32_to_cpu(v)); } /* Load mode to lxmod EA */ err = ntfs_get_ea(inode, "$LXMOD", sizeof("$LXMOD") - 1, &v, sizeof(v)); if (err == sizeof(v)) { inode->i_mode = le32_to_cpu(v); + *has_lxmod = true; } else { /* Everyone gets all permissions. */ inode->i_mode |= 0777; @@ -467,14 +528,24 @@ ssize_t ntfs_listxattr(struct dentry *dentry, char *buffer, size_t size) mutex_lock(&NTFS_I(inode)->mrec_lock); ea_info = ntfs_attr_readall(ni, AT_EA_INFORMATION, NULL, 0, &ea_info_size); - if (!ea_info || ea_info_size != sizeof(struct ea_information)) + if (IS_ERR(ea_info)) { + err = PTR_ERR(ea_info); + ea_info = NULL; + goto out; + } + if (ea_info_size != sizeof(struct ea_information)) { + err = -EIO; goto out; + } ea_info_qsize = le32_to_cpu(ea_info->ea_query_length); ea_buf = ntfs_attr_readall(ni, AT_EA, NULL, 0, &ea_buf_size); - if (!ea_buf) + if (IS_ERR(ea_buf)) { + err = PTR_ERR(ea_buf); + ea_buf = NULL; goto out; + } if (ea_info_qsize > ea_buf_size || ea_info_qsize == 0) goto out; @@ -581,7 +652,8 @@ static int ntfs_new_attr_flags(struct ntfs_inode *ni, __le32 fattr) struct mft_record *m; struct attr_record *a; __le16 new_aflags; - int mp_size, mp_ofs, name_ofs, arec_size, err; + u16 old_name_ofs, old_mp_ofs; + int mp_size, mp_ofs, name_ofs, old_arec_size, arec_size, err; m = map_mft_record(ni); if (IS_ERR(m)) @@ -613,8 +685,10 @@ static int ntfs_new_attr_flags(struct ntfs_inode *ni, __le32 fattr) else new_aflags &= ~ATTR_IS_COMPRESSED; - if (new_aflags == a->flags) - return 0; + if (new_aflags == a->flags) { + err = 0; + goto err_out; + } if ((new_aflags & (ATTR_IS_SPARSE | ATTR_IS_COMPRESSED)) == (ATTR_IS_SPARSE | ATTR_IS_COMPRESSED)) { @@ -623,15 +697,42 @@ static int ntfs_new_attr_flags(struct ntfs_inode *ni, __le32 fattr) goto err_out; } - if (!a->non_resident) - goto out; + if (!a->non_resident) { + if (!(new_aflags & (ATTR_IS_SPARSE | ATTR_IS_COMPRESSED))) { + err = 0; + goto err_out; + } - if (a->data.non_resident.data_size) { - pr_err("Can't change sparsed/compressed for non-empty file\n"); - err = -EOPNOTSUPP; - goto err_out; + if (le32_to_cpu(a->data.resident.value_length)) { + pr_err("Can't change sparse/compressed for non-empty file"); + err = -EOPNOTSUPP; + goto err_out; + } + + err = ntfs_attr_make_non_resident(ni, 0); + if (err) + goto err_out; + + ntfs_attr_reinit_search_ctx(ctx); + err = ntfs_attr_lookup(ni->type, ni->name, + ni->name_len, CASE_SENSITIVE, + 0, NULL, 0, ctx); + if (err) { + err = -EINVAL; + goto err_out; + } + a = ctx->attr; + } else { + if (a->data.non_resident.data_size) { + pr_err("Can't change sparsed/compressed for non-empty file"); + err = -EOPNOTSUPP; + goto err_out; + } } + old_name_ofs = le16_to_cpu(a->name_offset); + old_mp_ofs = le16_to_cpu(a->data.non_resident.mapping_pairs_offset); + if (new_aflags & (ATTR_IS_SPARSE | ATTR_IS_COMPRESSED)) name_ofs = (offsetof(struct attr_record, data.non_resident.compressed_size) + @@ -649,11 +750,37 @@ static int ntfs_new_attr_flags(struct ntfs_inode *ni, __le32 fattr) mp_ofs = (name_ofs + a->name_length * sizeof(__le16) + 7) & ~7; arec_size = (mp_ofs + mp_size + 7) & ~7; + old_arec_size = le32_to_cpu(a->length); + + /* + * Move payloads before shrinking the record. Otherwise resizing moves + * the following attribute over the old payload before it can be copied. + */ + if (arec_size < old_arec_size) { + if (a->name_length && name_ofs != old_name_ofs) + memmove((u8 *)a + name_ofs, (u8 *)a + old_name_ofs, + a->name_length * sizeof(__le16)); + if (mp_ofs != old_mp_ofs) + memmove((u8 *)a + mp_ofs, (u8 *)a + old_mp_ofs, mp_size); + } err = ntfs_attr_record_resize(m, a, arec_size); if (unlikely(err)) goto err_out; + /* + * When compressed/sparse state changes, the non-resident header grows or + * shrinks by the compressed_size field. Update the in-record payload layout + * to match the new offsets before exposing the new mapping_pairs_offset. + */ + if (arec_size > old_arec_size) { + if (mp_ofs != old_mp_ofs) + memmove((u8 *)a + mp_ofs, (u8 *)a + old_mp_ofs, mp_size); + if (a->name_length) + memmove((u8 *)a + name_ofs, (u8 *)a + old_name_ofs, + a->name_length * sizeof(__le16)); + } + if (new_aflags & (ATTR_IS_SPARSE | ATTR_IS_COMPRESSED)) { a->data.non_resident.compression_unit = 0; if (new_aflags & ATTR_IS_COMPRESSED || ni->vol->major_ver < 3) @@ -674,28 +801,31 @@ static int ntfs_new_attr_flags(struct ntfs_inode *ni, __le32 fattr) ni->itype.compressed.block_size_bits = 0; ni->itype.compressed.block_clusters = 0; } - - if (new_aflags & ATTR_IS_SPARSE) { - NInoSetSparse(ni); - ni->flags |= FILE_ATTR_SPARSE_FILE; - } - - if (new_aflags & ATTR_IS_COMPRESSED) { - NInoSetCompressed(ni); - ni->flags |= FILE_ATTR_COMPRESSED; - } } else { - ni->flags &= ~(FILE_ATTR_SPARSE_FILE | FILE_ATTR_COMPRESSED); a->data.non_resident.compression_unit = 0; - NInoClearSparse(ni); - NInoClearCompressed(ni); } a->name_offset = cpu_to_le16(name_ofs); a->data.non_resident.mapping_pairs_offset = cpu_to_le16(mp_ofs); -out: a->flags = new_aflags; + + if (new_aflags & ATTR_IS_SPARSE) { + NInoSetSparse(ni); + ni->flags |= FILE_ATTR_SPARSE_FILE; + } else { + NInoClearSparse(ni); + ni->flags &= ~FILE_ATTR_SPARSE_FILE; + } + + if (new_aflags & ATTR_IS_COMPRESSED) { + NInoSetCompressed(ni); + ni->flags |= FILE_ATTR_COMPRESSED; + } else { + NInoClearCompressed(ni); + ni->flags &= ~FILE_ATTR_COMPRESSED; + } + mark_mft_record_dirty(ctx->ntfs_ino); err_out: if (ctx) @@ -704,6 +834,23 @@ err_out: return err; } +static bool ntfs_is_reserved_lxattr(const char *name) +{ + return !strcmp(name, "$LXUID") || !strcmp(name, "$LXGID") || + !strcmp(name, "$LXMOD") || !strcmp(name, "$LXDEV"); +} + +static int ntfs_validate_fattr(struct ntfs_inode *ni, __le32 fattr) +{ + const __le32 wof_flags = FILE_ATTR_SPARSE_FILE | + FILE_ATTR_REPARSE_POINT; + + if (NInoWofCompressed(ni) && ((ni->flags ^ fattr) & wof_flags)) + return -EPERM; + + return 0; +} + static int ntfs_setxattr(const struct xattr_handler *handler, struct mnt_idmap *idmap, struct dentry *unused, struct inode *inode, const char *name, const void *value, @@ -716,12 +863,16 @@ static int ntfs_setxattr(const struct xattr_handler *handler, if (NVolShutdown(ni->vol)) return -EIO; + if (ntfs_is_reserved_lxattr(name) && !capable(CAP_SYS_ADMIN)) + return -EPERM; + if (!strcmp(name, SYSTEM_DOS_ATTRIB)) { if (sizeof(u8) != size) { err = -EINVAL; goto out; } - fattr = cpu_to_le32(*(u8 *)value); + fattr = cpu_to_le32((le32_to_cpu(ni->flags) & ~0xffU) | + *(u8 *)value); goto set_fattr; } @@ -736,6 +887,10 @@ static int ntfs_setxattr(const struct xattr_handler *handler, else fattr = cpu_to_le32(*(u32 *)value); + err = ntfs_validate_fattr(ni, fattr); + if (err) + goto out; + if (S_ISREG(inode->i_mode)) { mutex_lock(&ni->mrec_lock); err = ntfs_new_attr_flags(ni, fattr); @@ -750,6 +905,10 @@ set_fattr: else fattr &= ~FILE_ATTR_DIRECTORY; + err = ntfs_validate_fattr(ni, fattr); + if (err) + goto out; + if (ni->flags != fattr) { ni->flags = fattr; if (fattr & FILE_ATTR_READONLY) @@ -768,8 +927,10 @@ set_fattr: mutex_unlock(&ni->mrec_lock); out: - inode_set_ctime_current(inode); - mark_inode_dirty(inode); + if (!err) { + inode_set_ctime_current(inode); + mark_inode_dirty(inode); + } return err; } diff --git a/fs/ntfs/ea.h b/fs/ntfs/ea.h index 1f63bd55e057..acb39c2a6fbc 100644 --- a/fs/ntfs/ea.h +++ b/fs/ntfs/ea.h @@ -10,7 +10,8 @@ extern const struct xattr_handler *const ntfs_xattr_handlers[]; int ntfs_ea_set_wsl_not_symlink(struct ntfs_inode *ni, mode_t mode, dev_t dev); -int ntfs_ea_get_wsl_inode(struct inode *inode, dev_t *rdevp, unsigned int flags); +int ntfs_ea_get_wsl_inode(struct inode *inode, dev_t *rdevp, unsigned int flags, + bool *has_lxmod); int ntfs_ea_set_wsl_inode(struct inode *inode, dev_t rdev, __le16 *ea_size, unsigned int flags); ssize_t ntfs_listxattr(struct dentry *dentry, char *buffer, size_t size); diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c index 6a7b638e523d..88747217ba61 100644 --- a/fs/ntfs/file.c +++ b/fs/ntfs/file.c @@ -128,7 +128,8 @@ out_unlock: static int ntfs_file_release(struct inode *vi, struct file *filp) { - if (!NInoCompressed(NTFS_I(vi))) + if (!NInoCompressed(NTFS_I(vi)) && + !NInoWofCompressed(NTFS_I(vi))) return ntfs_trim_prealloc(vi); return 0; @@ -256,10 +257,11 @@ static int ntfs_setattr_size(struct inode *vi, struct iattr *attr) int err; loff_t old_size = vi->i_size; - if (NInoCompressed(ni) || NInoEncrypted(ni)) { - ntfs_warning(vi->i_sb, - "Changes in inode size are not supported yet for %s files, ignoring.", - NInoCompressed(ni) ? "compressed" : "encrypted"); + if (NInoCompressed(ni) || NInoEncrypted(ni) || NInoWofCompressed(ni)) { + ntfs_warning( + vi->i_sb, + "Changes in inode size are not supported yet for %s files.", + NInoEncrypted(ni) ? "encrypted" : "compressed"); return -EOPNOTSUPP; } @@ -268,23 +270,19 @@ static int ntfs_setattr_size(struct inode *vi, struct iattr *attr) return err; inode_dio_wait(vi); - truncate_setsize(vi, attr->ia_size); + if (attr->ia_size > old_size) { + truncate_pagecache(vi, old_size); + i_size_write(vi, attr->ia_size); + pagecache_isize_extended(vi, old_size, attr->ia_size); + } else + truncate_setsize(vi, attr->ia_size); + err = ntfs_truncate_vfs(vi, attr->ia_size, old_size); if (err) { i_size_write(vi, old_size); return err; } - if (NInoNonResident(ni) && attr->ia_size > old_size && - old_size % PAGE_SIZE != 0) { - loff_t len = min_t(loff_t, - round_up(old_size, PAGE_SIZE) - old_size, - attr->ia_size - old_size); - err = iomap_zero_range(vi, old_size, len, - NULL, &ntfs_seek_iomap_ops, - &ntfs_iomap_folio_ops, NULL); - } - return err; } @@ -312,6 +310,13 @@ int ntfs_setattr(struct mnt_idmap *idmap, struct dentry *dentry, if (err) goto out; + if ((ia_valid & ATTR_SIZE) && + (NInoCompressed(ni) || NInoEncrypted(ni) || + NInoWofCompressed(ni))) { + err = -EOPNOTSUPP; + goto out; + } + if (!(vol->vol_flags & VOLUME_IS_DIRTY)) ntfs_set_volume_flags(vol, VOLUME_IS_DIRTY); @@ -346,14 +351,12 @@ int ntfs_setattr(struct mnt_idmap *idmap, struct dentry *dentry, if (ia_valid & ATTR_MODE) flags |= NTFS_EA_MODE; - if (S_ISDIR(vi->i_mode)) - vi->i_mode &= ~vol->dmask; - else - vi->i_mode &= ~vol->fmask; - mutex_lock(&ni->mrec_lock); - ntfs_ea_set_wsl_inode(vi, 0, NULL, flags); + err = ntfs_ea_set_wsl_inode(vi, 0, NULL, flags); mutex_unlock(&ni->mrec_lock); + if (err) + goto out; + } mark_inode_dirty(vi); @@ -376,7 +379,7 @@ int ntfs_getattr(struct mnt_idmap *idmap, const struct path *path, stat->result_mask |= STATX_BTIME; stat->btime = NTFS_I(inode)->i_crtime; - if (NInoCompressed(ni)) + if (NInoCompressed(ni) || NInoWofCompressed(ni)) stat->attributes |= STATX_ATTR_COMPRESSED; if (NInoEncrypted(ni)) @@ -401,7 +404,8 @@ int ntfs_getattr(struct mnt_idmap *idmap, const struct path *path, bdev_logical_block_size(inode->i_sb->s_bdev); stat->result_mask |= STATX_DIOALIGN; - if (!NInoCompressed(ni) && !NInoEncrypted(ni)) { + if (!NInoCompressed(ni) && !NInoEncrypted(ni) && + !NInoWofCompressed(ni)) { stat->dio_mem_align = align; stat->dio_offset_align = align; } @@ -414,6 +418,10 @@ static loff_t ntfs_file_llseek(struct file *file, loff_t offset, int whence) { struct inode *inode = file->f_mapping->host; + if (NInoWofCompressed(NTFS_I(inode)) && + (whence == SEEK_HOLE || whence == SEEK_DATA)) + return -EOPNOTSUPP; + switch (whence) { case SEEK_HOLE: inode_lock_shared(inode); @@ -444,7 +452,8 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to) if (NVolShutdown(NTFS_SB(sb))) return -EIO; - if (NInoCompressed(NTFS_I(vi)) && iocb->ki_flags & IOCB_DIRECT) + if ((NInoCompressed(NTFS_I(vi)) || NInoWofCompressed(NTFS_I(vi))) && + iocb->ki_flags & IOCB_DIRECT) return -EOPNOTSUPP; inode_lock_shared(vi); @@ -535,6 +544,31 @@ out: return ret; } +static int ntfs_expand_for_write(struct ntfs_inode *ni, loff_t end) +{ + struct ntfs_volume *vol = ni->vol; + loff_t prealloc_size = 0; + int err; + + if (end <= ni->data_size) + return 0; + + if (NInoCompressed(ni)) { + if (end > ni->allocated_size) + prealloc_size = round_up(end, + ni->itype.compressed.block_size); + } else if (end > ni->allocated_size && + end < ni->allocated_size + vol->preallocated_size) { + prealloc_size = ni->allocated_size + vol->preallocated_size; + } + + mutex_lock(&ni->mrec_lock); + err = ntfs_attr_expand(ni, end, prealloc_size); + mutex_unlock(&ni->mrec_lock); + + return err; +} + static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) { struct file *file = iocb->ki_filp; @@ -543,13 +577,16 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) struct ntfs_volume *vol = ni->vol; ssize_t ret; ssize_t count; - loff_t pos; + loff_t pos, end; int err; loff_t old_data_size, old_init_size; if (NVolShutdown(vol)) return -EIO; + if (NInoWofCompressed(ni)) + return -EOPNOTSUPP; + if (NInoEncrypted(ni)) { ntfs_error(vi->i_sb, "Writing for %s files is not supported yet", NInoCompressed(ni) ? "Compressed" : "Encrypted"); @@ -580,10 +617,24 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) pos = iocb->ki_pos; count = ret; + end = pos + count; old_data_size = ni->data_size; old_init_size = ni->initialized_size; + if (end > old_data_size) { + ret = ntfs_expand_for_write(ni, end); + if (ret < 0) + goto out; + } + + if (NInoNonResident(ni) && !NInoCompressed(ni) && + end > old_init_size) { + ret = ntfs_extend_initialized_size(vi, pos, end); + if (ret < 0) + goto out; + } + if (NInoNonResident(ni) && NInoCompressed(ni)) { ret = ntfs_compress_write(ni, pos, count, from); if (ret > 0) @@ -620,6 +671,9 @@ static vm_fault_t ntfs_filemap_page_mkwrite(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); vm_fault_t ret; + if (NInoWofCompressed(NTFS_I(inode))) + return VM_FAULT_SIGBUS; + sb_start_pagefault(inode->i_sb); file_update_time(vmf->vma->vm_file); @@ -642,7 +696,7 @@ static int ntfs_file_mmap_prepare(struct vm_area_desc *desc) if (NVolShutdown(NTFS_SB(file->f_mapping->host->i_sb))) return -EIO; - if (NInoCompressed(NTFS_I(inode))) + if (NInoCompressed(NTFS_I(inode)) || NInoWofCompressed(NTFS_I(inode))) return -EOPNOTSUPP; if (vma_desc_test_all(desc, VMA_SHARED_BIT, VMA_MAYWRITE_BIT)) { @@ -655,7 +709,7 @@ static int ntfs_file_mmap_prepare(struct vm_area_desc *desc) from + desc->end - desc->start); if (NTFS_I(inode)->initialized_size < to) { - err = ntfs_extend_initialized_size(inode, to, to, false); + err = ntfs_extend_initialized_size(inode, to, to); if (err) return err; } @@ -670,6 +724,9 @@ static int ntfs_file_mmap_prepare(struct vm_area_desc *desc) static int ntfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, u64 start, u64 len) { + if (NInoWofCompressed(NTFS_I(inode))) + return -EOPNOTSUPP; + return iomap_fiemap(inode, fieinfo, start, len, &ntfs_read_iomap_ops); } @@ -1064,6 +1121,9 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t offset, loff_t le if (mode & ~(NTFS_FALLOC_FL_SUPPORTED)) return -EOPNOTSUPP; + if (NInoCompressed(ni) || NInoEncrypted(ni) || NInoWofCompressed(ni)) + return -EOPNOTSUPP; + if (!NVolFreeClusterKnown(vol)) wait_event(vol->free_waitq, NVolFreeClusterKnown(vol)); @@ -1087,7 +1147,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t offset, loff_t le old_size = i_size_read(vi); inode_lock(vi); - if (NInoCompressed(ni) || NInoEncrypted(ni)) { + if (NInoCompressed(ni) || NInoEncrypted(ni) || NInoWofCompressed(ni)) { err = -EOPNOTSUPP; goto out; } @@ -1126,13 +1186,9 @@ out: filemap_invalidate_unlock(vi->i_mapping); if (!err) { if (mode == 0 && NInoNonResident(ni) && - offset > old_size && old_size % PAGE_SIZE != 0) { - loff_t len = min_t(loff_t, - round_up(old_size, PAGE_SIZE) - old_size, - offset - old_size); - err = iomap_zero_range(vi, old_size, len, NULL, - &ntfs_seek_iomap_ops, - &ntfs_iomap_folio_ops, NULL); + offset > old_size) { + truncate_pagecache(vi, old_size); + pagecache_isize_extended(vi, old_size, offset); } NInoSetFileNameDirty(ni); inode_set_mtime_to_ts(vi, inode_set_ctime_current(vi)); diff --git a/fs/ntfs/index.c b/fs/ntfs/index.c index faa7ee920a3a..46a8b19c0723 100644 --- a/fs/ntfs/index.c +++ b/fs/ntfs/index.c @@ -616,6 +616,31 @@ static struct index_root *ntfs_ir_lookup2(struct ntfs_inode *ni, __le16 *name, u return ir; } +static int ntfs_ir_move_to_base(struct ntfs_index_context *icx) +{ + struct ntfs_attr_search_ctx *ctx = NULL; + struct index_root *ir; + bool moved = false; + int ret = 0; + + ir = ntfs_ir_lookup(icx->idx_ni, icx->name, icx->name_len, &ctx); + if (!ir) + return -ENOENT; + + if (ctx->ntfs_ino->mft_no != icx->idx_ni->mft_no) { + ret = ntfs_attr_record_move_to(ctx, icx->idx_ni); + if (!ret) { + moved = true; + ret = ntfs_attrlist_update(icx->idx_ni); + } + } + + ntfs_attr_put_search_ctx(ctx); + if (!ret && moved) + ret = ntfs_inode_free_empty_extents(icx->idx_ni); + return ret; +} + /* * Find a key in the index block. */ @@ -989,6 +1014,7 @@ static s64 ntfs_ibm_pos_to_vcn(struct ntfs_index_context *icx, s64 pos) static int ntfs_ibm_add(struct ntfs_index_context *icx) { u8 bmp[8]; + int ret; ntfs_debug("Entering\n"); @@ -998,10 +1024,11 @@ static int ntfs_ibm_add(struct ntfs_index_context *icx) * AT_BITMAP must be at least 8 bytes. */ memset(bmp, 0, sizeof(bmp)); - if (ntfs_attr_add(icx->idx_ni, AT_BITMAP, icx->name, icx->name_len, - bmp, sizeof(bmp))) { + ret = ntfs_attr_add(icx->idx_ni, AT_BITMAP, icx->name, icx->name_len, + bmp, sizeof(bmp)); + if (ret) { ntfs_error(icx->idx_ni->vol->sb, "Failed to add AT_BITMAP"); - return -EINVAL; + return ret; } return 0; @@ -1074,14 +1101,15 @@ static s64 ntfs_ibm_get_free(struct ntfs_index_context *icx) { u8 *bm; int bit; + int ret; s64 vcn, byte, size; ntfs_debug("Entering\n"); bm = ntfs_attr_readall(icx->idx_ni, AT_BITMAP, icx->name, icx->name_len, &size); - if (!bm) - return (s64)-1; + if (IS_ERR(bm)) + return PTR_ERR(bm); for (byte = 0; byte < size; byte++) { if (bm[byte] == 255) @@ -1099,10 +1127,12 @@ static s64 ntfs_ibm_get_free(struct ntfs_index_context *icx) out: ntfs_debug("allocated vcn: %lld\n", vcn); - if (ntfs_ibm_set(icx, vcn)) - vcn = (s64)-1; + ret = ntfs_ibm_set(icx, vcn); kvfree(bm); + if (ret) + return ret; + return vcn; } @@ -1112,6 +1142,7 @@ static struct index_block *ntfs_ir_to_ib(struct index_root *ir, s64 ib_vcn) struct index_entry *ie_last; char *ies_start, *ies_end; int i; + u32 ib_cap; ntfs_debug("Entering\n"); @@ -1127,6 +1158,16 @@ static struct index_block *ntfs_ir_to_ib(struct index_root *ir, s64 ib_vcn) * as well, which can never have any data. */ i = (char *)ie_last - ies_start + le16_to_cpu(ie_last->length); + + /* Entries must fit in the allocated index block */ + ib_cap = le32_to_cpu(ib->index.allocated_size) - + le32_to_cpu(ib->index.entries_offset); + if ((u32)i > ib_cap) { + ntfs_error(NULL, "Entries (%d B) exceed IB capacity", i); + kvfree(ib); + return NULL; + } + memcpy(ntfs_ie_get_first(&ib->index), ies_start, i); ib->index.flags = ir->index.flags; @@ -1264,7 +1305,7 @@ static int ntfs_ir_reparent(struct ntfs_index_context *icx) new_ib_vcn = ntfs_ibm_get_free(icx); if (new_ib_vcn < 0) { - ret = -EINVAL; + ret = (int)new_ib_vcn; goto out; } @@ -1335,19 +1376,42 @@ resize_failed: * When there is no space to build a non-resident * index, we may have to move the root to an extent */ - if ((ret == -ENOSPC) && (ctx->al_entry || !ntfs_inode_add_attrlist(icx->idx_ni))) { - ntfs_attr_put_search_ctx(ctx); - ctx = NULL; - ir = ntfs_ir_lookup(icx->idx_ni, icx->name, icx->name_len, &ctx); - if (ir && !ntfs_attr_record_move_away(ctx, ix_root_size - - le32_to_cpu(ctx->attr->data.resident.value_length))) { - if (ntfs_attrlist_update(ctx->base_ntfs_ino ? - ctx->base_ntfs_ino : ctx->ntfs_ino)) + if (ret == -ENOSPC) { + if (!ctx->al_entry) { + ret = ntfs_inode_add_attrlist(icx->idx_ni); + if (ret) goto clear_bmp; + ntfs_attr_put_search_ctx(ctx); ctx = NULL; goto retry; } + + if (ctx->ntfs_ino->mft_no != icx->idx_ni->mft_no) + goto clear_bmp; + + ret = ntfs_attr_record_move_away(ctx, ix_root_size - + le32_to_cpu(ctx->attr->data.resident.value_length)); + if (ret) + goto clear_bmp; + + ret = ntfs_attrlist_update(icx->idx_ni); + if (ret) { + int rollback_ret; + + ntfs_attr_put_search_ctx(ctx); + ctx = NULL; + rollback_ret = ntfs_ir_move_to_base(icx); + if (rollback_ret) + ntfs_error(icx->idx_ni->vol->sb, + "Failed to roll back INDEX_ROOT relocation: %d", + rollback_ret); + goto clear_bmp; + } + + ntfs_attr_put_search_ctx(ctx); + ctx = NULL; + goto retry; } clear_bmp: ntfs_ibm_clear(icx, new_ib_vcn); @@ -1579,7 +1643,7 @@ resplit: median = ntfs_ie_get_median(&ib->index); new_vcn = ntfs_ibm_get_free(icx); if (new_vcn < 0) { - ret = -EINVAL; + ret = (int)new_vcn; goto out; } diff --git a/fs/ntfs/inode.c b/fs/ntfs/inode.c index 7381a18cfadd..32edb4045178 100644 --- a/fs/ntfs/inode.c +++ b/fs/ntfs/inode.c @@ -173,10 +173,11 @@ struct inode *ntfs_iget(struct super_block *sb, u64 mft_no) unlock_new_inode(vi); } /* - * There is no point in keeping bad inodes around if the failure was - * due to ENOMEM. We want to be able to retry again later. + * There is no point in keeping bad inodes around. This also + * simplifies things in that we never need to check for bad inodes + * elsewhere. */ - if (unlikely(err == -ENOMEM)) { + if (unlikely(err)) { iput(vi); vi = ERR_PTR(err); } @@ -682,6 +683,7 @@ static int ntfs_read_locked_inode(struct inode *vi) unsigned int name_len = 4, flags = 0; int extend_sys = 0; dev_t dev = 0; + bool has_lxmod = false; bool vol_err = true; ntfs_debug("Entering for i_ino 0x%llx.", ni->mft_no); @@ -862,13 +864,15 @@ skip_attr_list_load: err = ntfs_attr_lookup(AT_EA_INFORMATION, NULL, 0, 0, 0, NULL, 0, ctx); if (!err) { NInoSetHasEA(ni); - ntfs_ea_get_wsl_inode(vi, &dev, flags); + ntfs_ea_get_wsl_inode(vi, &dev, flags, &has_lxmod); } if (ni->flags & FILE_ATTR_REPARSE_POINT) { unsigned int mode; - mode = ntfs_make_symlink(ni); + err = ntfs_parse_reparse(ni, &mode); + if (err) + goto unm_err_out; if (mode) vi->i_mode |= mode; else { @@ -886,16 +890,18 @@ skip_attr_list_load: if (S_ISDIR(vi->i_mode)) { /* - * Apply the directory permissions mask set in the mount - * options. + * Apply the directory permissions mask set in the mount options + * when no per-file WSL mode is present. */ - vi->i_mode &= ~vol->dmask; + if (!has_lxmod) + vi->i_mode &= ~vol->dmask; /* Things break without this kludge! */ if (vi->i_nlink > 1) set_nlink(vi, 1); } else { - /* Apply the file permissions mask set in the mount options. */ - vi->i_mode &= ~vol->fmask; + /* Apply the file permissions mask when no WSL mode is present. */ + if (!has_lxmod) + vi->i_mode &= ~vol->fmask; } /* @@ -1086,6 +1092,11 @@ view_index_meta: /* Setup the state. */ if (a->flags & (ATTR_COMPRESSION_MASK | ATTR_IS_SPARSE)) { if (a->flags & ATTR_COMPRESSION_MASK) { + if (NInoWofCompressed(ni)) { + ntfs_error(vi->i_sb, + "Found native compression on a WOF file."); + goto unm_err_out; + } NInoSetCompressed(ni); ni->flags |= FILE_ATTR_COMPRESSED; if (vol->cluster_size > 4096) { @@ -1116,7 +1127,7 @@ view_index_meta: } if (a->non_resident) { NInoSetNonResident(ni); - if (NInoCompressed(ni) || NInoSparse(ni)) { + if (NInoCompressed(ni) || (NInoSparse(ni) && !NInoWofCompressed(ni))) { if (NInoCompressed(ni) && a->data.non_resident.compression_unit != 4) { ntfs_error(vi->i_sb, @@ -1211,7 +1222,8 @@ no_data_attr_special_case: * sizes of all non-resident attributes present to give us the Linux * correct size that should go into i_blocks (after division by 512). */ - if (S_ISREG(vi->i_mode) && (NInoCompressed(ni) || NInoSparse(ni))) + if (S_ISREG(vi->i_mode) && + (NInoCompressed(ni) || (NInoSparse(ni) && !NInoWofCompressed(ni)))) vi->i_blocks = ni->itype.compressed.size >> 9; else vi->i_blocks = ni->allocated_size >> 9; @@ -1394,7 +1406,7 @@ static int ntfs_read_locked_attr_inode(struct inode *base_vi, struct inode *vi) "Attribute name is placed after the mapping pairs array."); goto unm_err_out; } - if (NInoCompressed(ni) || NInoSparse(ni)) { + if (NInoCompressed(ni) || (NInoSparse(ni) && !NInoWofCompressed(ni))) { if (NInoCompressed(ni) && a->data.non_resident.compression_unit != 4) { ntfs_error(vi->i_sb, "Found non-standard compression unit (%u instead of 4). Cannot handle this.", @@ -2401,7 +2413,7 @@ int ntfs_show_options(struct seq_file *sf, struct dentry *root) } int ntfs_extend_initialized_size(struct inode *vi, const loff_t offset, - const loff_t new_size, bool bsync) + const loff_t new_size) { struct ntfs_inode *ni = NTFS_I(vi); loff_t old_init_size; @@ -2428,10 +2440,6 @@ int ntfs_extend_initialized_size(struct inode *vi, const loff_t offset, &ntfs_iomap_folio_ops, NULL); if (err) return err; - if (bsync) - err = filemap_write_and_wait_range(vi->i_mapping, - old_init_size, - offset - 1); } @@ -3046,6 +3054,7 @@ int ntfs_inode_add_attrlist(struct ntfs_inode *ni) struct attr_list_entry *ale = NULL; struct mft_record *ni_mrec; u32 attr_al_len; + bool free_empty_extents = true; if (!ni) return -EINVAL; @@ -3145,6 +3154,7 @@ int ntfs_inode_add_attrlist(struct ntfs_inode *ni) ntfs_error(ni->vol->sb, "Couldn't add $ATTRIBUTE_LIST to MFT"); goto rollback; } + free_empty_extents = false; err = ntfs_attrlist_update(ni); if (err < 0) @@ -3164,6 +3174,8 @@ remove_attrlist_record: CASE_SENSITIVE, 0, NULL, 0, ctx)) { if (ntfs_attr_record_rm(ctx)) ntfs_error(ni->vol->sb, "Rollback failed to remove attrlist"); + else + free_empty_extents = true; } else { ntfs_error(ni->vol->sb, "Rollback failed to find attrlist"); } @@ -3202,6 +3214,11 @@ rollback: ni->attr_list_size = 0; NInoClearAttrList(ni); NInoClearAttrListDirty(ni); + ntfs_attr_put_search_ctx(ctx); + ctx = NULL; + if (free_empty_extents && ntfs_inode_free_empty_extents(ni)) + ntfs_error(ni->vol->sb, "Rollback failed to free empty extent"); + goto err_out; put_err_out: ntfs_attr_put_search_ctx(ctx); err_out: @@ -3288,6 +3305,55 @@ out: return err; } +/* + * ntfs_inode_free_empty_extents - free empty extent MFT records + * @ni: base inode whose empty extent records should be freed + * + * The caller must ensure that no on-disk attribute list references an empty + * extent record and must hold @ni->mrec_lock to serialize the extent array. + */ +int ntfs_inode_free_empty_extents(struct ntfs_inode *ni) +{ + int err = 0, i = 0; + + if (!ni || ni->nr_extents < 0) + return -EINVAL; + + mutex_lock(&ni->extent_lock); + while (i < ni->nr_extents) { + struct ntfs_inode *ext_ni = ni->ext.extent_ntfs_inos[i]; + struct mft_record *m; + int ret; + + m = map_mft_record(ext_ni); + if (IS_ERR(m)) { + if (!err) + err = PTR_ERR(m); + i++; + continue; + } + if (le32_to_cpu(m->bytes_in_use) - + le16_to_cpu(m->attrs_offset) != 8) { + unmap_mft_record(ext_ni); + i++; + continue; + } + unmap_mft_record(ext_ni); + + ret = ntfs_mft_record_free(ni->vol, ext_ni); + if (ret) { + if (!err) + err = ret; + i++; + continue; + } + ntfs_inode_close(ext_ni); + /* ntfs_inode_close() removed this entry from the extent array. */ + } + mutex_unlock(&ni->extent_lock); + return err; +} + void ntfs_destroy_ext_inode(struct ntfs_inode *ni) { ntfs_debug("Entering."); @@ -3387,6 +3453,9 @@ int ntfs_inode_free_space(struct ntfs_inode *ni, int size) * Chkdsk complain if $STANDARD_INFORMATION is not in the base MFT * record. * + * $INDEX_ROOT must remain resident, but its attribute record may be moved + * to an extent MFT record when the base record needs room for the list. + * * Also we can't move $ATTRIBUTE_LIST from base MFT_RECORD, so position * search context on first attribute after $STANDARD_INFORMATION and * $ATTRIBUTE_LIST. @@ -3428,9 +3497,6 @@ retry: ctx->attr->type == AT_DATA) goto retry; - if (ctx->attr->type == AT_INDEX_ROOT) - goto retry; - record_size = le32_to_cpu(ctx->attr->length); /* Move away attribute. */ diff --git a/fs/ntfs/inode.h b/fs/ntfs/inode.h index 9aacd5787ffe..ff61bd402df0 100644 --- a/fs/ntfs/inode.h +++ b/fs/ntfs/inode.h @@ -189,6 +189,7 @@ enum { NI_NonResident, NI_IndexAllocPresent, NI_Compressed, + NI_WofCompressed, NI_Encrypted, NI_Sparse, NI_SparseDisabled, @@ -248,6 +249,7 @@ NINO_FNS(MstProtected) NINO_FNS(NonResident) NINO_FNS(IndexAllocPresent) NINO_FNS(Compressed) +NINO_FNS(WofCompressed) NINO_FNS(Encrypted) NINO_FNS(Sparse) NINO_FNS(SparseDisabled) @@ -338,6 +340,7 @@ int ntfs_get_block_mft_record(struct ntfs_inode *mft_ni, struct ntfs_inode *ni); int __ntfs_write_inode(struct inode *vi, int sync); int ntfs_inode_attach_all_extents(struct ntfs_inode *ni); int ntfs_inode_add_attrlist(struct ntfs_inode *ni); +int ntfs_inode_free_empty_extents(struct ntfs_inode *ni); void ntfs_destroy_ext_inode(struct ntfs_inode *ni); int ntfs_inode_free_space(struct ntfs_inode *ni, int size); s64 ntfs_inode_attr_pread(struct inode *vi, s64 pos, s64 count, u8 *buf); @@ -352,7 +355,7 @@ static inline void ntfs_commit_inode(struct inode *vi) int ntfs_inode_sync_filename(struct ntfs_inode *ni); int ntfs_extend_initialized_size(struct inode *vi, const loff_t offset, - const loff_t new_size, bool bsync); + const loff_t new_size); void ntfs_set_vfs_operations(struct inode *inode, mode_t mode, dev_t dev); struct folio *ntfs_get_locked_folio(struct address_space *mapping, pgoff_t index, pgoff_t end_index, struct file_ra_state *ra); diff --git a/fs/ntfs/iomap.c b/fs/ntfs/iomap.c index d0964ac840d9..c812d7f19b36 100644 --- a/fs/ntfs/iomap.c +++ b/fs/ntfs/iomap.c @@ -81,7 +81,7 @@ const struct iomap_write_ops ntfs_iomap_folio_ops = { }; static int ntfs_read_iomap_begin_resident(struct inode *inode, loff_t offset, loff_t length, - unsigned int flags, struct iomap *iomap) + unsigned int flags, struct iomap *iomap, bool keep_mrec_lock) { struct ntfs_inode *base_ni, *ni = NTFS_I(inode); struct ntfs_attr_search_ctx *ctx; @@ -95,6 +95,8 @@ static int ntfs_read_iomap_begin_resident(struct inode *inode, loff_t offset, lo else base_ni = ni; + mutex_lock(&base_ni->mrec_lock); + ctx = ntfs_attr_get_search_ctx(base_ni, NULL); if (!ctx) { err = -ENOMEM; @@ -138,6 +140,13 @@ out: if (ctx) ntfs_attr_put_search_ctx(ctx); + if (!err && keep_mrec_lock && iomap->type == IOMAP_INLINE) { + iomap->private = base_ni; + return 0; + } + + mutex_unlock(&base_ni->mrec_lock); + return err; } @@ -261,23 +270,34 @@ static int ntfs_read_iomap_begin_non_resident(struct inode *inode, loff_t offset static int __ntfs_read_iomap_begin(struct inode *inode, loff_t offset, loff_t length, unsigned int flags, struct iomap *iomap, struct iomap *srcmap, - bool need_unwritten) + bool need_unwritten, bool keep_mrec_lock) { if (NInoNonResident(NTFS_I(inode))) return ntfs_read_iomap_begin_non_resident(inode, offset, length, flags, iomap, need_unwritten); return ntfs_read_iomap_begin_resident(inode, offset, length, - flags, iomap); + flags, iomap, keep_mrec_lock); } static int ntfs_read_iomap_begin(struct inode *inode, loff_t offset, loff_t length, unsigned int flags, struct iomap *iomap, struct iomap *srcmap) { return __ntfs_read_iomap_begin(inode, offset, length, flags, iomap, - srcmap, true); + srcmap, true, true); +} + +static int ntfs_read_iomap_end(struct inode *inode, loff_t pos, loff_t length, + ssize_t written, unsigned int flags, struct iomap *iomap) +{ + struct ntfs_inode *base_ni = iomap->private; + + if (base_ni) + mutex_unlock(&base_ni->mrec_lock); + return written; } -static DEFINE_IOMAP_ITER_NEXT(ntfs_read_iomap_next, ntfs_read_iomap_begin); +static DEFINE_IOMAP_ITER_NEXT_END(ntfs_read_iomap_next, + ntfs_read_iomap_begin, ntfs_read_iomap_end); const struct iomap_ops ntfs_read_iomap_ops = { .iomap_next = ntfs_read_iomap_next, @@ -320,7 +340,7 @@ static int ntfs_seek_iomap_begin(struct inode *inode, loff_t offset, loff_t leng unsigned int flags, struct iomap *iomap, struct iomap *srcmap) { return __ntfs_read_iomap_begin(inode, offset, length, flags, iomap, - srcmap, false); + srcmap, false, false); } static int ntfs_zero_read_iomap_end(struct inode *inode, loff_t pos, loff_t length, @@ -681,21 +701,7 @@ static int ntfs_write_iomap_begin_non_resident(struct inode *inode, loff_t offse loff_t length, unsigned int flags, struct iomap *iomap, int ntfs_iomap_flags) { - struct ntfs_inode *ni = NTFS_I(inode); - - if (ntfs_iomap_flags & (NTFS_IOMAP_FLAGS_BEGIN | NTFS_IOMAP_FLAGS_DIO) && - offset + length > ni->initialized_size) { - int ret; - - ret = ntfs_extend_initialized_size(inode, offset, - offset + length, - ntfs_iomap_flags & - NTFS_IOMAP_FLAGS_DIO); - if (ret < 0) - return ret; - } - - mutex_lock(&ni->mrec_lock); + mutex_lock(&NTFS_I(inode)->mrec_lock); if (ntfs_iomap_flags & NTFS_IOMAP_FLAGS_BEGIN) return ntfs_write_simple_iomap_begin_non_resident(inode, offset, length, iomap); @@ -711,28 +717,10 @@ static int __ntfs_write_iomap_begin(struct inode *inode, loff_t offset, struct iomap *iomap, int ntfs_iomap_flags) { struct ntfs_inode *ni = NTFS_I(inode); - loff_t end = offset + length; if (NVolShutdown(ni->vol)) return -EIO; - if (ntfs_iomap_flags & (NTFS_IOMAP_FLAGS_BEGIN | NTFS_IOMAP_FLAGS_DIO) && - end > ni->data_size) { - struct ntfs_volume *vol = ni->vol; - int ret; - - mutex_lock(&ni->mrec_lock); - if (end > ni->allocated_size && - end < ni->allocated_size + vol->preallocated_size) - ret = ntfs_attr_expand(ni, end, - ni->allocated_size + vol->preallocated_size); - else - ret = ntfs_attr_expand(ni, end, 0); - mutex_unlock(&ni->mrec_lock); - if (ret) - return ret; - } - if (!NInoNonResident(ni)) { mutex_lock(&ni->mrec_lock); return ntfs_write_iomap_begin_resident(inode, offset, iomap); diff --git a/fs/ntfs/lcnalloc.c b/fs/ntfs/lcnalloc.c index 835a041023a2..aa2e017a4384 100644 --- a/fs/ntfs/lcnalloc.c +++ b/fs/ntfs/lcnalloc.c @@ -298,7 +298,12 @@ struct runlist_element *ntfs_cluster_alloc(struct ntfs_volume *vol, const s64 st clusters = count; rlpos = rlsize = 0; mapping = lcnbmp_vi->i_mapping; - i_size = i_size_read(lcnbmp_vi); + /* + * lcn_empty_bits_per_page is sized from nr_clusters, but $Bitmap can + * cover more clusters than that; bound the scan by the array. + */ + i_size = min_t(s64, i_size_read(lcnbmp_vi), + ((s64)vol->nr_clusters + 7) >> 3); while (1) { ntfs_debug("Start of outer while loop: done_zones 0x%x, search_zone %i, pass %i, zone_start 0x%llx, zone_end 0x%llx, bmp_initial_pos 0x%llx, bmp_pos 0x%llx, rlpos %i, rlsize %i.", done_zones, search_zone, pass, diff --git a/fs/ntfs/lib/decompress_common.c b/fs/ntfs/lib/decompress_common.c new file mode 100644 index 000000000000..1705face42a3 --- /dev/null +++ b/fs/ntfs/lib/decompress_common.c @@ -0,0 +1,200 @@ +// SPDX-License-Identifier: MIT +/* + * decompress_common.c - Code shared by the XPRESS and LZX decompressors + * + * This is a port of the upstream wimlib "decompress_common.c" which builds + * subtable-based Huffman decode tables, as opposed to the older + * binary-tree-based format previously used in this library. The vectorized + * (SSE2/AVX2) fill paths are omitted for portability in the kernel. + * + * Copyright (C) 2022 Eric Biggers + */ + +#include "decompress_common.h" + +/* Compute the number of bits with which a subtable must be indexed for a + * codeword of length @codeword_len, given that the root table is indexed with + * @table_bits bits. + */ +static u32 compute_subtable_bits(u32 table_bits, + u32 codeword_len, u16 len_counts[]) +{ + u32 subtable_bits = codeword_len - table_bits; + s32 remainder = (s32)1 << subtable_bits; + + for (;;) { + remainder -= len_counts[table_bits + subtable_bits]; + if (remainder <= 0) + break; + subtable_bits++; + remainder <<= 1; + } + return subtable_bits; +} + +/* Build the subtables for codewords longer than table_bits. */ +static int build_subtables(u16 decode_table[], u32 num_syms, u32 table_bits, + u16 len_counts[], const u16 sorted_syms[], u32 sym_idx, + u32 decode_table_pos, u32 decode_table_size) +{ + u32 subtable_pos = 1U << table_bits; + u32 subtable_bits = table_bits; + u32 subtable_prefix = (u32)-1; + u32 codeword_len = table_bits + 1; + u32 codeword = decode_table_pos << 1; + u32 prefix; + u16 entry; + u32 n; + + for (; sym_idx < num_syms; sym_idx++) { + while (len_counts[codeword_len] == 0) { + codeword_len++; + codeword <<= 1; + } + + prefix = codeword >> (codeword_len - table_bits); + + if (prefix != subtable_prefix) { + subtable_prefix = prefix; + subtable_bits = compute_subtable_bits(table_bits, codeword_len, + len_counts); + decode_table[subtable_prefix] = + MAKE_DECODE_TABLE_ENTRY(subtable_pos, subtable_bits); + } + + entry = MAKE_DECODE_TABLE_ENTRY(sorted_syms[sym_idx], + codeword_len - table_bits); + n = 1U << (subtable_bits - (codeword_len - table_bits)); + + /* Defensive bound check: 'lens' is derived from untrusted + * on-disk compressed data, and subtable growth depends on + * its content. This should never trigger for a correctly + * sized DECODE_TABLE_ENOUGH() value, but turns a wrong value + * into a clean decode failure instead of writing past the + * caller's decode_table[]. + */ + if (unlikely(subtable_pos + n > decode_table_size)) + return -1; + + do { + decode_table[subtable_pos++] = entry; + } while (--n); + + len_counts[codeword_len]--; + codeword++; + } + + return 0; +} + +/* + * Given an alphabet of symbols and the length of each symbol's codeword in a + * canonical prefix code, build a table for quickly decoding symbols that were + * encoded with that code. + * + * The root table is indexed with 'table_bits' bits. Codewords not longer than + * 'table_bits' are decoded directly from the root table. Longer codewords are + * decoded via subtables: the corresponding root entry is a pointer (the index + * of the subtable plus the number of bits with which the subtable is indexed), + * and the subtable is indexed with the remaining bits of the codeword. + * + * Each entry stores both the symbol (high 12 bits) and the codeword length (low + * 4 bits), so a single lookup yields the symbol and lets the bitstream be + * advanced by the correct number of bits. + * + * @decode_table: array in which to build the table (declared with + * DECODE_TABLE()). May alias @lens. + * @num_syms: number of symbols in the alphabet. + * @table_bits: log2 of the number of root table entries. + * @lens: array of @num_syms codeword lengths, indexed by symbol. + * @max_codeword_len: longest codeword length allowed for this code. + * @working_space: temporary array declared with DECODE_TABLE_WORKING_SPACE(). + * @decode_table_size: number of u16 entries in @decode_table (i.e. + * ARRAY_SIZE(decode_table) at the call site). Used only as a + * defensive bound check against @lens-dependent subtable growth. + * + * Returns 0 on success, or -1 if the lengths do not form a valid prefix code, + * or if building the subtables would overflow @decode_table_size entries. + */ +int make_huffman_decode_table(u16 decode_table[], u32 num_syms, u32 table_bits, + const u8 lens[], u32 max_codeword_len, + u16 working_space[], u32 decode_table_size) +{ + u16 *const len_counts = &working_space[0]; + u16 *const offsets = &working_space[1 * (max_codeword_len + 1)]; + u16 *const sorted_syms = &working_space[2 * (max_codeword_len + 1)]; + u32 decode_table_pos = 0; + u32 sym_idx; + u32 codeword_len; + s32 remainder = 1; + void *entry_ptr = decode_table; + u32 len; + u32 sym; + + /* Count how many codewords have each length, including 0. */ + for (len = 0; len <= max_codeword_len; len++) + len_counts[len] = 0; + for (sym = 0; sym < num_syms; sym++) + len_counts[lens[sym]]++; + + /* A codeword of length n should require a proportion of the codespace + * equaling (1/2)^n. The code is complete iff the codespace is exactly + * filled by the lengths. + */ + for (len = 1; len <= max_codeword_len; len++) { + remainder = (remainder << 1) - len_counts[len]; + if (unlikely(remainder < 0)) + return -1; /* over-subscribed */ + } + + if (remainder != 0) { + /* Incomplete code. Permitted only if the code is empty. */ + if (unlikely(remainder != (s32)(1U << max_codeword_len))) + return -1; + + /* Empty code: zero the root table so lookups yield symbol 0 + * without consuming any bits. + */ + memset(decode_table, 0, sizeof(decode_table[0]) << table_bits); + return 0; + } + + /* Sort the symbols primarily by increasing codeword length and + * secondarily by increasing symbol value. + */ + offsets[0] = 0; + for (len = 0; len < max_codeword_len; len++) + offsets[len + 1] = offsets[len] + len_counts[len]; + for (sym = 0; sym < num_syms; sym++) + sorted_syms[offsets[lens[sym]]++] = sym; + + /* Fill the root table entries for codewords no longer than table_bits. */ + sym_idx = offsets[0]; + codeword_len = 1; + for (; codeword_len <= table_bits; codeword_len++) { + u32 stores_per_loop = 1U << (table_bits - codeword_len); + u32 end_sym_idx = sym_idx + len_counts[codeword_len]; + + for (; sym_idx < end_sym_idx; sym_idx++) { + u16 v = MAKE_DECODE_TABLE_ENTRY(sorted_syms[sym_idx], + codeword_len); + u32 n = stores_per_loop; + u16 *p = entry_ptr; + + do { + *p++ = v; + } while (--n); + entry_ptr = p; + } + } + decode_table_pos = (u16 *)entry_ptr - decode_table; + + /* If all symbols were processed, no subtables are required. */ + if (sym_idx == num_syms) + return 0; + + /* At least one subtable is required. Process the remaining symbols. */ + return build_subtables(decode_table, num_syms, table_bits, len_counts, + sorted_syms, sym_idx, decode_table_pos, + decode_table_size); +} diff --git a/fs/ntfs/lib/decompress_common.h b/fs/ntfs/lib/decompress_common.h new file mode 100644 index 000000000000..9bf85cd52e6d --- /dev/null +++ b/fs/ntfs/lib/decompress_common.h @@ -0,0 +1,444 @@ +/* SPDX-License-Identifier: MIT */ +/* + * decompress_common.h - Code shared by the XPRESS and LZX decompressors + * + * This is a port of the upstream wimlib "decompress_common.h" which uses a + * subtable-based Huffman decode table format, as opposed to the older + * binary-tree-based format previously used in this library. + * + * Copyright (C) 2022 Eric Biggers + */ + +#ifndef _LINUX_NTFS_LIB_DECOMPRESS_COMMON_H +#define _LINUX_NTFS_LIB_DECOMPRESS_COMMON_H + +#include <linux/compiler.h> +#include <linux/string.h> +#include <linux/types.h> +#include <linux/slab.h> +#include <linux/unaligned.h> + +/* "Force inline" macro (not required, but helpful for performance). */ +#define forceinline __always_inline + +/* Size of a machine word. */ +#define WORDBYTES sizeof(size_t) +#define WORDBITS (8 * WORDBYTES) + +/* UNALIGNED_ACCESS_IS_FAST should be 1 if unaligned memory accesses can be + * performed efficiently on the target platform. + */ +#ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS +# define UNALIGNED_ACCESS_IS_FAST 1 +#else +# define UNALIGNED_ACCESS_IS_FAST 0 +#endif + +/* Deprecated name kept for compatibility with the upstream source. */ +#define FAST_UNALIGNED_ACCESS UNALIGNED_ACCESS_IS_FAST + +/* likely()/unlikely() are provided by <linux/compiler.h>. */ + +/* STATIC_ASSERT() - verify the truth of an expression at compile time. */ +#define STATIC_ASSERT(expr) ((void)sizeof(char[1 - 2 * !(expr)])) + +/* STATIC_ASSERT_ZERO() - like STATIC_ASSERT() but evaluates to 0 so it can be + * used in constant expressions. + */ +#define STATIC_ASSERT_ZERO(expr) ((int)sizeof(char[-!(expr)])) + +/* Unaligned word load/store helpers. */ +static forceinline size_t load_word_unaligned(const void *p) +{ + size_t v; + + memcpy(&v, p, sizeof(v)); + return v; +} + +static forceinline void store_word_unaligned(size_t v, void *p) +{ + memcpy(p, &v, sizeof(v)); +} + +static forceinline void copy_word_unaligned(const void *src, void *dst) +{ + store_word_unaligned(load_word_unaligned(src), dst); +} + +static forceinline size_t repeat_u16(u16 b) +{ + size_t v = b; + + STATIC_ASSERT(WORDBITS == 32 || WORDBITS == 64); + v |= v << 16; + v |= v << ((WORDBITS == 64) ? 32 : 0); + return v; +} + +static forceinline size_t repeat_byte(u8 b) +{ + return repeat_u16(((u16)b << 8) | b); +} + +/******************************************************************************/ +/* Input bitstream for XPRESS and LZX */ +/*----------------------------------------------------------------------------*/ + +/* Structure that encapsulates a block of in-memory data being interpreted as a + * stream of bits, optionally with interwoven literal bytes. Bits are assumed + * to be stored in little endian 16-bit coding units, with the bits ordered high + * to low. + */ +struct input_bitstream { + /* Bits that have been read from the input buffer. The bits are + * left-justified; the next bit is always bit 31. + */ + u32 bitbuf; + + /* Number of bits currently held in @bitbuf. */ + u32 bitsleft; + + /* Pointer to the next byte to be retrieved from the input buffer. */ + const u8 *next; + + /* Pointer past the end of the input buffer. */ + const u8 *end; +}; + +/* Initialize a bitstream to read from the specified input buffer. */ +static forceinline void init_input_bitstream(struct input_bitstream *is, + const void *buffer, u32 size) +{ + is->bitbuf = 0; + is->bitsleft = 0; + is->next = buffer; + is->end = is->next + size; +} + +/* Note: for performance reasons, the following methods don't return error + * codes to the caller if the input buffer is overrun. Instead, they just + * assume that all overrun data is zeroes. + */ + +/* Ensure the bit buffer variable for the bitstream contains at least @num_bits + * bits. Following this, bitstream_peek_bits() and/or bitstream_remove_bits() + * may be called on the bitstream to peek or remove up to @num_bits bits. This + * works for at most 16 bits, which is sufficient for LZX (max codeword length + * 16) and XPRESS (max codeword length 15). + */ +static forceinline void bitstream_ensure_bits(struct input_bitstream *is, + unsigned int num_bits) +{ + if (is->bitsleft >= num_bits) + return; + + if (unlikely(is->end - is->next < 2)) + goto overflow; + + is->bitbuf |= (u32)get_unaligned_le16(is->next) << (16 - is->bitsleft); + is->next += 2; + is->bitsleft += 16; + return; + +overflow: + is->bitsleft = 32; +} + +/* Return the next @num_bits bits from the bitstream, without removing them. + * There must be at least @num_bits remaining in the buffer variable. + */ +static forceinline u32 bitstream_peek_bits(const struct input_bitstream *is, + unsigned int num_bits) +{ + return (is->bitbuf >> 1) >> (sizeof(is->bitbuf) * 8 - num_bits - 1); +} + +/* Remove @num_bits from the bitstream. */ +static forceinline void bitstream_remove_bits(struct input_bitstream *is, + unsigned int num_bits) +{ + is->bitbuf <<= num_bits; + is->bitsleft -= num_bits; +} + +/* Remove and return @num_bits bits from the bitstream. */ +static forceinline u32 bitstream_pop_bits(struct input_bitstream *is, + unsigned int num_bits) +{ + u32 bits = bitstream_peek_bits(is, num_bits); + + bitstream_remove_bits(is, num_bits); + return bits; +} + +/* Read and return the next @num_bits bits from the bitstream. */ +static forceinline u32 bitstream_read_bits(struct input_bitstream *is, + unsigned int num_bits) +{ + bitstream_ensure_bits(is, num_bits); + return bitstream_pop_bits(is, num_bits); +} + +/* Read and return the next literal byte embedded in the bitstream. */ +static forceinline u8 bitstream_read_byte(struct input_bitstream *is) +{ + if (unlikely(is->end == is->next)) + return 0; + return *is->next++; +} + +/* Read and return the next 16-bit integer embedded in the bitstream. */ +static forceinline u16 bitstream_read_u16(struct input_bitstream *is) +{ + u16 v; + + if (unlikely(is->end - is->next < 2)) + return 0; + v = get_unaligned_le16(is->next); + is->next += 2; + return v; +} + +/* Read and return the next 32-bit integer embedded in the bitstream. */ +static forceinline u32 bitstream_read_u32(struct input_bitstream *is) +{ + u32 v; + + if (unlikely(is->end - is->next < 4)) + return 0; + v = get_unaligned_le32(is->next); + is->next += 4; + return v; +} + +/* Read into @dst_buffer an array of literal bytes embedded in the bitstream. + * Return 0 if there were enough bytes remaining in the input, otherwise -1. + */ +static forceinline int bitstream_read_bytes(struct input_bitstream *is, + void *dst_buffer, size_t count) +{ + if (unlikely((size_t)(is->end - is->next) < count)) + return -1; + memcpy(dst_buffer, is->next, count); + is->next += count; + return 0; +} + +/* Align the input bitstream on a coding-unit boundary. */ +static forceinline void bitstream_align(struct input_bitstream *is) +{ + is->bitsleft = 0; + is->bitbuf = 0; +} + +/******************************************************************************/ +/* Huffman decoding */ +/*----------------------------------------------------------------------------*/ + +/* + * Required alignment for the Huffman decode tables. We require this alignment + * so that we can fill the entries with word instructions without having to deal + * with misaligned buffers. + */ +#define DECODE_TABLE_ALIGNMENT 16 + +/* + * Each decode table entry is 16 bits divided into two fields: 'symbol' (high 12 + * bits) and 'length' (low 4 bits). See the comments in decompress_common.c for + * the precise meaning of these fields depending on the entry type. + */ +#define DECODE_TABLE_SYMBOL_SHIFT 4 +#define DECODE_TABLE_MAX_SYMBOL ((1 << (16 - DECODE_TABLE_SYMBOL_SHIFT)) - 1) +#define DECODE_TABLE_MAX_LENGTH ((1 << DECODE_TABLE_SYMBOL_SHIFT) - 1) +#define DECODE_TABLE_LENGTH_MASK DECODE_TABLE_MAX_LENGTH +#define MAKE_DECODE_TABLE_ENTRY(symbol, length) \ + (((symbol) << DECODE_TABLE_SYMBOL_SHIFT) | (length)) + +/* + * Read and return the next Huffman-encoded symbol from the given bitstream + * using the given decode table. If the input data is exhausted, then the + * Huffman symbol will be decoded as if the missing bits were all zeroes. + */ +static forceinline unsigned int read_huffsym(struct input_bitstream *is, + const u16 decode_table[], + unsigned int table_bits, + unsigned int max_codeword_len) +{ + unsigned int entry; + unsigned int symbol; + unsigned int length; + + /* Preload the bitbuffer with 'max_codeword_len' bits. */ + bitstream_ensure_bits(is, max_codeword_len); + + /* Index the root table by the next 'table_bits' bits of input. */ + entry = decode_table[bitstream_peek_bits(is, table_bits)]; + + /* Extract the "symbol" and "length" from the entry. */ + symbol = entry >> DECODE_TABLE_SYMBOL_SHIFT; + length = entry & DECODE_TABLE_LENGTH_MASK; + + /* If the codeword is longer than 'table_bits', the root entry is a + * subtable pointer. Discard the bits used to index the root table and + * index the subtable by the next 'length' bits. + */ + if (max_codeword_len > table_bits && + entry >= (1U << (table_bits + DECODE_TABLE_SYMBOL_SHIFT))) { + bitstream_remove_bits(is, table_bits); + entry = decode_table[symbol + bitstream_peek_bits(is, length)]; + symbol = entry >> DECODE_TABLE_SYMBOL_SHIFT; + length = entry & DECODE_TABLE_LENGTH_MASK; + } + + /* Discard the (remaining) bits of the codeword. */ + bitstream_remove_bits(is, length); + + return symbol; +} + +/* + * DECODE_TABLE_ENOUGH() evaluates to the maximum number of decode table + * entries, including all subtable entries, that may be required for decoding a + * given Huffman code. It is a compile-time mapping computed by the zlib + * 'enough' utility. An unknown combination produces a build error. + */ +#define DECODE_TABLE_ENOUGH(num_syms, table_bits, max_codeword_len) ( \ + ((num_syms) == 8 && (table_bits) == 5 && (max_codeword_len) == 7) ? 36 : \ + ((num_syms) == 8 && (table_bits) == 6 && (max_codeword_len) == 7) ? 66 : \ + ((num_syms) == 8 && (table_bits) == 7 && (max_codeword_len) == 7) ? 128 : \ + ((num_syms) == 20 && (table_bits) == 5 && (max_codeword_len) == 15) ? 1062 : \ + ((num_syms) == 20 && (table_bits) == 6 && (max_codeword_len) == 15) ? 582 : \ + ((num_syms) == 20 && (table_bits) == 7 && (max_codeword_len) == 15) ? 390 : \ + ((num_syms) == 54 && (table_bits) == 9 && (max_codeword_len) == 15) ? 618 : \ + ((num_syms) == 54 && (table_bits) == 10 && (max_codeword_len) == 15) ? 1098 : \ + ((num_syms) == 249 && (table_bits) == 9 && (max_codeword_len) == 16) ? 878 : \ + ((num_syms) == 249 && (table_bits) == 10 && (max_codeword_len) == 16) ? 1326 : \ + ((num_syms) == 249 && (table_bits) == 11 && (max_codeword_len) == 16) ? 2318 : \ + ((num_syms) == 496 && (table_bits) == 11 && (max_codeword_len) == 16) ? 2566 : \ + ((num_syms) == 256 && (table_bits) == 9 && (max_codeword_len) == 15) ? 822 : \ + ((num_syms) == 256 && (table_bits) == 10 && (max_codeword_len) == 15) ? 1302 : \ + ((num_syms) == 256 && (table_bits) == 11 && (max_codeword_len) == 15) ? 2310 : \ + ((num_syms) == 512 && (table_bits) == 10 && (max_codeword_len) == 15) ? 1558 : \ + ((num_syms) == 512 && (table_bits) == 11 && (max_codeword_len) == 15) ? 2566 : \ + ((num_syms) == 512 && (table_bits) == 12 && (max_codeword_len) == 15) ? 4606 : \ + ((num_syms) == 656 && (table_bits) == 10 && (max_codeword_len) == 16) ? 1734 : \ + ((num_syms) == 656 && (table_bits) == 11 && (max_codeword_len) == 16) ? 2726 : \ + ((num_syms) == 656 && (table_bits) == 12 && (max_codeword_len) == 16) ? 4758 : \ + ((num_syms) == 799 && (table_bits) == 9 && (max_codeword_len) == 15) ? 1366 : \ + ((num_syms) == 799 && (table_bits) == 10 && (max_codeword_len) == 15) ? 1846 : \ + ((num_syms) == 799 && (table_bits) == 11 && (max_codeword_len) == 15) ? 2854 : \ + -1) + +/* Wrapper around DECODE_TABLE_ENOUGH() that does additional compile-time + * validation. + */ +#define DECODE_TABLE_SIZE(num_syms, table_bits, max_codeword_len) ( \ + STATIC_ASSERT_ZERO((num_syms) > 0) + \ + STATIC_ASSERT_ZERO((table_bits) > 0) + \ + STATIC_ASSERT_ZERO((max_codeword_len) > 0) + \ + STATIC_ASSERT_ZERO((num_syms) <= 1U << (max_codeword_len)) + \ + STATIC_ASSERT_ZERO((table_bits) <= (max_codeword_len)) + \ + STATIC_ASSERT_ZERO((num_syms) - 1 <= DECODE_TABLE_MAX_SYMBOL) + \ + STATIC_ASSERT_ZERO((table_bits) <= DECODE_TABLE_MAX_LENGTH) + \ + STATIC_ASSERT_ZERO((max_codeword_len) - (table_bits) <= \ + DECODE_TABLE_MAX_LENGTH) + \ + STATIC_ASSERT_ZERO((1U << table_bits) > (num_syms) - 1) + \ + STATIC_ASSERT_ZERO(DECODE_TABLE_ENOUGH( \ + (num_syms), (table_bits), \ + (max_codeword_len)) > 0) + \ + STATIC_ASSERT_ZERO(DECODE_TABLE_ENOUGH( \ + (num_syms), (table_bits), \ + (max_codeword_len)) - 1 <= \ + DECODE_TABLE_MAX_SYMBOL) + \ + DECODE_TABLE_ENOUGH((num_syms), (table_bits), \ + (max_codeword_len)) \ +) + +/* Declare the decode table for a Huffman code. */ +#define DECODE_TABLE(name, num_syms, table_bits, max_codeword_len) \ + u16 name[DECODE_TABLE_SIZE((num_syms), (table_bits), \ + (max_codeword_len))] \ + __aligned(DECODE_TABLE_ALIGNMENT) + +/* Declare the temporary "working_space" array needed for building the decode + * table for a Huffman code. + */ +#define DECODE_TABLE_WORKING_SPACE(name, num_syms, max_codeword_len) \ + u16 name[2 * ((max_codeword_len) + 1) + (num_syms)] + +int make_huffman_decode_table(u16 decode_table[], u32 num_syms, + u32 table_bits, const u8 lens[], + u32 max_codeword_len, u16 working_space[], + u32 decode_table_size); + +/******************************************************************************/ +/* LZ match copying */ +/*----------------------------------------------------------------------------*/ + +/* + * Copy an LZ77 match of 'length' bytes from the match source at 'out_next - + * offset' to the match destination at 'out_next'. The source and destination + * may overlap. This handles validating the length and offset; it returns 0 if + * the match was valid (and was copied), otherwise -1. + */ +static forceinline int lz_copy(u32 length, u32 offset, u8 *out_begin, + u8 *out_next, u8 *out_end, u32 min_length) +{ + const u8 *src; + u8 *end; + + /* Validate the offset. */ + if (unlikely(offset > (u32)(out_next - out_begin))) + return -1; + + src = out_next - offset; + + /* Fast path: copy a short, non-overlapping match whose end is not too + * close to the end of the buffer. + */ + if (UNALIGNED_ACCESS_IS_FAST && length <= 3 * WORDBYTES && + offset >= WORDBYTES && out_end - out_next >= 3 * WORDBYTES) { + copy_word_unaligned(src + WORDBYTES * 0, out_next + WORDBYTES * 0); + copy_word_unaligned(src + WORDBYTES * 1, out_next + WORDBYTES * 1); + copy_word_unaligned(src + WORDBYTES * 2, out_next + WORDBYTES * 2); + return 0; + } + + /* Validate the length. */ + if (unlikely(length > (u32)(out_end - out_next))) + return -1; + end = out_next + length; + + if (UNALIGNED_ACCESS_IS_FAST && likely(out_end - end >= WORDBYTES - 1)) { + if (offset >= WORDBYTES) { + do { + copy_word_unaligned(src, out_next); + src += WORDBYTES; + out_next += WORDBYTES; + } while (out_next < end); + return 0; + } else if (offset == 1) { + size_t v = repeat_byte(*(out_next - 1)); + + do { + store_word_unaligned(v, out_next); + src += WORDBYTES; + out_next += WORDBYTES; + } while (out_next < end); + return 0; + } + } + + /* Fall back to a bytewise copy. */ + if (min_length >= 2) + *out_next++ = *src++; + if (min_length >= 3) + *out_next++ = *src++; + do { + *out_next++ = *src++; + } while (out_next != end); + return 0; +} + +#endif /* _LINUX_NTFS_LIB_DECOMPRESS_COMMON_H */ diff --git a/fs/ntfs/lib/lib.h b/fs/ntfs/lib/lib.h new file mode 100644 index 000000000000..a684d600fd3c --- /dev/null +++ b/fs/ntfs/lib/lib.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * lib.h - Public declarations for the LZX and XPRESS decompressors. + * + * Adapted for the linux kernel. These are the low-level decompressor + * allocations; WOF (system-compressed) access goes through the + * ntfs_codec_ops interface declared in "../ntfs_codec.h". + */ + +#ifndef _LINUX_NTFS_LIB_LIB_H +#define _LINUX_NTFS_LIB_LIB_H + +#include <linux/types.h> + +/* globals from xpress_decompress.c */ +struct xpress_decompressor *xpress_allocate_decompressor(void); +void xpress_free_decompressor(struct xpress_decompressor *d); +int xpress_decompress(struct xpress_decompressor *d, + const void *compressed_data, size_t compressed_size, + void *uncompressed_data, size_t uncompressed_size); + +/* globals from lzx_decompress.c */ +struct lzx_decompressor *lzx_allocate_decompressor(void); +void lzx_free_decompressor(struct lzx_decompressor *d); +int lzx_decompress(struct lzx_decompressor *d, const void *compressed_data, + size_t compressed_size, void *uncompressed_data, + size_t uncompressed_size); + +#endif /* _LINUX_NTFS_LIB_LIB_H */ diff --git a/fs/ntfs/lib/lzx_decompress.c b/fs/ntfs/lib/lzx_decompress.c new file mode 100644 index 000000000000..be040dce6e23 --- /dev/null +++ b/fs/ntfs/lib/lzx_decompress.c @@ -0,0 +1,631 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * lzx_decompress.c - A decompressor for the LZX compression format + * + * This is a port of the upstream wimlib "lzx_decompress.c" which uses a + * subtable-based Huffman decode table format. The window size is fixed at + * 32768 bytes, which is the only size used in System-compressed (WOF) files. + * + * Copyright (C) 2012-2016 Eric Biggers + */ + +#include <linux/array_size.h> +#include <linux/bits.h> + +#include "decompress_common.h" +#include "lib.h" +#include "../ntfs_codec.h" + +/* Number of literal byte values. */ +#define LZX_NUM_CHARS 256 + +/* The smallest and largest allowed match lengths. */ +#define LZX_MIN_MATCH_LEN 2 +#define LZX_MAX_MATCH_LEN 257 + +/* Number of distinct match lengths that can be represented. */ +#define LZX_NUM_LENS (LZX_MAX_MATCH_LEN - LZX_MIN_MATCH_LEN + 1) + +/* Number of match lengths for which no length symbol is required. */ +#define LZX_NUM_PRIMARY_LENS 7 +#define LZX_NUM_LEN_HEADERS (LZX_NUM_PRIMARY_LENS + 1) + +/* Valid values of the 3-bit block type field. */ +#define LZX_BLOCKTYPE_VERBATIM 1 +#define LZX_BLOCKTYPE_ALIGNED 2 +#define LZX_BLOCKTYPE_UNCOMPRESSED 3 + +/* LZX window size is fixed at 32768 bytes for System-compressed files. */ + +/* Number of offset slots for a 32768-byte window. */ +#define LZX_NUM_OFFSET_SLOTS 30 + +/* Number of symbols in the main code. */ +#define LZX_MAINCODE_NUM_SYMBOLS \ + (LZX_NUM_CHARS + (LZX_NUM_OFFSET_SLOTS * LZX_NUM_LEN_HEADERS)) + +/* Number of symbols in the length code. */ +#define LZX_LENCODE_NUM_SYMBOLS (LZX_NUM_LENS - LZX_NUM_PRIMARY_LENS) + +/* Number of symbols in the precode. */ +#define LZX_PRECODE_NUM_SYMBOLS 20 + +/* Number of bits in which each precode codeword length is represented. */ +#define LZX_PRECODE_ELEMENT_SIZE 4 + +/* Number of low-order bits of each match offset that are entropy-encoded in + * aligned offset blocks. + */ +#define LZX_NUM_ALIGNED_OFFSET_BITS 3 + +/* Number of symbols in the aligned offset code. */ +#define LZX_ALIGNEDCODE_NUM_SYMBOLS BIT(LZX_NUM_ALIGNED_OFFSET_BITS) + +/* Mask for the match offset bits that are entropy-encoded in aligned offset + * blocks. + */ +#define LZX_ALIGNED_OFFSET_BITMASK (BIT(LZX_NUM_ALIGNED_OFFSET_BITS) - 1) + +/* Number of bits in which each aligned offset codeword length is represented. */ +#define LZX_ALIGNEDCODE_ELEMENT_SIZE 3 + +/* The first offset slot which requires an aligned offset symbol in aligned + * offset blocks. + */ +#define LZX_MIN_ALIGNED_OFFSET_SLOT 8 + +/* Maximum lengths (in bits) of the codewords in each Huffman code. */ +#define LZX_MAX_MAIN_CODEWORD_LEN 16 +#define LZX_MAX_LEN_CODEWORD_LEN 16 +#define LZX_MAX_PRE_CODEWORD_LEN ((1 << LZX_PRECODE_ELEMENT_SIZE) - 1) +#define LZX_MAX_ALIGNED_CODEWORD_LEN ((1 << LZX_ALIGNEDCODE_ELEMENT_SIZE) - 1) + +/* For LZX-compressed blocks in WIM/system-compressed files this value is + * always used as the filesize parameter for the E8 call preprocessing. + */ +#define LZX_WIM_MAGIC_FILESIZE 12000000 + +/* Assumed LZX block size when the encoded block size begins with a 0 bit. */ +#define LZX_DEFAULT_BLOCK_SIZE 32768 + +/* Number of offsets in the recent (or "repeat") offsets queue. */ +#define LZX_NUM_RECENT_OFFSETS 3 + +/* An offset of n bytes is actually encoded as (n + LZX_OFFSET_ADJUSTMENT). */ +#define LZX_OFFSET_ADJUSTMENT (LZX_NUM_RECENT_OFFSETS - 1) + +/* These values are chosen for fast decompression. */ +#define LZX_MAINCODE_TABLEBITS 11 +#define LZX_LENCODE_TABLEBITS 9 +#define LZX_PRECODE_TABLEBITS 6 +#define LZX_ALIGNEDCODE_TABLEBITS 7 + +#define LZX_READ_LENS_MAX_OVERRUN 50 + +/* Mapping: offset slot => first match offset that uses that offset slot. + * The offset slots for repeat offsets map to "fake" offsets < 1. + */ +static const s32 lzx_offset_slot_base[LZX_NUM_OFFSET_SLOTS + 1] = { + -2, -1, 0, 1, 2, /* 0 --- 4 */ + 4, 6, 10, 14, 22, /* 5 --- 9 */ + 30, 46, 62, 94, 126, /* 10 --- 14 */ + 190, 254, 382, 510, 766, /* 15 --- 19 */ + 1022, 1534, 2046, 3070, 4094, /* 20 --- 24 */ + 6142, 8190, 12286, 16382, 24574, /* 25 --- 29 */ + 32766, /* extra */ +}; + +/* Mapping: offset slot => how many extra bits must be read and added to the + * corresponding offset slot base to decode the match offset. + */ +static const u8 lzx_extra_offset_bits[LZX_NUM_OFFSET_SLOTS] = { + 0, 0, 0, 0, 1, + 1, 2, 2, 3, 3, + 4, 4, 5, 5, 6, + 6, 7, 7, 8, 8, + 9, 9, 10, 10, 11, + 11, 12, 12, 13, 13, +}; + +/* Like lzx_extra_offset_bits[], but with the entropy-coded aligned offset + * bits already subtracted. Valid only for offset slots that may appear in + * aligned offset blocks. + */ +static const u8 lzx_extra_offset_bits_minus_aligned[LZX_NUM_OFFSET_SLOTS] = { + 0, 0, 0, 0, 1, + 1, 2, 2, 0, 0, + 1, 1, 2, 2, 3, + 3, 4, 4, 5, 5, + 6, 6, 7, 7, 8, + 8, 9, 9, 10, 10, +}; + +/* Reusable heap-allocated memory for LZX decompression. The decode tables and + * their corresponding codeword length arrays are grouped in unions so the + * memory can be reused across phases, and the per-code working spaces share a + * single union since only one is needed at a time. + */ +struct lzx_decompressor { + DECODE_TABLE(maincode_decode_table, LZX_MAINCODE_NUM_SYMBOLS, + LZX_MAINCODE_TABLEBITS, LZX_MAX_MAIN_CODEWORD_LEN); + u8 maincode_lens[LZX_MAINCODE_NUM_SYMBOLS + LZX_READ_LENS_MAX_OVERRUN]; + + DECODE_TABLE(lencode_decode_table, LZX_LENCODE_NUM_SYMBOLS, + LZX_LENCODE_TABLEBITS, LZX_MAX_LEN_CODEWORD_LEN); + u8 lencode_lens[LZX_LENCODE_NUM_SYMBOLS + LZX_READ_LENS_MAX_OVERRUN]; + + union { + DECODE_TABLE(alignedcode_decode_table, + LZX_ALIGNEDCODE_NUM_SYMBOLS, + LZX_ALIGNEDCODE_TABLEBITS, + LZX_MAX_ALIGNED_CODEWORD_LEN); + u8 alignedcode_lens[LZX_ALIGNEDCODE_NUM_SYMBOLS]; + }; + + union { + DECODE_TABLE(precode_decode_table, LZX_PRECODE_NUM_SYMBOLS, + LZX_PRECODE_TABLEBITS, LZX_MAX_PRE_CODEWORD_LEN); + u8 precode_lens[LZX_PRECODE_NUM_SYMBOLS]; + /* extra_offset_bits[] is used as scratch in aligned blocks. */ + u8 extra_offset_bits[LZX_NUM_OFFSET_SLOTS]; + }; + + union { + DECODE_TABLE_WORKING_SPACE(maincode_working_space, + LZX_MAINCODE_NUM_SYMBOLS, + LZX_MAX_MAIN_CODEWORD_LEN); + DECODE_TABLE_WORKING_SPACE(lencode_working_space, + LZX_LENCODE_NUM_SYMBOLS, + LZX_MAX_LEN_CODEWORD_LEN); + DECODE_TABLE_WORKING_SPACE(alignedcode_working_space, + LZX_ALIGNEDCODE_NUM_SYMBOLS, + LZX_MAX_ALIGNED_CODEWORD_LEN); + DECODE_TABLE_WORKING_SPACE(precode_working_space, + LZX_PRECODE_NUM_SYMBOLS, + LZX_MAX_PRE_CODEWORD_LEN); + }; +} __aligned(DECODE_TABLE_ALIGNMENT); + +static forceinline unsigned int read_presym(const struct lzx_decompressor *d, + struct input_bitstream *is) +{ + return read_huffsym(is, d->precode_decode_table, LZX_PRECODE_TABLEBITS, + LZX_MAX_PRE_CODEWORD_LEN); +} + +static forceinline unsigned int read_mainsym(const struct lzx_decompressor *d, + struct input_bitstream *is) +{ + return read_huffsym(is, d->maincode_decode_table, + LZX_MAINCODE_TABLEBITS, LZX_MAX_MAIN_CODEWORD_LEN); +} + +static forceinline unsigned int read_lensym(const struct lzx_decompressor *d, + struct input_bitstream *is) +{ + return read_huffsym(is, d->lencode_decode_table, LZX_LENCODE_TABLEBITS, + LZX_MAX_LEN_CODEWORD_LEN); +} + +static forceinline unsigned int +read_alignedsym(const struct lzx_decompressor *d, struct input_bitstream *is) +{ + return read_huffsym(is, d->alignedcode_decode_table, + LZX_ALIGNEDCODE_TABLEBITS, + LZX_MAX_ALIGNED_CODEWORD_LEN); +} + +/* + * Read a precode from the compressed bitstream, then use it to decode + * @num_lens codeword length values and write them to @lens. + */ +static int lzx_read_codeword_lens(struct lzx_decompressor *d, + struct input_bitstream *is, u8 *lens, + u32 num_lens) +{ + u8 *len_ptr = lens; + u8 *lens_end = lens + num_lens; + u32 i; + + /* Read the lengths of the precode codewords. These are stored + * explicitly. + */ + for (i = 0; i < LZX_PRECODE_NUM_SYMBOLS; i++) { + d->precode_lens[i] = + bitstream_read_bits(is, LZX_PRECODE_ELEMENT_SIZE); + } + + /* Build the decoding table for the precode. */ + if (make_huffman_decode_table(d->precode_decode_table, + LZX_PRECODE_NUM_SYMBOLS, + LZX_PRECODE_TABLEBITS, + d->precode_lens, + LZX_MAX_PRE_CODEWORD_LEN, + d->precode_working_space, + ARRAY_SIZE(d->precode_decode_table))) + return -1; + + /* Decode the codeword lengths. */ + do { + u32 presym; + u8 len; + + presym = read_presym(d, is); + if (presym < 17) { + /* Difference from old length. */ + len = *len_ptr - presym; + if ((s8)len < 0) + len += 17; + *len_ptr++ = len; + } else { + /* Special RLE values. */ + u32 run_len; + + if (presym == 17) { + run_len = 4 + bitstream_read_bits(is, 4); + len = 0; + } else if (presym == 18) { + run_len = 20 + bitstream_read_bits(is, 5); + len = 0; + } else { + run_len = 4 + bitstream_read_bits(is, 1); + presym = read_presym(d, is); + if (unlikely(presym > 17)) + return -1; + len = *len_ptr - presym; + if ((s8)len < 0) + len += 17; + } + + do { + *len_ptr++ = len; + } while (--run_len); + /* The worst case overrun is when presym == 18, + * run_len == 20 + 31, and only 1 length was + * remaining, so LZX_READ_LENS_MAX_OVERRUN == 50. + * Overrun while reading the first half of + * maincode_lens can corrupt the previous values in + * the second half, but the resulting lengths will + * still be in range, and data that generates overruns + * is invalid anyway. + */ + } + } while (len_ptr < lens_end); + + return 0; +} + +static void undo_translate_target(void *target, s32 input_pos) +{ + s32 abs_offset, rel_offset; + + abs_offset = get_unaligned_le32(target); + if (abs_offset >= 0) { + if (abs_offset < LZX_WIM_MAGIC_FILESIZE) { + /* "good translation" */ + rel_offset = abs_offset - input_pos; + put_unaligned_le32(rel_offset, target); + } + } else { + if (abs_offset >= -input_pos) { + /* "compensating translation" */ + rel_offset = abs_offset + LZX_WIM_MAGIC_FILESIZE; + put_unaligned_le32(rel_offset, target); + } + } +} + +/* + * Undo the 'E8' preprocessing used in LZX. Before compression, the + * uncompressed data was preprocessed by changing the targets of suspected x86 + * CALL instructions from relative offsets to absolute offsets. After + * match/literal decoding, the decompressor must undo the translation. + * + * E8 preprocessing is disabled in the last 6 bytes of the data, which means + * the 5-byte call instruction cannot start in the last 10 bytes. The scalar + * implementation below exploits this by replacing the last 6 bytes with 0xE8 + * trap bytes, eliminating end-of-buffer checks from the inner loop. + */ +static void lzx_postprocess(u8 *data, u32 size) +{ + u8 *tail; + u8 saved_bytes[6]; + u8 *p; + + if (size <= 10) + return; + + tail = &data[size - 6]; + memcpy(saved_bytes, tail, 6); + memset(tail, 0xE8, 6); + p = data; + for (;;) { + while (*p != 0xE8) + p++; + if (p >= tail) + break; + undo_translate_target(p + 1, (s32)(p - data)); + p += 5; + } + memcpy(tail, saved_bytes, 6); +} + +static int lzx_read_block_header(struct lzx_decompressor *d, + struct input_bitstream *is, + u32 recent_offsets[], int *block_type_ret, + u32 *block_size_ret) +{ + int block_type; + u32 block_size; + u32 i; + + bitstream_ensure_bits(is, 4); + + /* Read the block type. */ + block_type = bitstream_pop_bits(is, 3); + + /* Read the block size. With the 32768-byte window used in system + * compression, block sizes are always encoded in 16 bits. + */ + if (bitstream_pop_bits(is, 1)) + block_size = LZX_DEFAULT_BLOCK_SIZE; + else + block_size = bitstream_read_bits(is, 16); + + switch (block_type) { + case LZX_BLOCKTYPE_ALIGNED: + /* Read the aligned offset codeword lengths. */ + for (i = 0; i < LZX_ALIGNEDCODE_NUM_SYMBOLS; i++) { + d->alignedcode_lens[i] = + bitstream_read_bits(is, + LZX_ALIGNEDCODE_ELEMENT_SIZE); + } + /* Fall though, since the rest of the header for aligned offset + * blocks is the same as that for verbatim blocks. + */ + fallthrough; + + case LZX_BLOCKTYPE_VERBATIM: + /* Read the main codeword lengths, which are divided into two + * parts: literal symbols and match headers. + */ + if (lzx_read_codeword_lens(d, is, d->maincode_lens, + LZX_NUM_CHARS)) + return -1; + if (lzx_read_codeword_lens(d, is, + d->maincode_lens + LZX_NUM_CHARS, + LZX_MAINCODE_NUM_SYMBOLS - LZX_NUM_CHARS)) + return -1; + + /* Read the length codeword lengths. */ + if (lzx_read_codeword_lens(d, is, d->lencode_lens, + LZX_LENCODE_NUM_SYMBOLS)) + return -1; + break; + + case LZX_BLOCKTYPE_UNCOMPRESSED: + /* The header of an uncompressed block contains new values for + * the recent offsets queue, starting on the next 16-bit + * boundary in the bitstream. If the stream is *already* + * aligned, the next 16 bits must be discarded. + */ + bitstream_ensure_bits(is, 1); + bitstream_align(is); + recent_offsets[0] = bitstream_read_u32(is); + recent_offsets[1] = bitstream_read_u32(is); + recent_offsets[2] = bitstream_read_u32(is); + + /* Offsets of 0 are invalid. */ + if (recent_offsets[0] == 0 || recent_offsets[1] == 0 || + recent_offsets[2] == 0) + return -1; + break; + + default: + /* Unrecognized block type. */ + return -1; + } + + *block_type_ret = block_type; + *block_size_ret = block_size; + return 0; +} + +static int lzx_decompress_block(struct lzx_decompressor *d, + struct input_bitstream *is, int block_type, + u32 block_size, u8 *const out_begin, + u8 *out_next, u32 recent_offsets[]) +{ + u8 *const block_end = out_next + block_size; + unsigned int min_aligned_offset_slot; + const u8 *extra_offset_bits; + + /* Build the Huffman decode tables. The main and length tables are + * always needed; for aligned blocks the aligned offset table is also + * needed. + */ + if (make_huffman_decode_table(d->maincode_decode_table, + LZX_MAINCODE_NUM_SYMBOLS, + LZX_MAINCODE_TABLEBITS, d->maincode_lens, + LZX_MAX_MAIN_CODEWORD_LEN, + d->maincode_working_space, + ARRAY_SIZE(d->maincode_decode_table))) + return -1; + + if (make_huffman_decode_table(d->lencode_decode_table, + LZX_LENCODE_NUM_SYMBOLS, + LZX_LENCODE_TABLEBITS, d->lencode_lens, + LZX_MAX_LEN_CODEWORD_LEN, + d->lencode_working_space, + ARRAY_SIZE(d->lencode_decode_table))) + return -1; + + if (block_type == LZX_BLOCKTYPE_ALIGNED) { + if (make_huffman_decode_table(d->alignedcode_decode_table, + LZX_ALIGNEDCODE_NUM_SYMBOLS, + LZX_ALIGNEDCODE_TABLEBITS, + d->alignedcode_lens, + LZX_MAX_ALIGNED_CODEWORD_LEN, + d->alignedcode_working_space, + ARRAY_SIZE(d->alignedcode_decode_table))) + return -1; + min_aligned_offset_slot = LZX_MIN_ALIGNED_OFFSET_SLOT; + extra_offset_bits = lzx_extra_offset_bits_minus_aligned; + } else { + min_aligned_offset_slot = LZX_NUM_OFFSET_SLOTS; + extra_offset_bits = lzx_extra_offset_bits; + } + + /* Decode the literals and matches. */ + do { + unsigned int mainsym; + unsigned int length; + u32 offset; + unsigned int offset_slot; + + mainsym = read_mainsym(d, is); + if (mainsym < LZX_NUM_CHARS) { + /* Literal */ + *out_next++ = mainsym; + continue; + } + + /* Match */ + + /* Decode the length header and offset slot. + */ + STATIC_ASSERT(LZX_NUM_CHARS % LZX_NUM_LEN_HEADERS == 0); + length = mainsym % LZX_NUM_LEN_HEADERS; + offset_slot = (mainsym - LZX_NUM_CHARS) / LZX_NUM_LEN_HEADERS; + + /* If needed, read a length symbol to decode the full length. */ + if (length == LZX_NUM_PRIMARY_LENS) + length += read_lensym(d, is); + length += LZX_MIN_MATCH_LEN; + + if (offset_slot < LZX_NUM_RECENT_OFFSETS) { + /* Repeat offset. This isn't a real LRU queue, since + * using the R2 offset doesn't bump the R1 offset down + * to R2. + */ + offset = recent_offsets[offset_slot]; + recent_offsets[offset_slot] = recent_offsets[0]; + } else { + /* Explicit offset. */ + offset = bitstream_read_bits(is, + extra_offset_bits[offset_slot]); + if (offset_slot >= min_aligned_offset_slot) { + offset = (offset << LZX_NUM_ALIGNED_OFFSET_BITS) | + read_alignedsym(d, is); + } + offset += lzx_offset_slot_base[offset_slot]; + + /* Update the match offset LRU queue. */ + STATIC_ASSERT(LZX_NUM_RECENT_OFFSETS == 3); + recent_offsets[2] = recent_offsets[1]; + recent_offsets[1] = recent_offsets[0]; + } + recent_offsets[0] = offset; + + /* Validate the match and copy it to the current position. */ + if (unlikely(lz_copy(length, offset, out_begin, out_next, + block_end, LZX_MIN_MATCH_LEN))) + return -1; + out_next += length; + } while (out_next != block_end); + + return 0; +} + +int lzx_decompress(struct lzx_decompressor *d, const void *compressed_data, + size_t compressed_size, void *uncompressed_data, + size_t uncompressed_size) +{ + u8 *const out_begin = uncompressed_data; + u8 *out_next = out_begin; + u8 *const out_end = out_begin + uncompressed_size; + struct input_bitstream is; + + STATIC_ASSERT(LZX_NUM_RECENT_OFFSETS == 3); + u32 recent_offsets[LZX_NUM_RECENT_OFFSETS] = {1, 1, 1}; + bool may_have_e8_byte = false; + + init_input_bitstream(&is, compressed_data, compressed_size); + + /* Codeword lengths begin as all 0's for delta encoding purposes. */ + memset(d->maincode_lens, 0, LZX_MAINCODE_NUM_SYMBOLS); + memset(d->lencode_lens, 0, LZX_LENCODE_NUM_SYMBOLS); + + /* Decompress blocks until we have all the uncompressed data. + */ + while (out_next != out_end) { + int block_type; + u32 block_size; + + if (lzx_read_block_header(d, &is, recent_offsets, &block_type, + &block_size)) + return -1; + + if (block_size < 1 || block_size > (u32)(out_end - out_next)) + return -1; + + if (likely(block_type != LZX_BLOCKTYPE_UNCOMPRESSED)) { + /* Compressed block. */ + if (lzx_decompress_block(d, &is, block_type, block_size, + out_begin, out_next, + recent_offsets)) + return -1; + + /* If the first E8 byte was in this block, then it + * must have been encoded as a literal (mainsym E8). + */ + if (d->maincode_lens[0xE8]) + may_have_e8_byte = true; + } else { + /* Uncompressed block. */ + if (bitstream_read_bytes(&is, out_next, block_size)) + return -1; + if (block_size & 1) + bitstream_read_byte(&is); + /* There may have been an E8 byte in the block. */ + may_have_e8_byte = true; + } + out_next += block_size; + } + + /* Postprocess the data unless it cannot possibly contain E8 bytes. */ + if (may_have_e8_byte) + lzx_postprocess(uncompressed_data, uncompressed_size); + + return 0; +} + +struct lzx_decompressor *lzx_allocate_decompressor(void) +{ + return kmalloc_obj(struct lzx_decompressor, GFP_NOFS); +} + +void lzx_free_decompressor(struct lzx_decompressor *d) +{ + kfree(d); +} + +static size_t lzx_scratch_size(u32 chunk_size) +{ + return sizeof(struct lzx_decompressor); +} + +static int lzx_decompress_chunk(void *scratch, const void *src, size_t src_len, + void *dst, size_t dst_len, u32 chunk_size) +{ + struct lzx_decompressor *d = scratch; + + return lzx_decompress(d, src, src_len, dst, dst_len); +} + +const struct ntfs_codec_ops ntfs_lzx32k_codec_ops = { + .id = NTFS_CODEC_LZX32K, + .name = "lzx32k", + .scratch_size = lzx_scratch_size, + .decompress_chunk = lzx_decompress_chunk, +}; diff --git a/fs/ntfs/lib/xpress_decompress.c b/fs/ntfs/lib/xpress_decompress.c new file mode 100644 index 000000000000..cb212543268d --- /dev/null +++ b/fs/ntfs/lib/xpress_decompress.c @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * xpress_decompress.c - A decompressor for the XPRESS compression format + * (Huffman variant), which can be used in "System Compressed" (WOF) files. + * + * This is a port of the upstream wimlib "xpress_decompress.c" which uses a + * subtable-based Huffman decode table format. The decode table and the + * codeword-length array share a union since the lengths are fully consumed + * before the table is written. + * + * Copyright (C) 2012-2016 Eric Biggers + */ + +#include <linux/array_size.h> + +#include "decompress_common.h" +#include "lib.h" +#include "../ntfs_codec.h" + +#define XPRESS_NUM_CHARS 256 +#define XPRESS_NUM_SYMBOLS 512 +#define XPRESS_MAX_CODEWORD_LEN 15 +#define XPRESS_MIN_MATCH_LEN 3 + +/* This value is chosen for fast decompression. */ +#define XPRESS_TABLEBITS 11 + +/* Reusable heap-allocated memory for XPRESS decompression. The decode table + * and the codeword-length array alias each other in a union: all lengths are + * consumed into the working space before any decode-table entry is written. + */ +struct xpress_decompressor { + union { + DECODE_TABLE(decode_table, XPRESS_NUM_SYMBOLS, XPRESS_TABLEBITS, + XPRESS_MAX_CODEWORD_LEN); + u8 lens[XPRESS_NUM_SYMBOLS]; + }; + DECODE_TABLE_WORKING_SPACE(working_space, XPRESS_NUM_SYMBOLS, + XPRESS_MAX_CODEWORD_LEN); +} __aligned(DECODE_TABLE_ALIGNMENT); + +int xpress_decompress(struct xpress_decompressor *d, + const void *compressed_data, size_t compressed_size, + void *uncompressed_data, size_t uncompressed_size) +{ + const u8 *const in_begin = compressed_data; + u8 *const out_begin = uncompressed_data; + u8 *out_next = out_begin; + u8 *const out_end = out_begin + uncompressed_size; + struct input_bitstream is; + u32 i; + + /* Read the Huffman codeword lengths (512 4-bit values packed into 256 + * bytes). + */ + if (compressed_size < XPRESS_NUM_SYMBOLS / 2) + return -1; + for (i = 0; i < XPRESS_NUM_SYMBOLS / 2; i++) { + d->lens[2 * i + 0] = in_begin[i] & 0xf; + d->lens[2 * i + 1] = in_begin[i] >> 4; + } + + /* Build a decoding table for the Huffman code. */ + if (make_huffman_decode_table(d->decode_table, XPRESS_NUM_SYMBOLS, + XPRESS_TABLEBITS, d->lens, + XPRESS_MAX_CODEWORD_LEN, + d->working_space, + ARRAY_SIZE(d->decode_table))) + return -1; + + /* Decode the matches and literals. */ + init_input_bitstream(&is, in_begin + XPRESS_NUM_SYMBOLS / 2, + compressed_size - XPRESS_NUM_SYMBOLS / 2); + + while (out_next != out_end) { + u32 sym; + u32 log2_offset; + u32 length; + u32 offset; + + sym = read_huffsym(&is, d->decode_table, XPRESS_TABLEBITS, + XPRESS_MAX_CODEWORD_LEN); + if (sym < XPRESS_NUM_CHARS) { + /* Literal */ + *out_next++ = sym; + } else { + /* Match */ + length = sym & 0xf; + log2_offset = (sym >> 4) & 0xf; + + bitstream_ensure_bits(&is, 16); + + offset = ((u32)1 << log2_offset) | + bitstream_pop_bits(&is, log2_offset); + + if (length == 0xf) { + length += bitstream_read_byte(&is); + if (length == 0xf + 0xff) + length = bitstream_read_u16(&is); + } + length += XPRESS_MIN_MATCH_LEN; + + if (unlikely(lz_copy(length, offset, out_begin, out_next, + out_end, XPRESS_MIN_MATCH_LEN))) + return -1; + + out_next += length; + } + } + return 0; +} + +struct xpress_decompressor *xpress_allocate_decompressor(void) +{ + return kmalloc_obj(struct xpress_decompressor, GFP_NOFS); +} + +void xpress_free_decompressor(struct xpress_decompressor *d) +{ + kfree(d); +} + +static size_t xpress_scratch_size(u32 chunk_size) +{ + return sizeof(struct xpress_decompressor); +} + +static int xpress_decompress_chunk(void *scratch, const void *src, + size_t src_len, void *dst, size_t dst_len, + u32 chunk_size) +{ + return xpress_decompress(scratch, src, src_len, dst, dst_len); +} + +const struct ntfs_codec_ops ntfs_xpress4k_codec_ops = { + .id = NTFS_CODEC_XPRESS4K, + .name = "xpress4k", + .scratch_size = xpress_scratch_size, + .decompress_chunk = xpress_decompress_chunk, +}; + +const struct ntfs_codec_ops ntfs_xpress8k_codec_ops = { + .id = NTFS_CODEC_XPRESS8K, + .name = "xpress8k", + .scratch_size = xpress_scratch_size, + .decompress_chunk = xpress_decompress_chunk, +}; + +const struct ntfs_codec_ops ntfs_xpress16k_codec_ops = { + .id = NTFS_CODEC_XPRESS16K, + .name = "xpress16k", + .scratch_size = xpress_scratch_size, + .decompress_chunk = xpress_decompress_chunk, +}; diff --git a/fs/ntfs/mft.c b/fs/ntfs/mft.c index fd20d7abd6f5..984a0827f9ac 100644 --- a/fs/ntfs/mft.c +++ b/fs/ntfs/mft.c @@ -2333,7 +2333,17 @@ mft_rec_already_initialized: * wrong with the previous mft record. */ seq_no = m->sequence_number; - usn = *(__le16 *)((u8 *)m + le16_to_cpu(m->usa_ofs)); + /* + * The mft record still holds unvalidated, MST-protected on-disk + * bytes, so m->usa_ofs is untrusted here. Only preserve the old + * update sequence number if that offset is in bounds; otherwise + * leave usn zero so it is not restored below. + */ + if (!(le16_to_cpu(m->usa_ofs) & 1) && + le16_to_cpu(m->usa_ofs) + sizeof(usn) <= vol->mft_record_size) + usn = *(__le16 *)((u8 *)m + le16_to_cpu(m->usa_ofs)); + else + usn = 0; err = ntfs_mft_record_layout(vol, bit, m); if (unlikely(err)) { ntfs_error(vol->sb, "Failed to layout allocated mft record 0x%llx.", @@ -2420,7 +2430,7 @@ mft_rec_already_initialized: * record. */ - (*ni)->mrec = kmalloc(vol->mft_record_size, GFP_NOFS); + (*ni)->mrec = kmemdup(m, vol->mft_record_size, GFP_NOFS); if (!(*ni)->mrec) { folio_unlock(folio); kunmap_local(m); @@ -2429,7 +2439,6 @@ mft_rec_already_initialized: goto undo_mftbmp_alloc; } - memcpy((*ni)->mrec, m, vol->mft_record_size); post_read_mst_fixup((struct ntfs_record *)(*ni)->mrec, vol->mft_record_size); ntfs_mft_mark_dirty(folio); folio_unlock(folio); diff --git a/fs/ntfs/namei.c b/fs/ntfs/namei.c index 8364d7bb571d..7091b2496fac 100644 --- a/fs/ntfs/namei.c +++ b/fs/ntfs/namei.c @@ -61,12 +61,12 @@ static int ntfs_check_bad_windows_name(struct ntfs_volume *vol, const __le16 *wc, unsigned int wc_len) { - if (ntfs_check_bad_char(wc, wc_len)) - return -EINVAL; - if (!NVolCheckWindowsNames(vol)) return 0; + if (ntfs_check_bad_char(wc, wc_len)) + return -EINVAL; + /* Check for trailing space or dot. */ if (wc_len > 0 && (wc[wc_len - 1] == cpu_to_le16(' ') || @@ -424,8 +424,6 @@ static struct ntfs_inode *__ntfs_create(struct mnt_idmap *idmap, struct inode *d * directories, also setup the index values to the defaults. */ if (S_ISDIR(mode)) { - mode &= ~vol->dmask; - NInoSetMstProtected(ni); ni->itype.index.block_size = 4096; ni->itype.index.block_size_bits = ntfs_ffs(4096) - 1; @@ -439,8 +437,6 @@ static struct ntfs_inode *__ntfs_create(struct mnt_idmap *idmap, struct inode *d ni->itype.index.vcn_size_bits = vol->sector_size_bits; } - } else { - mode &= ~vol->fmask; } if (IS_RDONLY(vi)) diff --git a/fs/ntfs/ntfs.h b/fs/ntfs/ntfs.h index 45064dbcc2e4..df5a75d506f6 100644 --- a/fs/ntfs/ntfs.h +++ b/fs/ntfs/ntfs.h @@ -202,6 +202,10 @@ static inline struct ntfs_volume *NTFS_SB(struct super_block *sb) /* From fs/ntfs/compress.c */ int ntfs_read_compressed_block(struct folio *folio); +#ifdef CONFIG_NTFS_FS_WOF_COMPRESSION +int ntfs_read_wof_compressed_block(struct folio *folio); +void ntfs_wof_free_workspaces(void); +#endif int allocate_compression_buffers(void); void free_compression_buffers(void); int ntfs_compress_write(struct ntfs_inode *ni, loff_t pos, size_t count, diff --git a/fs/ntfs/ntfs_codec.h b/fs/ntfs/ntfs_codec.h new file mode 100644 index 000000000000..6030055fb0c1 --- /dev/null +++ b/fs/ntfs/ntfs_codec.h @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Transparent compression codec interface. + * + * Copyright (c) 2026 LG Electronics Co., Ltd. + */ + +#ifndef _NTFS_CODEC_H +#define _NTFS_CODEC_H + +#include <linux/types.h> +#include <linux/fs.h> +#include <linux/mm.h> + +struct compress_context; + +enum ntfs_codec_id { + NTFS_CODEC_LZNT1, +#ifdef CONFIG_NTFS_FS_WOF_COMPRESSION + NTFS_CODEC_XPRESS4K, + NTFS_CODEC_XPRESS8K, + NTFS_CODEC_XPRESS16K, + NTFS_CODEC_LZX32K, +#endif +}; + +struct ntfs_codec_ops { + enum ntfs_codec_id id; + const char *name; + size_t (*scratch_size)(u32 chunk_size); + int (*decompress_chunk)(void *scratch, + const void *src, size_t src_len, + void *dst, size_t dst_len, + u32 chunk_size); + int (*decompress_pages)(struct page *dest_pages[], + int completed_pages[], + int *dest_index, int *dest_ofs, + int dest_max_index, int dest_max_ofs, + int xpage, char *xpage_done, + u8 *cb_start, u32 cb_size, + loff_t i_size, s64 initialized_size); + int (*compress_subblock)(struct compress_context *pctx, + const char *inbuf, int bufsize, char *outbuf); +}; + +extern const struct ntfs_codec_ops ntfs_lznt1_codec_ops; +#ifdef CONFIG_NTFS_FS_WOF_COMPRESSION +extern const struct ntfs_codec_ops ntfs_xpress4k_codec_ops; +extern const struct ntfs_codec_ops ntfs_xpress8k_codec_ops; +extern const struct ntfs_codec_ops ntfs_xpress16k_codec_ops; +extern const struct ntfs_codec_ops ntfs_lzx32k_codec_ops; +#endif + +#endif /* _NTFS_CODEC_H */ diff --git a/fs/ntfs/reparse.c b/fs/ntfs/reparse.c index fa523dc3691e..5e483a2f9060 100644 --- a/fs/ntfs/reparse.c +++ b/fs/ntfs/reparse.c @@ -24,6 +24,25 @@ struct wsl_link_reparse_data { char link[]; }; +struct wof_reparse_data { + __le32 version; + __le32 provider; + __le32 provider_version; + __le32 compression_format; +} __packed; + +#define WOF_CURRENT_VERSION cpu_to_le32(1) + +#define WOF_PROVIDER_WIM cpu_to_le32(1) +#define WOF_PROVIDER_FILE cpu_to_le32(2) + +#define WOF_PROVIDER_CURRENT_VERSION cpu_to_le32(1) + +#define WOF_COMPRESSION_XPRESS4K cpu_to_le32(0) +#define WOF_COMPRESSION_LZX cpu_to_le32(1) +#define WOF_COMPRESSION_XPRESS8K cpu_to_le32(2) +#define WOF_COMPRESSION_XPRESS16K cpu_to_le32(3) + static bool reparse_name_is_valid(size_t size, size_t name_off, u16 len) { if ((name_off | len) & 1) @@ -203,6 +222,12 @@ static bool valid_reparse_data(struct ntfs_inode *ni, !(ni->flags & FILE_ATTRIBUTE_RECALL_ON_OPEN)) return false; break; + case IO_REPARSE_TAG_WOF: { + if (!valid_reparse_buffer(ni, reparse_attr, size, + sizeof(struct wof_reparse_data))) + return false; + break; + } default: if (!valid_reparse_buffer(ni, reparse_attr, size, 0)) return false; @@ -239,88 +264,136 @@ static unsigned int ntfs_reparse_tag_mode(__le32 reparse_tag) } /* - * Get the target for symbolic link + * Parse reparse point data and initialize its in-memory representation. */ -unsigned int ntfs_make_symlink(struct ntfs_inode *ni) +int ntfs_parse_reparse(struct ntfs_inode *ni, unsigned int *mode) { s64 attr_size = 0; - int err; + int err = -EINVAL; unsigned int lth; struct reparse_point *reparse_attr; - unsigned int mode = 0; kvfree(ni->target); ni->target = NULL; ni->reparse_tag = 0; ni->reparse_flags = 0; + *mode = 0; reparse_attr = ntfs_attr_readall(ni, AT_REPARSE_POINT, NULL, 0, &attr_size); - if (reparse_attr && - valid_reparse_data(ni, reparse_attr, attr_size)) { - err = -EINVAL; + if (IS_ERR(reparse_attr)) { + err = PTR_ERR(reparse_attr); + ntfs_error(ni->vol->sb, + "Failed to read reparse point: %d.", err); + return err; + } + if (!valid_reparse_data(ni, reparse_attr, attr_size)) { + ntfs_error(ni->vol->sb, "Invalid reparse point."); + err = -EFSCORRUPTED; + goto out; + } - switch (reparse_attr->reparse_tag) { - case IO_REPARSE_TAG_MOUNT_POINT: - { - struct mount_point_reparse_data *data = - (struct mount_point_reparse_data *)reparse_attr->reparse_data; - const __le16 *name = (const __le16 *)((u8 *)data->path_buffer + - le16_to_cpu(data->substitute_name_offset)); - - err = ntfs_reparse_target_to_nls(ni->vol, - name, - le16_to_cpu(data->substitute_name_length), - &ni->target); - break; - } - case IO_REPARSE_TAG_SYMLINK: - { - struct symlink_reparse_data *data = - (struct symlink_reparse_data *)reparse_attr->reparse_data; - const __le16 *name = (const __le16 *)((u8 *)data->path_buffer + - le16_to_cpu(data->substitute_name_offset)); - - err = ntfs_reparse_target_to_nls(ni->vol, - name, - le16_to_cpu(data->substitute_name_length), - &ni->target); - if (!err) - ni->reparse_flags = data->flags; - break; - } - case IO_REPARSE_TAG_LX_SYMLINK: - { - struct wsl_link_reparse_data *wsl_link_data = - (struct wsl_link_reparse_data *)reparse_attr->reparse_data; - - if (wsl_link_data->type == cpu_to_le32(2)) { - lth = le16_to_cpu(reparse_attr->reparse_data_length) - - sizeof(wsl_link_data->type); - ni->target = kvzalloc(lth + 1, GFP_NOFS); - if (ni->target) { - memcpy(ni->target, wsl_link_data->link, lth); - ni->target[lth] = 0; - err = 0; - } + switch (reparse_attr->reparse_tag) { + case IO_REPARSE_TAG_MOUNT_POINT: + { + struct mount_point_reparse_data *data = + (struct mount_point_reparse_data *)reparse_attr->reparse_data; + const __le16 *name = (const __le16 *)((u8 *)data->path_buffer + + le16_to_cpu(data->substitute_name_offset)); + + err = ntfs_reparse_target_to_nls(ni->vol, + name, + le16_to_cpu(data->substitute_name_length), + &ni->target); + break; + } + case IO_REPARSE_TAG_SYMLINK: + { + struct symlink_reparse_data *data = + (struct symlink_reparse_data *)reparse_attr->reparse_data; + const __le16 *name = (const __le16 *)((u8 *)data->path_buffer + + le16_to_cpu(data->substitute_name_offset)); + + err = ntfs_reparse_target_to_nls(ni->vol, + name, + le16_to_cpu(data->substitute_name_length), + &ni->target); + if (!err) + ni->reparse_flags = data->flags; + break; + } + case IO_REPARSE_TAG_LX_SYMLINK: + { + struct wsl_link_reparse_data *wsl_link_data = + (struct wsl_link_reparse_data *)reparse_attr->reparse_data; + + if (wsl_link_data->type == cpu_to_le32(2)) { + lth = le16_to_cpu(reparse_attr->reparse_data_length) - + sizeof(wsl_link_data->type); + ni->target = kvzalloc(lth + 1, GFP_NOFS); + if (ni->target) { + memcpy(ni->target, wsl_link_data->link, lth); + ni->target[lth] = 0; + err = 0; } - break; } - default: - err = 0; + break; + } + case IO_REPARSE_TAG_WOF: + { +#ifdef CONFIG_NTFS_FS_WOF_COMPRESSION + const struct wof_reparse_data *wof_data = + (const struct wof_reparse_data *)reparse_attr->reparse_data; + + ni->itype.compressed.block_size_bits = 0; + ni->itype.compressed.block_size = 0; + if (wof_data->version == WOF_CURRENT_VERSION && + wof_data->provider == WOF_PROVIDER_FILE && + wof_data->provider_version == + WOF_PROVIDER_CURRENT_VERSION) { + switch (wof_data->compression_format) { + case WOF_COMPRESSION_XPRESS4K: + ni->itype.compressed.block_size_bits = + 12; + break; + case WOF_COMPRESSION_XPRESS8K: + ni->itype.compressed.block_size_bits = + 13; + break; + case WOF_COMPRESSION_XPRESS16K: + ni->itype.compressed.block_size_bits = + 14; + break; + case WOF_COMPRESSION_LZX: + ni->itype.compressed.block_size_bits = + 15; + break; + } } + if (ni->itype.compressed.block_size_bits) + ni->itype.compressed.block_size = + 1 + << ni->itype.compressed.block_size_bits; +#endif + NInoSetWofCompressed(ni); + VFS_I(ni)->i_mode &= ~0222; + err = 0; + break; + } + default: + err = 0; + } - if (!err) { - mode = ntfs_reparse_tag_mode(reparse_attr->reparse_tag); - ni->reparse_tag = reparse_attr->reparse_tag; - } - } else - ni->flags &= ~FILE_ATTR_REPARSE_POINT; + if (!err) { + *mode = ntfs_reparse_tag_mode( + reparse_attr->reparse_tag); + ni->reparse_tag = reparse_attr->reparse_tag; + } - if (reparse_attr) - kvfree(reparse_attr); +out: + kvfree(reparse_attr); - return mode; + return err; } unsigned int ntfs_reparse_tag_dt_types(struct ntfs_volume *vol, unsigned long mref) @@ -336,6 +409,8 @@ unsigned int ntfs_reparse_tag_dt_types(struct ntfs_volume *vol, unsigned long mr reparse_attr = (struct reparse_point *)ntfs_attr_readall(NTFS_I(vi), AT_REPARSE_POINT, NULL, 0, &attr_size); + if (IS_ERR(reparse_attr)) + reparse_attr = NULL; if (reparse_attr && attr_size >= sizeof(*reparse_attr)) { switch (reparse_attr->reparse_tag) { @@ -358,8 +433,7 @@ unsigned int ntfs_reparse_tag_dt_types(struct ntfs_volume *vol, unsigned long mr } } - if (reparse_attr) - kvfree(reparse_attr); + kvfree(reparse_attr); iput(vi); return dt_type; @@ -894,12 +968,7 @@ int ntfs_reparse_set_native_symlink(struct ntfs_inode *ni, err = ntfs_set_ntfs_reparse_data(ni, (char *)reparse, total_reparse_len); if (!err) { - int len = strlen(sub_name); - - for (i = 0; i < len; i++) { - if (sub_name[i] == '\\') - sub_name[i] = '/'; - } + strreplace(sub_name, '\\', '/'); ni->target = sub_name; sub_name = NULL; if (prt_sub_shared) diff --git a/fs/ntfs/reparse.h b/fs/ntfs/reparse.h index c11a5bb7e6a5..b6360b0452e6 100644 --- a/fs/ntfs/reparse.h +++ b/fs/ntfs/reparse.h @@ -9,7 +9,7 @@ extern __le16 reparse_index_name[]; -unsigned int ntfs_make_symlink(struct ntfs_inode *ni); +int ntfs_parse_reparse(struct ntfs_inode *ni, unsigned int *mode); unsigned int ntfs_reparse_tag_dt_types(struct ntfs_volume *vol, unsigned long mref); int ntfs_translate_symlink_path(struct dentry *dentry, const char *target, char **translated); diff --git a/fs/ntfs/runlist.c b/fs/ntfs/runlist.c index 8e0fd400e7f7..00373e450ea7 100644 --- a/fs/ntfs/runlist.c +++ b/fs/ntfs/runlist.c @@ -772,6 +772,25 @@ struct runlist_element *ntfs_mapping_pairs_decompress(const struct ntfs_volume * return ERR_PTR(-EIO); } + /* + * An empty mapping-pairs array is valid only for a zero-length + * attribute. + */ + if (!*buf && + (vcn || + le64_to_cpu(attr->data.non_resident.highest_vcn) != + (u64)(vcn - 1) || + le64_to_cpu(attr->data.non_resident.allocated_size) || + le64_to_cpu(attr->data.non_resident.data_size) || + le64_to_cpu(attr->data.non_resident.initialized_size))) { + ntfs_error(vol->sb, "Invalid empty mapping pairs array."); + return ERR_PTR(-EIO); + } + if (!vcn && !*buf && old_runlist && old_runlist->rl) { + *new_rl_count = old_runlist->count; + return old_runlist->rl; + } + /* Current position in runlist array. */ rlpos = 0; /* Allocate first page and set current runlist size to one page. */ @@ -884,12 +903,41 @@ struct runlist_element *ntfs_mapping_pairs_decompress(const struct ntfs_volume * ntfs_error(vol->sb, "lcn == -1"); } #endif + /* Check lcn is within the volume. */ + if (unlikely(lcn >= (s64)vol->nr_clusters)) { + ntfs_error(vol->sb, + "LCN >= nr_clusters in mapping pairs array."); + goto err_out; + } + /* Check lcn is not below -1. */ if (unlikely(lcn < -1)) { ntfs_error(vol->sb, "Invalid s64 < -1 in mapping pairs array."); goto err_out; } + if (lcn >= 0) { + s64 run_end; + + /* + * Ensure that the run stays within the volume. + * A valid starting LCN is not sufficient because + * the run length comes from disk. + */ + if (unlikely(check_add_overflow(lcn, + rl[rlpos].length, + &run_end))) { + ntfs_error(vol->sb, + "Run length overflow in mapping pairs array."); + goto err_out; + } + if (unlikely(run_end > (s64)vol->nr_clusters)) { + ntfs_error(vol->sb, + "Run extends beyond volume boundary."); + goto err_out; + } + } + /* chkdsk accepts zero-sized runs only for holes */ if ((lcn != -1) && !rl[rlpos].length) { ntfs_error(vol->sb, diff --git a/fs/ntfs/super.c b/fs/ntfs/super.c index 8abe7bee4c0d..30481e5d5dd4 100644 --- a/fs/ntfs/super.c +++ b/fs/ntfs/super.c @@ -645,7 +645,7 @@ static bool parse_ntfs_boot_sector(struct ntfs_volume *vol, { unsigned int sectors_per_cluster, sectors_per_cluster_bits, nr_hidden_sects; int clusters_per_mft_record, clusters_per_index_record; - s64 ll; + u64 ll; vol->sector_size = le16_to_cpu(b->bpb.bytes_per_sector); vol->sector_size_bits = ffs(vol->sector_size) - 1; @@ -755,23 +755,23 @@ static bool parse_ntfs_boot_sector(struct ntfs_volume *vol, * the same as it is much faster on 32-bit CPUs. */ ll = le64_to_cpu(b->number_of_sectors) >> sectors_per_cluster_bits; - if ((u64)ll >= 1ULL << 32) { + if (ll >= 1ULL << 32) { ntfs_error(vol->sb, "Cannot handle 64-bit clusters."); return false; } vol->nr_clusters = ll; ntfs_debug("vol->nr_clusters = 0x%llx", vol->nr_clusters); ll = le64_to_cpu(b->mft_lcn); - if (ll >= vol->nr_clusters) { - ntfs_error(vol->sb, "MFT LCN (%lli, 0x%llx) is beyond end of volume. Weird.", + if (ll >= (u64)vol->nr_clusters) { + ntfs_error(vol->sb, "MFT LCN (%llu, 0x%llx) is beyond end of volume. Weird.", ll, ll); return false; } vol->mft_lcn = ll; ntfs_debug("vol->mft_lcn = 0x%llx", vol->mft_lcn); ll = le64_to_cpu(b->mftmirr_lcn); - if (ll >= vol->nr_clusters) { - ntfs_error(vol->sb, "MFTMirr LCN (%lli, 0x%llx) is beyond end of volume. Weird.", + if (ll >= (u64)vol->nr_clusters) { + ntfs_error(vol->sb, "MFTMirr LCN (%llu, 0x%llx) is beyond end of volume. Weird.", ll, ll); return false; } @@ -2691,6 +2691,9 @@ static void __exit exit_ntfs_fs(void) * destroy cache. */ rcu_barrier(); +#ifdef CONFIG_NTFS_FS_WOF_COMPRESSION + ntfs_wof_free_workspaces(); +#endif kmem_cache_destroy(ntfs_big_inode_cache); kmem_cache_destroy(ntfs_inode_cache); kmem_cache_destroy(ntfs_name_cache); diff --git a/fs/ntfs/wof.c b/fs/ntfs/wof.c new file mode 100644 index 000000000000..8f84c2212eee --- /dev/null +++ b/fs/ntfs/wof.c @@ -0,0 +1,726 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Windows System Compression (WOF) decompression glue. + * + * Copyright (c) 2026 LG Electronics Co., Ltd. + */ + +#include <linux/fs.h> +#include <linux/blkdev.h> +#include <linux/overflow.h> +#include <linux/pagemap.h> +#include <linux/sched/mm.h> +#include <linux/slab.h> +#include <linux/unaligned.h> +#include <linux/vmalloc.h> + +#include "ntfs.h" +#include "inode.h" +#include "debug.h" +#include "ntfs_codec.h" +#include "attrib.h" + +static const __le16 WOF_NAME[] = { + cpu_to_le16('W'), cpu_to_le16('o'), cpu_to_le16('f'), + cpu_to_le16('C'), cpu_to_le16('o'), cpu_to_le16('m'), + cpu_to_le16('p'), cpu_to_le16('r'), cpu_to_le16('e'), + cpu_to_le16('s'), cpu_to_le16('s'), cpu_to_le16('e'), + cpu_to_le16('d'), cpu_to_le16('D'), cpu_to_le16('a'), + cpu_to_le16('t'), cpu_to_le16('a'), +}; + +#define WOF_NAME_LEN 17 + +#define NTFS_WOF_MAX_COMP_UNIT (1U << 15) +#define NTFS_WOF_MAX_PAGES \ + DIV_ROUND_UP(NTFS_WOF_MAX_COMP_UNIT + PAGE_SIZE - 1, PAGE_SIZE) + +struct ntfs_wof_workspace { + struct mutex *lock; + const struct ntfs_codec_ops *codec; + u32 comp_unit; + void *input; + size_t input_size; + void *output; + void *scratch; +}; + +static DEFINE_MUTEX(ntfs_wof_xpress4k_lock); +static DEFINE_MUTEX(ntfs_wof_xpress8k_lock); +static DEFINE_MUTEX(ntfs_wof_xpress16k_lock); +static DEFINE_MUTEX(ntfs_wof_lzx32k_lock); + +static struct ntfs_wof_workspace ntfs_wof_xpress4k_workspace = { + .lock = &ntfs_wof_xpress4k_lock, + .codec = &ntfs_xpress4k_codec_ops, + .comp_unit = 1U << 12, +}; + +static struct ntfs_wof_workspace ntfs_wof_xpress8k_workspace = { + .lock = &ntfs_wof_xpress8k_lock, + .codec = &ntfs_xpress8k_codec_ops, + .comp_unit = 1U << 13, +}; + +static struct ntfs_wof_workspace ntfs_wof_xpress16k_workspace = { + .lock = &ntfs_wof_xpress16k_lock, + .codec = &ntfs_xpress16k_codec_ops, + .comp_unit = 1U << 14, +}; + +static struct ntfs_wof_workspace ntfs_wof_lzx32k_workspace = { + .lock = &ntfs_wof_lzx32k_lock, + .codec = &ntfs_lzx32k_codec_ops, + .comp_unit = 1U << 15, +}; + +static struct ntfs_wof_workspace *const ntfs_wof_workspaces[] = { + &ntfs_wof_xpress4k_workspace, + &ntfs_wof_xpress8k_workspace, + &ntfs_wof_xpress16k_workspace, + &ntfs_wof_lzx32k_workspace, +}; + +static struct ntfs_wof_workspace *ntfs_wof_workspace(u8 block_size_bits) +{ + switch (block_size_bits) { + case 12: + return &ntfs_wof_xpress4k_workspace; + case 13: + return &ntfs_wof_xpress8k_workspace; + case 14: + return &ntfs_wof_xpress16k_workspace; + case 15: + return &ntfs_wof_lzx32k_workspace; + default: + return NULL; + } +} + +static int ntfs_wof_workspace_prepare(struct ntfs_wof_workspace *ws) +{ + void *input, *output, *scratch; + size_t scratch_size; + + if (ws->input) + return 0; + + ws->input_size = round_up((size_t)ws->comp_unit + 511, 512); + scratch_size = ws->codec->scratch_size(ws->comp_unit); + if (!scratch_size) + return -EINVAL; + + input = kvmalloc(ws->input_size, GFP_NOFS); + output = kvmalloc(ws->comp_unit, GFP_NOFS); + scratch = kvzalloc(scratch_size, GFP_NOFS); + if (!input || !output || !scratch) { + kvfree(input); + kvfree(output); + kvfree(scratch); + return -ENOMEM; + } + + ws->input = input; + ws->output = output; + ws->scratch = scratch; + return 0; +} + +void ntfs_wof_free_workspaces(void) +{ + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(ntfs_wof_workspaces); i++) { + struct ntfs_wof_workspace *ws = ntfs_wof_workspaces[i]; + + mutex_lock(ws->lock); + kvfree(ws->input); + kvfree(ws->output); + kvfree(ws->scratch); + ws->input = NULL; + ws->output = NULL; + ws->scratch = NULL; + mutex_unlock(ws->lock); + } +} + +static int ntfs_bdev_read_from_rl(struct ntfs_volume *vol, + struct runlist *runlist, + sector_t start_sector, u64 sector_count, + void *buf) +{ + struct runlist_element *rl; + u32 sec_per_clu_bits; + s64 vcn; + u64 sec_off; + size_t buf_off = 0; + unsigned int nofs_flags; + int err; + + if (vol->cluster_size_bits < 9) + return -EINVAL; + sec_per_clu_bits = vol->cluster_size_bits - 9; + vcn = start_sector >> sec_per_clu_bits; + sec_off = start_sector & ((1ULL << sec_per_clu_bits) - 1); + + nofs_flags = memalloc_nofs_save(); + down_read(&runlist->lock); + if (!runlist->rl) { + err = -EINVAL; + goto out_unlock; + } + + rl = __ntfs_attr_find_vcn_nolock(runlist, vcn); + if (IS_ERR(rl)) { + err = PTR_ERR(rl); + goto out_unlock; + } + + while (sector_count > 0) { + s64 lcn; + s64 rl_end; + u64 byte_off, byte_len, sectors, available; + + if (rl->length <= 0 || vcn < rl->vcn) { + err = -EINVAL; + goto out_unlock; + } + + lcn = ntfs_rl_vcn_to_lcn(rl, vcn); + if (lcn < 0 && lcn != LCN_HOLE) { + err = -EINVAL; + goto out_unlock; + } + + if (check_add_overflow(rl->vcn, rl->length, &rl_end) || + rl_end <= vcn || + (u64)(rl_end - vcn) > (U64_MAX >> sec_per_clu_bits)) { + err = -EOVERFLOW; + goto out_unlock; + } + available = (u64)(rl_end - vcn) << sec_per_clu_bits; + if (available <= sec_off) { + err = -EINVAL; + goto out_unlock; + } + available -= sec_off; + sectors = min_t(u64, sector_count, available); + if (check_mul_overflow(sectors, (u64)SECTOR_SIZE, &byte_len) || + byte_len > SIZE_MAX - buf_off) { + err = -EOVERFLOW; + goto out_unlock; + } + + if (lcn == LCN_HOLE) { + memset((u8 *)buf + buf_off, 0, byte_len); + } else { + byte_off = ntfs_cluster_to_bytes(vol, lcn); + if (check_add_overflow(byte_off, sec_off << 9, + &byte_off) || + byte_off > S64_MAX) { + err = -EOVERFLOW; + goto out_unlock; + } + err = ntfs_bdev_read(vol->sb->s_bdev, + (char *)buf + buf_off, + (loff_t)byte_off, byte_len); + if (err) + goto out_unlock; + } + + buf_off += byte_len; + sector_count -= sectors; + rl++; + vcn = rl->vcn; + sec_off = 0; + } + + err = 0; +out_unlock: + up_read(&runlist->lock); + memalloc_nofs_restore(nofs_flags); + return err; +} + +static int parse_wof_chunk_table(struct ntfs_inode *base_ni, + struct ntfs_inode *ni, u64 chunk_idx, + u64 chunk_count, u32 decomp_size, + u64 *chunk_offset, u32 *chunk_size, + void *table_buf, size_t table_buf_size) +{ + u8 bytes_per_off; + u8 *buf; + u64 off[2]; + u64 byte_off, chunk_data_size, table_size; + u32 bytes_to_read; + int ret = 0; + + if (i_size_read(VFS_I(base_ni)) < (1ULL << 32)) + bytes_per_off = sizeof(__le32); + else + bytes_per_off = sizeof(__le64); + + if (!chunk_count || chunk_idx >= chunk_count) + return -EINVAL; + + table_size = (chunk_count - 1) * bytes_per_off; + if (ni->data_size < 0 || (u64)ni->data_size < table_size) + return -EINVAL; + chunk_data_size = (u64)ni->data_size - table_size; + + if (chunk_count == 1) { + if (chunk_data_size > decomp_size) + return -EINVAL; + *chunk_offset = 0; + *chunk_size = chunk_data_size; + goto out; + } + + byte_off = chunk_idx ? (chunk_idx - 1) * bytes_per_off : 0; + bytes_to_read = chunk_idx + 1 == chunk_count ? + bytes_per_off : + (chunk_idx ? 2 : 1) * bytes_per_off; + + if (NInoNonResident(ni)) { + sector_t start_sector = byte_off >> 9; + u32 sector_off = byte_off & ((1 << 9) - 1); + u32 sectors = DIV_ROUND_UP(sector_off + bytes_to_read, 512); + + if ((size_t)sectors << 9 > table_buf_size) + return -EINVAL; + buf = table_buf; + ret = ntfs_bdev_read_from_rl(ni->vol, &ni->runlist, + start_sector, sectors, buf); + if (ret) + return -EIO; + buf += sector_off; + } else { + struct ntfs_attr_search_ctx *ctx; + u32 value_length; + u16 value_offset; + + if (bytes_to_read > table_buf_size) + return -EINVAL; + + mutex_lock(&base_ni->mrec_lock); + ctx = ntfs_attr_get_search_ctx(base_ni, NULL); + if (!ctx) { + ret = -ENOMEM; + goto out_unlock_mrec; + } + ret = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, + CASE_SENSITIVE, 0, NULL, 0, ctx); + if (ret) + goto out_put_ctx; + + value_length = + le32_to_cpu(ctx->attr->data.resident.value_length); + value_offset = + le16_to_cpu(ctx->attr->data.resident.value_offset); + if (byte_off + bytes_to_read > value_length) { + ret = -EINVAL; + goto out_put_ctx; + } + memcpy(table_buf, (u8 *)ctx->attr + value_offset + byte_off, + bytes_to_read); + buf = table_buf; +out_put_ctx: + ntfs_attr_put_search_ctx(ctx); +out_unlock_mrec: + mutex_unlock(&base_ni->mrec_lock); + if (ret) + return ret; + } + + if (bytes_per_off == sizeof(__le32)) { + off[0] = chunk_idx ? get_unaligned_le32(buf) : 0; + if (chunk_idx + 1 == chunk_count) + off[1] = chunk_data_size; + else if (chunk_idx) + off[1] = get_unaligned_le32(buf + bytes_per_off); + else + off[1] = get_unaligned_le32(buf); + } else { + off[0] = chunk_idx ? get_unaligned_le64(buf) : 0; + if (chunk_idx + 1 == chunk_count) + off[1] = chunk_data_size; + else if (chunk_idx) + off[1] = get_unaligned_le64(buf + bytes_per_off); + else + off[1] = get_unaligned_le64(buf); + } + + if (off[1] <= off[0] || off[1] > chunk_data_size || + off[1] - off[0] > decomp_size) + return -EINVAL; + + *chunk_offset = table_size + off[0]; + *chunk_size = off[1] - off[0]; +out: + if (!*chunk_size) + return -EINVAL; + return 0; +} + +static int ntfs_read_wof_chunk(struct ntfs_volume *vol, + struct ntfs_inode *wof_ni, u64 chunk_offset, + u32 chunk_size, void *input, size_t input_size, + char **chunk_mem) +{ + struct ntfs_inode *base_ni = wof_ni->ext.base_ntfs_ino; + struct ntfs_attr_search_ctx *ctx; + u32 input_offset = chunk_offset & 511; + u32 input_size_aligned; + u32 value_length; + u16 value_offset; + int err; + + input_size_aligned = round_up(chunk_size + input_offset, 512); + if (input_size_aligned > input_size) + return -EINVAL; + + if (NInoNonResident(wof_ni)) { + err = ntfs_bdev_read_from_rl(vol, &wof_ni->runlist, + chunk_offset >> 9, + input_size_aligned >> 9, input); + if (err) + return err; + *chunk_mem = (u8 *)input + input_offset; + return 0; + } + + mutex_lock(&base_ni->mrec_lock); + ctx = ntfs_attr_get_search_ctx(base_ni, NULL); + if (!ctx) { + err = -ENOMEM; + goto out_unlock_mrec; + } + + err = ntfs_attr_lookup(wof_ni->type, wof_ni->name, wof_ni->name_len, + CASE_SENSITIVE, 0, NULL, 0, ctx); + if (err) + goto out_put_ctx; + + value_length = le32_to_cpu(ctx->attr->data.resident.value_length); + value_offset = le16_to_cpu(ctx->attr->data.resident.value_offset); + if (chunk_offset + chunk_size > value_length) { + err = -EINVAL; + goto out_put_ctx; + } + memcpy(input, (u8 *)ctx->attr + value_offset + chunk_offset, + chunk_size); + *chunk_mem = input; +out_put_ctx: + ntfs_attr_put_search_ctx(ctx); +out_unlock_mrec: + mutex_unlock(&base_ni->mrec_lock); + return err; +} + +struct ntfs_wof_dest { + struct folio *folios[NTFS_WOF_MAX_PAGES]; + struct page *pages[NTFS_WOF_MAX_PAGES]; + unsigned int nr_folios; + unsigned int nr_pages; +}; + +static void ntfs_wof_release_dest(struct ntfs_wof_dest *dest, + struct folio *target, bool success) +{ + unsigned int i; + + for (i = 0; i < dest->nr_folios; i++) { + struct folio *folio = dest->folios[i]; + + if (folio == target) + continue; + if (success) { + flush_dcache_folio(folio); + folio_mark_uptodate(folio); + } else { + folio_clear_uptodate(folio); + } + folio_unlock(folio); + folio_put(folio); + } +} + +static int ntfs_wof_collect_dest(struct address_space *mapping, + struct folio *target, loff_t chunk_start, + loff_t chunk_end, struct ntfs_wof_dest *dest) +{ + pgoff_t index, last, page_index; + unsigned int i; + + memset(dest, 0, sizeof(*dest)); + index = chunk_start >> PAGE_SHIFT; + last = (chunk_end - 1) >> PAGE_SHIFT; + while (index <= last) { + struct folio *folio; + pgoff_t next; + bool is_target; + + if (folio_contains(target, index)) { + folio = target; + is_target = true; + } else { + folio = __filemap_get_folio( + mapping, index, + FGP_LOCK | FGP_CREAT | FGP_NOFS | FGP_NOWAIT, + GFP_NOFS); + if (IS_ERR(folio)) + return PTR_ERR(folio); + is_target = false; + if (folio_pos(folio) < chunk_start || + folio_next_pos(folio) > chunk_end) { + folio_unlock(folio); + folio_put(folio); + return -EAGAIN; + } + } + + if (dest->nr_folios == ARRAY_SIZE(dest->folios)) { + if (!is_target) { + folio_unlock(folio); + folio_put(folio); + } + return -EINVAL; + } + dest->folios[dest->nr_folios++] = folio; + next = folio->index + folio_nr_pages(folio); + if (next <= index) + return -EAGAIN; + index = next; + } + + for (page_index = chunk_start >> PAGE_SHIFT; page_index <= last; + page_index++) { + struct folio *folio = NULL; + + for (i = 0; i < dest->nr_folios; i++) { + if (folio_contains(dest->folios[i], page_index)) { + folio = dest->folios[i]; + break; + } + } + if (!folio || dest->nr_pages == ARRAY_SIZE(dest->pages)) + return -EAGAIN; + dest->pages[dest->nr_pages++] = + folio_page(folio, page_index - folio->index); + } + return 0; +} + +static int ntfs_wof_decode(struct ntfs_wof_workspace *ws, const void *src, + u32 src_len, void *dst, u32 dst_len) +{ + if (src_len == dst_len) { + memcpy(dst, src, dst_len); + return 0; + } + return ws->codec->decompress_chunk(ws->scratch, src, src_len, dst, + dst_len, ws->comp_unit); +} + +static int ntfs_wof_decode_page_direct(struct ntfs_wof_workspace *ws, + struct folio *target, loff_t chunk_start, + const void *src, u32 src_len, + u32 dst_len) +{ + unsigned int page_offset = offset_in_page(chunk_start); + struct page *page; + pgoff_t page_index; + void *addr; + int err; + + page_index = chunk_start >> PAGE_SHIFT; + if (!folio_contains(target, page_index)) + return -EAGAIN; + + page = folio_page(target, page_index - target->index); + addr = kmap_local_page(page); + err = ntfs_wof_decode(ws, src, src_len, (u8 *)addr + page_offset, + dst_len); + kunmap_local(addr); + if (err) + return -EINVAL; + return 0; +} + +static int ntfs_wof_decode_folios_direct(struct ntfs_wof_workspace *ws, + struct address_space *mapping, + struct folio *target, + loff_t chunk_start, loff_t chunk_end, + const void *src, u32 src_len, + u32 dst_len) +{ + unsigned int page_offset = offset_in_page(chunk_start); + struct ntfs_wof_dest dest; + void *addr; + unsigned int nofs_flags; + int err; + + err = ntfs_wof_collect_dest(mapping, target, chunk_start, chunk_end, + &dest); + if (err) { + ntfs_wof_release_dest(&dest, target, false); + return -EAGAIN; + } + + nofs_flags = memalloc_nofs_save(); + addr = vmap(dest.pages, dest.nr_pages, VM_MAP, PAGE_KERNEL); + memalloc_nofs_restore(nofs_flags); + if (!addr) { + ntfs_wof_release_dest(&dest, target, false); + return -EAGAIN; + } + + err = ntfs_wof_decode(ws, src, src_len, (u8 *)addr + page_offset, + dst_len); + vunmap(addr); + if (err) { + ntfs_wof_release_dest(&dest, target, false); + return -EINVAL; + } + ntfs_wof_release_dest(&dest, target, true); + return 0; +} + +static int ntfs_wof_try_direct(struct ntfs_wof_workspace *ws, + struct address_space *mapping, + struct folio *target, loff_t chunk_start, + loff_t chunk_end, const void *src, u32 src_len, + u32 dst_len) +{ + unsigned int page_offset = offset_in_page(chunk_start); + + if (dst_len <= PAGE_SIZE - page_offset) + return ntfs_wof_decode_page_direct(ws, target, chunk_start, src, + src_len, dst_len); + + return ntfs_wof_decode_folios_direct(ws, mapping, target, chunk_start, + chunk_end, src, src_len, dst_len); +} + +int ntfs_read_wof_compressed_block(struct folio *folio) +{ + struct address_space *mapping = folio->mapping; + struct ntfs_inode *ni = NTFS_I(mapping->host), *wof_ni; + struct inode *wof_inode; + struct ntfs_volume *vol = ni->vol; + struct ntfs_wof_workspace *ws; + loff_t i_size = i_size_read(VFS_I(ni)); + loff_t folio_start = folio_pos(folio); + loff_t folio_end = folio_next_pos(folio); + char *chunk_mem; + u32 decomp_size; + u64 chunk_count, chunk_idx, last_chunk, chunk_offset; + int err = 0; + + ws = ntfs_wof_workspace(ni->itype.compressed.block_size_bits); + if (!ws) { + err = -EOPNOTSUPP; + goto out; + } + + if (folio_start >= i_size) { + folio_zero_segment(folio, 0, folio_size(folio)); + goto out; + } + + wof_inode = ntfs_attr_iget(VFS_I(ni), AT_DATA, (__le16 *)WOF_NAME, + WOF_NAME_LEN); + if (IS_ERR(wof_inode)) { + err = PTR_ERR(wof_inode); + goto out; + } + + wof_ni = NTFS_I(wof_inode); + if (wof_ni->initialized_size != wof_ni->data_size) { + ntfs_error(vol->sb, + "WOF compressed stream is not fully initialized (init %lld, data %lld).", + wof_ni->initialized_size, wof_ni->data_size); + err = -EIO; + goto out_iput; + } + if (NInoNonResident(wof_ni) && !NInoFullyMapped(wof_ni)) { + down_write(&wof_ni->runlist.lock); + if (!NInoFullyMapped(wof_ni)) + err = ntfs_attr_map_whole_runlist(wof_ni); + up_write(&wof_ni->runlist.lock); + if (err) + goto out_iput; + } + + mutex_lock(ws->lock); + err = ntfs_wof_workspace_prepare(ws); + if (err) + goto out_unlock_ws; + + chunk_idx = div_u64(folio_start, ws->comp_unit); + last_chunk = + div_u64(min_t(loff_t, folio_end, i_size) - 1, ws->comp_unit); + chunk_count = DIV_ROUND_UP_ULL(i_size, ws->comp_unit); + for (; chunk_idx <= last_chunk; chunk_idx++) { + u32 chunk_size; + u64 chunk_file_offset; + loff_t chunk_end, copy_start, copy_end; + + decomp_size = chunk_idx + 1 == chunk_count ? + i_size - chunk_idx * ws->comp_unit : + ws->comp_unit; + err = parse_wof_chunk_table(ni, wof_ni, chunk_idx, chunk_count, + decomp_size, &chunk_offset, + &chunk_size, ws->input, + ws->input_size); + if (err) + goto out_unlock_ws; + + err = ntfs_read_wof_chunk(vol, wof_ni, chunk_offset, chunk_size, + ws->input, ws->input_size, + &chunk_mem); + if (err) + goto out_unlock_ws; + + chunk_file_offset = chunk_idx * ws->comp_unit; + chunk_end = chunk_file_offset + decomp_size; + err = ntfs_wof_try_direct(ws, mapping, folio, chunk_file_offset, + chunk_end, chunk_mem, chunk_size, + decomp_size); + if (!err) + continue; + if (err != -EAGAIN) + goto out_unlock_ws; + + err = ntfs_wof_decode(ws, chunk_mem, chunk_size, ws->output, + decomp_size); + if (err) { + ntfs_error(vol->sb, "Decompression failed: %d", err); + err = -EINVAL; + goto out_unlock_ws; + } + copy_start = max_t(loff_t, folio_start, chunk_file_offset); + copy_end = min_t(loff_t, folio_end, + chunk_file_offset + decomp_size); + memcpy_to_folio(folio, copy_start - folio_start, + ws->output + copy_start - chunk_file_offset, + copy_end - copy_start); + } + + if (folio_end > i_size) + folio_zero_segment(folio, i_size - folio_start, + folio_size(folio)); +out_unlock_ws: + mutex_unlock(ws->lock); +out_iput: + iput(wof_inode); +out: + if (!err) { + flush_dcache_folio(folio); + folio_mark_uptodate(folio); + } else { + folio_clear_uptodate(folio); + } + folio_unlock(folio); + return err; +} diff --git a/fs/quota/dquot.c b/fs/quota/dquot.c index 204afc5e984b..1c78c695d0dd 100644 --- a/fs/quota/dquot.c +++ b/fs/quota/dquot.c @@ -1240,7 +1240,7 @@ static int ignore_hardlimit(struct dquot *dquot) { struct mem_dqinfo *info = &sb_dqopt(dquot->dq_sb)->info[dquot->dq_id.type]; - return capable(CAP_SYS_RESOURCE) && + return capable_noaudit(CAP_SYS_RESOURCE) && (info->dqi_format->qf_fmt_id != QFMT_VFS_OLD || !(info->dqi_flags & DQF_ROOT_SQUASH)); } diff --git a/fs/smb/client/dir.c b/fs/smb/client/dir.c index 7803bd5bd01f..4a2122713bd6 100644 --- a/fs/smb/client/dir.c +++ b/fs/smb/client/dir.c @@ -1138,6 +1138,8 @@ int cifs_tmpfile(struct mnt_idmap *idmap, struct inode *dir, } while (unlikely(rc == -EEXIST) && ++retries < max_retries); if (rc) { + if (rc == -ENOENT) + rc = -EOPNOTSUPP; cifs_del_pending_open(&open); goto out; } diff --git a/fs/smb/client/smb2pdu.h b/fs/smb/client/smb2pdu.h index b9bf2fa989d5..ab6c667bebc0 100644 --- a/fs/smb/client/smb2pdu.h +++ b/fs/smb/client/smb2pdu.h @@ -21,30 +21,6 @@ /* The total header size for SMB2 read and write */ #define SMB2_READWRITE_PDU_HEADER_SIZE (48 + sizeof(struct smb2_hdr)) -/* See MS-SMB2 2.2.43 */ -struct smb2_rdma_transform { - __le16 RdmaDescriptorOffset; - __le16 RdmaDescriptorLength; - __le32 Channel; /* for values see channel description in smb2 read above */ - __le16 TransformCount; - __le16 Reserved1; - __le32 Reserved2; -} __packed; - -/* TransformType */ -#define SMB2_RDMA_TRANSFORM_TYPE_ENCRYPTION 0x0001 -#define SMB2_RDMA_TRANSFORM_TYPE_SIGNING 0x0002 - -struct smb2_rdma_crypto_transform { - __le16 TransformType; - __le16 SignatureLength; - __le16 NonceLength; - __u16 Reserved; - __u8 Signature[]; /* variable length */ - /* u8 Nonce[] */ - /* followed by padding */ -} __packed; - /* * Definitions for SMB2 Protocol Data Units (network frames) * diff --git a/fs/smb/common/fscc.h b/fs/smb/common/fscc.h index 941db5a95564..e46d3379b779 100644 --- a/fs/smb/common/fscc.h +++ b/fs/smb/common/fscc.h @@ -202,6 +202,21 @@ struct file_zero_data_information { __le64 BeyondFinalZero; } __packed; +struct file_level_trim_range { + __le64 Offset; + __le64 Length; +} __packed; + +struct file_level_trim { + __le32 Key; + __le32 NumRanges; + struct file_level_trim_range Ranges[]; +} __packed; + +struct file_level_trim_output { + __le32 NumRangesProcessed; +} __packed; + /* * This level 18, although with struct with same name is different from cifs * level 0x107. Level 0x107 has an extra u64 between AccessFlags and diff --git a/fs/smb/common/smb2pdu.h b/fs/smb/common/smb2pdu.h index e7ff52b8aba5..f9a8862cb3d4 100644 --- a/fs/smb/common/smb2pdu.h +++ b/fs/smb/common/smb2pdu.h @@ -370,7 +370,8 @@ struct smb2_tree_connect_req { #define SMB2_SHAREFLAG_FORCE_LEVELII_OPLOCK 0x00001000 #define SMB2_SHAREFLAG_ENABLE_HASH_V1 0x00002000 #define SMB2_SHAREFLAG_ENABLE_HASH_V2 0x00004000 -#define SHI1005_FLAGS_ENCRYPT_DATA 0x00008000 +#define SMB2_SHAREFLAG_ENCRYPT_DATA 0x00008000 +#define SHI1005_FLAGS_ENCRYPT_DATA SMB2_SHAREFLAG_ENCRYPT_DATA #define SMB2_SHAREFLAG_IDENTITY_REMOTING 0x00040000 /* 3.1.1 */ #define SMB2_SHAREFLAG_COMPRESS_DATA 0x00100000 /* 3.1.1 */ #define SMB2_SHAREFLAG_ISOLATED_TRANSPORT 0x00200000 @@ -742,6 +743,28 @@ struct smb2_close_rsp { #define SMB2_CHANNEL_RDMA_V1_INVALIDATE cpu_to_le32(0x00000002) #define SMB2_CHANNEL_RDMA_TRANSFORM cpu_to_le32(0x00000003) +/* See MS-SMB2 2.2.43. */ +struct smb2_rdma_transform { + __le16 RdmaDescriptorOffset; + __le16 RdmaDescriptorLength; + __le32 Channel; + __le16 TransformCount; + __le16 Reserved1; + __le32 Reserved2; +} __packed; + +#define SMB2_RDMA_TRANSFORM_TYPE_ENCRYPTION 0x0001 +#define SMB2_RDMA_TRANSFORM_TYPE_SIGNING 0x0002 + +struct smb2_rdma_crypto_transform { + __le16 TransformType; + __le16 SignatureLength; + __le16 NonceLength; + __le16 Reserved; + __u8 Signature[]; + /* Followed by Nonce[] and optional alignment padding. */ +} __packed; + /* SMB2 read request without RFC1001 length at the beginning */ struct smb2_read_req { struct smb2_hdr hdr; @@ -847,8 +870,8 @@ struct smb2_lock_req { __le16 StructureSize; /* Must be 48 */ __le16 LockCount; /* - * The least significant four bits are the index, the other 28 bits are - * the lock sequence number (0 to 64). See MS-SMB2 2.2.26 + * The least significant four bits are the lock sequence number. The + * other 28 bits are the index (0 to 64). See MS-SMB2 2.2.26. */ __le32 LockSequenceNumber; __u64 PersistentFileId; @@ -1261,6 +1284,14 @@ struct create_mxac_req { #define SMB2_CRTCTX_AAPL_SUPPORTS_OSX_COPYFILE 2 #define SMB2_CRTCTX_AAPL_UNIX_BASED 4 #define SMB2_CRTCTX_AAPL_SUPPORTS_NFS_ACE 8 +/* + * V2 extends the same inline-FinderInfo mechanism as + * SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR with an added flags field, + * confirmed byte-identical to V1 otherwise against AAPL's actual + * public client behavior. Mutually exclusive with the V1 bit on + * the wire, not both set together. + */ +#define SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR_V2 16 /* "AAPL" Volume Capabilities bitmap */ #define SMB2_CRTCTX_AAPL_SUPPORT_RESOLVE_ID 1 @@ -1452,7 +1483,7 @@ struct resume_key_ioctl_rsp { __u64 ResumeKeyU64[3]; }; __le32 ContextLength; /* MBZ */ - char Context[]; /* ignored, Windows sets to 4 bytes of zero */ + char Context[4]; /* ignored, Windows sets to 4 bytes of zero */ } __packed; struct smb2_ioctl_rsp { diff --git a/fs/smb/common/smb2status.h b/fs/smb/common/smb2status.h index b6421bc5113c..2989c3a5cb67 100644 --- a/fs/smb/common/smb2status.h +++ b/fs/smb/common/smb2status.h @@ -1049,6 +1049,7 @@ struct ntstatus { #define STATUS_WOW_ASSERTION cpu_to_le32(0xC0009898) // -EIO #define STATUS_INVALID_SIGNATURE cpu_to_le32(0xC000A000) // -EIO #define STATUS_HMAC_NOT_SUPPORTED cpu_to_le32(0xC000A001) // -EIO +#define STATUS_AUTH_TAG_MISMATCH cpu_to_le32(0xC000A002) // -EBADMSG #define STATUS_IPSEC_QUEUE_OVERFLOW cpu_to_le32(0xC000A010) // -EIO #define STATUS_ND_QUEUE_OVERFLOW cpu_to_le32(0xC000A011) // -EIO #define STATUS_HOPLIMIT_EXCEEDED cpu_to_le32(0xC000A012) // -EIO diff --git a/fs/smb/common/smbfsctl.h b/fs/smb/common/smbfsctl.h index d1fcb46a7cde..b1123b0f768d 100644 --- a/fs/smb/common/smbfsctl.h +++ b/fs/smb/common/smbfsctl.h @@ -119,6 +119,7 @@ #define FSCTL_SRV_ENUMERATE_SNAPSHOTS 0x00144064 /* Retrieve an opaque file reference for server-side data movement ie copy */ #define FSCTL_SRV_REQUEST_RESUME_KEY 0x00140078 +#define FSCTL_SRV_ENUM_SNAPS 0x00144064 #define FSCTL_LMR_REQUEST_RESILIENCY 0x001401D4 #define FSCTL_LMR_GET_LINK_TRACK_INF 0x001400E8 /* BB add struct */ #define FSCTL_LMR_SET_LINK_TRACK_INF 0x001400EC /* BB add struct */ diff --git a/fs/smb/common/smbglob.h b/fs/smb/common/smbglob.h index 4e33d91cdc9d..d9c7e6e7af29 100644 --- a/fs/smb/common/smbglob.h +++ b/fs/smb/common/smbglob.h @@ -39,6 +39,7 @@ struct smb_version_values { size_t create_mxac_size; size_t create_disk_id_size; size_t create_posix_size; + size_t create_aapl_size; }; static inline unsigned int get_rfc1002_len(void *buf) diff --git a/fs/smb/server/Kconfig b/fs/smb/server/Kconfig index 08d8b7a965a6..221ec9717a83 100644 --- a/fs/smb/server/Kconfig +++ b/fs/smb/server/Kconfig @@ -9,7 +9,6 @@ config SMB_SERVER select CRYPTO select CRYPTO_LIB_AES_CBC_MACS select CRYPTO_LIB_ARC4 - select CRYPTO_LIB_DES select CRYPTO_LIB_MD5 select CRYPTO_LIB_SHA256 select CRYPTO_LIB_SHA512 diff --git a/fs/smb/server/auth.c b/fs/smb/server/auth.c index e8d1c068a43e..1b24eb3f3998 100644 --- a/fs/smb/server/auth.c +++ b/fs/smb/server/auth.c @@ -24,7 +24,6 @@ #include <linux/fips.h> #include <crypto/arc4.h> -#include <crypto/des.h> #include "server.h" #include "smb_common.h" @@ -122,6 +121,8 @@ static int calc_ntlmv2_hash(struct ksmbd_conn *conn, struct ksmbd_session *sess, out: kfree(uniname); kfree(domain); + if (ret) /* Done by hmac_md5_final() already if ret == 0 */ + memzero_explicit(&ctx, sizeof(ctx)); return ret; } @@ -439,6 +440,7 @@ int ksmbd_krb5_authenticate(struct ksmbd_session *sess, char *in_blob, resp_ext = ksmbd_ipc_login_request_ext(resp->login_response.account); user = ksmbd_alloc_user(&resp->login_response, resp_ext); + kvfree(resp_ext); if (!user) { ksmbd_debug(AUTH, "login failure\n"); retval = -ENOMEM; @@ -462,9 +464,11 @@ int ksmbd_krb5_authenticate(struct ksmbd_session *sess, char *in_blob, memcpy(out_blob, resp->payload + resp->session_key_len, resp->spnego_blob_len); *out_len = resp->spnego_blob_len; + sess->kerberos_expiry = resp->session_expiry; retval = 0; out: - kvfree(resp); + kvfree_sensitive(resp, sizeof(*resp) + resp->session_key_len + + resp->spnego_blob_len); return retval; } #else @@ -556,6 +560,7 @@ static void generate_key(struct ksmbd_conn *conn, const char *sess_key, hmac_sha256_final(&ctx, prfhash); memcpy(key, prfhash, key_size); + memzero_explicit(prfhash, sizeof(prfhash)); } static int generate_smb3signingkey(struct ksmbd_session *sess, @@ -716,8 +721,21 @@ static int ksmbd_get_encryption_key(struct ksmbd_work *work, __u64 ses_id, if (enc) sess = work->sess; - else - sess = ksmbd_session_lookup_all(work->conn, ses_id); + else { + /* + * A previous-session replacement leaves the old encryption key in + * place. Use it to authenticate an encrypted request, then let + * session validation reject the expired session. This preserves the + * encrypted STATUS_USER_SESSION_DELETED response without reviving + * the session. + */ + sess = ksmbd_session_lookup_all_states(work->conn, ses_id); + if (sess && sess->state != SMB2_SESSION_VALID && + (sess->state != SMB2_SESSION_EXPIRED || !sess->enc)) { + ksmbd_user_session_put(sess); + sess = NULL; + } + } if (!sess) return -EINVAL; @@ -814,6 +832,189 @@ static struct scatterlist *ksmbd_init_sg(struct kvec *iov, unsigned int nvec, return sg; } +/** + * ksmbd_init_rdma_sg() - build an AEAD scatterlist for an RDMA payload + * @buf: payload buffer + * @buflen: payload length + * @tag: authentication tag buffer + * @taglen: authentication tag length + * + * Split vmalloc-backed payloads at page boundaries and append the detached + * authentication tag as the final scatterlist entry. + * + * Return: allocated scatterlist, or NULL on allocation failure + */ +static struct scatterlist *ksmbd_init_rdma_sg(void *buf, + unsigned int buflen, + u8 *tag, + unsigned int taglen) +{ + struct scatterlist *sg; + unsigned int nr_data = 1, nr_entries, i = 0; + void *data = buf; + int len = buflen; + + if (is_vmalloc_addr(buf)) + nr_data = DIV_ROUND_UP(offset_in_page(buf) + buflen, PAGE_SIZE); + nr_entries = nr_data + 1; + + sg = kmalloc_objs(struct scatterlist, nr_entries, KSMBD_DEFAULT_GFP); + if (!sg) + return NULL; + + sg_init_table(sg, nr_entries); + if (!is_vmalloc_addr(buf)) { + smb2_sg_set_buf(&sg[i++], buf, buflen); + } else { + while (len) { + unsigned int bytes = min_t(unsigned int, + PAGE_SIZE - offset_in_page(data), len); + + sg_set_page(&sg[i++], vmalloc_to_page(data), bytes, + offset_in_page(data)); + data += bytes; + len -= bytes; + } + } + smb2_sg_set_buf(&sg[i], tag, taglen); + return sg; +} + +/** + * ksmbd_crypt_rdma() - encrypt or decrypt an SMB Direct data buffer + * @conn: connection containing the negotiated cipher + * @key: session encryption or decryption key + * @buf: RDMA payload, transformed in place + * @buflen: payload length (the authentication tag is carried out of band) + * @nonce: transform nonce + * @nonce_len: nonce length + * @tag: authentication tag output for encryption, input for decryption + * @tag_len: authentication tag length + * @enc: true to encrypt, false to decrypt + * + * SMB2_RDMA_CRYPTO_TRANSFORM carries the nonce and authentication tag in the + * SMB2 message while only the payload is transferred through RDMA. Therefore + * this uses AEAD without the normal SMB3 transform header as associated data. + * + * Return: 0 on success, otherwise a negative errno + */ +int ksmbd_crypt_rdma(struct ksmbd_conn *conn, const u8 *key, + void *buf, unsigned int buflen, const u8 *nonce, + unsigned int nonce_len, u8 *tag, unsigned int tag_len, + bool enc) +{ + struct ksmbd_crypto_ctx *ctx; + struct crypto_aead *tfm; + struct aead_request *req = NULL; + struct scatterlist *sg = NULL; + unsigned int iv_len, crypt_len; + u8 auth_tag[SMB2_SIGNATURE_SIZE] = {}; + u8 *iv = NULL; + u16 cipher = le16_to_cpu(conn->cipher_type); + int rc; + DECLARE_CRYPTO_WAIT(wait); + + if (!buflen || !tag_len || tag_len > SMB2_SIGNATURE_SIZE) { + pr_err("RDMA %s rejected: cipher=0x%04x payload=%u nonce=%u tag=%u\n", + enc ? "encryption" : "decryption", cipher, buflen, + nonce_len, tag_len); + return -EINVAL; + } + if (!enc) + memcpy(auth_tag, tag, tag_len); + + if (conn->cipher_type == SMB2_ENCRYPTION_AES128_GCM || + conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM) { + if (nonce_len != SMB3_AES_GCM_NONCE) { + pr_err("RDMA %s rejected: cipher=0x%04x invalid nonce=%u expected=%u\n", + enc ? "encryption" : "decryption", cipher, + nonce_len, SMB3_AES_GCM_NONCE); + return -EINVAL; + } + ctx = ksmbd_crypto_ctx_find_gcm(); + } else { + if (nonce_len != SMB3_AES_CCM_NONCE) { + pr_err("RDMA %s rejected: cipher=0x%04x invalid nonce=%u expected=%u\n", + enc ? "encryption" : "decryption", cipher, + nonce_len, SMB3_AES_CCM_NONCE); + return -EINVAL; + } + ctx = ksmbd_crypto_ctx_find_ccm(); + } + if (!ctx) { + pr_err("RDMA %s failed: cipher=0x%04x crypto context unavailable\n", + enc ? "encryption" : "decryption", cipher); + return -ENOMEM; + } + + tfm = (conn->cipher_type == SMB2_ENCRYPTION_AES128_GCM || + conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM) ? + CRYPTO_GCM(ctx) : CRYPTO_CCM(ctx); + if (conn->cipher_type == SMB2_ENCRYPTION_AES256_CCM || + conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM) + rc = crypto_aead_setkey(tfm, key, SMB3_GCM256_CRYPTKEY_SIZE); + else + rc = crypto_aead_setkey(tfm, key, SMB3_GCM128_CRYPTKEY_SIZE); + if (rc) + goto out; + + rc = crypto_aead_setauthsize(tfm, tag_len); + if (rc) + goto out; + + req = aead_request_alloc(tfm, KSMBD_DEFAULT_GFP); + if (!req) { + rc = -ENOMEM; + goto out; + } + + sg = ksmbd_init_rdma_sg(buf, buflen, auth_tag, tag_len); + if (!sg) { + rc = -ENOMEM; + goto out; + } + + iv_len = crypto_aead_ivsize(tfm); + iv = kzalloc(iv_len, KSMBD_DEFAULT_GFP); + if (!iv) { + rc = -ENOMEM; + goto out; + } + if (conn->cipher_type == SMB2_ENCRYPTION_AES128_GCM || + conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM) { + memcpy(iv, nonce, nonce_len); + } else { + iv[0] = 3; + memcpy(iv + 1, nonce, nonce_len); + } + + crypt_len = buflen + (enc ? 0 : tag_len); + aead_request_set_crypt(req, sg, sg, crypt_len, iv); + aead_request_set_ad(req, 0); + aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG | + CRYPTO_TFM_REQ_MAY_SLEEP, + crypto_req_done, &wait); + rc = crypto_wait_req(enc ? crypto_aead_encrypt(req) : + crypto_aead_decrypt(req), &wait); + if (!rc && enc) + memcpy(tag, auth_tag, tag_len); +out: + kfree(iv); + kfree(sg); + aead_request_free(req); + ksmbd_release_crypto_ctx(ctx); + if (rc) + pr_err("RDMA %s failed: cipher=0x%04x payload=%u nonce=%u tag=%u rc=%d\n", + enc ? "encryption" : "decryption", cipher, buflen, + nonce_len, tag_len, rc); + else + ksmbd_debug(RDMA, + "RDMA %s completed: cipher=0x%04x payload=%u nonce=%u tag=%u\n", + enc ? "encryption" : "decryption", cipher, buflen, + nonce_len, tag_len); + return rc; +} + int ksmbd_crypt_message(struct ksmbd_work *work, struct kvec *iov, unsigned int nvec, int enc) { @@ -848,7 +1049,8 @@ int ksmbd_crypt_message(struct ksmbd_work *work, struct kvec *iov, ctx = ksmbd_crypto_ctx_find_ccm(); if (!ctx) { pr_err("crypto alloc failed\n"); - return -ENOMEM; + rc = -ENOMEM; + goto zeroize_key; } if (conn->cipher_type == SMB2_ENCRYPTION_AES128_GCM || @@ -928,5 +1130,8 @@ free_req: aead_request_free(req); free_ctx: ksmbd_release_crypto_ctx(ctx); +zeroize_key: + memzero_explicit(key, sizeof(key)); + memzero_explicit(sign, sizeof(sign)); return rc; } diff --git a/fs/smb/server/auth.h b/fs/smb/server/auth.h index f14b7c033264..7ce9c42d58f1 100644 --- a/fs/smb/server/auth.h +++ b/fs/smb/server/auth.h @@ -38,6 +38,10 @@ struct kvec; int ksmbd_crypt_message(struct ksmbd_work *work, struct kvec *iov, unsigned int nvec, int enc); +int ksmbd_crypt_rdma(struct ksmbd_conn *conn, const u8 *key, + void *buf, unsigned int buflen, const u8 *nonce, + unsigned int nonce_len, u8 *tag, unsigned int tag_len, + bool enc); void ksmbd_copy_gss_neg_header(void *buf); int ksmbd_auth_ntlmv2(struct ksmbd_conn *conn, struct ksmbd_session *sess, struct ntlmv2_resp *ntlmv2, int blen, char *domain_name, diff --git a/fs/smb/server/compress.c b/fs/smb/server/compress.c index 01d1771ff663..5162fb84c755 100644 --- a/fs/smb/server/compress.c +++ b/fs/smb/server/compress.c @@ -14,24 +14,14 @@ #define SMB_COMPRESS_MIN_LEN PAGE_SIZE -/** - * ksmbd_decompress_request() - replace a compressed request with its SMB2 PDU - * @conn: connection which owns the current RFC1002 request buffer - * - * Derive the uncompressed size from the transform variant, enforce ksmbd's - * normal message limits, and ask the common decoder to validate every payload. - * On success, replace conn->request_buf with a regular RFC1002-framed SMB2 - * message so the rest of the request path needs no compression awareness. - * - * Return: 0 on success, otherwise a negative errno. - */ -int ksmbd_decompress_request(struct ksmbd_conn *conn) +static int __ksmbd_decompress_request(struct ksmbd_conn *conn, + void *request_buf, void **out_buf) { struct smb2_compression_hdr *hdr; - unsigned int pdu_size = get_rfc1002_len(conn->request_buf); + unsigned int pdu_size = get_rfc1002_len(request_buf); u32 orig_size, offset, out_size; u32 max_allowed_pdu_size; - char *buf, *out; + char *out; int rc; if (pdu_size < sizeof(struct smb2_compression_hdr)) @@ -41,7 +31,7 @@ int ksmbd_decompress_request(struct ksmbd_conn *conn) conn->compress_algorithm == SMB3_COMPRESS_NONE) return -EINVAL; - hdr = smb_get_msg(conn->request_buf); + hdr = smb_get_msg(request_buf); if (hdr->ProtocolId != SMB2_COMPRESSION_TRANSFORM_ID) return -EINVAL; @@ -74,19 +64,69 @@ int ksmbd_decompress_request(struct ksmbd_conn *conn) if (!out) return -ENOMEM; - buf = (char *)hdr; *(__be32 *)out = cpu_to_be32(out_size); rc = smb_compression_decompress(conn->compress_algorithm, conn->compress_chained, conn->compress_pattern, - buf, pdu_size, out + 4, out_size); + (char *)hdr, pdu_size, out + 4, out_size); if (rc) { kvfree(out); return rc; } + *out_buf = out; + return 0; +} + +/** + * ksmbd_decompress_request() - replace a compressed request with its SMB2 PDU + * @conn: connection which owns the current RFC1002 request buffer + * + * Derive the uncompressed size from the transform variant, enforce ksmbd's + * normal message limits, and ask the common decoder to validate every payload. + * On success, replace conn->request_buf with a regular RFC1002-framed SMB2 + * message so the rest of the request path needs no compression awareness. + * + * Return: 0 on success, otherwise a negative errno. + */ +int ksmbd_decompress_request(struct ksmbd_conn *conn) +{ + void *out_buf; + int rc; + + rc = __ksmbd_decompress_request(conn, conn->request_buf, &out_buf); + if (rc) + return rc; + kvfree(conn->request_buf); - conn->request_buf = out; + conn->request_buf = out_buf; + return 0; +} + +/** + * ksmbd_decompress_work_request() - decompress an encrypted work request + * @work: work item whose request buffer contains a compression transform + * + * SMB3 encrypts a compressed message by applying compression first and + * encryption second. The receive loop can therefore only decode the + * compression transform before work allocation for an unencrypted request; + * an encrypted request must be decompressed after its encryption layer has + * been removed. + * + * Return: 0 on success, otherwise a negative errno. + */ +int ksmbd_decompress_work_request(struct ksmbd_work *work) +{ + void *out_buf; + int rc; + + rc = __ksmbd_decompress_request(work->conn, work->request_buf, + &out_buf); + if (rc) + return rc; + + kvfree(work->request_buf); + work->request_buf = out_buf; return 0; } diff --git a/fs/smb/server/compress.h b/fs/smb/server/compress.h index 663c6f44f09b..13df2eb221e8 100644 --- a/fs/smb/server/compress.h +++ b/fs/smb/server/compress.h @@ -11,6 +11,7 @@ #include "../common/compress/compress.h" int ksmbd_decompress_request(struct ksmbd_conn *conn); +int ksmbd_decompress_work_request(struct ksmbd_work *work); int ksmbd_compress_response(struct ksmbd_work *work); #endif /* __KSMBD_COMPRESS_H__ */ diff --git a/fs/smb/server/connection.c b/fs/smb/server/connection.c index ef6f202f4024..91fdd1ddc61f 100644 --- a/fs/smb/server/connection.c +++ b/fs/smb/server/connection.c @@ -11,6 +11,7 @@ #include "server.h" #include "smb_common.h" #include "mgmt/ksmbd_ida.h" +#include "mgmt/user_session.h" #include "connection.h" #include "compress.h" #include "transport_tcp.h" @@ -27,33 +28,107 @@ DECLARE_RWSEM(conn_list_lock); #ifdef CONFIG_PROC_FS static struct proc_dir_entry *proc_clients; +static const char *ksmbd_conn_state_string(struct ksmbd_conn *conn) +{ + switch (READ_ONCE(conn->status)) { + case KSMBD_SESS_NEW: + return "new"; + case KSMBD_SESS_GOOD: + return "good"; + case KSMBD_SESS_EXITING: + return "exiting"; + case KSMBD_SESS_NEED_RECONNECT: + return "reconnect"; + case KSMBD_SESS_NEED_NEGOTIATE: + return "negotiate"; + case KSMBD_SESS_NEED_SETUP: + return "setup"; + case KSMBD_SESS_RELEASING: + return "releasing"; + default: + return "unknown"; + } +} + +static const char *ksmbd_conn_transport_string(struct ksmbd_conn *conn) +{ + if (conn->transport->ops->rdma_read || conn->transport->ops->rdma_write) + return "smbdirect"; + return "tcp"; +} + +static void proc_show_conn_feature(struct seq_file *m, bool *separator, + bool enabled, const char *name) +{ + if (!enabled) + return; + seq_printf(m, "%s%s", *separator ? "," : "", name); + *separator = true; +} + +static void proc_show_conn_features(struct seq_file *m, + struct ksmbd_conn *conn) +{ + bool separator = false; + + proc_show_conn_feature(m, &separator, + conn->sign || conn->signing_negotiated, "sign"); + proc_show_conn_feature(m, &separator, conn->cipher_type, "encrypt"); + proc_show_conn_feature(m, &separator, + conn->compress_algorithm != SMB3_COMPRESS_NONE, + "compress"); + proc_show_conn_feature(m, &separator, conn->rdma_transform_ids, + "rdma-transform"); + proc_show_conn_feature(m, &separator, conn->posix_ext_supported, "posix"); + if (!separator) + seq_puts(m, "none"); +} + static int proc_show_clients(struct seq_file *m, void *v) { struct ksmbd_conn *conn; struct timespec64 now, t; int i; - seq_printf(m, "#%-20s %-10s %-10s %-10s %-10s %-10s\n", - "<name>", "<dialect>", "<credits>", "<open files>", - "<requests>", "<last active>"); - down_read(&conn_list_lock); hash_for_each(conn_list, i, conn, hlist) { + unsigned int outstanding_credits, total_credits; + unsigned long id; + void *entry; + unsigned int sessions = 0; + jiffies_to_timespec64(jiffies - conn->last_active, &t); ktime_get_real_ts64(&now); t = timespec64_sub(now, t); + + spin_lock(&conn->credits_lock); + outstanding_credits = conn->outstanding_credits; + total_credits = conn->total_credits; + spin_unlock(&conn->credits_lock); + + rcu_read_lock(); + xa_for_each(&conn->sessions, id, entry) + sessions++; + rcu_read_unlock(); #if IS_ENABLED(CONFIG_IPV6) if (!conn->inet_addr) - seq_printf(m, "%-20pI6c", &conn->inet6_addr); + seq_printf(m, "client:\t%pI6c\n", &conn->inet6_addr); else #endif - seq_printf(m, "%-20pI4", &conn->inet_addr); - seq_printf(m, " 0x%-10x %-10u %-12d %-10d %ptT\n", - conn->dialect, - conn->total_credits, - atomic_read(&conn->stats.open_files_count), - atomic_read(&conn->req_running), - &t); + seq_printf(m, "client:\t%pI4\n", &conn->inet_addr); + seq_printf(m, "transport:\t%s\n", ksmbd_conn_transport_string(conn)); + seq_printf(m, "state:\t%s\n", ksmbd_conn_state_string(conn)); + seq_printf(m, "dialect:\t0x%04x\n", conn->dialect); + seq_printf(m, "credits:\t%u/%u\n", outstanding_credits, + total_credits); + seq_printf(m, "sessions:\t%u\n", sessions); + seq_printf(m, "open_files:\t%d\n", + atomic_read(&conn->stats.open_files_count)); + seq_printf(m, "requests:\t%lld\n", + atomic64_read(&conn->stats.request_served)); + seq_puts(m, "features:\t"); + proc_show_conn_features(m, conn); + seq_printf(m, "\nlast_active:\t%ptT\n\n", &t); } up_read(&conn_list_lock); return 0; @@ -117,7 +192,7 @@ static void __ksmbd_conn_release_work(struct work_struct *work) ida_destroy(&conn->async_ida); conn->transport->ops->free_transport(conn->transport); - kfree(conn); + kfree_sensitive(conn); } /** @@ -183,8 +258,9 @@ void ksmbd_conn_free(struct ksmbd_conn *conn) */ xa_destroy(&conn->sessions); kvfree(conn->request_buf); - kfree(conn->preauth_info); + kfree_sensitive(conn->preauth_info); kfree(conn->mechToken); + ksmbd_preauth_session_destroy(conn); ksmbd_conn_put(conn); } @@ -219,10 +295,19 @@ struct ksmbd_conn *ksmbd_conn_alloc(void) conn->total_credits = 1; conn->outstanding_credits = 0; + /* + * The command sequence window starts as the set { 0 } when the + * connection is established. + */ + conn->seq_low = 0; + conn->seq_high = 1; + __set_bit(0, conn->seq_bitmap); + init_waitqueue_head(&conn->req_running_q); init_waitqueue_head(&conn->r_count_q); INIT_LIST_HEAD(&conn->requests); INIT_LIST_HEAD(&conn->async_requests); + INIT_LIST_HEAD(&conn->preauth_sess_table); spin_lock_init(&conn->request_lock); spin_lock_init(&conn->credits_lock); ida_init(&conn->async_ida); @@ -291,6 +376,26 @@ void ksmbd_conn_try_dequeue_request(struct ksmbd_work *work) wake_up_all(&conn->req_running_q); } +static void ksmbd_conn_cancel_async_requests(struct ksmbd_conn *conn) +{ + struct ksmbd_work *work, *tmp; + + ksmbd_debug(CONN, "Cancel pending async requests on releasing connection\n"); + spin_lock(&conn->request_lock); + list_for_each_entry_safe(work, tmp, &conn->async_requests, + async_request_entry) { + if (work->state != KSMBD_WORK_ACTIVE) + continue; + + ksmbd_debug(CONN, "Cancel async request id %d\n", + work->async_id); + work->state = KSMBD_WORK_CANCELLED; + if (work->cancel_fn) + work->cancel_fn(work->cancel_argv); + } + spin_unlock(&conn->request_lock); +} + void ksmbd_conn_lock(struct ksmbd_conn *conn) { mutex_lock(&conn->srv_mutex); @@ -301,25 +406,64 @@ void ksmbd_conn_unlock(struct ksmbd_conn *conn) mutex_unlock(&conn->srv_mutex); } -void ksmbd_all_conn_set_status(u64 sess_id, u32 status) +static bool ksmbd_session_is_bound_to_conn(struct ksmbd_session *sess, + struct ksmbd_conn *conn) +{ + bool found; + + rcu_read_lock(); + found = xa_load(&conn->sessions, sess->id) == sess; + rcu_read_unlock(); + if (found) + return true; + + down_read(&sess->chann_lock); + found = xa_load(&sess->ksmbd_chann_list, (long)conn); + up_read(&sess->chann_lock); + return found; +} + +void ksmbd_all_conn_set_status(struct ksmbd_session *sess, u32 status) { struct ksmbd_conn *conn; int bkt; down_read(&conn_list_lock); hash_for_each(conn_list, bkt, conn, hlist) { - if (conn->binding || xa_load(&conn->sessions, sess_id)) - WRITE_ONCE(conn->status, status); + if (ksmbd_session_is_bound_to_conn(sess, conn)) { + spin_lock(&conn->request_lock); + if (!ksmbd_conn_exiting(conn) && + !ksmbd_conn_releasing(conn)) + WRITE_ONCE(conn->status, status); + spin_unlock(&conn->request_lock); + } } up_read(&conn_list_lock); } +void ksmbd_conn_abort(struct ksmbd_conn *conn) +{ + bool shutdown = false; + + spin_lock(&conn->request_lock); + if (!ksmbd_conn_exiting(conn) && !ksmbd_conn_releasing(conn)) { + ksmbd_conn_set_exiting(conn); + shutdown = true; + } + spin_unlock(&conn->request_lock); + wake_up_all(&conn->req_running_q); + + if (shutdown && conn->transport->ops->shutdown) + conn->transport->ops->shutdown(conn->transport); +} + void ksmbd_conn_wait_idle(struct ksmbd_conn *conn) { wait_event(conn->req_running_q, atomic_read(&conn->req_running) < 2); } -int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id) +int ksmbd_conn_wait_idle_sess(struct ksmbd_conn *curr_conn, + struct ksmbd_session *sess) { struct ksmbd_conn *conn; int rc, retry_count = 0, max_timeout = 120; @@ -331,7 +475,7 @@ retry_idle: down_read(&conn_list_lock); hash_for_each(conn_list, bkt, conn, hlist) { - if (conn->binding || xa_load(&conn->sessions, sess_id)) { + if (ksmbd_session_is_bound_to_conn(sess, conn)) { rcount = (conn == curr_conn) ? 2 : 1; if (atomic_read(&conn->req_running) >= rcount) { rc = wait_event_timeout(conn->req_running_q, @@ -350,7 +494,8 @@ retry_idle: return 0; } -int ksmbd_conn_write(struct ksmbd_work *work) +static int __ksmbd_conn_write(struct ksmbd_work *work, + struct ksmbd_transport_write *tx) { struct ksmbd_conn *conn = work->conn; int sent; @@ -366,12 +511,14 @@ int ksmbd_conn_write(struct ksmbd_work *work) if (!work->iov_idx) return -EINVAL; + tx->iov = work->iov; + tx->iov_cnt = work->iov_cnt; + tx->size = get_rfc1002_len(work->iov[0].iov_base) + 4; + tx->need_invalidate_rkey = work->need_invalidate_rkey; + tx->remote_key = work->remote_key; + ksmbd_conn_lock(conn); - sent = conn->transport->ops->writev(conn->transport, work->iov, - work->iov_cnt, - get_rfc1002_len(work->iov[0].iov_base) + 4, - work->need_invalidate_rkey, - work->remote_key); + sent = conn->transport->ops->writev(conn->transport, tx); ksmbd_conn_unlock(conn); if (sent < 0) { @@ -382,6 +529,22 @@ int ksmbd_conn_write(struct ksmbd_work *work) return 0; } +int ksmbd_conn_write(struct ksmbd_work *work) +{ + struct ksmbd_transport_write tx = {}; + + return __ksmbd_conn_write(work, &tx); +} + +int ksmbd_conn_write_eor(struct ksmbd_work *work) +{ + struct ksmbd_transport_write tx = { + .msg_flags = MSG_EOR, + }; + + return __ksmbd_conn_write(work, &tx); +} + int ksmbd_conn_rdma_read(struct ksmbd_conn *conn, void *buf, unsigned int buflen, struct smbdirect_buffer_descriptor_v1 *desc, @@ -566,6 +729,7 @@ recheck: } ksmbd_conn_set_releasing(conn); + ksmbd_conn_cancel_async_requests(conn); /* Wait till all reference dropped to the Server object*/ ksmbd_debug(CONN, "Wait for all pending requests(%d)\n", atomic_read(&conn->r_count)); wait_event(conn->r_count_q, atomic_read(&conn->r_count) == 0); @@ -623,7 +787,8 @@ int ksmbd_conn_transport_init(void) } out: mutex_unlock(&init_lock); - create_proc_clients(); + if (create_proc_clients()) + pr_warn("Unable to create clients procfs entry\n"); return ret; } @@ -657,8 +822,10 @@ again: * handler exited its receive loop for an unrelated * reason). */ - if (READ_ONCE(conn->status) != KSMBD_SESS_RELEASING) + spin_lock(&conn->request_lock); + if (!ksmbd_conn_releasing(conn)) ksmbd_conn_set_exiting(conn); + spin_unlock(&conn->request_lock); target = conn; break; } @@ -671,7 +838,7 @@ again: if (atomic_dec_and_test(&target->refcnt)) { ida_destroy(&target->async_ida); t->ops->free_transport(t); - kfree(target); + kfree_sensitive(target); } goto again; } diff --git a/fs/smb/server/connection.h b/fs/smb/server/connection.h index 0e4ebfac5558..63484c8efbbd 100644 --- a/fs/smb/server/connection.h +++ b/fs/smb/server/connection.h @@ -17,14 +17,25 @@ #include <linux/nls.h> #include <linux/unicode.h> #include <linux/workqueue.h> +#include <linux/bitmap.h> #include "smb_common.h" #include "ksmbd_work.h" struct smbdirect_buffer_descriptor_v1; +struct ksmbd_session; #define KSMBD_SOCKET_BACKLOG 16 +/* + * Size of the per-connection SMB2 command sequence window. This mirrors + * SMB2_MAX_CREDITS, the maximum number of credits (and therefore the + * maximum number of outstanding sequence numbers) that can be granted on + * a connection. It must be a power of two so the window can be indexed as + * a ring. + */ +#define KSMBD_CMD_SEQ_WINDOW 8192 + enum { KSMBD_SESS_NEW = 0, KSMBD_SESS_GOOD, @@ -74,6 +85,16 @@ struct ksmbd_conn { unsigned int total_credits; unsigned int outstanding_credits; spinlock_t credits_lock; + /* + * Connection command sequence window. [seq_low, seq_high) is the + * range of granted sequence numbers (message IDs). seq_bitmap marks + * the ones in that range that have been granted but + * not yet consumed by a received request. All three are protected by + * credits_lock. + */ + u64 seq_low; + u64 seq_high; + DECLARE_BITMAP(seq_bitmap, KSMBD_CMD_SEQ_WINDOW); wait_queue_head_t req_running_q; wait_queue_head_t r_count_q; /* Lock to protect requests list*/ @@ -118,12 +139,17 @@ struct ksmbd_conn { /* Negotiated SMB 3.1.1 compression capabilities. */ bool compress_chained; bool compress_pattern; + /* Bitmap indexed by SMB2_RDMA_TRANSFORM_* IDs. */ + unsigned long rdma_transform_ids; + bool rdma_transform_negotiated; bool posix_ext_supported; bool signing_negotiated; __le16 signing_algorithm; bool binding; atomic_t refcnt; bool is_aapl; + bool aapl_readdir_attr; /* READDIR_ATTR negotiated */ + bool aapl_readdir_attr_v2; /* V2 specifically */ struct work_struct release_work; }; @@ -132,14 +158,22 @@ struct ksmbd_conn_ops { int (*terminate_fn)(struct ksmbd_conn *conn); }; +struct ksmbd_transport_write { + struct kvec *iov; + int iov_cnt; + int size; + bool need_invalidate_rkey; + unsigned int remote_key; + int msg_flags; +}; + struct ksmbd_transport_ops { void (*disconnect)(struct ksmbd_transport *t); void (*shutdown)(struct ksmbd_transport *t); int (*read)(struct ksmbd_transport *t, char *buf, unsigned int size, int max_retries); - int (*writev)(struct ksmbd_transport *t, struct kvec *iovs, int niov, - int size, bool need_invalidate_rkey, - unsigned int remote_key); + int (*writev)(struct ksmbd_transport *t, + const struct ksmbd_transport_write *tx); int (*rdma_read)(struct ksmbd_transport *t, void *buf, unsigned int len, struct smbdirect_buffer_descriptor_v1 *desc, @@ -166,15 +200,18 @@ extern struct rw_semaphore conn_list_lock; bool ksmbd_conn_alive(struct ksmbd_conn *conn); void ksmbd_conn_wait_idle(struct ksmbd_conn *conn); -int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id); +int ksmbd_conn_wait_idle_sess(struct ksmbd_conn *curr_conn, + struct ksmbd_session *sess); struct ksmbd_conn *ksmbd_conn_alloc(void); void ksmbd_conn_free(struct ksmbd_conn *conn); struct ksmbd_conn *ksmbd_conn_get(struct ksmbd_conn *conn); void ksmbd_conn_put(struct ksmbd_conn *conn); +void ksmbd_conn_abort(struct ksmbd_conn *conn); int ksmbd_conn_wq_init(void); void ksmbd_conn_wq_destroy(void); bool ksmbd_conn_lookup_dialect(struct ksmbd_conn *c); int ksmbd_conn_write(struct ksmbd_work *work); +int ksmbd_conn_write_eor(struct ksmbd_work *work); int ksmbd_conn_rdma_read(struct ksmbd_conn *conn, void *buf, unsigned int buflen, struct smbdirect_buffer_descriptor_v1 *desc, @@ -279,5 +316,5 @@ static inline void ksmbd_conn_set_releasing(struct ksmbd_conn *conn) WRITE_ONCE(conn->status, KSMBD_SESS_RELEASING); } -void ksmbd_all_conn_set_status(u64 sess_id, u32 status); +void ksmbd_all_conn_set_status(struct ksmbd_session *sess, u32 status); #endif /* __CONNECTION_H__ */ diff --git a/fs/smb/server/ksmbd_netlink.h b/fs/smb/server/ksmbd_netlink.h index 8ccd57fd904b..2673522c76bc 100644 --- a/fs/smb/server/ksmbd_netlink.h +++ b/fs/smb/server/ksmbd_netlink.h @@ -113,7 +113,8 @@ struct ksmbd_startup_request { __u32 max_connections; /* Number of maximum simultaneous connections */ __s8 bind_interfaces_only; __u32 max_ip_connections; /* Number of maximum connection per ip address */ - __s8 reserved[499]; /* Reserved room */ + __s8 aapl_model[32]; /* AAPL model string for Finder icon, e.g. "Xserve" */ + __s8 reserved[467]; /* Reserved room */ __u32 ifc_list_sz; /* interfaces list size */ __s8 ____payload[]; } __packed; @@ -285,6 +286,7 @@ struct ksmbd_spnego_authen_response { * stored in SecurityBuffer of SMB2 SESSION * SETUP response */ + __u64 session_expiry; /* Kerberos ticket expiry time */ __u8 payload[]; /* session key + AP_REP */ }; @@ -377,6 +379,10 @@ enum KSMBD_TREE_CONN_STATUS { #define KSMBD_SHARE_FLAG_UPDATE BIT(14) #define KSMBD_SHARE_FLAG_CROSSMNT BIT(15) #define KSMBD_SHARE_FLAG_CONTINUOUS_AVAILABILITY BIT(16) +#define KSMBD_SHARE_FLAG_HIDE_UNREADABLE BIT(17) +#define KSMBD_SHARE_FLAG_TIME_MACHINE BIT(18) +/* Keep BIT(19) reserved for the existing ksmbd-tools WIDE_LINKS flag. */ +#define KSMBD_SHARE_FLAG_ENCRYPT_DATA BIT(20) /* * Tree connect request flags. diff --git a/fs/smb/server/ksmbd_work.c b/fs/smb/server/ksmbd_work.c index e2c2f45264be..f35335307670 100644 --- a/fs/smb/server/ksmbd_work.c +++ b/fs/smb/server/ksmbd_work.c @@ -11,6 +11,7 @@ #include "server.h" #include "connection.h" #include "ksmbd_work.h" +#include "vfs_cache.h" #include "mgmt/ksmbd_ida.h" static struct kmem_cache *work_cache; @@ -56,6 +57,7 @@ struct ksmbd_work *ksmbd_alloc_work_struct(void) INIT_LIST_HEAD(&work->request_entry); INIT_LIST_HEAD(&work->async_request_entry); INIT_LIST_HEAD(&work->fp_entry); + INIT_LIST_HEAD(&work->notify_entry); INIT_LIST_HEAD(&work->aux_read_list); work->iov_alloc_cnt = ARRAY_SIZE(work->iov_inline); work->iov = work->iov_inline; @@ -85,6 +87,9 @@ void ksmbd_free_work_struct(struct ksmbd_work *work) if (work->async_id) ksmbd_release_id(&work->conn->async_ida, work->async_id); + if (work->owns_conn_ref) + ksmbd_conn_put(work->conn); + ksmbd_fd_put(work, work->request_open); kmem_cache_free(work_cache, work); } diff --git a/fs/smb/server/ksmbd_work.h b/fs/smb/server/ksmbd_work.h index 88104f0cf363..5f1d3ebab4fb 100644 --- a/fs/smb/server/ksmbd_work.h +++ b/fs/smb/server/ksmbd_work.h @@ -12,6 +12,7 @@ struct ksmbd_conn; struct ksmbd_session; struct ksmbd_tree_connect; +struct ksmbd_file; #define KSMBD_WORK_INLINE_IOVS 4 @@ -90,7 +91,11 @@ struct ksmbd_work { bool compress_response:1; /* Is this SYNC or ASYNC ksmbd_work */ bool asynchronous:1; + /* Work owns a reference to @conn. */ + bool owns_conn_ref:1; bool need_invalidate_rkey:1; + bool request_open_chseq_tracked:1; + bool session_setup_reauth:1; unsigned int remote_key; /* cancel works */ @@ -98,12 +103,21 @@ struct ksmbd_work { void **cancel_argv; void (*cancel_fn)(void **argv); + /* + * Refcounted open associated with the SMB2 command currently being + * processed. + */ + struct ksmbd_file *request_open; + __le16 request_open_chseq; + struct work_struct work; /* List head at conn->requests */ struct list_head request_entry; /* List head at conn->async_requests */ struct list_head async_request_entry; struct list_head fp_entry; + /* List head at ksmbd_file->notify_pendings */ + struct list_head notify_entry; }; /** diff --git a/fs/smb/server/mgmt/share_config.c b/fs/smb/server/mgmt/share_config.c index e00aee155935..b2d9580bddc6 100644 --- a/fs/smb/server/mgmt/share_config.c +++ b/fs/smb/server/mgmt/share_config.c @@ -29,6 +29,63 @@ struct ksmbd_veto_pattern { struct list_head list; }; +#ifdef CONFIG_PROC_FS +static const struct ksmbd_const_name ksmbd_share_flag_names[] = { + {KSMBD_SHARE_FLAG_AVAILABLE, "available"}, + {KSMBD_SHARE_FLAG_BROWSEABLE, "browseable"}, + {KSMBD_SHARE_FLAG_WRITEABLE, "writeable"}, + {KSMBD_SHARE_FLAG_READONLY, "read-only"}, + {KSMBD_SHARE_FLAG_GUEST_OK, "guest-ok"}, + {KSMBD_SHARE_FLAG_GUEST_ONLY, "guest-only"}, + {KSMBD_SHARE_FLAG_STORE_DOS_ATTRS, "store-dos-attrs"}, + {KSMBD_SHARE_FLAG_OPLOCKS, "oplocks"}, + {KSMBD_SHARE_FLAG_PIPE, "pipe"}, + {KSMBD_SHARE_FLAG_HIDE_DOT_FILES, "hide-dot-files"}, + {KSMBD_SHARE_FLAG_INHERIT_OWNER, "inherit-owner"}, + {KSMBD_SHARE_FLAG_STREAMS, "streams"}, + {KSMBD_SHARE_FLAG_FOLLOW_SYMLINKS, "follow-symlinks"}, + {KSMBD_SHARE_FLAG_ACL_XATTR, "acl-xattr"}, + {KSMBD_SHARE_FLAG_UPDATE, "update"}, + {KSMBD_SHARE_FLAG_CROSSMNT, "crossmnt"}, + {KSMBD_SHARE_FLAG_CONTINUOUS_AVAILABILITY, "continuous-availability"}, + {KSMBD_SHARE_FLAG_ENCRYPT_DATA, "encrypt-data"}, +}; + +static int proc_show_shares(struct seq_file *m, void *v) +{ + struct ksmbd_share_config *share; + int i; + + down_read(&shares_table_lock); + hash_for_each(shares_table, i, share, hlist) { + seq_printf(m, "name:\t%s\n", share->name); + seq_printf(m, "type:\t%s\n", + test_share_config_flag(share, KSMBD_SHARE_FLAG_PIPE) ? + "pipe" : "disk"); + seq_printf(m, "tree_connects:\t%d\n", + atomic_read(&share->tree_connections)); + seq_printf(m, "file_mask:\t0%07o\n", share->create_mask); + seq_printf(m, "directory_mask:\t0%07o\n", share->directory_mask); + seq_puts(m, "flags:\t"); + ksmbd_proc_show_flag_names(m, ksmbd_share_flag_names, + ARRAY_SIZE(ksmbd_share_flag_names), + share->flags); + seq_puts(m, "\n\n"); + } + up_read(&shares_table_lock); + return 0; +} + +int create_proc_shares(void) +{ + if (!ksmbd_proc_create("shares", proc_show_shares, NULL)) + return -ENOMEM; + return 0; +} +#else +int create_proc_shares(void) { return 0; } +#endif + static unsigned int share_name_hash(const char *name) { return jhash(name, strlen(name), 0); @@ -157,8 +214,14 @@ static struct ksmbd_share_config *share_config_request(struct ksmbd_work *work, share->flags = resp->flags; atomic_set(&share->refcount, 1); + ksmbd_share_tree_conn_init(share); INIT_LIST_HEAD(&share->veto_list); share->name = kstrdup(name, KSMBD_DEFAULT_GFP); + if (!share->name) { + kill_share(share); + share = NULL; + goto out; + } if (!test_share_config_flag(share, KSMBD_SHARE_FLAG_PIPE)) { int path_len = PATH_MAX; @@ -205,7 +268,7 @@ static struct ksmbd_share_config *share_config_request(struct ksmbd_work *work, share->path = NULL; } } - if (ret || !share->name) { + if (ret) { kill_share(share); share = NULL; goto out; diff --git a/fs/smb/server/mgmt/share_config.h b/fs/smb/server/mgmt/share_config.h index d4ac2dd4de20..d157545fe7d1 100644 --- a/fs/smb/server/mgmt/share_config.h +++ b/fs/smb/server/mgmt/share_config.h @@ -24,6 +24,9 @@ struct ksmbd_share_config { struct path vfs_path; atomic_t refcount; +#ifdef CONFIG_PROC_FS + atomic_t tree_connections; +#endif struct hlist_node hlist; unsigned short create_mask; unsigned short directory_mask; @@ -60,6 +63,27 @@ static inline int test_share_config_flag(struct ksmbd_share_config *share, return share->flags & flag; } +#ifdef CONFIG_PROC_FS +static inline void ksmbd_share_tree_conn_init(struct ksmbd_share_config *share) +{ + atomic_set(&share->tree_connections, 0); +} + +static inline void ksmbd_share_tree_conn_inc(struct ksmbd_share_config *share) +{ + atomic_inc(&share->tree_connections); +} + +static inline void ksmbd_share_tree_conn_dec(struct ksmbd_share_config *share) +{ + atomic_dec(&share->tree_connections); +} +#else +static inline void ksmbd_share_tree_conn_init(struct ksmbd_share_config *share) {} +static inline void ksmbd_share_tree_conn_inc(struct ksmbd_share_config *share) {} +static inline void ksmbd_share_tree_conn_dec(struct ksmbd_share_config *share) {} +#endif + void ksmbd_share_config_del(struct ksmbd_share_config *share); void __ksmbd_share_config_put(struct ksmbd_share_config *share); @@ -74,4 +98,5 @@ struct ksmbd_share_config *ksmbd_share_config_get(struct ksmbd_work *work, const char *name); bool ksmbd_share_veto_filename(struct ksmbd_share_config *share, const char *filename); +int create_proc_shares(void); #endif /* __SHARE_CONFIG_MANAGEMENT_H__ */ diff --git a/fs/smb/server/mgmt/tree_connect.c b/fs/smb/server/mgmt/tree_connect.c index 58e5b8592da4..5f63e236267a 100644 --- a/fs/smb/server/mgmt/tree_connect.c +++ b/fs/smb/server/mgmt/tree_connect.c @@ -88,6 +88,7 @@ ksmbd_tree_conn_connect(struct ksmbd_work *work, const char *share_name) goto out_error; } ksmbd_counter_inc(KSMBD_COUNTER_TREE_CONNS); + ksmbd_share_tree_conn_inc(sc); kvfree(resp); return status; @@ -116,6 +117,7 @@ static int __ksmbd_tree_conn_disconnect(struct ksmbd_session *sess, ret = ksmbd_ipc_tree_disconnect_request(sess->id, tree_conn->id); ksmbd_release_tree_conn_id(sess, tree_conn->id); ksmbd_counter_dec(KSMBD_COUNTER_TREE_CONNS); + ksmbd_share_tree_conn_dec(tree_conn->share_conf); if (atomic_dec_and_test(&tree_conn->refcount)) { ksmbd_share_config_put(tree_conn->share_conf); kfree(tree_conn); diff --git a/fs/smb/server/mgmt/user_config.c b/fs/smb/server/mgmt/user_config.c index cf45841d9d1b..5efc3d7455b9 100644 --- a/fs/smb/server/mgmt/user_config.c +++ b/fs/smb/server/mgmt/user_config.c @@ -26,8 +26,9 @@ struct ksmbd_user *ksmbd_login_user(const char *account) resp_ext = ksmbd_ipc_login_request_ext(account); user = ksmbd_alloc_user(resp, resp_ext); + kvfree(resp_ext); out: - kvfree(resp); + kvfree_sensitive(resp, sizeof(*resp)); return user; } @@ -36,6 +37,17 @@ struct ksmbd_user *ksmbd_alloc_user(struct ksmbd_login_response *resp, { struct ksmbd_user *user; + /* + * resp->hash_sz is a __u16 taken from the mountd IPC login response but + * resp->hash[] is only KSMBD_REQ_MAX_HASH_SZ bytes. A malformed or + * malicious response can set hash_sz far beyond that (up to 65535), + * making the memcpy() below read past the response object + * (slab-out-of-bounds in ksmbd_alloc_user()). Reject any oversized + * hash rather than trust the length. + */ + if (resp->hash_sz > sizeof(resp->hash)) + return NULL; + user = kmalloc_obj(struct ksmbd_user, KSMBD_DEFAULT_GFP); if (!user) return NULL; @@ -70,7 +82,7 @@ struct ksmbd_user *ksmbd_alloc_user(struct ksmbd_login_response *resp, err_free: kfree(user->name); - kfree(user->passkey); + kfree_sensitive(user->passkey); kfree(user); return NULL; } @@ -80,7 +92,7 @@ void ksmbd_free_user(struct ksmbd_user *user) ksmbd_ipc_logout_request(user->name, user->flags); kfree(user->sgid); kfree(user->name); - kfree(user->passkey); + kfree_sensitive(user->passkey); kfree(user); } diff --git a/fs/smb/server/mgmt/user_session.c b/fs/smb/server/mgmt/user_session.c index f99c86284ba3..7022d5d656b4 100644 --- a/fs/smb/server/mgmt/user_session.c +++ b/fs/smb/server/mgmt/user_session.c @@ -78,6 +78,28 @@ static const char *session_user_name(struct ksmbd_session *session) return session->user->name; } +static const char *session_account_type(struct ksmbd_session *session) +{ + if (user_guest(session->user)) + return "guest"; + if (ksmbd_anonymous_user(session->user)) + return "anonymous"; + return "user"; +} + +static unsigned int session_open_file_count(struct ksmbd_session *session) +{ + struct ksmbd_file *fp; + unsigned int count = 0; + unsigned int id; + + read_lock(&session->file_table.lock); + idr_for_each_entry(session->file_table.idr, fp, id) + count++; + read_unlock(&session->file_table.lock); + return count; +} + static int show_proc_session(struct seq_file *m, void *v) { struct ksmbd_session *sess; @@ -90,97 +112,91 @@ static int show_proc_session(struct seq_file *m, void *v) sess = (struct ksmbd_session *)m->private; ksmbd_user_session_get(sess); + seq_printf(m, "user:\t%s\n", session_user_name(sess)); + seq_printf(m, "account_type:\t%s\n", + session_account_type(sess)); + seq_printf(m, "id:\t%llu\n", sess->id); + seq_printf(m, "state:\t%s\n", session_state_string(sess)); + seq_printf(m, "dialect:\t0x%04x\n", sess->dialect); + seq_printf(m, "last_active_seconds:\t%lu\n", + jiffies_to_msecs(jiffies - sess->last_active) / MSEC_PER_SEC); + seq_printf(m, "open_files:\t%u\n", + session_open_file_count(sess)); + i = 0; down_read(&sess->chann_lock); xa_for_each(&sess->ksmbd_chann_list, id, chan) { + const char *name; + #if IS_ENABLED(CONFIG_IPV6) if (chan->conn->inet_addr) - seq_printf(m, "%-20s\t%pI4\n", "client", + seq_printf(m, "client:\t%pI4\n", &chan->conn->inet_addr); else - seq_printf(m, "%-20s\t%pI6c\n", "client", + seq_printf(m, "client:\t%pI6c\n", &chan->conn->inet6_addr); #else - seq_printf(m, "%-20s\t%pI4\n", "client", + seq_printf(m, "client:\t%pI4\n", &chan->conn->inet_addr); #endif - seq_printf(m, "%-20s\t%s\n", "user", session_user_name(sess)); - seq_printf(m, "%-20s\t%llu\n", "id", sess->id); - seq_printf(m, "%-20s\t%s\n", "state", - session_state_string(sess)); - - seq_printf(m, "%-20s\t", "capabilities"); + seq_puts(m, "capabilities:\t"); ksmbd_proc_show_flag_names(m, ksmbd_sess_cap_const_names, ARRAY_SIZE(ksmbd_sess_cap_const_names), chan->conn->vals->req_capabilities); + seq_putc(m, '\n'); + seq_printf(m, "posix_extensions:\t%s\n", + chan->conn->posix_ext_supported ? "yes" : "no"); if (sess->sign) { - seq_printf(m, "%-20s\t", "signing"); - ksmbd_proc_show_const_name(m, "%s\t", - ksmbd_signing_const_names, - ARRAY_SIZE(ksmbd_signing_const_names), - le16_to_cpu(chan->conn->signing_algorithm)); - } else if (sess->enc) { - seq_printf(m, "%-20s\t", "encryption"); - ksmbd_proc_show_const_name(m, "%s\t", - ksmbd_cipher_const_names, - ARRAY_SIZE(ksmbd_cipher_const_names), - le16_to_cpu(chan->conn->cipher_type)); + unsigned int algorithm = + le16_to_cpu(chan->conn->signing_algorithm); + + name = ksmbd_proc_const_name(ksmbd_signing_const_names, + ARRAY_SIZE(ksmbd_signing_const_names), + algorithm); + if (name) + seq_printf(m, "signing:\t%s\n", name); + else + seq_printf(m, "signing:\t0x%04x\n", + algorithm); + } + if (sess->enc) { + unsigned int cipher = le16_to_cpu(chan->conn->cipher_type); + + name = ksmbd_proc_const_name(ksmbd_cipher_const_names, + ARRAY_SIZE(ksmbd_cipher_const_names), + cipher); + if (name) + seq_printf(m, "encryption:\t%s\n", name); + else + seq_printf(m, "encryption:\t0x%04x\n", + cipher); } i++; } up_read(&sess->chann_lock); - seq_printf(m, "%-20s\t%d\n", "channels", i); + seq_printf(m, "channels:\t%d\n", i); i = 0; down_read(&sess->tree_conns_lock); xa_for_each(&sess->tree_conns, id, tree_conn) { share_conf = tree_conn->share_conf; - seq_printf(m, "%-20s\t%s\t%8d", "share", - share_conf->name, tree_conn->id); - if (test_share_config_flag(share_conf, KSMBD_SHARE_FLAG_PIPE)) - seq_printf(m, " %s ", "pipe"); - else - seq_printf(m, " %s ", "disk"); - seq_putc(m, '\n'); + seq_printf(m, "share:\t%s\n", share_conf->name); + seq_printf(m, "tree_id:\t%d\n", tree_conn->id); + seq_printf(m, "share_type:\t%s\n", + test_share_config_flag(share_conf, KSMBD_SHARE_FLAG_PIPE) ? + "pipe" : "disk"); + i++; } up_read(&sess->tree_conns_lock); + seq_printf(m, "tree_connects:\t%d\n", i); ksmbd_user_session_put(sess); return 0; } -void ksmbd_proc_show_flag_names(struct seq_file *m, - const struct ksmbd_const_name *table, - int count, - unsigned int flags) -{ - int i; - - for (i = 0; i < count; i++) { - if (table[i].const_value & flags) - seq_printf(m, "0x%08x\t", table[i].const_value); - } - seq_putc(m, '\n'); -} - -void ksmbd_proc_show_const_name(struct seq_file *m, - const char *format, - const struct ksmbd_const_name *table, - int count, - unsigned int const_value) -{ - int i; - - for (i = 0; i < count; i++) { - if (table[i].const_value & const_value) - seq_printf(m, format, table[i].name); - } - seq_putc(m, '\n'); -} - static int create_proc_session(struct ksmbd_session *sess) { char name[30]; @@ -188,6 +204,8 @@ static int create_proc_session(struct ksmbd_session *sess) snprintf(name, sizeof(name), "sessions/%llu", sess->id); sess->proc_entry = ksmbd_proc_create(name, show_proc_session, sess); + if (!sess->proc_entry) + return -ENOMEM; return 0; } @@ -204,9 +222,6 @@ static int show_proc_sessions(struct seq_file *m, void *v) int i; unsigned long id; - seq_printf(m, "#%-40s %-15s %-10s %-10s\n", - "<client>", "<user>", "<sess_id>", "<state>"); - down_read(&sessions_table_lock); hash_for_each(sessions_table, i, session, hlist) { down_read(&session->chann_lock); @@ -216,13 +231,13 @@ static int show_proc_sessions(struct seq_file *m, void *v) #if IS_ENABLED(CONFIG_IPV6) if (!chan->conn->inet_addr) - seq_printf(m, " %-40pI6c", &chan->conn->inet6_addr); + seq_printf(m, "client:\t%pI6c\n", &chan->conn->inet6_addr); else #endif - seq_printf(m, " %-40pI4", &chan->conn->inet_addr); - seq_printf(m, " %-15s %-10llu %-10s\n", - session_user_name(session), - session->id, + seq_printf(m, "client:\t%pI4\n", &chan->conn->inet_addr); + seq_printf(m, "user:\t%s\n", session_user_name(session)); + seq_printf(m, "id:\t%llu\n", session->id); + seq_printf(m, "state:\t%s\n\n", session_state_string(session)); ksmbd_user_session_put(session); @@ -308,8 +323,11 @@ static int __rpc_method(char *rpc_name) if (!strcmp(rpc_name, "\\lsarpc") || !strcmp(rpc_name, "lsarpc")) return KSMBD_RPC_LSARPC_METHOD_INVOKE; + if (!strcmp(rpc_name, "\\mdssvc") || !strcmp(rpc_name, "mdssvc")) + return -ENOENT; + pr_err("Unsupported RPC: %s\n", rpc_name); - return 0; + return -ENOENT; } int ksmbd_session_rpc_open(struct ksmbd_session *sess, char *rpc_name) @@ -319,8 +337,8 @@ int ksmbd_session_rpc_open(struct ksmbd_session *sess, char *rpc_name) int method, id; method = __rpc_method(rpc_name); - if (!method) - return -EINVAL; + if (method < 0) + return method; entry = kzalloc_obj(struct ksmbd_session_rpc, KSMBD_DEFAULT_GFP); if (!entry) @@ -389,10 +407,16 @@ void ksmbd_session_destroy(struct ksmbd_session *sess) ksmbd_launch_ksmbd_durable_scavenger(); ksmbd_session_rpc_clear_list(sess); free_channel_list(sess); - kfree(sess->Preauth_HashValue); + kfree_sensitive(sess->Preauth_HashValue); ksmbd_release_id(&session_ida, sess->id); ida_destroy(&sess->tree_conn_ida); - kfree(sess); + kfree_sensitive(sess); +} + +static void ksmbd_session_remove_from_table(struct ksmbd_session *sess) +{ + hash_del(&sess->hlist); + ksmbd_counter_dec(KSMBD_COUNTER_SESSIONS); } struct ksmbd_session *__session_lookup(unsigned long long id) @@ -421,7 +445,7 @@ static void ksmbd_expire_session(struct ksmbd_conn *conn) time_after(jiffies, sess->last_active + SMB2_SESSION_TIMEOUT))) { xa_erase(&conn->sessions, sess->id); - hash_del(&sess->hlist); + ksmbd_session_remove_from_table(sess); ksmbd_session_destroy(sess); continue; } @@ -433,10 +457,21 @@ static void ksmbd_expire_session(struct ksmbd_conn *conn) int ksmbd_session_register(struct ksmbd_conn *conn, struct ksmbd_session *sess) { + int ret; + sess->dialect = conn->dialect; memcpy(sess->ClientGUID, conn->ClientGUID, SMB2_CLIENT_GUID_SIZE); ksmbd_expire_session(conn); - return xa_err(xa_store(&conn->sessions, sess->id, sess, KSMBD_DEFAULT_GFP)); + ret = xa_err(xa_store(&conn->sessions, sess->id, sess, + KSMBD_DEFAULT_GFP)); + if (ret) { + down_write(&sessions_table_lock); + ksmbd_session_remove_from_table(sess); + up_write(&sessions_table_lock); + ksmbd_user_session_put(sess); + } + + return ret; } static int ksmbd_chann_del(struct ksmbd_conn *conn, struct ksmbd_session *sess) @@ -464,7 +499,7 @@ void ksmbd_sessions_deregister(struct ksmbd_conn *conn) hash_for_each_safe(sessions_table, bkt, tmp, sess, hlist) { if (!ksmbd_chann_del(conn, sess) && xa_empty(&sess->ksmbd_chann_list)) { - hash_del(&sess->hlist); + ksmbd_session_remove_from_table(sess); down_write(&conn->session_lock); xa_erase(&conn->sessions, sess->id); up_write(&conn->session_lock); @@ -475,18 +510,10 @@ void ksmbd_sessions_deregister(struct ksmbd_conn *conn) down_write(&conn->session_lock); xa_for_each(&conn->sessions, id, sess) { - unsigned long chann_id; - struct channel *chann; - - xa_for_each(&sess->ksmbd_chann_list, chann_id, chann) { - if (chann->conn != conn) - ksmbd_conn_set_exiting(chann->conn); - } - ksmbd_chann_del(conn, sess); if (xa_empty(&sess->ksmbd_chann_list)) { xa_erase(&conn->sessions, sess->id); - hash_del(&sess->hlist); + ksmbd_session_remove_from_table(sess); if (atomic_dec_and_test(&sess->refcnt)) ksmbd_session_destroy(sess); } @@ -539,19 +566,35 @@ struct ksmbd_session *ksmbd_session_lookup_slowpath(unsigned long long id) return sess; } -struct ksmbd_session *ksmbd_session_lookup_all(struct ksmbd_conn *conn, - unsigned long long id) +struct ksmbd_session *ksmbd_session_lookup_all_states(struct ksmbd_conn *conn, + unsigned long long id) { struct ksmbd_session *sess; + bool channel_found; sess = ksmbd_session_lookup(conn, id); - if (!sess && conn->binding) { + if (!sess) { sess = ksmbd_session_lookup_slowpath(id); - if (sess && !xa_load(&sess->ksmbd_chann_list, (long)conn)) { + if (!sess) + return NULL; + + down_read(&sess->chann_lock); + channel_found = xa_load(&sess->ksmbd_chann_list, (long)conn); + up_read(&sess->chann_lock); + if (!channel_found) { ksmbd_user_session_put(sess); sess = NULL; } } + return sess; +} + +struct ksmbd_session *ksmbd_session_lookup_all(struct ksmbd_conn *conn, + unsigned long long id) +{ + struct ksmbd_session *sess; + + sess = ksmbd_session_lookup_all_states(conn, id); if (sess && sess->state != SMB2_SESSION_VALID) { ksmbd_user_session_put(sess); sess = NULL; @@ -592,6 +635,17 @@ struct preauth_session *ksmbd_preauth_session_alloc(struct ksmbd_conn *conn, return sess; } +void ksmbd_preauth_session_destroy(struct ksmbd_conn *conn) +{ + struct preauth_session *sess, *tmp; + + list_for_each_entry_safe(sess, tmp, &conn->preauth_sess_table, + preauth_entry) { + list_del(&sess->preauth_entry); + kfree(sess); + } +} + void destroy_previous_session(struct ksmbd_conn *conn, struct ksmbd_user *user, u64 id) { @@ -612,16 +666,17 @@ void destroy_previous_session(struct ksmbd_conn *conn, memcmp(user->passkey, prev_user->passkey, user->passkey_sz)) goto out; - ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_RECONNECT); - err = ksmbd_conn_wait_idle_sess_id(conn, id); + ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_RECONNECT); + err = ksmbd_conn_wait_idle_sess(conn, prev_sess); if (err) { - ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_SETUP); + ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_SETUP); goto out; } ksmbd_destroy_file_table(prev_sess); + prev_sess->kerberos_expiry = 0; prev_sess->state = SMB2_SESSION_EXPIRED; - ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_SETUP); + ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_SETUP); ksmbd_launch_ksmbd_durable_scavenger(); out: up_write(&conn->session_lock); @@ -691,10 +746,11 @@ static struct ksmbd_session *__session_create(int protocol) down_write(&sessions_table_lock); hash_add(sessions_table, &sess->hlist, sess->id); + ksmbd_counter_inc(KSMBD_COUNTER_SESSIONS); up_write(&sessions_table_lock); - create_proc_session(sess); - ksmbd_counter_inc(KSMBD_COUNTER_SESSIONS); + if (create_proc_session(sess)) + pr_warn_ratelimited("Unable to create session %llu procfs entry\n", sess->id); return sess; error: diff --git a/fs/smb/server/mgmt/user_session.h b/fs/smb/server/mgmt/user_session.h index 4637a8c8436d..f8a24c33f7fe 100644 --- a/fs/smb/server/mgmt/user_session.h +++ b/fs/smb/server/mgmt/user_session.h @@ -47,6 +47,7 @@ struct ksmbd_session { __u8 *Preauth_HashValue; char sess_key[CIFS_KEY_SIZE]; + u64 kerberos_expiry; struct hlist_node hlist; struct rw_semaphore chann_lock; @@ -100,10 +101,13 @@ void ksmbd_sessions_deregister(struct ksmbd_conn *conn); struct ksmbd_session *__session_lookup(unsigned long long id); struct ksmbd_session *ksmbd_session_lookup_all(struct ksmbd_conn *conn, unsigned long long id); +struct ksmbd_session *ksmbd_session_lookup_all_states(struct ksmbd_conn *conn, + unsigned long long id); void destroy_previous_session(struct ksmbd_conn *conn, struct ksmbd_user *user, u64 id); struct preauth_session *ksmbd_preauth_session_alloc(struct ksmbd_conn *conn, u64 sess_id); +void ksmbd_preauth_session_destroy(struct ksmbd_conn *conn); struct preauth_session *ksmbd_preauth_session_lookup(struct ksmbd_conn *conn, unsigned long long id); diff --git a/fs/smb/server/misc.h b/fs/smb/server/misc.h index 3909104e18ad..1faaddd0f5f7 100644 --- a/fs/smb/server/misc.h +++ b/fs/smb/server/misc.h @@ -43,7 +43,7 @@ struct ksmbd_const_name { const char *name; }; -void ksmbd_proc_init(void); +int ksmbd_proc_init(void); void ksmbd_proc_cleanup(void); void ksmbd_proc_reset(void); struct proc_dir_entry *ksmbd_proc_create(const char *name, @@ -53,13 +53,10 @@ void ksmbd_proc_show_flag_names(struct seq_file *m, const struct ksmbd_const_name *table, int count, unsigned int flags); -void ksmbd_proc_show_const_name(struct seq_file *m, - const char *format, - const struct ksmbd_const_name *table, - int count, - unsigned int const_value); +const char *ksmbd_proc_const_name(const struct ksmbd_const_name *table, + int count, unsigned int const_value); #else -static inline void ksmbd_proc_init(void) {} +static inline int ksmbd_proc_init(void) { return 0; } static inline void ksmbd_proc_cleanup(void) {} static inline void ksmbd_proc_reset(void) {} #endif diff --git a/fs/smb/server/oplock.c b/fs/smb/server/oplock.c index 79787099afdc..58af0fddf39f 100644 --- a/fs/smb/server/oplock.c +++ b/fs/smb/server/oplock.c @@ -16,6 +16,7 @@ #include "mgmt/user_session.h" #include "mgmt/share_config.h" #include "mgmt/tree_connect.h" +#include "server.h" static LIST_HEAD(lease_table_list); static DEFINE_RWLOCK(lease_list_lock); @@ -89,6 +90,7 @@ static struct oplock_info *alloc_opinfo(struct ksmbd_work *work, opinfo->conn = ksmbd_conn_get(work->conn); opinfo->level = SMB2_OPLOCK_LEVEL_NONE; opinfo->op_state = OPLOCK_STATE_NONE; + spin_lock_init(&opinfo->state_lock); opinfo->pending_break = 0; opinfo->fid = id; opinfo->Tid = Tid; @@ -545,14 +547,23 @@ void close_id_del_oplock(struct ksmbd_file *fp) opinfo_del(opinfo); rcu_assign_pointer(fp->f_opinfo, NULL); - if (opinfo->op_state == OPLOCK_ACK_WAIT) { - opinfo->op_state = OPLOCK_CLOSING; - wake_up_interruptible_all(&opinfo->oplock_q); - if (opinfo->is_lease) { - atomic_set(&opinfo->breaking_cnt, 0); - wake_up_interruptible_all(&opinfo->oplock_brk); - } - } + spin_lock(&opinfo->state_lock); + if (opinfo->op_state == OPLOCK_ACK_WAIT && opinfo->is_lease) + atomic_set(&opinfo->breaking_cnt, 0); + /* + * An opinfo that has been removed from the inode list is terminal. Keep + * this transition and releasing pending_break under state_lock. a breaker + * takes the same lock before it acquires pending_break or sets ACK_WAIT. + */ + opinfo->op_state = OPLOCK_CLOSING; + clear_bit_unlock(0, &opinfo->pending_break); + spin_unlock(&opinfo->state_lock); + wake_up_interruptible_all(&opinfo->oplock_q); + if (opinfo->is_lease) + wake_up_interruptible_all(&opinfo->oplock_brk); + /* memory barrier is needed for wake_up_bit() */ + smp_mb__after_atomic(); + wake_up_bit(&opinfo->pending_break, 0); opinfo_count_dec(fp); atomic_dec(&opinfo->refcount); @@ -734,12 +745,18 @@ static bool wait_for_break_ack(struct oplock_info *opinfo) /* is this a timeout ? */ if (!rc) { + spin_lock(&opinfo->state_lock); + if (opinfo->op_state == OPLOCK_CLOSING) { + spin_unlock(&opinfo->state_lock); + return false; + } if (opinfo->is_lease) { opinfo->o_lease->state = SMB2_LEASE_NONE_LE; lease_update_oplock_levels(opinfo->o_lease); } opinfo->level = SMB2_OPLOCK_LEVEL_NONE; opinfo->op_state = OPLOCK_STATE_NONE; + spin_unlock(&opinfo->state_lock); return true; } @@ -754,9 +771,35 @@ static void wake_up_oplock_break(struct oplock_info *opinfo) wake_up_bit(&opinfo->pending_break, 0); } +static bool oplock_break_set_ack_wait(struct oplock_info *opinfo) +{ + bool ret = false; + + spin_lock(&opinfo->state_lock); + if (opinfo->op_state != OPLOCK_CLOSING) { + opinfo->op_state = OPLOCK_ACK_WAIT; + ret = true; + } + spin_unlock(&opinfo->state_lock); + + return ret; +} + static int oplock_break_pending(struct oplock_info *opinfo, int req_op_level) { - while (test_and_set_bit(0, &opinfo->pending_break)) { + for (;;) { + bool closing; + + spin_lock(&opinfo->state_lock); + closing = opinfo->op_state == OPLOCK_CLOSING; + if (!closing && !test_and_set_bit(0, &opinfo->pending_break)) { + spin_unlock(&opinfo->state_lock); + break; + } + spin_unlock(&opinfo->state_lock); + if (closing) + return -ENOENT; + if (opinfo->is_lease) opinfo->o_lease->reuse_epoch = true; @@ -765,9 +808,12 @@ static int oplock_break_pending(struct oplock_info *opinfo, int req_op_level) /* Not immediately break to none. */ opinfo->open_trunc = 0; - if (opinfo->op_state == OPLOCK_CLOSING) + spin_lock(&opinfo->state_lock); + closing = opinfo->op_state == OPLOCK_CLOSING; + spin_unlock(&opinfo->state_lock); + if (closing) return -ENOENT; - else if (opinfo->level <= req_op_level) { + if (opinfo->level <= req_op_level) { if (opinfo->is_lease == false) return 1; @@ -989,38 +1035,71 @@ out: ksmbd_conn_put(conn); } +/* + * Select and pin the connection used for a lease break before doing any + * allocations which may sleep. opinfo->conn is cleared under ci->m_lock, + * while lease->l_lb and the lease table lifetime are protected by + * lease_list_lock. + */ +static struct ksmbd_conn *smb2_lease_break_conn_get(struct oplock_info *opinfo) +{ + struct lease *lease = opinfo->o_lease; + struct lease_table *lb; + struct ksmbd_conn *conn; + + /* Keep the connection which owns the open, when it is still active. */ + down_read(&lease->ci->m_lock); + conn = READ_ONCE(opinfo->conn); + if (conn && !ksmbd_conn_releasing(conn)) + conn = ksmbd_conn_get(conn); + else + conn = NULL; + up_read(&lease->ci->m_lock); + + if (conn || lease->version != 2) + return conn; + + /* Otherwise route v2 lease breaks through the shared lease channel. */ + read_lock(&lease_list_lock); + lb = lease->l_lb; + if (lb && lb->conn && !ksmbd_conn_releasing(lb->conn)) + conn = ksmbd_conn_get(lb->conn); + read_unlock(&lease_list_lock); + + return conn; +} + /** * smb2_lease_break_noti() - break lease when a new client request * write lease * @opinfo: contains lease state information - * @wait_ack: wait for lease break acknowledgment from the client + * @sync: send the lease break notification synchronously * @inc_epoch: increment the lease epoch before sending the break * * Return: 0 on success, otherwise error */ -static int smb2_lease_break_noti(struct oplock_info *opinfo, bool wait_ack, +static int smb2_lease_break_noti(struct oplock_info *opinfo, bool sync, bool inc_epoch) { struct ksmbd_conn *conn; struct ksmbd_work *work; struct lease_break_info *br_info; struct lease *lease = opinfo->o_lease; - int ret = 0; - conn = READ_ONCE(opinfo->conn); - if (lease->version == 2 && lease->l_lb && lease->l_lb->conn && - !ksmbd_conn_releasing(lease->l_lb->conn)) - conn = lease->l_lb->conn; + conn = smb2_lease_break_conn_get(opinfo); if (!conn) return ksmbd_invalidate_durable_fd(opinfo->fid); work = ksmbd_alloc_work_struct(); - if (!work) + if (!work) { + ksmbd_conn_put(conn); return -ENOMEM; + } br_info = kmalloc_obj(struct lease_break_info, KSMBD_DEFAULT_GFP); if (!br_info) { ksmbd_free_work_struct(work); + ksmbd_conn_put(conn); return -ENOMEM; } @@ -1036,16 +1115,17 @@ static int smb2_lease_break_noti(struct oplock_info *opinfo, bool wait_ack, memcpy(br_info->lease_key, lease->lease_key, SMB2_LEASE_KEY_SIZE); work->request_buf = (char *)br_info; - work->conn = ksmbd_conn_get(conn); + /* Transfer the reference acquired by smb2_lease_break_conn_get(). */ + work->conn = conn; work->sess = opinfo->sess; ksmbd_conn_r_count_inc(conn); if (opinfo->op_state == OPLOCK_ACK_WAIT) { - INIT_WORK(&work->work, __smb2_lease_break_noti); - ksmbd_queue_work(work); - if (wait_ack) { - if (wait_for_break_ack(opinfo)) - ret = ksmbd_invalidate_durable_fd(opinfo->fid); + if (sync) { + __smb2_lease_break_noti(&work->work); + } else { + INIT_WORK(&work->work, __smb2_lease_break_noti); + ksmbd_queue_work(work); } } else { __smb2_lease_break_noti(&work->work); @@ -1054,7 +1134,7 @@ static int smb2_lease_break_noti(struct oplock_info *opinfo, bool wait_ack, lease_update_oplock_levels(opinfo->o_lease); } } - return ret; + return 0; } static void wait_lease_breaking(struct oplock_info *opinfo) @@ -1075,7 +1155,8 @@ static void wait_lease_breaking(struct oplock_info *opinfo) } static int oplock_break(struct oplock_info *brk_opinfo, int req_op_level, - struct ksmbd_work *in_work, bool share_break) + struct ksmbd_work *in_work, bool share_break, + bool sync_lease_break) { int err = 0; bool sent_interim = false; @@ -1136,16 +1217,13 @@ again: } } - if (in_work && !sent_interim) { - setup_async_work(in_work, NULL, NULL); - smb2_send_interim_resp(in_work, STATUS_PENDING); - release_async_work(in_work); - sent_interim = true; - } - if (lease->state & (SMB2_LEASE_WRITE_CACHING_LE | SMB2_LEASE_HANDLE_CACHING_LE)) { - brk_opinfo->op_state = OPLOCK_ACK_WAIT; + if (!oplock_break_set_ack_wait(brk_opinfo)) { + atomic_dec_if_positive(&brk_opinfo->breaking_cnt); + wake_up_oplock_break(brk_opinfo); + return -ENOENT; + } } else atomic_dec(&brk_opinfo->breaking_cnt); @@ -1156,8 +1234,16 @@ again: inc_epoch = false; lease->reuse_epoch = false; } - err = smb2_lease_break_noti(brk_opinfo, wait_ack, inc_epoch); + err = smb2_lease_break_noti(brk_opinfo, sync_lease_break, inc_epoch); inc_epoch = false; + if (in_work && !sent_interim) { + setup_async_work(in_work, NULL, NULL); + smb2_send_interim_resp(in_work, STATUS_PENDING); + release_async_work(in_work); + sent_interim = true; + } + if (wait_ack && !err && wait_for_break_ack(brk_opinfo)) + err = ksmbd_invalidate_durable_fd(brk_opinfo->fid); ksmbd_debug(OPLOCK, "oplock granted = %d\n", brk_opinfo->level); if (brk_opinfo->op_state == OPLOCK_CLOSING) @@ -1192,8 +1278,24 @@ again: return err < 0 ? err : 0; if (brk_opinfo->level == SMB2_OPLOCK_LEVEL_BATCH || - brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) - brk_opinfo->op_state = OPLOCK_ACK_WAIT; + brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) { + if (!oplock_break_set_ack_wait(brk_opinfo)) { + wake_up_oplock_break(brk_opinfo); + return -ENOENT; + } + } + + /* + * Keep a conflicting CREATE asynchronous while waiting for an + * oplock-break acknowledgement. Besides avoiding a blocked client + * request, this lets a replay arrive while the original CREATE is + * still pending and be rejected with FILE_NOT_AVAILABLE. + */ + if (in_work) { + setup_async_work(in_work, NULL, NULL); + smb2_send_interim_resp(in_work, STATUS_PENDING); + release_async_work(in_work); + } } err = smb2_oplock_break_noti(brk_opinfo); @@ -1229,7 +1331,8 @@ static void oplock_break_drain_none(struct list_head *head) struct oplock_break_entry *ent, *tmp; list_for_each_entry_safe(ent, tmp, head, list) { - oplock_break(ent->opinfo, SMB2_OPLOCK_LEVEL_NONE, NULL, false); + oplock_break(ent->opinfo, SMB2_OPLOCK_LEVEL_NONE, NULL, false, + false); list_del(&ent->list); opinfo_put(ent->opinfo); kfree(ent); @@ -1347,7 +1450,7 @@ void smb_send_parent_lease_break_noti(struct ksmbd_file *fp, struct ksmbd_inode *p_ci = NULL; LIST_HEAD(brk_list); - if (lctx->version != 2) + if (lctx && lctx->version != 2) return; p_ci = ksmbd_inode_lookup_lock(fp->filp->f_path.dentry->d_parent); @@ -1360,9 +1463,10 @@ void smb_send_parent_lease_break_noti(struct ksmbd_file *fp, continue; if (opinfo->o_lease->state != SMB2_OPLOCK_LEVEL_NONE && - (!(lctx->flags & SMB2_LEASE_FLAG_PARENT_LEASE_KEY_SET_LE) || - !compare_guid_key(opinfo, fp->conn->ClientGUID, - lctx->parent_lease_key))) { + (!lctx || + (!(lctx->flags & SMB2_LEASE_FLAG_PARENT_LEASE_KEY_SET_LE) || + !compare_guid_key(opinfo, fp->conn->ClientGUID, + lctx->parent_lease_key)))) { if (!atomic_inc_not_zero(&opinfo->refcount)) continue; @@ -1435,12 +1539,13 @@ void smb_lazy_parent_lease_break_close(struct ksmbd_file *fp) * @tid: Tree id of connection * @lctx: lease context information on file open * @share_ret: share mode + * @replay: whether this is a replayed CREATE request * * Return: 0 on success, otherwise error */ int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid, struct ksmbd_file *fp, __u16 tid, - struct lease_ctx_info *lctx, int share_ret) + struct lease_ctx_info *lctx, int share_ret, bool replay) { int err = 0; int break_level = SMB2_OPLOCK_LEVEL_II; @@ -1453,6 +1558,7 @@ int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid, bool prev_durable_detached = false; unsigned long long prev_fid = KSMBD_NO_FID; bool new_lease = false; + bool break_needed; __le32 prev_op_state = 0; /* Only v2 leases handle the directory */ @@ -1524,6 +1630,21 @@ int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid, prev_op_has_lease = prev_opinfo->is_lease; if (prev_op_has_lease) prev_op_state = prev_opinfo->o_lease->state; + /* + * A replay received while this open is waiting for an oplock or lease + * break must not observe an intermediate level and proceed as a new + * open. This check has to precede break_needed. an oplock may already + * have been downgraded from Batch to II while its acknowledgement is + * still pending. + */ + if (replay && + (test_bit(0, &prev_opinfo->pending_break) || + prev_opinfo->op_state == OPLOCK_ACK_WAIT)) { + err = -EINPROGRESS; + opinfo_put(prev_opinfo); + goto err_out; + } + if (share_ret < 0 && prev_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) { err = share_ret; @@ -1531,8 +1652,11 @@ int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid, goto err_out; } - if (prev_opinfo->level != SMB2_OPLOCK_LEVEL_BATCH && - prev_opinfo->level != SMB2_OPLOCK_LEVEL_EXCLUSIVE) { + break_needed = prev_opinfo->level == SMB2_OPLOCK_LEVEL_BATCH || + prev_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE || + (share_ret < 0 && prev_op_has_lease && + (prev_op_state & SMB2_LEASE_HANDLE_CACHING_LE)); + if (!break_needed) { opinfo_put(prev_opinfo); goto op_break_not_needed; } @@ -1542,7 +1666,7 @@ int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid, prev_fid = prev_op_snapshot.fid; err = oplock_break(prev_opinfo, break_level, work, - share_ret < 0 && prev_opinfo->is_lease); + share_ret < 0 && prev_opinfo->is_lease, false); if (prev_durable_detached || (prev_durable_open && err == -ENOENT)) ksmbd_invalidate_durable_fd(prev_fid); opinfo_put(prev_opinfo); @@ -1555,7 +1679,14 @@ int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid, goto set_lev; } if (err == -ENOENT) { - if (req_op_level != SMB2_OPLOCK_LEVEL_NONE) + /* + * A pending durable CREATE can lose the previous oplock when + * its holder closes the file. In that case grant the original + * request its full caching state. Other opens still need the + * normal shared-open downgrade below. + */ + if (!prev_durable_open && + req_op_level != SMB2_OPLOCK_LEVEL_NONE) req_op_level = SMB2_OPLOCK_LEVEL_II; goto set_lev; } @@ -1640,7 +1771,7 @@ static bool smb_break_all_write_oplock(struct ksmbd_work *work, } brk_opinfo->open_trunc = is_trunc; - oplock_break(brk_opinfo, SMB2_OPLOCK_LEVEL_II, work, false); + oplock_break(brk_opinfo, SMB2_OPLOCK_LEVEL_II, work, false, false); sent_break = true; opinfo_put(brk_opinfo); @@ -1655,10 +1786,12 @@ static bool smb_break_all_write_oplock(struct ksmbd_work *work, * @is_trunc: truncate on open * @send_interim: send interim response to the client * @send_oplock_break: send oplock break notification to the client + * @sync_lease_break: send the lease break notification synchronously */ static void __smb_break_all_levII_oplock(struct ksmbd_work *work, struct ksmbd_file *fp, int is_trunc, - bool send_interim, bool send_oplock_break) + bool send_interim, bool send_oplock_break, + bool sync_lease_break) { struct oplock_info *op, *brk_op; struct oplock_break_entry *ent, *tmp; @@ -1725,13 +1858,16 @@ next: if (!brk_op->is_lease && !send_oplock_break) { brk_op->level = SMB2_OPLOCK_LEVEL_NONE; - brk_op->op_state = OPLOCK_STATE_NONE; + spin_lock(&brk_op->state_lock); + if (brk_op->op_state != OPLOCK_CLOSING) + brk_op->op_state = OPLOCK_STATE_NONE; + spin_unlock(&brk_op->state_lock); } else { oplock_break(brk_op, brk_op->is_lease && !is_trunc ? SMB2_OPLOCK_LEVEL_II : SMB2_OPLOCK_LEVEL_NONE, send_interim && !sent_interim ? work : NULL, - false); + false, sync_lease_break); } sent_interim = true; list_del(&ent->list); @@ -1746,19 +1882,24 @@ next: void smb_break_all_levII_oplock(struct ksmbd_work *work, struct ksmbd_file *fp, int is_trunc) { - __smb_break_all_levII_oplock(work, fp, is_trunc, true, true); + __smb_break_all_levII_oplock(work, fp, is_trunc, true, true, false); +} + +void smb_break_all_levII_oplock_rename(struct ksmbd_work *work, struct ksmbd_file *fp) +{ + __smb_break_all_levII_oplock(work, fp, 0, true, true, true); } void smb_break_all_levII_oplock_no_interim(struct ksmbd_work *work, struct ksmbd_file *fp, int is_trunc) { - __smb_break_all_levII_oplock(work, fp, is_trunc, false, true); + __smb_break_all_levII_oplock(work, fp, is_trunc, false, true, false); } void smb_break_all_levII_oplock_for_delete(struct ksmbd_work *work, struct ksmbd_file *fp) { - __smb_break_all_levII_oplock(work, fp, 0, false, false); + __smb_break_all_levII_oplock(work, fp, 0, false, false, false); } /** @@ -1775,7 +1916,7 @@ void smb_break_all_oplock(struct ksmbd_work *work, struct ksmbd_file *fp) return; sent_break = smb_break_all_write_oplock(work, fp, 1); - __smb_break_all_levII_oplock(work, fp, 1, !sent_break, true); + __smb_break_all_levII_oplock(work, fp, 1, !sent_break, true, false); } /** @@ -2012,12 +2153,12 @@ void create_durable_v2_rsp_buf(char *cc, struct ksmbd_file *fp) struct create_durable_rsp_v2 *buf; buf = (struct create_durable_rsp_v2 *)cc; - memset(buf, 0, sizeof(struct create_durable_rsp)); + memset(buf, 0, sizeof(*buf)); buf->ccontext.DataOffset = cpu_to_le16(offsetof - (struct create_durable_rsp, Data)); + (struct create_durable_rsp_v2, dcontext)); buf->ccontext.DataLength = cpu_to_le32(8); buf->ccontext.NameOffset = cpu_to_le16(offsetof - (struct create_durable_rsp, Name)); + (struct create_durable_rsp_v2, Name)); buf->ccontext.NameLength = cpu_to_le16(4); /* SMB2_CREATE_DURABLE_HANDLE_RESPONSE_V2 is "DH2Q" */ buf->Name[0] = 'D'; @@ -2138,6 +2279,90 @@ void create_posix_rsp_buf(char *cc, struct ksmbd_file *fp) SIDUNIX_GROUP, (struct smb_sid *)&buf->SidBuffer[28]); } +/** + * create_aapl_rsp_buf() - build AAPL kAAPL_SERVER_QUERY response + * @cc: buffer to write the create context into (AAPL_RSP_MAX_SIZE bytes) + * @vol_caps: volume capability flags (SMB2_CRTCTX_AAPL_* volume bits) + * @req_bitmap: the client's request bitmap, echoed back in reply_bitmap + * + * Response format follows the layout observed from macOS's own smbd, and + * matches the client-side parsing in AAPL's published public client kernel + * source (public client behavior reference, kAAPL_SERVER_QUERY + * case): reply_bitmap, then server_caps/vol_caps/model-info fields present + * only when their reply_bitmap bit is set: + * reply_bitmap = req_bitmap masked to the fields we support + * server_caps = AAPL_SERVER_CAPS_KSMBD when requested + * vol_caps = caller-supplied + * model string = server_conf.aapl_model (default "Xserve") in UTF-16LE, + * when SMB2_CRTCTX_AAPL_MODEL_INFO requested + * + * Sending reply_bitmap with MODEL_INFO set but no model string causes + * smbfs.kext to enter a broken disconnect path requiring a macOS reboot. + * @readdir_attr_v2: advertise SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR_V2 + * instead of the V1 bit + */ +void create_aapl_rsp_buf(char *cc, __u64 vol_caps, __u64 req_bitmap, + bool readdir_attr_v2) +{ + struct create_aapl_rsp *buf; + u64 reply_bitmap; + u64 server_caps; + u32 data_len; + + buf = (struct create_aapl_rsp *)cc; + memset(buf, 0, AAPL_RSP_MAX_SIZE); + + reply_bitmap = req_bitmap & (SMB2_CRTCTX_AAPL_SERVER_CAPS | + SMB2_CRTCTX_AAPL_VOLUME_CAPS | + SMB2_CRTCTX_AAPL_MODEL_INFO); + + /* base data: cmd(4)+reserved(4)+reply_bitmap(8)+server_caps(8)+vol_caps(8) */ + data_len = 32; + if (reply_bitmap & SMB2_CRTCTX_AAPL_MODEL_INFO) + data_len += 4 + 4 + AAPL_MODEL_UTF16_BYTES; /* pad2+model_bytes+string */ + + buf->ccontext.DataOffset = cpu_to_le16(offsetof(struct create_aapl_rsp, cmd)); + buf->ccontext.DataLength = cpu_to_le32(data_len); + buf->ccontext.NameOffset = cpu_to_le16(offsetof(struct create_aapl_rsp, Name)); + buf->ccontext.NameLength = cpu_to_le16(SMB2_CREATE_AAPL_LEN); + buf->Name[0] = 'A'; + buf->Name[1] = 'A'; + buf->Name[2] = 'P'; + buf->Name[3] = 'L'; + + buf->cmd = cpu_to_le32(SMB2_CRTCTX_AAPL_SERVER_QUERY); + buf->reply_bitmap = cpu_to_le64(reply_bitmap); + server_caps = AAPL_SERVER_CAPS_KSMBD; + if (readdir_attr_v2) + server_caps = (server_caps & ~SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR) | + SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR_V2; + buf->server_caps = (reply_bitmap & SMB2_CRTCTX_AAPL_SERVER_CAPS) ? + cpu_to_le64(server_caps) : 0; + buf->vol_caps = (reply_bitmap & SMB2_CRTCTX_AAPL_VOLUME_CAPS) ? + cpu_to_le64(vol_caps) : 0; + + if (reply_bitmap & SMB2_CRTCTX_AAPL_MODEL_INFO) { + __le32 *p = (__le32 *)((u8 *)buf + sizeof(*buf)); + __le16 *model_str = (__le16 *)(p + 2); + const char *src = server_conf.aapl_model[0] ? + server_conf.aapl_model : "Xserve"; + int i, model_bytes = 0; + + /* Convert ASCII model string to UTF-16LE in-place */ + for (i = 0; src[i] && i < AAPL_MODEL_MAX_CHARS; i++) { + model_str[i] = cpu_to_le16((unsigned char)src[i]); + model_bytes += 2; + } + + p[0] = 0; /* pad2 */ + p[1] = cpu_to_le32(model_bytes); + + /* Update DataLength to reflect actual model string size */ + buf->ccontext.DataLength = + cpu_to_le32(data_len - AAPL_MODEL_UTF16_BYTES + model_bytes); + } +} + /* * Find lease object(opinfo) for given lease key/fid from lease * break/file close path. @@ -2182,7 +2407,6 @@ found: if (!atomic_inc_not_zero(&opinfo->refcount)) continue; ret_op = opinfo; - break; } spin_unlock(&lease->lock); if (ret_op) { diff --git a/fs/smb/server/oplock.h b/fs/smb/server/oplock.h index 3f581d22bb67..b08d21758e07 100644 --- a/fs/smb/server/oplock.h +++ b/fs/smb/server/oplock.h @@ -66,6 +66,7 @@ struct oplock_info { struct ksmbd_file *o_fp; int level; int op_state; + spinlock_t state_lock; unsigned long pending_break; u64 fid; atomic_t breaking_cnt; @@ -96,9 +97,10 @@ struct oplock_break_info { int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid, struct ksmbd_file *fp, __u16 tid, - struct lease_ctx_info *lctx, int share_ret); + struct lease_ctx_info *lctx, int share_ret, bool replay); void smb_break_all_levII_oplock(struct ksmbd_work *work, - struct ksmbd_file *fp, int is_trunc); + struct ksmbd_file *fp, int is_trunc); +void smb_break_all_levII_oplock_rename(struct ksmbd_work *work, struct ksmbd_file *fp); void smb_break_all_levII_oplock_no_interim(struct ksmbd_work *work, struct ksmbd_file *fp, int is_trunc); void smb_break_all_levII_oplock_for_delete(struct ksmbd_work *work, @@ -125,6 +127,8 @@ void create_durable_v2_rsp_buf(char *cc, struct ksmbd_file *fp); void create_mxac_rsp_buf(char *cc, int maximal_access); void create_disk_id_rsp_buf(char *cc, __u64 file_id, __u64 vol_id); void create_posix_rsp_buf(char *cc, struct ksmbd_file *fp); +void create_aapl_rsp_buf(char *cc, __u64 vol_caps, __u64 req_bitmap, + bool readdir_attr_v2); struct create_context *smb2_find_context_vals(void *open_req, const char *tag, int tag_len); struct oplock_info *lookup_lease_in_table(struct ksmbd_conn *conn, char *lease_key); diff --git a/fs/smb/server/proc.c b/fs/smb/server/proc.c index 101a2cc45a44..826353ed0553 100644 --- a/fs/smb/server/proc.c +++ b/fs/smb/server/proc.c @@ -11,10 +11,12 @@ #include <linux/seq_file.h> #include "misc.h" +#include "connection.h" #include "server.h" #include "stats.h" #include "smb_common.h" #include "smb2pdu.h" +#include "vfs_cache.h" static struct proc_dir_entry *ksmbd_proc_fs; struct ksmbd_counters ksmbd_counters; @@ -27,6 +29,42 @@ struct proc_dir_entry *ksmbd_proc_create(const char *name, show, v); } +void ksmbd_proc_show_flag_names(struct seq_file *m, + const struct ksmbd_const_name *table, + int count, unsigned int flags) +{ + unsigned int remaining = flags; + bool separator = false; + int i; + + for (i = 0; i < count; i++) { + unsigned int flag = table[i].const_value; + + if (!flag || (remaining & flag) != flag) + continue; + seq_printf(m, "%s%s", separator ? "," : "", table[i].name); + separator = true; + remaining &= ~flag; + } + + if (remaining) + seq_printf(m, "%s0x%08x", separator ? "," : "", remaining); + else if (!separator) + seq_puts(m, "none"); +} + +const char *ksmbd_proc_const_name(const struct ksmbd_const_name *table, + int count, unsigned int const_value) +{ + int i; + + for (i = 0; i < count; i++) { + if (table[i].const_value == const_value) + return table[i].name; + } + return NULL; +} + struct ksmbd_const_smb2_process_req { unsigned int const_value; const char *name; @@ -54,32 +92,127 @@ static const struct ksmbd_const_smb2_process_req smb2_process_req[KSMBD_COUNTER_ {le16_to_cpu(SMB2_OPLOCK_BREAK), "SMB2_OPLOCK_BREAK"}, }; +static const char *ksmbd_server_state_string(void) +{ + switch (READ_ONCE(server_conf.state)) { + case SERVER_STATE_STARTING_UP: + return "starting"; + case SERVER_STATE_RUNNING: + return "running"; + case SERVER_STATE_RESETTING: + return "resetting"; + case SERVER_STATE_SHUTTING_DOWN: + return "shutdown"; + default: + return "unknown"; + } +} + +static const char *ksmbd_signing_mode_string(void) +{ + switch (server_conf.signing) { + case KSMBD_CONFIG_OPT_DISABLED: + return "disabled"; + case KSMBD_CONFIG_OPT_MANDATORY: + return "mandatory"; + case KSMBD_CONFIG_OPT_AUTO: + return "auto"; + default: + return "unknown"; + } +} + +static void proc_show_runtime_totals(struct seq_file *m) +{ + struct ksmbd_conn *conn; + unsigned int clients = 0; + unsigned int open_files = 0; + int i; + + down_read(&conn_list_lock); + hash_for_each(conn_list, i, conn, hlist) { + clients++; + open_files += atomic_read(&conn->stats.open_files_count); + } + up_read(&conn_list_lock); + + seq_printf(m, "clients:\t%u\n", clients); + seq_printf(m, "open_files:\t%u\n", open_files); +} + static int proc_show_ksmbd_stats(struct seq_file *m, void *v) { int i; seq_puts(m, "Server\n"); - seq_printf(m, "name: %s\n", ksmbd_server_string()); - seq_printf(m, "netbios: %s\n", ksmbd_netbios_name()); - seq_printf(m, "work group: %s\n", ksmbd_work_group()); - seq_printf(m, "min protocol: %s\n", ksmbd_get_protocol_string(server_conf.min_protocol)); - seq_printf(m, "max protocol: %s\n", ksmbd_get_protocol_string(server_conf.max_protocol)); - seq_printf(m, "flags: 0x%08x\n", server_conf.flags); - seq_printf(m, "share_fake_fscaps: 0x%08x\n", + seq_printf(m, "state:\t%s\n", ksmbd_server_state_string()); + seq_printf(m, "name:\t%s\n", ksmbd_server_string()); + seq_printf(m, "netbios:\t%s\n", ksmbd_netbios_name()); + seq_printf(m, "work_group:\t%s\n", ksmbd_work_group()); + seq_printf(m, "min_protocol:\t%s\n", ksmbd_get_protocol_string(server_conf.min_protocol)); + seq_printf(m, "max_protocol:\t%s\n", ksmbd_get_protocol_string(server_conf.max_protocol)); + seq_printf(m, "flags:\t0x%08x\n", server_conf.flags); + seq_printf(m, "tcp_port:\t%u\n", server_conf.tcp_port); + seq_printf(m, "signing:\t%s\n", ksmbd_signing_mode_string()); + seq_printf(m, "signing_enforced:\t%s\n", + server_conf.enforced_signing ? "yes" : "no"); + seq_printf(m, "bind_interfaces_only:\t%s\n", + server_conf.bind_interfaces_only ? "yes" : "no"); + seq_printf(m, "max_connections:\t%u\n", server_conf.max_connections); + seq_printf(m, "max_connections_per_ip:\t%u\n", + server_conf.max_ip_connections); + seq_printf(m, "max_inflight_requests:\t%u\n", + server_conf.max_inflight_req); + seq_printf(m, "deadtime_seconds:\t%lu\n", server_conf.deadtime / HZ); + seq_printf(m, "ipc_timeout_seconds:\t%u\n", server_conf.ipc_timeout / HZ); + if (server_conf.ipc_last_active) + seq_printf(m, "ipc_last_active_seconds:\t%lu\n", + jiffies_to_msecs(jiffies - server_conf.ipc_last_active) / + MSEC_PER_SEC); + else + seq_puts(m, "ipc_last_active_seconds:\tnever\n"); + seq_printf(m, "durable_scavenger:\t%s\n", + ksmbd_durable_scavenger_active() ? "running" : "stopped"); + seq_printf(m, "share_fake_fscaps:\t0x%08x\n", server_conf.share_fake_fscaps); - seq_printf(m, "sessions: %lld\n", + proc_show_runtime_totals(m); + seq_printf(m, "sessions:\t%lld\n", ksmbd_counter_sum(KSMBD_COUNTER_SESSIONS)); - seq_printf(m, "tree connects: %lld\n", + seq_printf(m, "tree_connects:\t%lld\n", ksmbd_counter_sum(KSMBD_COUNTER_TREE_CONNS)); - seq_printf(m, "read bytes: %lld\n", + seq_printf(m, "requests:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_REQUESTS)); + seq_printf(m, "read_bytes:\t%lld\n", ksmbd_counter_sum(KSMBD_COUNTER_READ_BYTES)); - seq_printf(m, "written bytes: %lld\n", + seq_printf(m, "written_bytes:\t%lld\n", ksmbd_counter_sum(KSMBD_COUNTER_WRITE_BYTES)); seq_puts(m, "\nSMB2\n"); for (i = 0; i < KSMBD_COUNTER_MAX_REQS; i++) - seq_printf(m, "%-20s:\t%lld\n", smb2_process_req[i].name, + seq_printf(m, "%s:\t%lld\n", smb2_process_req[i].name, ksmbd_counter_sum(KSMBD_COUNTER_FIRST_REQ + i)); + + seq_puts(m, "\nSMB2 status\n"); + seq_printf(m, "success:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_STATUS_SUCCESS)); + seq_printf(m, "informational:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_STATUS_INFORMATIONAL)); + seq_printf(m, "warning:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_STATUS_WARNING)); + seq_printf(m, "error:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_STATUS_ERROR)); + seq_printf(m, "access_denied:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_ERROR_ACCESS_DENIED)); + seq_printf(m, "not_found:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_ERROR_NOT_FOUND)); + seq_printf(m, "invalid_parameter:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_ERROR_INVALID_PARAMETER)); + seq_printf(m, "sharing_violation:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_ERROR_SHARING_VIOLATION)); + seq_printf(m, "not_supported:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_ERROR_NOT_SUPPORTED)); + seq_printf(m, "other:\t%lld\n", + ksmbd_counter_sum(KSMBD_COUNTER_ERROR_OTHER)); return 0; } @@ -106,14 +239,14 @@ void ksmbd_proc_reset(void) percpu_counter_set(&ksmbd_counters.counters[i], 0); } -void ksmbd_proc_init(void) +int ksmbd_proc_init(void) { int i; - int retval; + int retval = -ENOMEM; ksmbd_proc_fs = proc_mkdir("fs/ksmbd", NULL); if (!ksmbd_proc_fs) - return; + return retval; if (!proc_mkdir_mode("sessions", 0400, ksmbd_proc_fs)) goto err_out; @@ -124,11 +257,14 @@ void ksmbd_proc_init(void) goto err_out; } - if (!ksmbd_proc_create("server", proc_show_ksmbd_stats, NULL)) + if (!ksmbd_proc_create("server", proc_show_ksmbd_stats, NULL)) { + retval = -ENOMEM; goto err_out; + } ksmbd_proc_reset(); - return; + return 0; err_out: ksmbd_proc_cleanup(); + return retval; } diff --git a/fs/smb/server/server.c b/fs/smb/server/server.c index 960c4c897c11..0069d4e6a60a 100644 --- a/fs/smb/server/server.c +++ b/fs/smb/server/server.c @@ -15,6 +15,7 @@ #include "server.h" #include "smb_common.h" +#include "smb2pdu.h" #include "../common/smb2status.h" #include "connection.h" #include "transport_ipc.h" @@ -23,6 +24,8 @@ #include "auth.h" #include "stats.h" #include "compress.h" +#include "mgmt/share_config.h" +#include "mgmt/tree_connect.h" int ksmbd_debug_types; @@ -155,8 +158,11 @@ andx_again: } ret = cmds->proc(work); - if (conn->ops->inc_reqs) - conn->ops->inc_reqs(command); + if (conn->ops->inc_reqs) { + struct smb2_hdr *rsp = ksmbd_resp_buf_curr(work); + + conn->ops->inc_reqs(command, rsp->Status); + } if (ret < 0) ksmbd_debug(CONN, "Failed to process %u [%d]\n", command, ret); @@ -182,9 +188,33 @@ static void __handle_ksmbd_work(struct ksmbd_work *work, if (conn->ops->is_transform_hdr && conn->ops->is_transform_hdr(work->request_buf)) { rc = conn->ops->decrypt_req(work); - if (rc < 0) + if (rc < 0) { + ksmbd_conn_abort(conn); return; + } work->encrypted = true; + + /* + * SMB3 applies compression before encryption. The receive loop + * handles a plain compression transform before allocating work, but + * an encrypted request exposes that transform only after decryption. + */ + if (((struct smb2_hdr *)smb_get_msg(work->request_buf))->ProtocolId == + SMB2_COMPRESSION_TRANSFORM_ID) { + rc = ksmbd_decompress_work_request(work); + if (rc < 0) { + ksmbd_conn_abort(conn); + return; + } + } + + /* The decrypted payload must now be a complete SMB2 request. */ + if (((struct smb2_hdr *)smb_get_msg(work->request_buf))->ProtocolId != + SMB2_PROTO_NUMBER || + get_rfc1002_len(work->request_buf) < sizeof(struct smb2_pdu)) { + ksmbd_conn_abort(conn); + return; + } } if (conn->ops->allocate_rsp_buf(work)) @@ -204,6 +234,9 @@ static void __handle_ksmbd_work(struct ksmbd_work *work, if (rc == -EINVAL) conn->ops->set_rsp_status(work, STATUS_INVALID_PARAMETER); + else if (rc == -EKEYEXPIRED) + conn->ops->set_rsp_status(work, + STATUS_NETWORK_SESSION_EXPIRED); else conn->ops->set_rsp_status(work, STATUS_USER_SESSION_DELETED); @@ -211,7 +244,11 @@ static void __handle_ksmbd_work(struct ksmbd_work *work, struct smb2_hdr *rsp_hdr; rsp_hdr = ksmbd_resp_buf_curr(work); - rsp_hdr->Flags |= SMB2_FLAGS_SIGNED; + if (rc == -EKEYEXPIRED && work->sess && + conn->ops->set_sign_rsp) + conn->ops->set_sign_rsp(work); + else + rsp_hdr->Flags |= SMB2_FLAGS_SIGNED; } goto send; } else if (rc > 0) { @@ -225,12 +262,23 @@ static void __handle_ksmbd_work(struct ksmbd_work *work, STATUS_NETWORK_NAME_DELETED); goto send; } + + if (work->tcon && + test_share_config_flag(work->tcon->share_conf, + KSMBD_SHARE_FLAG_ENCRYPT_DATA) && + !work->encrypted) { + conn->ops->set_rsp_status(work, + STATUS_ACCESS_DENIED); + goto send; + } } } rc = __process_request(work, conn, &command); - if (rc == SERVER_HANDLER_ABORT) + if (rc == SERVER_HANDLER_ABORT) { + smb2_complete_request_open(work); break; + } /* * Call smb2_set_rsp_credits() function to set number of credits @@ -243,10 +291,13 @@ static void __handle_ksmbd_work(struct ksmbd_work *work, if (rc < 0) { conn->ops->set_rsp_status(work, STATUS_INVALID_PARAMETER); + smb2_complete_request_open(work); goto send; } } + smb2_complete_request_open(work); + is_chained = is_chained_smb2_message(work); if (work->sess && @@ -262,6 +313,7 @@ static void __handle_ksmbd_work(struct ksmbd_work *work, } while (is_chained == true); send: + smb2_complete_request_open(work); /* * Release any credit charge still outstanding for this request. On * the normal path smb2_set_rsp_credits() already returned it, but the @@ -579,11 +631,16 @@ static int ksmbd_server_shutdown(void) { WRITE_ONCE(server_conf.state, SERVER_STATE_SHUTTING_DOWN); - ksmbd_proc_cleanup(); class_unregister(&ksmbd_control_class); ksmbd_workqueue_destroy(); ksmbd_ipc_release(); ksmbd_conn_transport_destroy(); + /* + * ksmbd_conn_transport_destroy() calls delete_proc_clients() and destroys + * sessions. ksmbd_session_destroy() removes each session's proc entry. + * Keep the procfs tree alive until these entries have been removed. + */ + ksmbd_proc_cleanup(); ksmbd_crypto_destroy(); ksmbd_free_global_file_table(); destroy_lease_table(NULL); @@ -603,18 +660,25 @@ static int __init ksmbd_server_init(void) return ret; } - ksmbd_proc_init(); - create_proc_sessions(); + ret = ksmbd_proc_init(); + if (ret) + goto err_unregister; + + if (create_proc_sessions()) + pr_warn("Unable to create sessions procfs entry\n"); + + if (create_proc_shares()) + pr_warn("Unable to create shares procfs entry\n"); ksmbd_server_tcp_callbacks_init(); ret = server_conf_init(); if (ret) - goto err_unregister; + goto err_proc_cleanup; ret = ksmbd_work_pool_init(); if (ret) - goto err_unregister; + goto err_proc_cleanup; ret = ksmbd_init_file_cache(); if (ret) @@ -660,6 +724,8 @@ err_exit_file_cache: ksmbd_exit_file_cache(); err_destroy_work_pools: ksmbd_work_pool_destroy(); +err_proc_cleanup: + ksmbd_proc_cleanup(); err_unregister: class_unregister(&ksmbd_control_class); diff --git a/fs/smb/server/server.h b/fs/smb/server/server.h index b8a7317be86b..4d4d268b59d5 100644 --- a/fs/smb/server/server.h +++ b/fs/smb/server/server.h @@ -48,6 +48,8 @@ struct ksmbd_server_config { char *conf[SERVER_CONF_WORK_GROUP + 1]; struct task_struct *dh_task; bool bind_interfaces_only; + /* AAPL model string for Finder icon, e.g. "Xserve" */ + char aapl_model[32]; }; extern struct ksmbd_server_config server_conf; diff --git a/fs/smb/server/smb2misc.c b/fs/smb/server/smb2misc.c index 9f3629c86291..532dea7be0b3 100644 --- a/fs/smb/server/smb2misc.c +++ b/fs/smb/server/smb2misc.c @@ -372,6 +372,75 @@ static int smb2_validate_credit_charge(struct ksmbd_work *work, return ret; } +/* + * Verify that the sequence number(s) consumed by an incoming request fall + * within the connection's command sequence window and are not a replay, then + * remove them from the window. Returns 0 if the request + * may proceed, or 1 if it is invalid and the connection must be torn down. + */ +static int smb2_check_sequence_number(struct ksmbd_work *work, + struct smb2_hdr *hdr) +{ + struct ksmbd_conn *conn = work->conn; + u64 mid = le64_to_cpu(hdr->MessageId); + unsigned short charge; + u64 i; + int ret = 0; + + /* An SMB2 CANCEL consumes no sequence number. */ + if (hdr->Command == SMB2_CANCEL) + return 0; + + /* + * A multi-credit request consumes CreditCharge consecutive sequence + * numbers; every other request consumes exactly one. + */ + charge = le16_to_cpu(hdr->CreditCharge); + if (!(conn->vals->req_capabilities & SMB2_GLOBAL_CAP_LARGE_MTU) || + charge == 0) + charge = 1; + + /* The 64-bit sequence number space must not wrap. */ + if (mid + charge < mid) { + pr_err("SMB2 sequence number wrapped (mid %llu charge %u)\n", + mid, charge); + return 1; + } + + spin_lock(&conn->credits_lock); + + /* The whole range must lie within the granted window... */ + if (mid < conn->seq_low || mid + charge > conn->seq_high) { + ksmbd_debug(SMB, + "MessageId %llu (charge %u) outside command window [%llu, %llu)\n", + mid, charge, conn->seq_low, conn->seq_high); + ret = 1; + goto out; + } + + /* ...and none of it may have been consumed already (replay). */ + for (i = mid; i < mid + charge; i++) { + if (!test_bit(i & (KSMBD_CMD_SEQ_WINDOW - 1), conn->seq_bitmap)) { + ksmbd_debug(SMB, + "replayed sequence number %llu (mid %llu charge %u)\n", + i, mid, charge); + ret = 1; + goto out; + } + } + + /* Consume the sequence numbers and slide the low edge forward. */ + for (i = mid; i < mid + charge; i++) + __clear_bit(i & (KSMBD_CMD_SEQ_WINDOW - 1), conn->seq_bitmap); + while (conn->seq_low < conn->seq_high && + !test_bit(conn->seq_low & (KSMBD_CMD_SEQ_WINDOW - 1), + conn->seq_bitmap)) + conn->seq_low++; +out: + spin_unlock(&conn->credits_lock); + return ret; +} + int ksmbd_smb2_check_message(struct ksmbd_work *work) { struct smb2_pdu *pdu = ksmbd_req_buf_next(work); @@ -476,6 +545,16 @@ validate_credit: smb2_validate_credit_charge(work, hdr)) return 1; + /* + * A sequence number violation (out of window or a replay) is a + * protocol error. tear the connection down rather than + * keep accepting requests on it. + */ + if (smb2_check_sequence_number(work, hdr)) { + ksmbd_conn_set_exiting(work->conn); + return 1; + } + return 0; } diff --git a/fs/smb/server/smb2ops.c b/fs/smb/server/smb2ops.c index c9a32ee096b5..4578291fb172 100644 --- a/fs/smb/server/smb2ops.c +++ b/fs/smb/server/smb2ops.c @@ -37,6 +37,7 @@ static struct smb_version_values smb21_server_values = { .create_mxac_size = sizeof(struct create_mxac_rsp), .create_disk_id_size = sizeof(struct create_disk_id_rsp), .create_posix_size = sizeof(struct create_posix_rsp), + .create_aapl_size = AAPL_RSP_MAX_SIZE, }; static struct smb_version_values smb30_server_values = { @@ -64,6 +65,7 @@ static struct smb_version_values smb30_server_values = { .create_mxac_size = sizeof(struct create_mxac_rsp), .create_disk_id_size = sizeof(struct create_disk_id_rsp), .create_posix_size = sizeof(struct create_posix_rsp), + .create_aapl_size = AAPL_RSP_MAX_SIZE, }; static struct smb_version_values smb302_server_values = { @@ -91,6 +93,7 @@ static struct smb_version_values smb302_server_values = { .create_mxac_size = sizeof(struct create_mxac_rsp), .create_disk_id_size = sizeof(struct create_disk_id_rsp), .create_posix_size = sizeof(struct create_posix_rsp), + .create_aapl_size = AAPL_RSP_MAX_SIZE, }; static struct smb_version_values smb311_server_values = { @@ -118,6 +121,7 @@ static struct smb_version_values smb311_server_values = { .create_mxac_size = sizeof(struct create_mxac_rsp), .create_disk_id_size = sizeof(struct create_disk_id_rsp), .create_posix_size = sizeof(struct create_posix_rsp), + .create_aapl_size = AAPL_RSP_MAX_SIZE, }; static struct smb_version_ops smb2_0_server_ops = { @@ -266,8 +270,10 @@ void init_smb3_02_server(struct ksmbd_conn *conn) if (server_conf.flags & KSMBD_GLOBAL_FLAG_SMB3_MULTICHANNEL) conn->vals->req_capabilities |= SMB2_GLOBAL_CAP_MULTI_CHANNEL; - if (server_conf.flags & KSMBD_GLOBAL_FLAG_DURABLE_HANDLE) - conn->vals->req_capabilities |= SMB2_GLOBAL_CAP_PERSISTENT_HANDLES; + /* + * Durable handles are in-memory only. Do not advertise persistent + * handles until CA recovery and fencing are implemented. + */ } /** @@ -290,10 +296,7 @@ int init_smb3_11_server(struct ksmbd_conn *conn) if (server_conf.flags & KSMBD_GLOBAL_FLAG_SMB3_MULTICHANNEL) conn->vals->req_capabilities |= SMB2_GLOBAL_CAP_MULTI_CHANNEL; - if (server_conf.flags & KSMBD_GLOBAL_FLAG_DURABLE_HANDLE) - conn->vals->req_capabilities |= SMB2_GLOBAL_CAP_PERSISTENT_HANDLES; - - INIT_LIST_HEAD(&conn->preauth_sess_table); + /* See init_smb3_02_server(): persistent handles require CA recovery. */ return 0; } @@ -326,6 +329,13 @@ void init_smb2_max_trans_size(unsigned int sz) void init_smb2_max_credits(unsigned int sz) { + /* + * The command sequence window (and its backing bitmap) can track at + * most SMB2_MAX_CREDITS outstanding sequence numbers, so the number of + * credits granted on a connection must not exceed that. + */ + if (sz > SMB2_MAX_CREDITS) + sz = SMB2_MAX_CREDITS; smb21_server_values.max_credits = sz; smb30_server_values.max_credits = sz; smb302_server_values.max_credits = sz; diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c index d092709734e2..a8046f477d54 100644 --- a/fs/smb/server/smb2pdu.c +++ b/fs/smb/server/smb2pdu.c @@ -16,6 +16,8 @@ #include <linux/mount.h> #include <linux/filelock.h> #include <linux/fileattr.h> +#include <linux/timekeeping.h> +#include <linux/unaligned.h> #include "glob.h" #include "../common/smbfsctl.h" @@ -56,6 +58,10 @@ static void __wbuf(struct ksmbd_work *work, void **req, void **rsp) } } +static struct ksmbd_work *smb2_notify_cancel_claim(void **argv); +static void smb2_notify_cancel_fn(void **argv); +static void smb2_complete_notify_cancel(struct ksmbd_work *in_work); + #define WORK_BUFFERS(w, rq, rs) __wbuf((w), (void **)&(rq), (void **)&(rs)) #define SMB2_CREATE_FILE_ATTRIBUTE_MASK \ @@ -65,28 +71,8 @@ static void __wbuf(struct ksmbd_work *work, void **req, void **rsp) /* Windows reports automatic write-time updates at roughly 15 ms resolution. */ #define KSMBD_WRITE_TIME_RESOLUTION (15ULL * 10000) -/** - * check_session_id() - check for valid session id in smb header - * @conn: connection instance - * @id: session id from smb header - * - * Return: 1 if valid session id, otherwise 0 - */ -static inline bool check_session_id(struct ksmbd_conn *conn, u64 id) -{ - struct ksmbd_session *sess; - - if (id == 0 || id == -1) - return false; - - sess = ksmbd_session_lookup_all(conn, id); - if (sess) { - ksmbd_user_session_put(sess); - return true; - } - pr_err("Invalid user session id: %llu\n", id); - return false; -} +/* MAXFILESIZE in [MS-FSA] 2.1.5.3 Server Requests a Write. */ +#define SMB2_MAX_FILE_SIZE 0xfffffff0000ULL struct channel *lookup_chann_list(struct ksmbd_session *sess, struct ksmbd_conn *conn) { @@ -386,6 +372,7 @@ int smb2_set_rsp_credits(struct ksmbd_work *work) struct ksmbd_conn *conn = work->conn; unsigned short credits_requested, aux_max; unsigned short credit_charge, credits_granted = 0; + u64 window_room, i; if (work->send_no_response) return 0; @@ -422,11 +409,26 @@ int smb2_set_rsp_credits(struct ksmbd_work *work) aux_max = 1; else aux_max = conn->vals->max_credits - conn->total_credits; + + /* + * The command sequence window must not grow beyond + * KSMBD_CMD_SEQ_WINDOW sequence numbers ahead of the oldest one still + * outstanding. Cap the grant by the room left in the window so that + * credits are withheld until the client consumes the low end (and so + * that seq_bitmap stays usable as a ring). + */ + window_room = conn->seq_low + KSMBD_CMD_SEQ_WINDOW - conn->seq_high; + aux_max = min_t(unsigned short, aux_max, window_room); credits_granted = min_t(unsigned short, credits_requested, aux_max); conn->total_credits += credits_granted; work->credits_granted += credits_granted; + /* Extend the sequence window to cover the newly granted credits. */ + for (i = conn->seq_high; i < conn->seq_high + credits_granted; i++) + __set_bit(i & (KSMBD_CMD_SEQ_WINDOW - 1), conn->seq_bitmap); + conn->seq_high += credits_granted; + if (!req_hdr->NextCommand) { /* Update CreditRequest in last request */ hdr->CreditRequest = cpu_to_le16(work->credits_granted); @@ -552,6 +554,8 @@ static void init_chained_smb2_rsp(struct ksmbd_work *work) */ rsp_hdr->Flags = (SMB2_FLAGS_SERVER_TO_REDIR | SMB2_FLAGS_RELATED_OPERATIONS); + if (rcv_hdr->Flags & SMB2_FLAGS_REPLAY_OPERATION) + rsp_hdr->Flags |= SMB2_FLAGS_REPLAY_OPERATION; rsp_hdr->NextCommand = 0; rsp_hdr->MessageId = rcv_hdr->MessageId; rsp_hdr->Id.SyncId.ProcessId = rcv_hdr->Id.SyncId.ProcessId; @@ -644,6 +648,8 @@ int init_smb2_rsp_hdr(struct ksmbd_work *work) * Message is response. We don't grant oplock yet. */ rsp_hdr->Flags = (SMB2_FLAGS_SERVER_TO_REDIR); + if (rcv_hdr->Flags & SMB2_FLAGS_REPLAY_OPERATION) + rsp_hdr->Flags |= SMB2_FLAGS_REPLAY_OPERATION; rsp_hdr->NextCommand = 0; rsp_hdr->MessageId = rcv_hdr->MessageId; rsp_hdr->Id.SyncId.ProcessId = rcv_hdr->Id.SyncId.ProcessId; @@ -654,6 +660,192 @@ int init_smb2_rsp_hdr(struct ksmbd_work *work) return 0; } +static __le16 smb3_hdr_channel_sequence(struct smb2_hdr *hdr) +{ + return ((struct smb3_hdr_req *)hdr)->ChannelSequence; +} + +static bool smb3_hdr_replay(struct smb2_hdr *hdr) +{ + return hdr->Flags & SMB2_FLAGS_REPLAY_OPERATION; +} + +static int smb3_verify_channel_sequence(struct ksmbd_work *work, + struct ksmbd_file *fp, + struct smb2_hdr *hdr, + bool allow_stale) +{ + __le16 chseq_le; + u16 chseq, old_chseq; + int ret = 0; + + if (work->conn->dialect < SMB30_PROT_ID) + return 0; + + chseq_le = smb3_hdr_channel_sequence(hdr); + chseq = le16_to_cpu(chseq_le); + + spin_lock(&fp->f_lock); + old_chseq = le16_to_cpu(fp->channel_sequence); + if (smb3_hdr_replay(hdr)) { + if (chseq == old_chseq && fp->outstanding_pre_requests == 0) { + fp->outstanding_requests++; + } else if ((u16)(chseq - old_chseq) <= 0x7fff && + fp->outstanding_pre_requests == 0) { + fp->outstanding_pre_requests += fp->outstanding_requests; + fp->outstanding_requests = 1; + fp->channel_sequence = chseq_le; + } else if (allow_stale) { + fp->outstanding_pre_requests++; + } else { + ret = -EAGAIN; + } + } else { + if (chseq == old_chseq) { + fp->outstanding_requests++; + } else if ((u16)(chseq - old_chseq) <= 0x7fff) { + fp->outstanding_pre_requests += fp->outstanding_requests; + fp->outstanding_requests = 1; + fp->channel_sequence = chseq_le; + } else if (allow_stale) { + fp->outstanding_pre_requests++; + } else { + ret = -EAGAIN; + } + } + spin_unlock(&fp->f_lock); + + return ret; +} + +static void smb3_complete_channel_sequence(struct ksmbd_work *work, + struct ksmbd_file *fp, + __le16 chseq_le) +{ + u16 chseq; + + if (work->conn->dialect < SMB30_PROT_ID) + return; + + chseq = le16_to_cpu(chseq_le); + + spin_lock(&fp->f_lock); + if (chseq == le16_to_cpu(fp->channel_sequence)) { + if (fp->outstanding_requests) + fp->outstanding_requests--; + } else { + if (fp->outstanding_pre_requests) + fp->outstanding_pre_requests--; + } + spin_unlock(&fp->f_lock); +} + +static int smb2_set_request_open(struct ksmbd_work *work, struct ksmbd_file *fp, + struct smb2_hdr *hdr, bool verify_chseq, + bool allow_stale_chseq) +{ + struct ksmbd_file *open; + int ret; + + smb2_complete_request_open(work); + + open = ksmbd_file_get(fp); + if (!open) + return -ESTALE; + + if (verify_chseq) { + ret = smb3_verify_channel_sequence(work, fp, hdr, + allow_stale_chseq); + if (ret) { + ksmbd_fd_put(work, open); + return ret; + } + work->request_open_chseq_tracked = true; + } + + work->request_open = open; + work->request_open_chseq = smb3_hdr_channel_sequence(hdr); + return 0; +} + +void smb2_complete_request_open(struct ksmbd_work *work) +{ + struct ksmbd_file *open = work->request_open; + + if (!open) + return; + + if (work->request_open_chseq_tracked) + smb3_complete_channel_sequence(work, open, + work->request_open_chseq); + + work->request_open = NULL; + work->request_open_chseq_tracked = false; + ksmbd_fd_put(work, open); +} + +static bool smb2_lock_sequence_applicable(struct ksmbd_work *work, + struct ksmbd_file *fp) +{ + return fp->is_resilient || fp->is_durable || fp->is_persistent || + (work->conn->dialect >= SMB30_PROT_ID && + (work->conn->vals->req_capabilities & + SMB2_GLOBAL_CAP_MULTI_CHANNEL)); +} + +static bool smb2_verify_lock_sequence(struct ksmbd_work *work, + struct ksmbd_file *fp, + struct smb2_lock_req *req) +{ + u32 val, index; + u8 sequence; + bool replay = false; + + if (work->conn->dialect == SMB20_PROT_ID || + !smb2_lock_sequence_applicable(work, fp)) + return false; + + val = le32_to_cpu(req->LockSequenceNumber); + sequence = val & 0xf; + index = val >> 4; + if (!index || index > KSMBD_LOCK_SEQ_ARRAY_SIZE) + return false; + + spin_lock(&fp->f_lock); + if (fp->lock_seq[index - 1].valid) { + if (fp->lock_seq[index - 1].sequence == sequence) + replay = true; + else + fp->lock_seq[index - 1].valid = false; + } + spin_unlock(&fp->f_lock); + + return replay; +} + +static void smb2_update_lock_sequence(struct ksmbd_work *work, + struct ksmbd_file *fp, + struct smb2_lock_req *req) +{ + u32 val, index; + u8 sequence; + + if (work->conn->dialect == SMB20_PROT_ID || + !smb2_lock_sequence_applicable(work, fp)) + return; + + val = le32_to_cpu(req->LockSequenceNumber); + sequence = val & 0xf; + index = val >> 4; + if (!index || index > KSMBD_LOCK_SEQ_ARRAY_SIZE) + return; + + spin_lock(&fp->f_lock); + fp->lock_seq[index - 1].valid = true; + fp->lock_seq[index - 1].sequence = sequence; + spin_unlock(&fp->f_lock); +} + /** * smb2_allocate_rsp_buf() - allocate smb2 response buffer * @work: smb work containing smb request buffer @@ -698,6 +890,47 @@ int smb2_allocate_rsp_buf(struct ksmbd_work *work) return 0; } +static bool smb2_session_expired_cmd_allowed(struct ksmbd_work *work, + unsigned int cmd) +{ + struct smb2_lock_req *req; + unsigned int len, lock_count, i; + + if (cmd == SMB2_CANCEL_HE || cmd == SMB2_CLOSE_HE || + cmd == SMB2_LOGOFF_HE) + return true; + if (cmd != SMB2_LOCK_HE) + return false; + + req = ksmbd_req_buf_next(work); + if (req->hdr.NextCommand) + len = le32_to_cpu(req->hdr.NextCommand); + else { + len = get_rfc1002_len(work->request_buf); + if (len < work->next_smb2_rcv_hdr_off) + return false; + len -= work->next_smb2_rcv_hdr_off; + } + + lock_count = le16_to_cpu(req->LockCount); + if (!lock_count || len < offsetof(struct smb2_lock_req, locks) || + lock_count > (len - offsetof(struct smb2_lock_req, locks)) / + sizeof(struct smb2_lock_element)) + return false; + + for (i = 0; i < lock_count; i++) { + if (le32_to_cpu(req->locks[i].Flags) != SMB2_LOCKFLAG_UNLOCK) + return false; + } + return true; +} + +static bool smb2_session_kerberos_expired(struct ksmbd_session *sess) +{ + return sess->kerberos_expiry && + ktime_get_real_seconds() >= sess->kerberos_expiry; +} + /** * smb2_check_user_session() - check for valid session for a user * @work: smb work containing smb request buffer @@ -712,19 +945,37 @@ int smb2_check_user_session(struct ksmbd_work *work) unsigned long long sess_id; /* - * SMB2_ECHO, SMB2_NEGOTIATE, SMB2_SESSION_SETUP command do not - * require a session id, so no need to validate user session's for - * these commands. + * SMB2_NEGOTIATE and SMB2_SESSION_SETUP do not require a session id. + * SMB2_ECHO may omit it, but an echo carrying a session id still needs + * the session attached to work so that its signature can be checked and + * the response can be signed, including after Kerberos expiry. */ - if (cmd == SMB2_ECHO_HE || cmd == SMB2_NEGOTIATE_HE || - cmd == SMB2_SESSION_SETUP_HE) + if (cmd == SMB2_NEGOTIATE_HE || cmd == SMB2_SESSION_SETUP_HE) + return 0; + + sess_id = le64_to_cpu(req_hdr->SessionId); + if (cmd == SMB2_ECHO_HE) { + /* + * ECHO remains valid without a live session, including after + * LOGOFF. Attach an existing session only to authenticate a signed + * ECHO and sign its response; a stale SessionId is not an error. + */ + if (!work->next_smb2_rcv_hdr_off && sess_id) + work->sess = ksmbd_session_lookup_all_states(conn, sess_id); + if (work->sess) { + if (smb2_session_kerberos_expired(work->sess)) { + work->sess->state = SMB2_SESSION_EXPIRED; + } else if (work->sess->state != SMB2_SESSION_VALID) { + ksmbd_user_session_put(work->sess); + work->sess = NULL; + } + } return 0; + } if (!ksmbd_conn_good(conn)) return -EIO; - sess_id = le64_to_cpu(req_hdr->SessionId); - /* * If request is not the first in Compound request, * Just validate session id in header with work->sess->id. @@ -739,18 +990,43 @@ int smb2_check_user_session(struct ksmbd_work *work) sess_id, work->sess->id); return -EINVAL; } + if (smb2_session_kerberos_expired(work->sess)) + work->sess->state = SMB2_SESSION_EXPIRED; if (work->sess->state != SMB2_SESSION_VALID) { pr_err("compound request on a non-valid session (state %d)\n", work->sess->state); - return -EINVAL; + if (smb2_session_kerberos_expired(work->sess) && + smb2_session_expired_cmd_allowed(work, cmd)) + return 1; + return smb2_session_kerberos_expired(work->sess) ? + -EKEYEXPIRED : -EINVAL; } return 1; } /* Check for validity of user session */ - work->sess = ksmbd_session_lookup_all(conn, sess_id); - if (work->sess) + work->sess = ksmbd_session_lookup_all_states(conn, sess_id); + if (work->sess) { + if (smb2_session_kerberos_expired(work->sess)) { + work->sess->state = SMB2_SESSION_EXPIRED; + return smb2_session_expired_cmd_allowed(work, cmd) ? + 1 : -EKEYEXPIRED; + } + if (work->sess->state != SMB2_SESSION_VALID) { + /* + * Keep the reference for an encrypted request so the caller can + * return STATUS_USER_SESSION_DELETED encrypted with the old key. + */ + if (work->encrypted && + work->sess->state == SMB2_SESSION_EXPIRED && + work->sess->enc) + return -ENOENT; + ksmbd_user_session_put(work->sess); + work->sess = NULL; + return -ENOENT; + } return 1; + } ksmbd_debug(SMB, "Invalid user session, Uid %llu\n", sess_id); return -ENOENT; } @@ -790,6 +1066,24 @@ smb2_get_name(const char *src, const int maxlen, struct nls_table *local_nls) return name; } +/* Link a fully initialized async work item unless the connection is closing. */ +static bool ksmbd_conn_link_async_request(struct ksmbd_conn *conn, + struct ksmbd_work *work) +{ + bool linked = false; + + spin_lock(&conn->request_lock); + if (!ksmbd_conn_exiting(conn) && !ksmbd_conn_releasing(conn)) { + if (list_empty(&work->async_request_entry)) + list_add_tail(&work->async_request_entry, + &conn->async_requests); + linked = true; + } + spin_unlock(&conn->request_lock); + + return linked; +} + int setup_async_work(struct ksmbd_work *work, void (*fn)(void **), void **arg) { struct ksmbd_conn *conn = work->conn; @@ -802,20 +1096,22 @@ int setup_async_work(struct ksmbd_work *work, void (*fn)(void **), void **arg) } work->asynchronous = true; work->async_id = id; - - ksmbd_debug(SMB, - "Send interim Response to inform async request id : %d\n", - work->async_id); - work->cancel_fn = fn; work->cancel_argv = arg; - if (list_empty(&work->async_request_entry)) { - spin_lock(&conn->request_lock); - list_add_tail(&work->async_request_entry, &conn->async_requests); - spin_unlock(&conn->request_lock); + if (!ksmbd_conn_link_async_request(conn, work)) { + work->asynchronous = false; + work->async_id = 0; + work->cancel_fn = NULL; + work->cancel_argv = NULL; + ksmbd_release_id(&conn->async_ida, id); + return -ESHUTDOWN; } + ksmbd_debug(SMB, + "Send interim Response to inform async request id : %d\n", + work->async_id); + return 0; } @@ -837,6 +1133,108 @@ void release_async_work(struct ksmbd_work *work) } } +static int smb2_send_interim_work(struct ksmbd_work *in_work, + struct ksmbd_work *work, bool eor) +{ + int err = 0; + + in_work->encrypted = work->encrypted; + if (work->encrypted && work->sess && work->sess->enc && + work->conn->ops->encrypt_resp) { + in_work->sess = work->sess; + err = work->conn->ops->encrypt_resp(in_work); + in_work->sess = NULL; + } + if (err) + return err; + + return eor ? ksmbd_conn_write_eor(in_work) : + ksmbd_conn_write(in_work); +} + +static int smb2_send_interim_prefix_work(struct ksmbd_work *work) +{ + struct ksmbd_work *in_work; + unsigned int len, copied = 0; + char *dst; + int err = -ENOMEM; + int i; + + len = get_rfc1002_len(work->iov[0].iov_base); + in_work = ksmbd_alloc_work_struct(); + if (!in_work) + return err; + + in_work->response_buf = kvzalloc(len + 4, KSMBD_DEFAULT_GFP); + if (!in_work->response_buf) + goto out; + in_work->response_sz = len + 4; + in_work->conn = work->conn; + dst = in_work->response_buf + 4; + for (i = 1; i <= work->iov_idx; i++) { + if (work->iov[i].iov_len > len - copied) { + err = -EINVAL; + goto out; + } + memcpy(dst + copied, work->iov[i].iov_base, + work->iov[i].iov_len); + copied += work->iov[i].iov_len; + } + if (copied != len) { + err = -EINVAL; + goto out; + } + + err = ksmbd_iov_pin_rsp(in_work, dst, len); + if (!err) + err = smb2_send_interim_work(in_work, work, true); +out: + ksmbd_free_work_struct(in_work); + return err; +} + +static void smb2_send_interim_compound_prefix(struct ksmbd_work *work) +{ + struct smb2_hdr *req_hdr; + struct smb2_hdr *rsp_hdr; + int err; + + if (!work->next_smb2_rcv_hdr_off || + !work->next_smb2_rsp_hdr_off || + work->curr_smb2_rsp_hdr_off == work->next_smb2_rsp_hdr_off || + !work->iov_idx) + return; + + req_hdr = ksmbd_req_buf_next(work); + /* Detach only the final async command from the completed prefix. */ + if (req_hdr->NextCommand) + return; + + /* + * The responses before the async command are sent as a standalone + * compound response. The last response in this prefix must terminate + * the chain. + */ + rsp_hdr = ksmbd_resp_buf_curr(work); + rsp_hdr->NextCommand = 0; + if ((rsp_hdr->Flags & SMB2_FLAGS_SIGNED) && work->sess && + work->conn->ops->set_sign_rsp) + work->conn->ops->set_sign_rsp(work); + + err = smb2_send_interim_prefix_work(work); + if (err) + ksmbd_debug(SMB, "failed to send compound interim prefix: %d\n", + err); + + work->iov_idx = 0; + work->iov_cnt = 0; + work->curr_smb2_rsp_hdr_off = work->next_smb2_rsp_hdr_off; + *(__be32 *)work->response_buf = 0; + + rsp_hdr = ksmbd_resp_buf_next(work); + rsp_hdr->Flags &= ~SMB2_FLAGS_RELATED_OPERATIONS; +} + void smb2_send_interim_resp(struct ksmbd_work *work, __le32 status) { struct smb2_hdr *rsp_hdr; @@ -851,6 +1249,9 @@ void smb2_send_interim_resp(struct ksmbd_work *work, __le32 status) return; } + if (status == STATUS_PENDING) + smb2_send_interim_compound_prefix(work); + in_work->conn = work->conn; memcpy(smb_get_msg(in_work->response_buf), ksmbd_resp_buf_next(work), __SMB2_HEADER_STRUCTURE_SIZE); @@ -861,7 +1262,8 @@ void smb2_send_interim_resp(struct ksmbd_work *work, __le32 status) smb2_set_err_rsp(in_work); rsp_hdr->Status = status; - ksmbd_conn_write(in_work); + if (smb2_send_interim_work(in_work, work, true)) + ksmbd_debug(SMB, "failed to send interim response\n"); ksmbd_free_work_struct(in_work); } @@ -901,9 +1303,6 @@ static int smb2_get_dos_mode(struct kstat *stat, int attribute) } else { attr = (attribute & 0x00005137) | FILE_ATTRIBUTE_ARCHIVE; attr &= ~(FILE_ATTRIBUTE_DIRECTORY); - if (S_ISREG(stat->mode) && (server_conf.share_fake_fscaps & - FILE_SUPPORTS_SPARSE_FILES)) - attr |= FILE_ATTRIBUTE_SPARSE_FILE; if (smb2_get_reparse_tag_special_file(stat->mode)) attr |= FILE_ATTRIBUTE_REPARSE_POINT; @@ -958,6 +1357,37 @@ static void build_compress_ctxt(struct smb2_compression_capabilities_context *pn pneg_ctxt->CompressionAlgorithms[3] = 0; } +/** + * build_rdma_ctx() - build an RDMA transform negotiate response context + * @ctxt: response context header to populate + * @transform_ids: bitmap of transforms common to the client and server + * + * Return: encoded negotiate context length + */ +static int build_rdma_ctx(struct smb2_neg_context *ctxt, + unsigned long transform_ids) +{ + struct smb2_rdma_transform_capabilities_context *pneg_ctxt; + int count = 0; + + pneg_ctxt = (void *)ctxt; + pneg_ctxt->ContextType = SMB2_RDMA_TRANSFORM_CAPABILITIES; + pneg_ctxt->Reserved = 0; + pneg_ctxt->Reserved1 = 0; + pneg_ctxt->Reserved2 = 0; + if (transform_ids & BIT(SMB2_RDMA_TRANSFORM_ENCRYPTION)) + pneg_ctxt->RDMATransformIds[count++] = + cpu_to_le16(SMB2_RDMA_TRANSFORM_ENCRYPTION); + if (!count) + pneg_ctxt->RDMATransformIds[count++] = + cpu_to_le16(SMB2_RDMA_TRANSFORM_NONE); + + pneg_ctxt->TransformCount = cpu_to_le16(count); + pneg_ctxt->DataLength = cpu_to_le16(8 + count * sizeof(__le16)); + return sizeof(struct smb2_neg_context) + + le16_to_cpu(pneg_ctxt->DataLength); +} + static void build_sign_cap_ctxt(struct smb2_signing_capabilities *pneg_ctxt, __le16 sign_algo) { @@ -1033,6 +1463,18 @@ static unsigned int assemble_neg_contexts(struct ksmbd_conn *conn, (conn->compress_pattern ? 12 : 10); } + if (conn->rdma_transform_negotiated) { + struct smb2_neg_context *rdma_ctxt; + + ctxt_size = round_up(ctxt_size, 8); + ksmbd_debug(SMB, + "assemble SMB2_RDMA_TRANSFORM_CAPABILITIES context\n"); + rdma_ctxt = (void *)(pneg_ctxt + ctxt_size); + ctxt_size += build_rdma_ctx(rdma_ctxt, + conn->rdma_transform_ids); + neg_ctxt_cnt++; + } + if (conn->posix_ext_supported) { ctxt_size = round_up(ctxt_size, 8); ksmbd_debug(SMB, @@ -1233,6 +1675,46 @@ static void decode_sign_cap_ctxt(struct ksmbd_conn *conn, } } +/** + * decode_rdma_ctx() - decode an RDMA transform negotiate request context + * @conn: connection being negotiated + * @ctxt: request context header to decode + * @ctxt_len: total context length, including the negotiate context header + * + * Record transforms supported by both peers only for SMB Direct connections. + * + * Return: NT status describing the decode result + */ +static __le32 decode_rdma_ctx(struct ksmbd_conn *conn, + struct smb2_neg_context *ctxt, int ctxt_len) +{ + struct smb2_rdma_transform_capabilities_context *pneg_ctxt; + unsigned int count, i; + + pneg_ctxt = (void *)ctxt; + /* RDMA transforms are a node capability, not just a transport capability. */ + if (!ksmbd_rdma_enabled()) + return STATUS_SUCCESS; + + if (ctxt_len < sizeof(*pneg_ctxt)) + return STATUS_INVALID_PARAMETER; + + count = le16_to_cpu(pneg_ctxt->TransformCount); + if (!count || count > + (ctxt_len - sizeof(*pneg_ctxt)) / sizeof(__le16)) + return STATUS_INVALID_PARAMETER; + + conn->rdma_transform_negotiated = true; + conn->rdma_transform_ids = 0; + for (i = 0; i < count; i++) { + u16 id = le16_to_cpu(pneg_ctxt->RDMATransformIds[i]); + + if (id == SMB2_RDMA_TRANSFORM_ENCRYPTION) + conn->rdma_transform_ids |= BIT(id); + } + return STATUS_SUCCESS; +} + static __le32 deassemble_neg_contexts(struct ksmbd_conn *conn, struct smb2_negotiate_req *req, unsigned int len_of_smb) @@ -1243,7 +1725,7 @@ static __le32 deassemble_neg_contexts(struct ksmbd_conn *conn, unsigned int offset = le32_to_cpu(req->NegotiateContextOffset); unsigned int neg_ctxt_cnt = le16_to_cpu(req->NegotiateContextCount); __le32 status = STATUS_INVALID_PARAMETER; - int compress_ctxt_cnt = 0; + int compress_ctxt_cnt = 0, rdma_transform_ctxt_cnt = 0; ksmbd_debug(SMB, "decoding %d negotiate contexts\n", neg_ctxt_cnt); if (len_of_smb <= offset) { @@ -1302,6 +1784,17 @@ static __le32 deassemble_neg_contexts(struct ksmbd_conn *conn, } else if (pctx->ContextType == SMB2_NETNAME_NEGOTIATE_CONTEXT_ID) { ksmbd_debug(SMB, "deassemble SMB2_NETNAME_NEGOTIATE_CONTEXT_ID context\n"); + } else if (pctx->ContextType == SMB2_RDMA_TRANSFORM_CAPABILITIES) { + ksmbd_debug(SMB, + "deassemble SMB2_RDMA_TRANSFORM_CAPABILITIES context\n"); + if (ksmbd_rdma_enabled() && + rdma_transform_ctxt_cnt++) { + status = STATUS_INVALID_PARAMETER; + break; + } + status = decode_rdma_ctx(conn, pctx, ctxt_len); + if (status != STATUS_SUCCESS) + break; } else if (pctx->ContextType == SMB2_POSIX_EXTENSIONS_AVAILABLE) { ksmbd_debug(SMB, "deassemble SMB2_POSIX_EXTENSIONS_AVAILABLE context\n"); @@ -1394,7 +1887,7 @@ int smb2_handle_negotiate(struct ksmbd_work *work) KSMBD_DEFAULT_GFP); if (!conn->preauth_info) { rc = -ENOMEM; - rsp->hdr.Status = STATUS_INVALID_PARAMETER; + rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; goto err_out; } @@ -1409,6 +1902,16 @@ int smb2_handle_negotiate(struct ksmbd_work *work) conn->preauth_info = NULL; goto err_out; } + if (!conn->cipher_type) + conn->rdma_transform_ids &= + ~BIT(SMB2_RDMA_TRANSFORM_ENCRYPTION); + ksmbd_debug(RDMA, + "RDMA transform negotiation: transport=%s context=%s encryption=%s cipher=0x%04x\n", + conn->transport->ops->rdma_read ? "rdma" : "tcp", + conn->rdma_transform_negotiated ? "present" : "absent", + conn->rdma_transform_ids & + BIT(SMB2_RDMA_TRANSFORM_ENCRYPTION) ? "enabled" : "disabled", + le16_to_cpu(conn->cipher_type)); rc = init_smb3_11_server(conn); if (rc < 0) { @@ -1477,11 +1980,9 @@ int smb2_handle_negotiate(struct ksmbd_work *work) rsp->SecurityMode = SMB2_NEGOTIATE_SIGNING_ENABLED_LE; conn->use_spnego = true; - if ((server_conf.signing == KSMBD_CONFIG_OPT_AUTO || - server_conf.signing == KSMBD_CONFIG_OPT_DISABLED) && - req->SecurityMode & SMB2_NEGOTIATE_SIGNING_REQUIRED_LE) + if (req->SecurityMode & SMB2_NEGOTIATE_SIGNING_REQUIRED_LE) conn->sign = true; - else if (server_conf.signing == KSMBD_CONFIG_OPT_MANDATORY) { + if (server_conf.signing == KSMBD_CONFIG_OPT_MANDATORY) { server_conf.enforced_signing = true; rsp->SecurityMode |= SMB2_NEGOTIATE_SIGNING_REQUIRED_LE; conn->sign = true; @@ -1491,7 +1992,7 @@ int smb2_handle_negotiate(struct ksmbd_work *work) ksmbd_conn_set_need_setup(conn); err_out: - if (rc) + if (rc && rsp->hdr.Status == STATUS_SUCCESS) rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; if (!rc) @@ -1821,7 +2322,9 @@ static int krb5_authenticate(struct ksmbd_work *work, struct ksmbd_session *sess = work->sess; char *in_blob, *out_blob; char channel_key[CIFS_KEY_SIZE] = {}; - char *auth_key = conn->binding ? channel_key : sess->sess_key; + char reauth_key[CIFS_KEY_SIZE] = {}; + char *auth_key = conn->binding ? channel_key : + (work->session_setup_reauth ? reauth_key : sess->sess_key); u64 prev_sess_id; bool binding = conn->binding; int in_len, out_len; @@ -1832,7 +2335,7 @@ static int krb5_authenticate(struct ksmbd_work *work, in_len = le16_to_cpu(req->SecurityBufferLength); out_blob = (char *)&rsp->hdr.ProtocolId + le16_to_cpu(rsp->SecurityBufferOffset); - out_len = work->response_sz - + out_len = work->response_sz - work->next_smb2_rsp_hdr_off - (le16_to_cpu(rsp->SecurityBufferOffset) + 4); retval = ksmbd_krb5_authenticate(sess, in_blob, in_len, @@ -1840,7 +2343,7 @@ static int krb5_authenticate(struct ksmbd_work *work, if (retval) { ksmbd_debug(SMB, "krb5 authentication failed\n"); if (retval != -EKEYREJECTED) - retval = -EINVAL; + retval = -EPERM; goto out; } @@ -1856,12 +2359,21 @@ static int krb5_authenticate(struct ksmbd_work *work, * that it is reauthentication. And the user/password * has been verified, so return it here. */ - if (sess->state == SMB2_SESSION_VALID) { + if (sess->state == SMB2_SESSION_VALID && !work->session_setup_reauth) { if (conn->binding) goto binding_session; return 0; } + /* + * Reauthentication verifies the new Kerberos credentials but keeps + * the established SMB session keys. + */ + if (work->session_setup_reauth) { + retval = 0; + goto out; + } + if ((rsp->SessionFlags != SMB2_SESSION_FLAG_IS_GUEST_LE && (conn->sign || server_conf.enforced_signing)) || (req->SecurityMode & SMB2_NEGOTIATE_SIGNING_REQUIRED)) @@ -1899,6 +2411,7 @@ binding_session: } retval = 0; out: + memzero_explicit(reauth_key, sizeof(reauth_key)); if (binding) memzero_explicit(channel_key, sizeof(channel_key)); return retval; @@ -1965,6 +2478,21 @@ int smb2_sess_setup(struct ksmbd_work *work) goto out_err; } + if (conn->dialect == SMB311_PROT_ID) { + struct channel *chann; + unsigned long index; + + down_read(&sess->chann_lock); + xa_for_each(&sess->ksmbd_chann_list, index, chann) { + if (conn->cipher_type != chann->conn->cipher_type) + rc = -EINVAL; + break; + } + up_read(&sess->chann_lock); + if (rc) + goto out_err; + } + if (!(req->hdr.Flags & SMB2_FLAGS_SIGNED)) { rc = -EINVAL; goto out_err; @@ -2042,8 +2570,13 @@ int smb2_sess_setup(struct ksmbd_work *work) } if (sess->state == SMB2_SESSION_EXPIRED) { - rc = -EFAULT; - goto out_err; + if (sess->kerberos_expiry && + ktime_get_real_seconds() >= sess->kerberos_expiry) { + work->session_setup_reauth = true; + } else { + rc = -EFAULT; + goto out_err; + } } if (ksmbd_conn_need_reconnect(conn)) { @@ -2087,10 +2620,8 @@ int smb2_sess_setup(struct ksmbd_work *work) if (conn->preferred_auth_mech & (KSMBD_AUTH_KRB5 | KSMBD_AUTH_MSKRB5)) { rc = krb5_authenticate(work, req, rsp); - if (rc) { - rc = -EINVAL; + if (rc) goto out_err; - } if (!ksmbd_conn_need_reconnect(conn)) { ksmbd_conn_set_good(conn); @@ -2119,7 +2650,7 @@ int smb2_sess_setup(struct ksmbd_work *work) ksmbd_preauth_session_lookup(conn, sess->id); if (preauth_sess) { list_del(&preauth_sess->preauth_entry); - kfree(preauth_sess); + kfree_sensitive(preauth_sess); } } } else { @@ -2173,7 +2704,7 @@ out_err: preauth_sess = ksmbd_preauth_session_lookup(conn, sess->id); if (preauth_sess) { list_del(&preauth_sess->preauth_entry); - kfree(preauth_sess); + kfree_sensitive(preauth_sess); } } @@ -2201,6 +2732,7 @@ out_err: */ if (!(req->Flags & SMB2_SESSION_REQ_FLAG_BINDING)) { sess->last_active = jiffies; + sess->kerberos_expiry = 0; sess->state = SMB2_SESSION_EXPIRED; } /* @@ -2278,12 +2810,22 @@ int smb2_tree_connect(struct ksmbd_work *work) name, treename); status = ksmbd_tree_conn_connect(work, name); - if (status.ret == KSMBD_TREE_CONN_STATUS_OK) + if (status.ret == KSMBD_TREE_CONN_STATUS_OK) { rsp->hdr.Id.SyncId.TreeId = cpu_to_le32(status.tree_conn->id); - else + share = status.tree_conn->share_conf; + + /* A share that requires encryption needs a negotiated SMB3 cipher. */ + if (test_share_config_flag(share, KSMBD_SHARE_FLAG_ENCRYPT_DATA) && + !smb3_encryption_negotiated(conn)) { + ksmbd_tree_conn_disconnect(sess, status.tree_conn); + status.tree_conn = NULL; + share = NULL; + status.ret = KSMBD_TREE_CONN_STATUS_ERROR; + goto out_err1; + } + } else goto out_err1; - share = status.tree_conn->share_conf; if (test_share_config_flag(share, KSMBD_SHARE_FLAG_PIPE)) { ksmbd_debug(SMB, "IPC share path request\n"); rsp->ShareType = SMB2_SHARE_TYPE_PIPE; @@ -2316,12 +2858,11 @@ int smb2_tree_connect(struct ksmbd_work *work) up_write(&sess->tree_conns_lock); rsp->StructureSize = cpu_to_le16(16); out_err1: - if (server_conf.flags & KSMBD_GLOBAL_FLAG_DURABLE_HANDLE && share && - test_share_config_flag(share, - KSMBD_SHARE_FLAG_CONTINUOUS_AVAILABILITY)) - rsp->Capabilities = SMB2_SHARE_CAP_CONTINUOUS_AVAILABILITY; - else - rsp->Capabilities = 0; + /* + * A configured CA share is not continuously available until persistent + * open recovery, ownership fencing, and failover are implemented. + */ + rsp->Capabilities = 0; rsp->Reserved = 0; /* default manual caching */ rsp->ShareFlags = SMB2_SHAREFLAG_MANUAL_CACHING; @@ -2329,10 +2870,26 @@ out_err1: if (conn->dialect == SMB311_PROT_ID && conn->compress_algorithm != SMB3_COMPRESS_NONE) rsp->ShareFlags |= cpu_to_le32(SMB2_SHAREFLAG_COMPRESS_DATA); + if (share && test_share_config_flag(share, + KSMBD_SHARE_FLAG_HIDE_UNREADABLE)) + rsp->ShareFlags |= + cpu_to_le32(SMB2_SHAREFLAG_ACCESS_BASED_DIRECTORY_ENUM); + if (share && test_share_config_flag(share, + KSMBD_SHARE_FLAG_ENCRYPT_DATA)) + rsp->ShareFlags |= + cpu_to_le32(SMB2_SHAREFLAG_ENCRYPT_DATA); rc = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_tree_connect_rsp)); - if (rc) + if (rc) { + if (status.ret == KSMBD_TREE_CONN_STATUS_OK) { + down_write(&sess->tree_conns_lock); + status.tree_conn->t_state = TREE_DISCONNECTED; + up_write(&sess->tree_conns_lock); + ksmbd_tree_conn_disconnect(sess, status.tree_conn); + status.tree_conn = NULL; + } status.ret = KSMBD_TREE_CONN_STATUS_NOMEM; + } if (!IS_ERR(treename)) kfree(treename); @@ -2516,7 +3073,6 @@ int smb2_session_logoff(struct ksmbd_work *work) struct ksmbd_session *sess = work->sess; struct smb2_logoff_req *req; struct smb2_logoff_rsp *rsp; - u64 sess_id; int err; WORK_BUFFERS(work, req, rsp); @@ -2530,8 +3086,7 @@ int smb2_session_logoff(struct ksmbd_work *work) smb2_set_err_rsp(work); return -ENOENT; } - sess_id = le64_to_cpu(req->hdr.SessionId); - ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_RECONNECT); + ksmbd_all_conn_set_status(sess, KSMBD_SESS_NEED_RECONNECT); ksmbd_conn_unlock(conn); ksmbd_close_session_fds(work); @@ -2545,10 +3100,11 @@ int smb2_session_logoff(struct ksmbd_work *work) } down_write(&conn->session_lock); + sess->kerberos_expiry = 0; sess->state = SMB2_SESSION_EXPIRED; up_write(&conn->session_lock); - ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_SETUP); + ksmbd_all_conn_set_status(sess, KSMBD_SESS_NEED_SETUP); rsp->StructureSize = cpu_to_le16(4); err = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_logoff_rsp)); @@ -2586,7 +3142,16 @@ static noinline int create_smb2_pipe(struct ksmbd_work *work) id = ksmbd_session_rpc_open(work->sess, name); if (id < 0) { - pr_err("Unable to open RPC pipe: %d\n", id); + /* + * mdssvc (Spotlight) is a routine, expected probe from macOS + * that we deliberately don't support -- it's disabled at the + * __rpc_method() level (mgmt/user_session.c), but this + * generic failure log would otherwise still fire on every + * single probe regardless. + */ + if (!(id == -ENOENT && (!strcmp(name, "\\mdssvc") || + !strcmp(name, "mdssvc")))) + pr_err("Unable to open RPC pipe: %d\n", id); err = id; goto out; } @@ -2621,6 +3186,9 @@ out: case -EINVAL: rsp->hdr.Status = STATUS_INVALID_PARAMETER; break; + case -ENOENT: + rsp->hdr.Status = STATUS_OBJECT_NAME_NOT_FOUND; + break; case -ENOSPC: case -ENOMEM: rsp->hdr.Status = STATUS_NO_MEMORY; @@ -2637,6 +3205,22 @@ out: return err; } +static bool smb2_is_private_ea(const char *name, size_t name_len) +{ + if (name_len == SD_PREFIX_LEN && + !strncasecmp(name, SD_PREFIX, SD_PREFIX_LEN)) + return true; + if (name_len == DOS_ATTRIBUTE_PREFIX_LEN && + !strncasecmp(name, DOS_ATTRIBUTE_PREFIX, + DOS_ATTRIBUTE_PREFIX_LEN)) + return true; + if (name_len >= STREAM_PREFIX_LEN && + !strncasecmp(name, STREAM_PREFIX, STREAM_PREFIX_LEN)) + return true; + + return false; +} + /** * smb2_set_ea() - handler for setting extended attributes using set * info command @@ -2678,6 +3262,10 @@ static int smb2_set_ea(struct smb2_ea_info *eabuf, unsigned int buf_len, rc = -EINVAL; break; } + if (smb2_is_private_ea(eabuf->name, eabuf->EaNameLength)) { + rc = -EACCES; + break; + } memcpy(attr_name, XATTR_USER_PREFIX, XATTR_USER_PREFIX_LEN); memcpy(&attr_name[XATTR_USER_PREFIX_LEN], eabuf->name, @@ -2771,6 +3359,30 @@ static noinline int smb2_set_stream_name_xattr(const struct path *path, return 0; if (fp->cdoption == FILE_OPEN_LE) { + if (!strcmp(stream_name, "AFP_AfpInfo") && + test_share_config_flag(fp->tcon->share_conf, + KSMBD_SHARE_FLAG_TIME_MACHINE)) { + /* + * Synthesize an empty AFP_AfpInfo xattr on first access. + * type=0/creator=0 tells macOS to use the file extension + * for icon and type detection. + * + * Scoped to TIME_MACHINE shares, matching the rest of + * the AAPL series -- conn->is_aapl alone isn't a safe + * gate here, since the pre-existing narrow UniqueId=0 + * path can also set it on ordinary, non-Time-Machine + * shares whenever a Mac client happens to negotiate + * AAPL there too. + */ + static const u8 afpinfo_empty[60] = { + 0x00, 0x05, 0x16, 0x07, /* magic 0x00051607 BE */ + 0x00, 0x02, 0x00, 0x00, /* version 0x00020000 BE */ + }; + rc = ksmbd_vfs_setxattr(idmap, path, xattr_stream_name, + (void *)afpinfo_empty, + sizeof(afpinfo_empty), 0, false); + return rc < 0 ? rc : 0; + } ksmbd_debug(SMB, "XATTR stream name lookup failed: %d\n", rc); return -EBADF; } @@ -2781,6 +3393,22 @@ static noinline int smb2_set_stream_name_xattr(const struct path *path, return 0; } +/* + * fp->stream.size is the byte length of the mangled xattr *name* + * (used as attr_name_len when looking the xattr up), not the size of + * the xattr's value. Reporting it as EndOfFile/AllocationSize for a + * stream handle is wrong -- query the xattr's actual value length + * instead. + */ +static loff_t ksmbd_stream_eof(struct ksmbd_file *fp) +{ + ssize_t slen = ksmbd_vfs_casexattr_len(file_mnt_idmap(fp->filp), + fp->filp->f_path.dentry, + fp->stream.name, + fp->stream.size); + return slen < 0 ? 0 : (loff_t)slen; +} + static int smb2_remove_smb_xattrs(const struct path *path) { struct mnt_idmap *idmap = mnt_idmap(path->mnt); @@ -2855,25 +3483,57 @@ static void smb2_new_xattrs(struct ksmbd_tree_connect *tcon, const struct path * ksmbd_debug(SMB, "failed to store file attribute into xattr\n"); } +static bool smb2_parent_compressed(struct ksmbd_tree_connect *tcon, + const struct path *path) +{ + struct dentry *parent = dget_parent(path->dentry); + struct file_kattr fa = { .flags_valid = true }; + struct xattr_dos_attrib da; + bool compressed = false; + int rc; + + rc = vfs_fileattr_get(parent, &fa); + if (!rc && fa.flags & FS_COMPR_FL) { + compressed = true; + goto out; + } + + rc = ksmbd_vfs_get_dos_attrib_xattr(mnt_idmap(path->mnt), parent, &da); + if (rc > 0 && da.attr & FILE_ATTRIBUTE_COMPRESSED) + compressed = true; + +out: + dput(parent); + return compressed; +} + static void smb2_update_xattrs(struct ksmbd_tree_connect *tcon, const struct path *path, struct ksmbd_file *fp) { - struct xattr_dos_attrib da; + struct xattr_dos_attrib da = {}; + bool store_dos_attrs = test_share_config_flag(tcon->share_conf, + KSMBD_SHARE_FLAG_STORE_DOS_ATTRS); int rc; fp->f_ci->m_fattr &= ~(FILE_ATTRIBUTE_HIDDEN_LE | FILE_ATTRIBUTE_SYSTEM_LE); /* get FileAttributes from XATTR_NAME_DOS_ATTRIBUTE */ - if (!test_share_config_flag(tcon->share_conf, - KSMBD_SHARE_FLAG_STORE_DOS_ATTRS)) - return; - rc = ksmbd_vfs_get_dos_attrib_xattr(mnt_idmap(path->mnt), path->dentry, &da); if (rc > 0) { - fp->f_ci->m_fattr = cpu_to_le32(da.attr); - fp->create_time = da.create_time; - fp->itime = da.itime; + if (store_dos_attrs) { + fp->f_ci->m_fattr = cpu_to_le32(da.attr); + fp->create_time = da.create_time; + fp->itime = da.itime; + } else { + fp->f_ci->m_fattr &= + ~(FILE_ATTRIBUTE_COMPRESSED_LE | + FILE_ATTRIBUTE_SPARSE_FILE_LE); + fp->f_ci->m_fattr |= + cpu_to_le32(da.attr & + (FILE_ATTRIBUTE_COMPRESSED | + FILE_ATTRIBUTE_SPARSE_FILE)); + } } } @@ -2975,12 +3635,89 @@ struct durable_info { unsigned short int type; bool persistent; bool reconnected; + bool replay; + bool replay_consumed; bool app_instance_id; + bool app_instance_version_valid; unsigned int timeout; char *CreateGuid; char AppInstanceId[SMB2_CREATE_GUID_SIZE]; + u64 app_instance_version_high; + u64 app_instance_version_low; }; +static int smb2_check_durable_replay(struct ksmbd_work *work, + struct ksmbd_file *fp, + struct lease_ctx_info *lc, + bool persistent) +{ + struct oplock_info *opinfo; + int ret = 0; + + if (!fp->is_durable && !fp->is_persistent) + return -EACCES; + + if (ksmbd_vfs_compare_durable_owner(fp, work->sess->user) == false) + return -EACCES; + + if (fp->is_persistent && !persistent) + return -EINVAL; + + opinfo = opinfo_get(fp); + if (!opinfo) + return 0; + + if (opinfo->sess && opinfo->sess->id != work->sess->id) { + ret = -ENOEXEC; + goto out; + } + + if (opinfo->is_lease) { + if (!lc || + memcmp(opinfo->o_lease->lease_key, lc->lease_key, + SMB2_LEASE_KEY_SIZE)) { + ret = -EACCES; + goto out; + } + } else { + if (lc) { + ret = -EACCES; + goto out; + } + + if (fp->is_durable && opinfo->level != SMB2_OPLOCK_LEVEL_BATCH) + ret = -EACCES; + } +out: + opinfo_put(opinfo); + return ret; +} + +static bool smb2_durable_replay_consumed(struct ksmbd_file *fp) +{ + bool consumed; + + spin_lock(&fp->f_lock); + consumed = fp->durable_replay_consumed; + spin_unlock(&fp->f_lock); + + return consumed; +} + +static void smb2_mark_durable_replay_consumed(struct ksmbd_file *fp) +{ + spin_lock(&fp->f_lock); + fp->durable_replay_consumed = true; + spin_unlock(&fp->f_lock); +} + +static bool smb2_durable_replay_differs(struct ksmbd_file *fp, + struct smb2_create_req *req) +{ + return fp->cdoption != req->CreateDisposition || + fp->create_file_attributes != req->FileAttributes; +} + static int parse_durable_handle_context(struct ksmbd_work *work, struct smb2_create_req *req, struct lease_ctx_info *lc, @@ -3008,6 +3745,7 @@ static int parse_durable_handle_context(struct ksmbd_work *work, case DURABLE_RECONN_V2: { struct create_durable_handle_reconnect_v2 *recon_v2; + u32 flags; if (dh_info->type == DURABLE_RECONN || dh_info->type == DURABLE_REQ_V2) { @@ -3022,6 +3760,12 @@ static int parse_durable_handle_context(struct ksmbd_work *work, } recon_v2 = (struct create_durable_handle_reconnect_v2 *)context; + flags = le32_to_cpu(recon_v2->dcontext.Flags); + if (flags & ~SMB2_DHANDLE_FLAG_PERSISTENT) { + err = -EINVAL; + goto out; + } + dh_info->persistent = flags & SMB2_DHANDLE_FLAG_PERSISTENT; persistent_id = recon_v2->dcontext.Fid.PersistentFileId; dh_info->fp = ksmbd_lookup_durable_fd(persistent_id); if (!dh_info->fp) { @@ -3030,7 +3774,9 @@ static int parse_durable_handle_context(struct ksmbd_work *work, goto out; } - if (dh_info->fp->durable_volatile_id != + /* A zero VolatileFileId means that the client did not specify it. */ + if (recon_v2->dcontext.Fid.VolatileFileId && + dh_info->fp->durable_volatile_id != recon_v2->dcontext.Fid.VolatileFileId) { err = -EBADF; ksmbd_put_durable_fd(dh_info->fp); @@ -3044,6 +3790,13 @@ static int parse_durable_handle_context(struct ksmbd_work *work, goto out; } + /* A persistent reconnect must match the original open type. */ + if (dh_info->fp->is_persistent != dh_info->persistent) { + err = dh_info->persistent ? -EINVAL : -EBADF; + ksmbd_put_durable_fd(dh_info->fp); + goto out; + } + dh_info->type = dh_idx; dh_info->reconnected = true; ksmbd_debug(SMB, @@ -3076,7 +3829,9 @@ static int parse_durable_handle_context(struct ksmbd_work *work, goto out; } - if (dh_info->fp->durable_volatile_id != + /* A zero VolatileFileId means that the client did not specify it. */ + if (recon->Data.Fid.VolatileFileId && + dh_info->fp->durable_volatile_id != recon->Data.Fid.VolatileFileId) { err = -EBADF; ksmbd_put_durable_fd(dh_info->fp); @@ -3107,7 +3862,16 @@ static int parse_durable_handle_context(struct ksmbd_work *work, durable_v2_blob = (struct create_durable_req_v2 *)context; + if (le32_to_cpu(durable_v2_blob->dcontext.Flags) & + ~SMB2_DHANDLE_FLAG_PERSISTENT) { + err = -EINVAL; + goto out; + } ksmbd_debug(SMB, "Request for durable v2 open\n"); + dh_info->CreateGuid = durable_v2_blob->dcontext.CreateGuid; + dh_info->persistent = + le32_to_cpu(durable_v2_blob->dcontext.Flags) & + SMB2_DHANDLE_FLAG_PERSISTENT; dh_info->fp = ksmbd_lookup_fd_cguid(durable_v2_blob->dcontext.CreateGuid); if (dh_info->fp) { if (!memcmp(conn->ClientGUID, dh_info->fp->client_guid, @@ -3118,12 +3882,73 @@ static int parse_durable_handle_context(struct ksmbd_work *work, goto out; } - if (dh_info->fp->conn) { + if (dh_info->fp->f_state == FP_NEW) { + /* Original CREATE is still pending. */ ksmbd_put_durable_fd(dh_info->fp); - err = -EBADF; + err = -EAGAIN; goto out; } - dh_info->reconnected = true; + + if (!dh_info->fp->is_durable && + !dh_info->fp->is_persistent) { + /* + * A DurableHandleReqV2 CREATE can complete + * without granting durability (for example, if + * it requested no oplock). Its CreateGuid still + * identifies a completed CREATE for replay. + */ + if (dh_info->fp->conn && + ksmbd_vfs_compare_durable_owner( + dh_info->fp, work->sess->user)) { + if (smb2_durable_replay_consumed( + dh_info->fp)) { + ksmbd_put_durable_fd(dh_info->fp); + dh_info->fp = NULL; + dh_info->type = dh_idx; + dh_info->replay_consumed = true; + break; + } + if (smb2_durable_replay_differs( + dh_info->fp, req)) + smb2_mark_durable_replay_consumed( + dh_info->fp); + dh_info->replay = true; + dh_info->type = dh_idx; + goto out; + } + ksmbd_put_durable_fd(dh_info->fp); + err = -EACCES; + goto out; + } + + if (dh_info->fp->conn && + smb2_durable_replay_consumed(dh_info->fp)) { + ksmbd_put_durable_fd(dh_info->fp); + dh_info->fp = NULL; + dh_info->type = dh_idx; + dh_info->replay_consumed = true; + break; + } + + err = smb2_check_durable_replay(work, + dh_info->fp, + lc, + dh_info->persistent); + if (err) { + ksmbd_put_durable_fd(dh_info->fp); + goto out; + } + + if (dh_info->fp->conn) { + if (smb2_durable_replay_differs(dh_info->fp, + req)) + smb2_mark_durable_replay_consumed( + dh_info->fp); + dh_info->replay = true; + } else { + dh_info->reconnected = true; + } + dh_info->type = dh_idx; goto out; } ksmbd_put_durable_fd(dh_info->fp); @@ -3132,10 +3957,6 @@ static int parse_durable_handle_context(struct ksmbd_work *work, if ((lc && (lc->req_state & SMB2_LEASE_HANDLE_CACHING_LE)) || req_op_level == SMB2_OPLOCK_LEVEL_BATCH) { - dh_info->CreateGuid = - durable_v2_blob->dcontext.CreateGuid; - dh_info->persistent = - le32_to_cpu(durable_v2_blob->dcontext.Flags); dh_info->timeout = le32_to_cpu(durable_v2_blob->dcontext.Timeout); dh_info->type = dh_idx; @@ -3188,6 +4009,68 @@ static int parse_app_instance_id(struct smb2_create_req *req, return 0; } +static int parse_app_instance_version(struct smb2_create_req *req, + struct durable_info *dh_info) +{ + struct create_context *context; + char *data; + + context = smb2_find_context_vals(req, SMB2_CREATE_APP_INSTANCE_VERSION, + SMB2_CREATE_GUID_SIZE); + if (IS_ERR(context)) + return PTR_ERR(context); + if (!context) + return 0; + + if (le32_to_cpu(context->DataLength) < 24) + return -EINVAL; + + data = (char *)context + le16_to_cpu(context->DataOffset); + if (get_unaligned_le16(data) != 24 || + get_unaligned_le16(data + 2) != 0) + return -EINVAL; + + dh_info->app_instance_version_high = get_unaligned_le64(data + 8); + dh_info->app_instance_version_low = get_unaligned_le64(data + 16); + dh_info->app_instance_version_valid = true; + return 0; +} + +static int smb2_handle_app_instance_id(struct smb2_create_rsp *rsp, + struct durable_info *dh_info) +{ + struct ksmbd_file *old_fp; + bool reject = false; + + if (!dh_info->app_instance_id) + return 0; + + old_fp = ksmbd_lookup_fd_app_instance_id(dh_info->AppInstanceId); + if (!old_fp) + return 0; + + if (dh_info->app_instance_version_valid) { + if (old_fp->app_instance_version_valid && + (dh_info->app_instance_version_high < + old_fp->app_instance_version_high || + (dh_info->app_instance_version_high == + old_fp->app_instance_version_high && + dh_info->app_instance_version_low <= + old_fp->app_instance_version_low))) + reject = true; + } else if (old_fp->app_instance_version_valid) { + reject = true; + } + + ksmbd_put_durable_fd(old_fp); + if (reject) { + rsp->hdr.Status = STATUS_FILE_FORCED_CLOSED; + return -EIO; + } + + return ksmbd_close_fd_app_instance_id(dh_info->AppInstanceId); +} + /** * smb2_open() - handler for smb file open request * @work: smb work containing request buffer @@ -3217,6 +4100,9 @@ int smb2_open(struct ksmbd_work *work) int rc = 0; int contxt_cnt = 0, query_disk_id = 0; bool maximal_access_ctxt = false, posix_ctxt = false; + bool aapl_ctxt = false; + bool durable_rsp = true; + __u64 aapl_req_bitmap = 0, aapl_client_caps = 0; int s_type = 0; int next_off = 0; char *name = NULL; @@ -3226,6 +4112,7 @@ int smb2_open(struct ksmbd_work *work) u64 time, alloc_size = 0; umode_t posix_mode = 0; __le32 daccess, maximal_access = 0; + u32 dos_attr; int iov_len = 0; ksmbd_debug(SMB, "Received smb2 create request\n"); @@ -3311,6 +4198,15 @@ int smb2_open(struct ksmbd_work *work) req_op_level = req->RequestedOplockLevel; + if (req->CreateContextsOffset) { + rc = parse_app_instance_id(req, &dh_info); + if (rc) + goto err_out2; + rc = parse_app_instance_version(req, &dh_info); + if (rc) + goto err_out2; + } + if (server_conf.flags & KSMBD_GLOBAL_FLAG_DURABLE_HANDLE && req->CreateContextsOffset) { lc = parse_lease_state(req); @@ -3330,9 +4226,21 @@ int smb2_open(struct ksmbd_work *work) ksmbd_debug(SMB, "error parsing durable handle context\n"); goto err_out2; } - rc = parse_app_instance_id(req, &dh_info); - if (rc) - goto err_out2; + + if (dh_info.replay == true) { + fp = dh_info.fp; + if (ksmbd_override_fsids(work)) { + rc = -ENOMEM; + goto err_out2; + } + + file_info = FILE_OPENED; + rc = ksmbd_vfs_getattr(&fp->filp->f_path, &stat); + if (rc) + goto err_out2; + + goto reconnected_fp; + } if (dh_info.reconnected == true) { rc = smb2_check_durable_oplock(conn, share, dh_info.fp, @@ -3360,8 +4268,6 @@ int smb2_open(struct ksmbd_work *work) goto reconnected_fp; } - if (dh_info.type == DURABLE_REQ_V2 && dh_info.app_instance_id) - ksmbd_close_fd_app_instance_id(dh_info.AppInstanceId); } else if (req_op_level == SMB2_OPLOCK_LEVEL_LEASE) { lc = parse_lease_state(req); if (IS_ERR(lc)) { @@ -3376,6 +4282,13 @@ int smb2_open(struct ksmbd_work *work) } } + if (dh_info.app_instance_id && !dh_info.reconnected && + !dh_info.replay) { + rc = smb2_handle_app_instance_id(rsp, &dh_info); + if (rc) + goto err_out2; + } + if (le32_to_cpu(req->ImpersonationLevel) > le32_to_cpu(IL_DELEGATE)) { pr_err("Invalid impersonationlevel : 0x%x\n", le32_to_cpu(req->ImpersonationLevel)); @@ -3405,8 +4318,6 @@ int smb2_open(struct ksmbd_work *work) if (req->CreateOptions & FILE_NON_DIRECTORY_FILE_LE) { rc = -EINVAL; goto err_out2; - } else if (req->CreateOptions & FILE_NO_COMPRESSION_LE) { - req->CreateOptions &= ~FILE_NO_COMPRESSION_LE; } } } @@ -3513,6 +4424,8 @@ int smb2_open(struct ksmbd_work *work) file_present = true; if (req->CreateOptions & FILE_DELETE_ON_CLOSE_LE) { + struct xattr_dos_attrib da; + /* * If file exists with under flags, return access * denied error. @@ -3526,6 +4439,16 @@ int smb2_open(struct ksmbd_work *work) if (!test_tree_conn_flag(tcon, KSMBD_TREE_CONN_FLAG_WRITABLE)) { ksmbd_debug(SMB, "User does not have write permission\n"); + rc = -EACCES; + goto err_out; + } + + if (test_share_config_flag(tcon->share_conf, + KSMBD_SHARE_FLAG_STORE_DOS_ATTRS) && + ksmbd_vfs_get_dos_attrib_xattr(mnt_idmap(path.mnt), + path.dentry, &da) > 0 && + da.attr & FILE_ATTRIBUTE_READONLY) { + rsp->hdr.Status = STATUS_CANNOT_DELETE; rc = -EACCES; goto err_out; } @@ -3543,6 +4466,13 @@ int smb2_open(struct ksmbd_work *work) rc = 0; } + if (!file_present && req->CreateOptions & FILE_DELETE_ON_CLOSE_LE && + req->FileAttributes & FILE_ATTRIBUTE_READONLY_LE) { + rsp->hdr.Status = STATUS_CANNOT_DELETE; + rc = -EACCES; + goto err_out; + } + /* * An explicit ::$DATA suffix names the unnamed data stream and is * canonicalized to a NULL stream name (base file), but the request @@ -3558,7 +4488,7 @@ int smb2_open(struct ksmbd_work *work) } } else { if (file_present && S_ISDIR(d_inode(path.dentry)->i_mode) && - s_type == DATA_STREAM) { + !stream_name && s_type == DATA_STREAM) { rc = -EIO; rsp->hdr.Status = STATUS_FILE_IS_A_DIRECTORY; } @@ -3602,9 +4532,29 @@ int smb2_open(struct ksmbd_work *work) if (file_present && !(req->CreateOptions & FILE_DELETE_ON_CLOSE_LE)) { rc = smb_check_perm_dacl(conn, &path, &daccess, - sess->user->uid); + req->DesiredAccess, + sess->user->uid, false); if (rc) goto err_out; + + if (maximal_access_ctxt) { + maximal_access = FILE_MAXIMAL_ACCESS_LE; + rc = smb_check_perm_dacl(conn, &path, &maximal_access, + 0, sess->user->uid, false); + if (rc) + goto err_out; + + /* + * smb_check_perm_dacl() returns success without + * touching *pdaccess when the object has no stored + * NT ACL, leaving maximal_access as the + * FILE_MAXIMAL_ACCESS_LE request sentinel instead of + * a real access mask. + */ + if (maximal_access == FILE_MAXIMAL_ACCESS_LE) + ksmbd_vfs_query_maximal_access(idmap, path.dentry, + &maximal_access); + } } if (daccess & FILE_MAXIMAL_ACCESS_LE) { @@ -3732,7 +4682,34 @@ int smb2_open(struct ksmbd_work *work) goto err_out; } + /* + * Publish the client and create GUID before an oplock/lease break can + * make this CREATE pending. A replay of that in-flight CREATE must find + * this FP_NEW handle and fail with STATUS_FILE_NOT_AVAILABLE instead of + * waiting on the same break again. + */ + memcpy(fp->client_guid, conn->ClientGUID, SMB2_CLIENT_GUID_SIZE); + if (dh_info.app_instance_id) { + memcpy(fp->app_instance_id, dh_info.AppInstanceId, + SMB2_CREATE_GUID_SIZE); + fp->has_app_instance_id = true; + } + if (dh_info.app_instance_version_valid) { + fp->app_instance_version_high = + dh_info.app_instance_version_high; + fp->app_instance_version_low = dh_info.app_instance_version_low; + fp->app_instance_version_valid = true; + } + if (dh_info.CreateGuid) { + memcpy(fp->create_guid, dh_info.CreateGuid, SMB2_CREATE_GUID_SIZE); + fp->durable_replay_consumed = dh_info.replay_consumed; + rc = ksmbd_vfs_set_durable_owner(fp, sess->user); + if (rc) + goto err_out; + } + fp->cdoption = req->CreateDisposition; + fp->create_file_attributes = req->FileAttributes; fp->daccess = daccess; fp->saccess = req->ShareAccess; fp->coption = req->CreateOptions; @@ -3872,6 +4849,9 @@ int smb2_open(struct ksmbd_work *work) goto err_out1; } } else { + if (created && !lc) + smb_send_parent_lease_break_noti(fp, NULL); + if (req_op_level == SMB2_OPLOCK_LEVEL_LEASE && lc) { if (S_ISDIR(file_inode(filp)->i_mode)) { lc->req_state &= ~SMB2_LEASE_WRITE_CACHING_LE; @@ -3900,7 +4880,8 @@ int smb2_open(struct ksmbd_work *work) rc = smb_grant_oplock(work, req_op_level, fp->persistent_id, fp, le32_to_cpu(req->hdr.Id.SyncId.TreeId), - lc, share_ret); + lc, share_ret, + smb3_hdr_replay(&req->hdr)); if (rc < 0) goto err_out1; } @@ -3934,16 +4915,26 @@ int smb2_open(struct ksmbd_work *work) goto err_out1; } alloc_size = le64_to_cpu(az_req->AllocationSize); + fp->allocation_size_set = true; ksmbd_debug(SMB, "request smb2 create allocate size : %llu\n", alloc_size); - smb_break_all_levII_oplock(work, fp, 1); - err = vfs_fallocate(fp->filp, FALLOC_FL_KEEP_SIZE, 0, - alloc_size); - if (err < 0) - ksmbd_debug(SMB, - "vfs_fallocate is failed : %d\n", - err); + /* + * fp->filp is the base file's data fork for a stream + * handle (streams are xattr-backed on the same + * underlying file) -- fallocate has no meaning for a + * stream and would otherwise pre-allocate storage on + * the base file's data instead. + */ + if (!ksmbd_stream_fd(fp)) { + smb_break_all_levII_oplock(work, fp, 1); + err = vfs_fallocate(fp->filp, FALLOC_FL_KEEP_SIZE, 0, + alloc_size); + if (err < 0) + ksmbd_debug(SMB, + "vfs_fallocate is failed : %d\n", + err); + } } context = smb2_find_context_vals(req, SMB2_CREATE_QUERY_ON_DISK_ID, 4); @@ -3955,7 +4946,32 @@ int smb2_open(struct ksmbd_work *work) query_disk_id = 1; } - if (conn->is_aapl == false) { + if (test_share_config_flag(share, KSMBD_SHARE_FLAG_TIME_MACHINE)) { + context = smb2_find_context_vals(req, SMB2_CREATE_AAPL, 4); + if (IS_ERR(context)) { + rc = PTR_ERR(context); + goto err_out1; + } else if (context) { + struct aapl_server_query_req *aapl_req; + + if (le32_to_cpu(context->DataLength) < + sizeof(struct aapl_server_query_req)) { + rc = -EINVAL; + goto err_out1; + } + + aapl_req = (struct aapl_server_query_req *) + ((char *)context + + le16_to_cpu(context->DataOffset)); + if (le32_to_cpu(aapl_req->cmd) == + SMB2_CRTCTX_AAPL_SERVER_QUERY) { + conn->is_aapl = true; + aapl_ctxt = true; + aapl_req_bitmap = le64_to_cpu(aapl_req->req_bitmap); + aapl_client_caps = le64_to_cpu(aapl_req->client_caps); + } + } + } else if (conn->is_aapl == false) { context = smb2_find_context_vals(req, SMB2_CREATE_AAPL, 4); if (IS_ERR(context)) { rc = PTR_ERR(context); @@ -3976,31 +4992,55 @@ int smb2_open(struct ksmbd_work *work) fp->change_time = ksmbd_UnixTimeToNT(stat.ctime); fp->allocation_size = S_ISDIR(stat.mode) ? 0 : (alloc_size ?: stat.blocks << 9); - if (req->FileAttributes || fp->f_ci->m_fattr == 0) + if (created || fp->f_ci->m_fattr == 0) fp->f_ci->m_fattr = cpu_to_le32(smb2_get_dos_mode(&stat, le32_to_cpu(req->FileAttributes))); if (!created) smb2_update_xattrs(tcon, &path, fp); + if (need_truncate && req->FileAttributes) { + dos_attr = le32_to_cpu(req->FileAttributes); + fp->f_ci->m_fattr = + cpu_to_le32(smb2_get_dos_mode(&stat, dos_attr)); + smb2_new_xattrs(tcon, &path, fp); + } ksmbd_vfs_update_compressed_fattr(path.dentry, &fp->f_ci->m_fattr); + if (created) { + if (fp->coption & FILE_NO_COMPRESSION_LE) { + rc = ksmbd_vfs_set_compression_create(work, fp, + COMPRESSION_FORMAT_NONE); + if (rc) + fp->f_ci->m_fattr &= ~FILE_ATTRIBUTE_COMPRESSED_LE; + rc = 0; + } else if (smb2_parent_compressed(tcon, &path)) { + rc = ksmbd_vfs_set_compression_create(work, fp, + COMPRESSION_FORMAT_LZNT1); + if (rc) + fp->f_ci->m_fattr |= FILE_ATTRIBUTE_COMPRESSED_LE; + rc = 0; + } + } + if (created) smb2_new_xattrs(tcon, &path, fp); - memcpy(fp->client_guid, conn->ClientGUID, SMB2_CLIENT_GUID_SIZE); + fp->create_action = cpu_to_le32(file_info); if (dh_info.type == DURABLE_REQ_V2 || dh_info.type == DURABLE_REQ) { if (dh_info.type == DURABLE_REQ_V2 && dh_info.persistent && test_share_config_flag(work->tcon->share_conf, - KSMBD_SHARE_FLAG_CONTINUOUS_AVAILABILITY)) + KSMBD_SHARE_FLAG_CONTINUOUS_AVAILABILITY) && + (conn->vals->req_capabilities & + SMB2_GLOBAL_CAP_PERSISTENT_HANDLES)) { + /* MS-SMB2 3.3.5.9.10: a persistent open is durable too. */ + fp->is_durable = true; fp->is_persistent = true; - else + } else { fp->is_durable = true; - + } if (dh_info.type == DURABLE_REQ_V2) { - memcpy(fp->create_guid, dh_info.CreateGuid, - SMB2_CREATE_GUID_SIZE); if (dh_info.app_instance_id) memcpy(fp->app_instance_id, dh_info.AppInstanceId, @@ -4010,14 +5050,33 @@ int smb2_open(struct ksmbd_work *work) min_t(unsigned int, dh_info.timeout, DURABLE_HANDLE_MAX_TIMEOUT); else - fp->durable_timeout = 60; + fp->durable_timeout = 60000; } } -reconnected_fp: + /* + * conn->is_aapl detection above (this function's create-context + * parsing) is skipped on the reconnect path below, since a + * reconnect always arrives on a fresh connection -- if the client + * cares, it sends its own AAPL context on this same CREATE, which + * this function's normal (non-reconnect) parsing already handles. + */ + reconnected_fp: + if (dh_info.replay) + file_info = le32_to_cpu(fp->create_action); rsp->StructureSize = cpu_to_le16(89); opinfo = opinfo_get(fp); rsp->OplockLevel = opinfo != NULL ? opinfo->level : 0; + /* + * A durable CREATE replay does not modify the existing open. When + * replayed without an oplock, however, its response reflects that + * request and cannot include a new durable-handle response context. + */ + if (dh_info.replay && !lc && + req_op_level == SMB2_OPLOCK_LEVEL_NONE) { + rsp->OplockLevel = SMB2_OPLOCK_LEVEL_NONE; + durable_rsp = false; + } rsp->Flags = 0; rsp->CreateAction = cpu_to_le32(file_info); rsp->CreationTime = cpu_to_le64(fp->create_time); @@ -4037,10 +5096,17 @@ reconnected_fp: * using the raw on-disk block count, which can include filesystem * preallocation and metadata rounding. */ - if (!S_ISDIR(stat.mode) && stat.size > fp->allocation_size) - fp->allocation_size = round_up(stat.size, stat.blksize); - rsp->AllocationSize = cpu_to_le64(fp->allocation_size); - rsp->EndofFile = S_ISDIR(stat.mode) ? 0 : cpu_to_le64(stat.size); + if (ksmbd_stream_fd(fp)) { + loff_t seof = ksmbd_stream_eof(fp); + + rsp->AllocationSize = cpu_to_le64((u64)seof); + rsp->EndofFile = cpu_to_le64((u64)seof); + } else { + if (!S_ISDIR(stat.mode) && stat.size > fp->allocation_size) + fp->allocation_size = round_up(stat.size, stat.blksize); + rsp->AllocationSize = cpu_to_le64(fp->allocation_size); + rsp->EndofFile = S_ISDIR(stat.mode) ? 0 : cpu_to_le64(stat.size); + } rsp->FileAttributes = fp->f_ci->m_fattr; rsp->Reserved2 = 0; @@ -4111,7 +5177,8 @@ reconnected_fp: next_off = conn->vals->create_disk_id_size; } - if (dh_info.type == DURABLE_REQ || dh_info.type == DURABLE_REQ_V2) { + if (durable_rsp && + (dh_info.type == DURABLE_REQ || dh_info.type == DURABLE_REQ_V2)) { struct create_context *durable_ccontext; durable_ccontext = (struct create_context *)(rsp->Buffer + @@ -4135,10 +5202,16 @@ reconnected_fp: if (next_ptr) *next_ptr = cpu_to_le32(next_off); next_ptr = &durable_ccontext->Next; - next_off = conn->vals->create_durable_size; + next_off = dh_info.type == DURABLE_REQ ? + conn->vals->create_durable_size : + conn->vals->create_durable_v2_size; } if (posix_ctxt) { + struct create_context *posix_ccontext; + + posix_ccontext = (struct create_context *)(rsp->Buffer + + le32_to_cpu(rsp->CreateContextsLength)); contxt_cnt++; create_posix_rsp_buf(rsp->Buffer + le32_to_cpu(rsp->CreateContextsLength), @@ -4148,6 +5221,40 @@ reconnected_fp: iov_len += conn->vals->create_posix_size; if (next_ptr) *next_ptr = cpu_to_le32(next_off); + next_ptr = &posix_ccontext->Next; + next_off = conn->vals->create_posix_size; + } + + /* + * AAPL create context response: see smb2pdu.h for the capability + * rationale. Scoped to TIME_MACHINE shares only. + */ + if (aapl_ctxt) { + if (aapl_client_caps & SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR) + conn->aapl_readdir_attr = true; + /* + * V2 extends the same inline-FinderInfo mechanism (see + * smb2pdu.h), so a V2-requesting client also gets + * aapl_readdir_attr treatment -- the reply just advertises + * the V2 bit instead of the V1 one (create_aapl_rsp_buf). + */ + if (aapl_client_caps & SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR_V2) { + conn->aapl_readdir_attr = true; + conn->aapl_readdir_attr_v2 = true; + } + + contxt_cnt++; + create_aapl_rsp_buf(rsp->Buffer + + le32_to_cpu(rsp->CreateContextsLength), + SMB2_CRTCTX_AAPL_FULL_SYNC, + aapl_req_bitmap, + conn->aapl_readdir_attr_v2); + le32_add_cpu(&rsp->CreateContextsLength, + conn->vals->create_aapl_size); + iov_len += conn->vals->create_aapl_size; + if (next_ptr) + *next_ptr = cpu_to_le32(next_off); + /* AAPL is last; next_ptr need not be updated */ } if (contxt_cnt > 0) { @@ -4164,8 +5271,11 @@ err_out1: err_out2: if (!rc) { - rc = ksmbd_update_fstate(&work->sess->file_table, fp, - FP_INITED); + if (!dh_info.replay) + rc = ksmbd_update_fstate(&work->sess->file_table, fp, + FP_INITED); + if (!rc) + rc = smb2_set_request_open(work, fp, &req->hdr, false, false); if (!rc) rc = ksmbd_iov_pin_rsp(work, (void *)rsp, iov_len); } @@ -4174,8 +5284,13 @@ err_out2: rsp->hdr.Status = STATUS_INVALID_PARAMETER; else if (rc == -EOPNOTSUPP) rsp->hdr.Status = STATUS_NOT_SUPPORTED; - else if (rc == -EACCES || rc == -ESTALE || rc == -EXDEV) - rsp->hdr.Status = STATUS_ACCESS_DENIED; + else if ((rc == -EACCES || rc == -ESTALE || rc == -EXDEV) && + !rsp->hdr.Status) { + if (req->DesiredAccess & FILE_ACCESS_SYSTEM_SECURITY_LE) + rsp->hdr.Status = STATUS_PRIVILEGE_NOT_HELD; + else + rsp->hdr.Status = STATUS_ACCESS_DENIED; + } else if (rc == -ENOENT) rsp->hdr.Status = STATUS_OBJECT_NAME_INVALID; else if (rc == -EPERM) @@ -4192,15 +5307,22 @@ err_out2: rsp->hdr.Status = STATUS_OBJECT_NAME_COLLISION; else if (rc == -EMFILE) rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; + else if (rc == -EINPROGRESS) + rsp->hdr.Status = STATUS_FILE_NOT_AVAILABLE; + else if (rc == -EAGAIN) + rsp->hdr.Status = STATUS_FILE_NOT_AVAILABLE; if (!rsp->hdr.Status) rsp->hdr.Status = STATUS_UNEXPECTED_IO_ERROR; - if (fp) + if (fp && !dh_info.replay) ksmbd_fd_put(work, fp); smb2_set_err_rsp(work); ksmbd_debug(SMB, "Error response: %x\n", rsp->hdr.Status); } + if (dh_info.replay) + ksmbd_put_durable_fd(dh_info.fp); + if (dh_info.reconnected) { /* * If reconnect succeeded, fp was republished in the @@ -4460,17 +5582,83 @@ static int smb2_populate_readdir_entry(struct ksmbd_conn *conn, int info_level, fibdinfo = (struct file_id_both_directory_info *)kstat; fibdinfo->FileNameLength = cpu_to_le32(conv_len); - fibdinfo->EaSize = - smb2_get_reparse_tag_special_file(ksmbd_kstat->kstat->mode); - if (fibdinfo->EaSize) - fibdinfo->ExtFileAttributes = FILE_ATTRIBUTE_REPARSE_POINT_LE; if (conn->is_aapl) fibdinfo->UniqueId = 0; else fibdinfo->UniqueId = cpu_to_le64(ksmbd_kstat->kstat->ino); fibdinfo->ShortNameLength = 0; fibdinfo->Reserved = 0; - fibdinfo->Reserved2 = cpu_to_le16(0); + if (conn->aapl_readdir_attr) { + /* + * READDIR_ATTR wire format, confirmed against reference server's + * reference implementation marshalling (reference implementation behavior): + * EaSize = max_access (expanded specific + * rights, simplified to "grant all") + * ShortNameLength = 24 (fixed; not 0, despite the spec) + * ShortName[0..7] = resource fork size (uint64 LE, 0 = no rfork) + * ShortName[8..23] = compressed FinderInfo (type+creator+flags+ + * ext_flags+date_added, 16 bytes LE; all + * zeros means type=0/creator=0, i.e. use + * the file extension for icon lookup) + * Reserved2 = Unix mode bits (uint16 LE) + * Reparse-point tag is indicated via ExtFileAttributes, not EaSize. + * + * V2 (conn->aapl_readdir_attr_v2): ShortNameLength+Reserved + * are read as a single flags field instead of being ignored + * -- see smb2pdu.h for the wire-format confirmation and + * AAPL_READDIR_ATTR_V2_NO_XATTR's meaning. + */ + __le32 reparse_tag = + smb2_get_reparse_tag_special_file(ksmbd_kstat->kstat->mode); + + if (reparse_tag) + fibdinfo->ExtFileAttributes = FILE_ATTRIBUTE_REPARSE_POINT_LE; + /* + * FILE_GENERIC_ALL_LE (0x10000000) is the raw + * "generic all" meta-bit -- valid only in a + * client's requested access mask, for the server + * to expand. It has none of the specific FILE_* + * rights bits set (FILE_LIST_DIRECTORY, FILE_TRAVERSE, + * etc.), so reporting it here as max_access would make + * macOS's bit-by-bit access checks fail on every + * entry -> permanent "no entry" badges in Finder. + * Report the actual expanded rights instead, same + * as smb_map_generic_desired_access() does when + * translating a client's GENERIC_ALL request. + */ + fibdinfo->EaSize = cpu_to_le32(GENERIC_ALL_FLAGS); + /* + * The spec says ShortNameLength should be 0 when + * there's no short name; 24 here instead matches + * reference implementation marshalling (reference + * behavior) for server-to-server wire parity. + * V2 repurposes it as a flags field that is + * interpreted; V1 doesn't. Either value is safe + * here, so keep 24 for parity. + */ + if (conn->aapl_readdir_attr_v2) { + /* + * V2 repurposes this field as flags (see comment + * above) -- 24 is a V1-only convention that real + * macOS clients ignore outright, so don't reuse it + * here as a base value for a field V2 clients + * actually interpret. + */ + fibdinfo->ShortNameLength = 0; + if (!ksmbd_kstat->has_ads_stream) + fibdinfo->ShortNameLength = AAPL_READDIR_ATTR_V2_NO_XATTR; + } else { + fibdinfo->ShortNameLength = 24; + } + memset(fibdinfo->ShortName, 0, sizeof(fibdinfo->ShortName)); + fibdinfo->Reserved2 = cpu_to_le16(ksmbd_kstat->kstat->mode & 0xffff); + } else { + fibdinfo->EaSize = + smb2_get_reparse_tag_special_file(ksmbd_kstat->kstat->mode); + if (fibdinfo->EaSize) + fibdinfo->ExtFileAttributes = FILE_ATTRIBUTE_REPARSE_POINT_LE; + fibdinfo->Reserved2 = cpu_to_le16(0); + } if (d_info->hide_dot_file && d_info->name[0] == '.') fibdinfo->ExtFileAttributes |= FILE_ATTRIBUTE_HIDDEN_LE; memcpy(fibdinfo->FileName, conv_name, conv_len); @@ -4573,6 +5761,7 @@ static int process_query_dir_entries(struct smb2_query_dir_private *priv) for (i = 0; i < priv->d_info->num_entry; i++) { struct dentry *dent; + struct path path; if (dentry_name(priv->d_info, priv->info_level)) return -EINVAL; @@ -4595,6 +5784,23 @@ static int process_query_dir_entries(struct smb2_query_dir_private *priv) continue; } + if (test_share_config_flag(priv->work->tcon->share_conf, + KSMBD_SHARE_FLAG_HIDE_UNREADABLE)) { + __le32 daccess = FILE_READ_DATA_LE | FILE_READ_EA_LE | + FILE_READ_ATTRIBUTES_LE; + + path.mnt = priv->dir_fp->filp->f_path.mnt; + path.dentry = dent; + rc = smb_check_perm_dacl(priv->work->conn, &path, + &daccess, daccess, + priv->work->sess->user->uid, + true); + if (rc) { + dput(dent); + continue; + } + } + ksmbd_kstat.kstat = &kstat; if (priv->info_level != FILE_NAMES_INFORMATION) { rc = ksmbd_vfs_fill_dentry_attrs(priv->work, @@ -5023,21 +6229,30 @@ err_out2: /** * buffer_check_err() - helper function to check buffer errors * @reqOutputBufferLength: max buffer length expected in command response + * @fixed_len: minimum fixed response length * @rsp: query info response buffer contains output buffer length * @rsp_org: base response buffer pointer in case of chained response * * Return: 0 on success, otherwise error */ static int buffer_check_err(int reqOutputBufferLength, + unsigned int fixed_len, struct smb2_query_info_rsp *rsp, void *rsp_org) { - if (reqOutputBufferLength < le32_to_cpu(rsp->OutputBufferLength)) { + unsigned int output_len = le32_to_cpu(rsp->OutputBufferLength); + + if (reqOutputBufferLength < fixed_len) { pr_err("Invalid Buffer Size Requested\n"); rsp->hdr.Status = STATUS_INFO_LENGTH_MISMATCH; *(__be32 *)rsp_org = cpu_to_be32(sizeof(struct smb2_hdr)); return -EINVAL; } + + if (reqOutputBufferLength < output_len) { + rsp->hdr.Status = STATUS_BUFFER_OVERFLOW; + rsp->OutputBufferLength = cpu_to_le32(reqOutputBufferLength); + } return 0; } @@ -5100,11 +6315,13 @@ static int smb2_get_info_file_pipe(struct ksmbd_session *sess, case FILE_STANDARD_INFORMATION: get_standard_info_pipe(rsp, rsp_org); rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength), + le32_to_cpu(rsp->OutputBufferLength), rsp, rsp_org); break; case FILE_INTERNAL_INFORMATION: get_internal_info_pipe(rsp, id, rsp_org); rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength), + le32_to_cpu(rsp->OutputBufferLength), rsp, rsp_org); break; default: @@ -5202,17 +6419,13 @@ static int smb2_get_ea(struct ksmbd_work *work, struct ksmbd_file *fp, if (strncmp(name, XATTR_USER_PREFIX, XATTR_USER_PREFIX_LEN)) continue; - if (!strncmp(&name[XATTR_USER_PREFIX_LEN], STREAM_PREFIX, - STREAM_PREFIX_LEN)) - continue; - if (req->InputBufferLength && strncmp(&name[XATTR_USER_PREFIX_LEN], ea_req->name, ea_req->EaNameLength)) continue; - if (!strncmp(&name[XATTR_USER_PREFIX_LEN], - DOS_ATTRIBUTE_PREFIX, DOS_ATTRIBUTE_PREFIX_LEN)) + if (smb2_is_private_ea(&name[XATTR_USER_PREFIX_LEN], + name_len - XATTR_USER_PREFIX_LEN)) continue; if (!strncmp(name, XATTR_USER_PREFIX, XATTR_USER_PREFIX_LEN)) @@ -5331,6 +6544,30 @@ static int get_file_basic_info(struct smb2_query_info_rsp *rsp, return 0; } +static int get_file_allocation_stat(struct ksmbd_file *fp, struct kstat *stat) +{ + int ret; + + /* + * Buffered writes can leave delayed allocation in a state where two + * consecutive queries report different block counts even when the + * second write only overwrites the first one. Complete writeback before + * reporting the filesystem allocation for an ordinary open. + */ + if (!fp->allocation_size_set) { + ret = file_write_and_wait(fp->filp); + if (ret) + return ret; + } + + ret = vfs_getattr(&fp->filp->f_path, stat, STATX_BASIC_STATS, + AT_STATX_SYNC_AS_STAT); + if (!ret && !fp->allocation_size_set) + fp->allocation_size = S_ISDIR(stat->mode) ? 0 : stat->blocks << 9; + + return ret; +} + static int get_file_standard_info(struct smb2_query_info_rsp *rsp, struct ksmbd_file *fp, void *rsp_org) { @@ -5339,8 +6576,7 @@ static int get_file_standard_info(struct smb2_query_info_rsp *rsp, struct kstat stat; int ret; - ret = vfs_getattr(&fp->filp->f_path, &stat, STATX_BASIC_STATS, - AT_STATX_SYNC_AS_STAT); + ret = get_file_allocation_stat(fp, &stat); if (ret) return ret; @@ -5351,8 +6587,10 @@ static int get_file_standard_info(struct smb2_query_info_rsp *rsp, sinfo->AllocationSize = cpu_to_le64(fp->allocation_size); sinfo->EndOfFile = S_ISDIR(stat.mode) ? 0 : cpu_to_le64(stat.size); } else { - sinfo->AllocationSize = cpu_to_le64(fp->stream.size); - sinfo->EndOfFile = cpu_to_le64(fp->stream.size); + loff_t seof = ksmbd_stream_eof(fp); + + sinfo->AllocationSize = cpu_to_le64((u64)seof); + sinfo->EndOfFile = cpu_to_le64((u64)seof); } sinfo->NumberOfLinks = cpu_to_le32(get_nlink(&stat) - delete_pending); sinfo->DeletePending = delete_pending; @@ -5387,7 +6625,6 @@ static int get_file_all_info(struct ksmbd_work *work, char *filename; u64 time; int ret, buf_free_len, filename_len; - struct smb2_query_info_req *req = ksmbd_req_buf_next(work); if (!(fp->daccess & FILE_READ_ATTRIBUTES_LE)) { ksmbd_debug(SMB, "no right to read the attributes : 0x%x\n", @@ -5400,17 +6637,15 @@ static int get_file_all_info(struct ksmbd_work *work, return PTR_ERR(filename); filename_len = strlen(filename); - buf_free_len = smb2_calc_max_out_buf_len(work, + buf_free_len = smb2_resp_buf_len(work, offsetof(struct smb2_query_info_rsp, Buffer) + - offsetof(struct smb2_file_all_info, FileName), - le32_to_cpu(req->OutputBufferLength)); + offsetof(struct smb2_file_all_info, FileName)); if (buf_free_len < (filename_len + 1) * 2) { kfree(filename); return -EINVAL; } - ret = vfs_getattr(&fp->filp->f_path, &stat, STATX_BASIC_STATS, - AT_STATX_SYNC_AS_STAT); + ret = get_file_allocation_stat(fp, &stat); if (ret) { kfree(filename); return ret; @@ -5432,8 +6667,10 @@ static int get_file_all_info(struct ksmbd_work *work, file_info->AllocationSize = cpu_to_le64(fp->allocation_size); file_info->EndOfFile = S_ISDIR(stat.mode) ? 0 : cpu_to_le64(stat.size); } else { - file_info->AllocationSize = cpu_to_le64(fp->stream.size); - file_info->EndOfFile = cpu_to_le64(fp->stream.size); + loff_t seof = ksmbd_stream_eof(fp); + + file_info->AllocationSize = cpu_to_le64((u64)seof); + file_info->EndOfFile = cpu_to_le64((u64)seof); } file_info->NumberOfLinks = cpu_to_le32(get_nlink(&stat) - delete_pending); @@ -5481,6 +6718,80 @@ static void get_file_alternate_info(struct ksmbd_work *work, cpu_to_le32(struct_size(file_info, FileName, conv_len)); } +static char *smb2_get_normalized_stream_name(struct ksmbd_file *fp) +{ + char *name, *stream_name = NULL, *xattr_list = NULL; + ssize_t xattr_list_len; + + if (!ksmbd_stream_fd(fp)) + return NULL; + + xattr_list_len = ksmbd_vfs_listxattr(fp->filp->f_path.dentry, + &xattr_list); + if (xattr_list_len <= 0) + goto out; + + for (name = xattr_list; name - xattr_list < xattr_list_len; + name += strlen(name) + 1) { + char *type; + + if (strlen(name) + 1 != fp->stream.size || + strncasecmp(name, fp->stream.name, fp->stream.size - 1)) + continue; + + name += XATTR_NAME_STREAM_LEN; + type = strrchr(name, ':'); + if (type) + stream_name = kstrndup(name, type - name, + KSMBD_DEFAULT_GFP); + break; + } +out: + kvfree(xattr_list); + return stream_name; +} + +static int get_file_normalized_name_info(struct ksmbd_work *work, + struct smb2_query_info_rsp *rsp, + struct ksmbd_file *fp) +{ + struct smb2_file_alt_name_info *file_info; + char *filename, *normalized, *stream_name; + int conv_len, filename_len; + + if (work->conn->dialect < SMB311_PROT_ID) { + rsp->hdr.Status = STATUS_NOT_SUPPORTED; + return -EOPNOTSUPP; + } + + filename = convert_to_nt_pathname(work->tcon->share_conf, + &fp->filp->f_path); + if (IS_ERR(filename)) + return PTR_ERR(filename); + if (filename[0] == '\\') + memmove(filename, filename + 1, strlen(filename)); + + stream_name = smb2_get_normalized_stream_name(fp); + normalized = kasprintf(KSMBD_DEFAULT_GFP, "%s%s%s", filename, + stream_name ? ":" : "", + stream_name ? stream_name : ""); + kfree(stream_name); + kfree(filename); + if (!normalized) + return -ENOMEM; + + filename_len = strlen(normalized); + file_info = (struct smb2_file_alt_name_info *)rsp->Buffer; + conv_len = smbConvertToUTF16((__le16 *)file_info->FileName, + normalized, filename_len, + work->conn->local_nls, 0); + kfree(normalized); + conv_len *= 2; + file_info->FileNameLength = cpu_to_le32(conv_len); + rsp->OutputBufferLength = cpu_to_le32(sizeof(*file_info) + conv_len); + return 0; +} + static int get_file_stream_info(struct ksmbd_work *work, struct smb2_query_info_rsp *rsp, struct ksmbd_file *fp, @@ -5492,9 +6803,10 @@ static int get_file_stream_info(struct ksmbd_work *work, struct kstat stat; const struct path *path = &fp->filp->f_path; ssize_t xattr_list_len; + ssize_t slen; + loff_t ssize; int nbytes = 0, streamlen, stream_name_len, next, idx = 0; int buf_free_len; - struct smb2_query_info_req *req = ksmbd_req_buf_next(work); int ret; ret = vfs_getattr(&fp->filp->f_path, &stat, STATX_BASIC_STATS, @@ -5504,10 +6816,8 @@ static int get_file_stream_info(struct ksmbd_work *work, file_info = (struct smb2_file_stream_info *)rsp->Buffer; - buf_free_len = - smb2_calc_max_out_buf_len(work, - offsetof(struct smb2_query_info_rsp, Buffer), - le32_to_cpu(req->OutputBufferLength)); + buf_free_len = smb2_resp_buf_len(work, + offsetof(struct smb2_query_info_rsp, Buffer)); if (buf_free_len < 0) goto out; @@ -5556,8 +6866,20 @@ static int get_file_stream_info(struct ksmbd_work *work, streamlen *= 2; kfree(stream_buf); file_info->StreamNameLength = cpu_to_le32(streamlen); - file_info->StreamSize = cpu_to_le64(stream_name_len); - file_info->StreamAllocationSize = cpu_to_le64(stream_name_len); + /* + * stream_name_len is the byte length of the xattr's *name*, + * not its value -- same class of bug ksmbd_stream_eof() + * (smb2pdu.c) already fixes for EndOfFile/AllocationSize on + * a stream handle; this enumeration path needs the same + * real xattr value length, not the name length reused as a + * size. + */ + slen = ksmbd_vfs_casexattr_len(file_mnt_idmap(fp->filp), + path->dentry, stream_name, + strlen(stream_name) + 1); + ssize = slen < 0 ? 0 : (loff_t)slen; + file_info->StreamSize = cpu_to_le64(ssize); + file_info->StreamAllocationSize = cpu_to_le64(ssize); nbytes += next; buf_free_len -= next; @@ -5621,8 +6943,7 @@ static int get_file_network_open_info(struct smb2_query_info_rsp *rsp, return -EACCES; } - ret = vfs_getattr(&fp->filp->f_path, &stat, STATX_BASIC_STATS, - AT_STATX_SYNC_AS_STAT); + ret = get_file_allocation_stat(fp, &stat); if (ret) return ret; @@ -5639,8 +6960,10 @@ static int get_file_network_open_info(struct smb2_query_info_rsp *rsp, file_info->AllocationSize = cpu_to_le64(fp->allocation_size); file_info->EndOfFile = S_ISDIR(stat.mode) ? 0 : cpu_to_le64(stat.size); } else { - file_info->AllocationSize = cpu_to_le64(fp->stream.size); - file_info->EndOfFile = cpu_to_le64(fp->stream.size); + loff_t seof = ksmbd_stream_eof(fp); + + file_info->AllocationSize = cpu_to_le64((u64)seof); + file_info->EndOfFile = cpu_to_le64((u64)seof); } file_info->Reserved = cpu_to_le32(0); rsp->OutputBufferLength = @@ -5753,8 +7076,7 @@ static int find_file_posix_info(struct smb2_query_info_rsp *rsp, return -EACCES; } - ret = vfs_getattr(&fp->filp->f_path, &stat, STATX_BASIC_STATS, - AT_STATX_SYNC_AS_STAT); + ret = get_file_allocation_stat(fp, &stat); if (ret) return ret; @@ -5771,8 +7093,10 @@ static int find_file_posix_info(struct smb2_query_info_rsp *rsp, file_info->EndOfFile = cpu_to_le64(stat.size); file_info->AllocationSize = cpu_to_le64(fp->allocation_size); } else { - file_info->EndOfFile = cpu_to_le64(fp->stream.size); - file_info->AllocationSize = cpu_to_le64(fp->stream.size); + loff_t seof = ksmbd_stream_eof(fp); + + file_info->EndOfFile = cpu_to_le64((u64)seof); + file_info->AllocationSize = cpu_to_le64((u64)seof); } file_info->HardLinks = cpu_to_le32(stat.nlink); file_info->Mode = cpu_to_le32(stat.mode & 0777); @@ -5820,6 +7144,7 @@ static int smb2_get_info_file(struct ksmbd_work *work, struct ksmbd_file *fp; int fileinfoclass = 0; int rc = 0; + unsigned int fixed_len; unsigned int id = KSMBD_NO_FID, pid = KSMBD_NO_FID; if (test_share_config_flag(work->tcon->share_conf, @@ -5874,6 +7199,9 @@ static int smb2_get_info_file(struct ksmbd_work *work, case FILE_ALTERNATE_NAME_INFORMATION: get_file_alternate_info(work, rsp, fp, work->response_buf); break; + case FILE_NORMALIZED_NAME_INFORMATION: + rc = get_file_normalized_name_info(work, rsp, fp); + break; case FILE_STREAM_INFORMATION: rc = get_file_stream_info(work, rsp, fp, work->response_buf); @@ -5923,9 +7251,23 @@ static int smb2_get_info_file(struct ksmbd_work *work, fileinfoclass); rc = -EOPNOTSUPP; } - if (!rc) + if (!rc) { + fixed_len = le32_to_cpu(rsp->OutputBufferLength); + switch (fileinfoclass) { + case FILE_ALL_INFORMATION: + fixed_len = FILE_ALL_INFORMATION_SIZE; + break; + case FILE_ALTERNATE_NAME_INFORMATION: + fixed_len = FILE_ALTERNATE_NAME_INFORMATION_SIZE; + break; + case FILE_STREAM_INFORMATION: + fixed_len = FILE_STREAM_INFORMATION_SIZE; + break; + } rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength), + fixed_len, rsp, work->response_buf); + } ksmbd_fd_put(work, fp); iov_pin_out: @@ -5946,6 +7288,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, struct kstatfs stfs; struct path path; int rc = 0, len; + unsigned int fixed_len = 0; if (!share->path) return -EIO; @@ -5981,6 +7324,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->DeviceCharacteristics |= cpu_to_le32(FILE_READ_ONLY_DEVICE); rsp->OutputBufferLength = cpu_to_le32(8); + fixed_len = 8; break; } case FS_ATTRIBUTE_INFORMATION: @@ -5995,6 +7339,8 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, attrs = FILE_SUPPORTS_OBJECT_IDS | FILE_PERSISTENT_ACLS | FILE_UNICODE_ON_DISK | + FILE_FILE_COMPRESSION | + FILE_SUPPORTS_SPARSE_FILES | FILE_SUPPORTS_BLOCK_REFCOUNTING; err = vfs_fileattr_get(path.dentry, &fa); @@ -6033,6 +7379,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->FileSystemNameLen = cpu_to_le32(len); sz = sizeof(FILE_SYSTEM_ATTRIBUTE_INFO) + len; rsp->OutputBufferLength = cpu_to_le32(sz); + fixed_len = 16; break; } case FS_VOLUME_INFORMATION: @@ -6060,6 +7407,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->SupportsObjects = 0; sz = sizeof(struct filesystem_vol_info) + len; rsp->OutputBufferLength = cpu_to_le32(sz); + fixed_len = 24; break; } case FS_SIZE_INFORMATION: @@ -6072,6 +7420,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->SectorsPerAllocationUnit = cpu_to_le32(1); info->BytesPerSector = cpu_to_le32(stfs.f_bsize); rsp->OutputBufferLength = cpu_to_le32(24); + fixed_len = 24; break; } case FS_FULL_SIZE_INFORMATION: @@ -6087,6 +7436,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->SectorsPerAllocationUnit = cpu_to_le32(1); info->BytesPerSector = cpu_to_le32(stfs.f_bsize); rsp->OutputBufferLength = cpu_to_le32(32); + fixed_len = 32; break; } case FS_OBJECT_ID_INFORMATION: @@ -6107,6 +7457,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->extended_info.rel_date = 0; memcpy(info->extended_info.version_string, "1.1.0", strlen("1.1.0")); rsp->OutputBufferLength = cpu_to_le32(64); + fixed_len = 64; break; } case FS_SECTOR_SIZE_INFORMATION: @@ -6124,10 +7475,12 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->FSEffPhysicalBytesPerSectorForAtomicity = cpu_to_le32(sector_size); info->Flags = cpu_to_le32(SSINFO_FLAGS_ALIGNED_DEVICE | - SSINFO_FLAGS_PARTITION_ALIGNED_ON_DEVICE); + SSINFO_FLAGS_PARTITION_ALIGNED_ON_DEVICE | + SSINFO_FLAGS_TRIM_ENABLED); info->ByteOffsetForSectorAlignment = 0; info->ByteOffsetForPartitionAlignment = 0; rsp->OutputBufferLength = cpu_to_le32(28); + fixed_len = 28; break; } case FS_CONTROL_INFORMATION: @@ -6148,6 +7501,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->DefaultQuotaLimit = cpu_to_le64(SMB2_NO_FID); info->Padding = 0; rsp->OutputBufferLength = cpu_to_le32(48); + fixed_len = 48; break; } case FS_POSIX_INFORMATION: @@ -6168,6 +7522,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->TotalFileNodes = cpu_to_le64(stfs.f_files); info->FreeFileNodes = cpu_to_le64(stfs.f_ffree); rsp->OutputBufferLength = cpu_to_le32(56); + fixed_len = 56; } break; } @@ -6176,6 +7531,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, return -EOPNOTSUPP; } rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength), + fixed_len, rsp, work->response_buf); path_put(&path); @@ -6229,6 +7585,18 @@ static int smb2_get_info_sec(struct ksmbd_work *work, if (!fp) return -ENOENT; + if (addition_info & (OWNER_SECINFO | GROUP_SECINFO | DACL_SECINFO) && + !(fp->daccess & FILE_READ_CONTROL_LE)) { + ksmbd_fd_put(work, fp); + return -EACCES; + } + + if (le32_to_cpu(req->OutputBufferLength) < sizeof(struct smb_ntsd)) { + rsp->hdr.Status = STATUS_BUFFER_TOO_SMALL; + ksmbd_fd_put(work, fp); + return -ENOSPC; + } + idmap = file_mnt_idmap(fp->filp); inode = file_inode(fp->filp); ksmbd_acls_fattr(&fattr, idmap, inode); @@ -6277,6 +7645,7 @@ release_acl: rsp->OutputBufferLength = cpu_to_le32(secdesclen); rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength), + le32_to_cpu(rsp->OutputBufferLength), rsp, work->response_buf); if (rc) goto err_out; @@ -6352,7 +7721,7 @@ err_out: rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; else if (rc == -EINVAL && rsp->hdr.Status == 0) rsp->hdr.Status = STATUS_INVALID_PARAMETER; - else if (rc == -EOPNOTSUPP || rsp->hdr.Status == 0) + else if (rsp->hdr.Status == 0) rsp->hdr.Status = STATUS_INVALID_INFO_CLASS; smb2_set_err_rsp(work); @@ -6407,7 +7776,6 @@ int smb2_close(struct ksmbd_work *work) u64 sess_id; struct smb2_close_req *req; struct smb2_close_rsp *rsp; - struct ksmbd_conn *conn = work->conn; struct ksmbd_file *fp; u64 time; int err = 0; @@ -6430,7 +7798,7 @@ int smb2_close(struct ksmbd_work *work) sess_id = work->compound_sid; work->compound_sid = 0; - if (check_session_id(conn, sess_id)) { + if (work->sess && work->sess->id == sess_id) { work->compound_sid = sess_id; } else { rsp->hdr.Status = STATUS_USER_SESSION_DELETED; @@ -6607,9 +7975,12 @@ static int smb2_rename(struct ksmbd_work *work, if (!file_info->ReplaceIfExists) flags = RENAME_NOREPLACE; - rc = ksmbd_vfs_rename(work, &fp->filp->f_path, new_name, flags); - if (!rc) - smb_break_all_levII_oplock(work, fp, 0); + rc = ksmbd_vfs_check_rename_share(work, &fp->filp->f_path); + if (rc) + goto out; + + smb_break_all_levII_oplock_rename(work, fp); + rc = ksmbd_vfs_rename(work, fp, new_name, flags); out: kfree(new_name); return rc; @@ -6690,6 +8061,8 @@ static int set_file_basic_info(struct ksmbd_file *fp, struct file *filp; struct inode *inode; struct mnt_idmap *idmap; + __le32 attrs_mask = FILE_ATTRIBUTE_DIRECTORY_LE | + FILE_ATTRIBUTE_COMPRESSED_LE; int rc = 0; if (!(fp->daccess & FILE_WRITE_ATTRIBUTES_LE)) @@ -6727,8 +8100,9 @@ static int set_file_basic_info(struct ksmbd_file *fp, } if (!(S_ISDIR(inode->i_mode) && file_info->Attributes == FILE_ATTRIBUTE_NORMAL_LE)) - fp->f_ci->m_fattr = file_info->Attributes | - (fp->f_ci->m_fattr & FILE_ATTRIBUTE_DIRECTORY_LE); + fp->f_ci->m_fattr = + (file_info->Attributes & ~FILE_ATTRIBUTE_COMPRESSED_LE) | + (fp->f_ci->m_fattr & attrs_mask); } if (test_share_config_flag(share, KSMBD_SHARE_FLAG_STORE_DOS_ATTRS) && @@ -6835,6 +8209,7 @@ static int set_file_allocation_info(struct ksmbd_work *work, } fp->allocation_size = le64_to_cpu(file_alloc_info->AllocationSize); + fp->allocation_size_set = true; return 0; } @@ -6902,18 +8277,21 @@ static int set_file_disposition_info(struct ksmbd_work *work, return -EACCES; } + if (fp->f_ci->m_fattr & FILE_ATTRIBUTE_READONLY_LE) + return -EACCES; + inode = file_inode(fp->filp); if (file_info->DeletePending) { if (ksmbd_has_stream_without_delete_share(fp)) return -ESHARE; - if (S_ISDIR(inode->i_mode) && + if (S_ISDIR(inode->i_mode) && !ksmbd_stream_fd(fp) && ksmbd_vfs_empty_dir(fp) == -ENOTEMPTY) return -EBUSY; smb_break_all_levII_oplock_for_delete(work, fp); - ksmbd_set_inode_pending_delete(fp); + ksmbd_fd_set_delete_pending(fp); } else { - ksmbd_clear_inode_pending_delete(fp); + ksmbd_fd_clear_delete_pending(fp); } return 0; } @@ -6930,7 +8308,7 @@ static int set_file_position_info(struct ksmbd_file *fp, sector_size = inode->i_sb->s_blocksize; if (current_byte_offset < 0 || - (fp->coption == FILE_NO_INTERMEDIATE_BUFFERING_LE && + (fp->coption & FILE_NO_INTERMEDIATE_BUFFERING_LE && current_byte_offset & (sector_size - 1))) { pr_err("CurrentByteOffset is not valid : %llu\n", current_byte_offset); @@ -7103,6 +8481,7 @@ int smb2_set_info(struct ksmbd_work *work) struct smb2_set_info_rsp *rsp; struct ksmbd_file *fp = NULL; int rc = 0; + bool chseq_err = false; unsigned int id = KSMBD_NO_FID, pid = KSMBD_NO_FID; ksmbd_debug(SMB, "Received smb2 set info request\n"); @@ -7142,6 +8521,13 @@ int smb2_set_info(struct ksmbd_work *work) goto err_out; } + rc = smb2_set_request_open(work, fp, &req->hdr, true, false); + if (rc) { + rsp->hdr.Status = STATUS_FILE_NOT_AVAILABLE; + chseq_err = true; + goto err_out; + } + saved_cred = override_creds(fp->filp->f_cred); switch (req->InfoType) { case SMB2_O_INFO_FILE: @@ -7172,8 +8558,14 @@ int smb2_set_info(struct ksmbd_work *work) return 0; err_out: - if (rc == -EACCES || rc == -EPERM || rc == -EXDEV) - rsp->hdr.Status = STATUS_ACCESS_DENIED; + if (rc == -EACCES || rc == -EPERM || rc == -EXDEV) { + if (fp && req->InfoType == SMB2_O_INFO_FILE && + req->FileInfoClass == FILE_DISPOSITION_INFORMATION && + fp->f_ci->m_fattr & FILE_ATTRIBUTE_READONLY_LE) + rsp->hdr.Status = STATUS_CANNOT_DELETE; + else + rsp->hdr.Status = STATUS_ACCESS_DENIED; + } else if (rc == -EINVAL) rsp->hdr.Status = STATUS_INVALID_PARAMETER; else if (rc == -EMSGSIZE) @@ -7186,7 +8578,7 @@ err_out: rsp->hdr.Status = STATUS_OBJECT_NAME_INVALID; else if (rc == -EBUSY || rc == -ENOTEMPTY) rsp->hdr.Status = STATUS_DIRECTORY_NOT_EMPTY; - else if (rc == -EAGAIN) + else if (rc == -EAGAIN && !chseq_err) rsp->hdr.Status = STATUS_FILE_LOCK_CONFLICT; else if (rc == -EBADF || rc == -ESTALE) rsp->hdr.Status = STATUS_INVALID_HANDLE; @@ -7267,18 +8659,31 @@ out: return err; } -static int smb2_set_remote_key_for_rdma(struct ksmbd_work *work, - struct smbdirect_buffer_descriptor_v1 *desc, - __le32 Channel, - __le16 ChannelInfoLength) +/** + * smb2_set_rdma_key() - validate descriptors and save invalidation state + * @work: request work item + * @desc: first RDMA buffer descriptor + * @Channel: nested RDMA channel type + * @channel_info_len: descriptor array length + * + * Return: 0 on success, otherwise -EINVAL + */ +static int smb2_set_rdma_key(struct ksmbd_work *work, + struct smbdirect_buffer_descriptor_v1 *desc, + __le32 Channel, __le16 channel_info_len) { unsigned int i, ch_count; + if (Channel != SMB2_CHANNEL_RDMA_V1 && + Channel != SMB2_CHANNEL_RDMA_V1_INVALIDATE) + return -EINVAL; if (work->conn->dialect == SMB30_PROT_ID && Channel != SMB2_CHANNEL_RDMA_V1) return -EINVAL; + if (le16_to_cpu(channel_info_len) % sizeof(*desc)) + return -EINVAL; - ch_count = le16_to_cpu(ChannelInfoLength) / sizeof(*desc); + ch_count = le16_to_cpu(channel_info_len) / sizeof(*desc); if (ksmbd_debug_types & KSMBD_DEBUG_RDMA) { for (i = 0; i < ch_count; i++) { pr_info("RDMA r/w request %#x: token %#x, length %#x\n", @@ -7297,9 +8702,240 @@ static int smb2_set_remote_key_for_rdma(struct ksmbd_work *work, return 0; } -static ssize_t smb2_read_rdma_channel(struct ksmbd_work *work, - struct smb2_read_req *req, void *data_buf, - size_t length) +/** + * smb2_prep_rdma_read() - transform an RDMA READ payload + * @work: request work item + * @req: READ request controlling encryption or signing + * @rsp: READ response receiving transform metadata + * @data: data that will be transferred through RDMA + * @datalen: data length + * + * Encrypt the payload in place and encode the detached crypto metadata in + * the response buffer. + * + * Return: metadata length, zero when no transform applies, or negative errno + */ +static int smb2_prep_rdma_read(struct ksmbd_work *work, + struct smb2_read_req *req, + struct smb2_read_rsp *rsp, + void *data, unsigned int datalen) +{ + struct ksmbd_conn *conn = work->conn; + struct smb2_rdma_transform *transform; + struct smb2_rdma_crypto_transform *crypto; + u8 *nonce; + unsigned int nonce_len = 0, transform_len; + u16 transform_type; + int err; + + if (!work->encrypted || + !(conn->rdma_transform_ids & BIT(SMB2_RDMA_TRANSFORM_ENCRYPTION))) + return 0; + + transform_type = SMB2_RDMA_TRANSFORM_TYPE_ENCRYPTION; + nonce_len = (conn->cipher_type == SMB2_ENCRYPTION_AES128_GCM || + conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM) ? + SMB3_AES_GCM_NONCE : SMB3_AES_CCM_NONCE; + + transform = (struct smb2_rdma_transform *)rsp->Buffer; + crypto = (struct smb2_rdma_crypto_transform *)(transform + 1); + memset(transform, 0, sizeof(*transform) + sizeof(*crypto) + + SMB2_SIGNATURE_SIZE + nonce_len); + transform->Channel = SMB2_CHANNEL_NONE; + transform->TransformCount = cpu_to_le16(1); + + crypto->TransformType = cpu_to_le16(transform_type); + crypto->SignatureLength = cpu_to_le16(SMB2_SIGNATURE_SIZE); + crypto->NonceLength = cpu_to_le16(nonce_len); + nonce = crypto->Signature + SMB2_SIGNATURE_SIZE; + + get_random_bytes(nonce, nonce_len); + err = ksmbd_crypt_rdma(conn, + work->sess->smb3encryptionkey, + data, datalen, nonce, nonce_len, + crypto->Signature, + SMB2_SIGNATURE_SIZE, true); + if (err) { + pr_err("RDMA READ encryption failed: session=%llu payload=%u rc=%d\n", + work->sess->id, datalen, err); + return err; + } + + transform_len = sizeof(*transform) + sizeof(*crypto) + + SMB2_SIGNATURE_SIZE + nonce_len; + rsp->Flags = SMB2_READFLAG_RESPONSE_RDMA_TRANSFORM; + rsp->DataLength = cpu_to_le32(transform_len); + ksmbd_debug(RDMA, + "RDMA READ encryption prepared: session=%llu cipher=0x%04x payload=%u transform=%u nonce=%u tag=%u\n", + work->sess->id, le16_to_cpu(conn->cipher_type), datalen, + transform_len, nonce_len, SMB2_SIGNATURE_SIZE); + return transform_len; +} + +struct smb2_rdma_write_transform { + struct smbdirect_buffer_descriptor_v1 *desc; + struct smb2_rdma_crypto_transform *crypto; + u8 *nonce; + unsigned int desc_len; + unsigned int nonce_len; + unsigned int signature_len; + u16 type; + __le32 channel; +}; + +/** + * smb2_current_req_len() - return the current compound request element size + * @work: request work item + * @hdr: current SMB2 header + * + * Return: current request element length measured from the SMB2 header + */ +static unsigned int smb2_current_req_len(struct ksmbd_work *work, + struct smb2_hdr *hdr) +{ + if (hdr->NextCommand) + return le32_to_cpu(hdr->NextCommand); + return get_rfc1002_len(work->request_buf) - + work->next_smb2_rcv_hdr_off; +} + +/** + * check_rdma_desc() - validate an RDMA descriptor array + * @desc: descriptor array + * @desc_len: descriptor array length + * @required_len: minimum aggregate buffer length + * + * Return: 0 when the descriptors cover the transfer, otherwise -EINVAL + */ +static int check_rdma_desc(struct smbdirect_buffer_descriptor_v1 *desc, + unsigned int desc_len, + unsigned int required_len) +{ + unsigned int i, count; + u64 described_len = 0; + + if (!desc_len || desc_len % sizeof(*desc)) + return -EINVAL; + count = desc_len / sizeof(*desc); + if (!le32_to_cpu(desc[0].length)) + return -EINVAL; + for (i = 0; i < count; i++) + described_len += le32_to_cpu(desc[i].length); + return described_len < required_len ? -EINVAL : 0; +} + +/** + * smb2_parse_rdma_write_transform() - validate RDMA WRITE transform metadata + * @work: request work item + * @req: WRITE request containing the transform + * @info: parsed transform information + * + * Validate transform counts, crypto fields, descriptor alignment and bounds, + * negotiated algorithms, and the nested RDMA channel. + * + * Return: 0 on success, otherwise a negative errno + */ +static int smb2_parse_rdma_write_transform(struct ksmbd_work *work, + struct smb2_write_req *req, + struct smb2_rdma_write_transform *info) +{ + struct smb2_rdma_transform *transform; + struct smb2_rdma_crypto_transform *crypto; + unsigned int req_len = smb2_current_req_len(work, &req->hdr); + unsigned int offset = le16_to_cpu(req->WriteChannelInfoOffset); + unsigned int length = le16_to_cpu(req->WriteChannelInfoLength); + unsigned int desc_offset, desc_len, crypto_len, expected_desc_offset; + int err; + + if (!work->conn->rdma_transform_ids || + offset < offsetof(struct smb2_write_req, Buffer) || + length < sizeof(*transform) || offset > req_len || + length > req_len - offset) + return -EINVAL; + + transform = (struct smb2_rdma_transform *)((char *)req + offset); + if (le16_to_cpu(transform->TransformCount) != 1 || + (transform->Channel != SMB2_CHANNEL_RDMA_V1 && + transform->Channel != SMB2_CHANNEL_RDMA_V1_INVALIDATE)) + return -EINVAL; + + desc_offset = le16_to_cpu(transform->RdmaDescriptorOffset); + desc_len = le16_to_cpu(transform->RdmaDescriptorLength); + if (!desc_len || desc_len % sizeof(*info->desc) || + desc_offset < sizeof(*transform) || desc_offset > length || + desc_len > length - desc_offset) + return -EINVAL; + + crypto = (struct smb2_rdma_crypto_transform *)(transform + 1); + if (length - sizeof(*transform) < sizeof(*crypto)) + return -EINVAL; + info->type = le16_to_cpu(crypto->TransformType); + info->signature_len = le16_to_cpu(crypto->SignatureLength); + info->nonce_len = le16_to_cpu(crypto->NonceLength); + if (!info->signature_len) + return info->type == SMB2_RDMA_TRANSFORM_TYPE_ENCRYPTION ? + -EBADMSG : -EINVAL; + if (info->signature_len > SMB2_SIGNATURE_SIZE) + return info->type == SMB2_RDMA_TRANSFORM_TYPE_ENCRYPTION ? + -EBADMSG : -EINVAL; + if (info->signature_len > length - sizeof(*transform) - sizeof(*crypto) || + info->nonce_len > length - sizeof(*transform) - sizeof(*crypto) - + info->signature_len) + return -EINVAL; + + crypto_len = sizeof(*crypto) + info->signature_len + info->nonce_len; + expected_desc_offset = ALIGN(sizeof(*transform) + crypto_len, 8); + if (desc_offset != expected_desc_offset) + return -EINVAL; + + if (info->type == SMB2_RDMA_TRANSFORM_TYPE_ENCRYPTION) { + unsigned int expected_nonce_len; + + if (!(work->conn->rdma_transform_ids & + BIT(SMB2_RDMA_TRANSFORM_ENCRYPTION)) || !work->encrypted) + return -EINVAL; + expected_nonce_len = + (work->conn->cipher_type == SMB2_ENCRYPTION_AES128_GCM || + work->conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM) ? + SMB3_AES_GCM_NONCE : SMB3_AES_CCM_NONCE; + if (info->nonce_len != expected_nonce_len) + return -EBADMSG; + } else { + return -EINVAL; + } + + info->desc = (struct smbdirect_buffer_descriptor_v1 *) + ((char *)transform + desc_offset); + info->desc_len = desc_len; + info->crypto = crypto; + info->nonce = crypto->Signature + info->signature_len; + info->channel = transform->Channel; + err = check_rdma_desc(info->desc, info->desc_len, + le32_to_cpu(req->RemainingBytes)); + if (err) + return err; + + ksmbd_debug(RDMA, + "RDMA WRITE encryption metadata: session=%llu cipher=0x%04x payload=%u channel=0x%x descriptors=%zu nonce=%u tag=%u\n", + work->sess->id, le16_to_cpu(work->conn->cipher_type), + le32_to_cpu(req->RemainingBytes), le32_to_cpu(info->channel), + info->desc_len / sizeof(*info->desc), info->nonce_len, + info->signature_len); + return 0; +} + +/** + * smb2_read_rdma() - transfer READ data to client RDMA buffers + * @work: request work item + * @req: READ request containing client descriptors + * @data_buf: data to transfer + * @length: data length + * + * Return: transferred length on success, otherwise a negative errno + */ +static ssize_t smb2_read_rdma(struct ksmbd_work *work, + struct smb2_read_req *req, void *data_buf, + size_t length) { int err; @@ -7329,6 +8965,7 @@ int smb2_read(struct ksmbd_work *work) size_t length, mincount; ssize_t nbytes = 0, remain_bytes = 0; int err = 0; + int rdma_transform_len = 0; bool is_rdma_channel = false, async_interim = false; unsigned int max_read_size = conn->vals->max_read_size; unsigned int id = KSMBD_NO_FID, pid = KSMBD_NO_FID; @@ -7363,6 +9000,12 @@ int smb2_read(struct ksmbd_work *work) pid = req->PersistentFileId; } + if (req->Channel != SMB2_CHANNEL_NONE && + req->Channel != SMB2_CHANNEL_RDMA_V1 && + req->Channel != SMB2_CHANNEL_RDMA_V1_INVALIDATE) { + err = -EINVAL; + goto out; + } if (req->Channel == SMB2_CHANNEL_RDMA_V1_INVALIDATE || req->Channel == SMB2_CHANNEL_RDMA_V1) { is_rdma_channel = true; @@ -7375,16 +9018,24 @@ int smb2_read(struct ksmbd_work *work) if (is_rdma_channel == true) { unsigned int ch_offset = le16_to_cpu(req->ReadChannelInfoOffset); + unsigned int ch_len = le16_to_cpu(req->ReadChannelInfoLength); + unsigned int req_len = smb2_current_req_len(work, &req->hdr); + struct smbdirect_buffer_descriptor_v1 *desc; - if (ch_offset < offsetof(struct smb2_read_req, Buffer)) { + if (!le32_to_cpu(req->Length) || + ch_offset < offsetof(struct smb2_read_req, Buffer) || + ch_offset > req_len || ch_len > req_len - ch_offset) { err = -EINVAL; goto out; } - err = smb2_set_remote_key_for_rdma(work, - (struct smbdirect_buffer_descriptor_v1 *) - ((char *)req + ch_offset), - req->Channel, - req->ReadChannelInfoLength); + desc = (struct smbdirect_buffer_descriptor_v1 *) + ((char *)req + ch_offset); + err = check_rdma_desc(desc, ch_len, le32_to_cpu(req->Length)); + if (err) + goto out; + err = smb2_set_rdma_key(work, desc, + req->Channel, + req->ReadChannelInfoLength); if (err) goto out; } @@ -7395,6 +9046,10 @@ int smb2_read(struct ksmbd_work *work) goto out; } + err = smb2_set_request_open(work, fp, &req->hdr, true, true); + if (err) + goto out; + if (!(fp->daccess & (FILE_READ_DATA_LE | FILE_READ_ATTRIBUTES_LE))) { pr_err("Not permitted to read : 0x%x\n", fp->daccess); err = -EACCES; @@ -7463,10 +9118,28 @@ int smb2_read(struct ksmbd_work *work) nbytes, offset, mincount); if (is_rdma_channel == true) { + rdma_transform_len = smb2_prep_rdma_read(work, req, + rsp, + aux_payload_buf, + nbytes); + if (rdma_transform_len < 0) { + kvfree(aux_payload_buf); + err = rdma_transform_len; + goto out; + } /* write data to the client using rdma channel */ - remain_bytes = smb2_read_rdma_channel(work, req, - aux_payload_buf, - nbytes); + remain_bytes = smb2_read_rdma(work, req, + aux_payload_buf, + nbytes); + if (remain_bytes < 0) + pr_err("RDMA READ transfer failed: session=%llu payload=%zu transform=%d rc=%zd\n", + work->sess ? work->sess->id : 0, nbytes, + rdma_transform_len, remain_bytes); + else + ksmbd_debug(RDMA, + "RDMA READ transfer completed: session=%llu payload=%zu transform=%d\n", + work->sess ? work->sess->id : 0, nbytes, + rdma_transform_len); kvfree(aux_payload_buf); aux_payload_buf = NULL; nbytes = 0; @@ -7479,11 +9152,13 @@ int smb2_read(struct ksmbd_work *work) rsp->StructureSize = cpu_to_le16(17); rsp->DataOffset = 80; rsp->Reserved = 0; - rsp->DataLength = cpu_to_le32(nbytes); + rsp->DataLength = cpu_to_le32(rdma_transform_len ?: nbytes); rsp->DataRemaining = cpu_to_le32(remain_bytes); - rsp->Flags = 0; + rsp->Flags = rdma_transform_len ? + SMB2_READFLAG_RESPONSE_RDMA_TRANSFORM : 0; err = ksmbd_iov_pin_rsp_read(work, (void *)rsp, - offsetof(struct smb2_read_rsp, Buffer), + offsetof(struct smb2_read_rsp, Buffer) + + rdma_transform_len, aux_payload_buf, nbytes); if (err) { kvfree(aux_payload_buf); @@ -7595,10 +9270,28 @@ out: return err; } -static ssize_t smb2_write_rdma_channel(struct ksmbd_work *work, - struct smb2_write_req *req, - struct ksmbd_file *fp, - loff_t offset, size_t length, bool sync) +/** + * smb2_write_rdma() - receive and store an RDMA WRITE payload + * @work: request work item + * @desc: client RDMA buffer descriptors + * @desc_len: descriptor array length + * @transform: parsed transform, or NULL for an untransformed transfer + * @fp: target open file + * @offset: target file offset + * @length: transfer length + * @sync: request synchronous storage completion + * + * Receive the payload, authenticate or decrypt it when required, and write it + * to the target file. + * + * Return: written byte count on success, otherwise a negative errno + */ +static ssize_t smb2_write_rdma(struct ksmbd_work *work, + struct smbdirect_buffer_descriptor_v1 *desc, + unsigned int desc_len, + struct smb2_rdma_write_transform *transform, + struct ksmbd_file *fp, loff_t offset, + size_t length, bool sync) { char *data_buf; int ret; @@ -7608,19 +9301,42 @@ static ssize_t smb2_write_rdma_channel(struct ksmbd_work *work, if (!data_buf) return -ENOMEM; - ret = ksmbd_conn_rdma_read(work->conn, data_buf, length, - (struct smbdirect_buffer_descriptor_v1 *) - ((char *)req + le16_to_cpu(req->WriteChannelInfoOffset)), - le16_to_cpu(req->WriteChannelInfoLength)); + ret = ksmbd_conn_rdma_read(work->conn, data_buf, length, desc, + desc_len); if (ret < 0) { + if (transform) + pr_err("RDMA WRITE encrypted transfer failed: session=%llu payload=%zu rdma_read_rc=%d\n", + work->sess->id, length, ret); kvfree(data_buf); return ret; } + if (transform && + transform->type == SMB2_RDMA_TRANSFORM_TYPE_ENCRYPTION) { + ret = ksmbd_crypt_rdma(work->conn, + work->sess->smb3decryptionkey, + data_buf, length, transform->nonce, + transform->nonce_len, + transform->crypto->Signature, + transform->signature_len, false); + if (ret) { + pr_err("RDMA WRITE decryption failed: session=%llu payload=%zu rc=%d\n", + work->sess->id, length, ret); + kvfree(data_buf); + return ret == -ENOMEM ? ret : -EBADMSG; + } + } ret = ksmbd_vfs_write(work, fp, data_buf, length, &offset, sync, &nbytes); kvfree(data_buf); - if (ret < 0) + if (ret < 0) { + if (transform) + pr_err("RDMA WRITE encrypted file write failed: session=%llu payload=%zu rc=%d\n", + work->sess->id, length, ret); return ret; + } + ksmbd_debug(RDMA, + "RDMA WRITE transfer completed: session=%llu payload=%zu transformed=%u written=%zd\n", + work->sess ? work->sess->id : 0, length, !!transform, nbytes); return nbytes; } @@ -7635,6 +9351,10 @@ int smb2_write(struct ksmbd_work *work) { struct smb2_write_req *req; struct smb2_write_rsp *rsp; + struct smb2_rdma_write_transform rdma_transform = {}; + struct smb2_rdma_write_transform *rdma_info = NULL; + struct smbdirect_buffer_descriptor_v1 *rdma_desc = NULL; + unsigned int rdma_desc_len = 0; struct ksmbd_file *fp = NULL; loff_t offset; size_t length; @@ -7642,6 +9362,7 @@ int smb2_write(struct ksmbd_work *work) char *data_buf; bool writethrough = false, is_rdma_channel = false; bool async_interim = false; + bool chseq_err = false; int err = 0; unsigned int max_write_size = work->conn->vals->max_write_size; unsigned int id = KSMBD_NO_FID, pid = KSMBD_NO_FID; @@ -7672,12 +9393,27 @@ int smb2_write(struct ksmbd_work *work) } offset = le64_to_cpu(req->Offset); - if (offset < 0) - return -EINVAL; + if (offset < 0) { + err = -EINVAL; + goto out; + } length = le32_to_cpu(req->Length); + if (req->Channel != SMB2_CHANNEL_NONE && + req->Channel != SMB2_CHANNEL_RDMA_V1 && + req->Channel != SMB2_CHANNEL_RDMA_V1_INVALIDATE && + req->Channel != SMB2_CHANNEL_RDMA_TRANSFORM) { + err = -EINVAL; + goto out; + } + if (req->Channel == SMB2_CHANNEL_RDMA_TRANSFORM && + work->conn->dialect != SMB311_PROT_ID) { + err = -EINVAL; + goto out; + } if (req->Channel == SMB2_CHANNEL_RDMA_V1 || - req->Channel == SMB2_CHANNEL_RDMA_V1_INVALIDATE) { + req->Channel == SMB2_CHANNEL_RDMA_V1_INVALIDATE || + req->Channel == SMB2_CHANNEL_RDMA_TRANSFORM) { is_rdma_channel = true; max_write_size = get_smbd_max_read_write_size(work->conn->transport); if (max_write_size == 0) { @@ -7687,19 +9423,55 @@ int smb2_write(struct ksmbd_work *work) length = le32_to_cpu(req->RemainingBytes); } + if (length) { + u64 end = (u64)offset + length; + + if (end > SMB2_MAX_FILE_SIZE) { + err = -EINVAL; + goto out; + } + if (end == SMB2_MAX_FILE_SIZE) { + err = -EFBIG; + goto out; + } + } + if (is_rdma_channel == true) { unsigned int ch_offset = le16_to_cpu(req->WriteChannelInfoOffset); + unsigned int ch_len = le16_to_cpu(req->WriteChannelInfoLength); + unsigned int req_len = smb2_current_req_len(work, &req->hdr); - if (req->Length != 0 || req->DataOffset != 0 || - ch_offset < offsetof(struct smb2_write_req, Buffer)) { + if (!length || req->Length != 0 || req->DataOffset != 0 || + ch_offset < offsetof(struct smb2_write_req, Buffer) || + ch_offset > req_len || ch_len > req_len - ch_offset) { err = -EINVAL; goto out; } - err = smb2_set_remote_key_for_rdma(work, - (struct smbdirect_buffer_descriptor_v1 *) - ((char *)req + ch_offset), - req->Channel, - req->WriteChannelInfoLength); + if (req->Channel == SMB2_CHANNEL_RDMA_TRANSFORM) { + err = smb2_parse_rdma_write_transform(work, req, + &rdma_transform); + if (err) { + pr_err("RDMA WRITE encryption metadata rejected: session=%llu rc=%d\n", + work->sess ? work->sess->id : 0, err); + goto out; + } + rdma_desc = rdma_transform.desc; + rdma_desc_len = rdma_transform.desc_len; + rdma_info = &rdma_transform; + err = smb2_set_rdma_key(work, rdma_desc, + rdma_transform.channel, + cpu_to_le16(rdma_desc_len)); + } else { + rdma_desc = (struct smbdirect_buffer_descriptor_v1 *) + ((char *)req + ch_offset); + rdma_desc_len = ch_len; + err = check_rdma_desc(rdma_desc, rdma_desc_len, length); + if (err) + goto out; + err = smb2_set_rdma_key(work, rdma_desc, + req->Channel, + req->WriteChannelInfoLength); + } if (err) goto out; } @@ -7716,6 +9488,13 @@ int smb2_write(struct ksmbd_work *work) goto out; } + err = smb2_set_request_open(work, fp, &req->hdr, true, false); + if (err) { + rsp->hdr.Status = STATUS_FILE_NOT_AVAILABLE; + chseq_err = true; + goto out; + } + if (!(fp->daccess & (FILE_WRITE_DATA_LE | FILE_READ_ATTRIBUTES_LE))) { pr_err("Not permitted to write : 0x%x\n", fp->daccess); err = -EACCES; @@ -7761,8 +9540,9 @@ int smb2_write(struct ksmbd_work *work) /* read data from the client using rdma channel, and * write the data. */ - nbytes = smb2_write_rdma_channel(work, req, fp, offset, length, - writethrough); + nbytes = smb2_write_rdma(work, rdma_desc, rdma_desc_len, + rdma_info, fp, offset, length, + writethrough); if (nbytes < 0) { err = (int)nbytes; goto out; @@ -7787,7 +9567,7 @@ out: if (async_interim) release_async_work(work); - if (err == -EAGAIN) + if (err == -EAGAIN && !chseq_err) rsp->hdr.Status = STATUS_FILE_LOCK_CONFLICT; else if (err == -ENOSPC || err == -EFBIG) rsp->hdr.Status = STATUS_DISK_FULL; @@ -7799,7 +9579,11 @@ out: rsp->hdr.Status = STATUS_SHARING_VIOLATION; else if (err == -EINVAL) rsp->hdr.Status = STATUS_INVALID_PARAMETER; - else + else if (err == -EBADMSG) + rsp->hdr.Status = STATUS_AUTH_TAG_MISMATCH; + else if (err == -EKEYREJECTED) + rsp->hdr.Status = STATUS_INVALID_SIGNATURE; + else if (rsp->hdr.Status == 0) rsp->hdr.Status = STATUS_INVALID_HANDLE; smb2_set_err_rsp(work); @@ -7866,6 +9650,7 @@ int smb2_cancel(struct ksmbd_work *work) struct smb2_hdr *hdr = smb_get_msg(work->request_buf); struct smb2_hdr *chdr; struct ksmbd_work *iter; + struct ksmbd_work *cancelled_notify = NULL; struct list_head *command_list; if (work->next_smb2_rcv_hdr_off) @@ -7903,11 +9688,23 @@ int smb2_cancel(struct ksmbd_work *work) le64_to_cpu(hdr->Id.AsyncId), le16_to_cpu(chdr->Command)); iter->state = KSMBD_WORK_CANCELLED; - if (iter->cancel_fn) + if (iter->cancel_fn == smb2_notify_cancel_fn) + cancelled_notify = + smb2_notify_cancel_claim(iter->cancel_argv); + else if (iter->cancel_fn) iter->cancel_fn(iter->cancel_argv); break; } spin_unlock(&conn->request_lock); + + /* + * Complete a cancelled notify before this CANCEL handler returns. + * Deferring it to the system workqueue lets a following request and + * its response overtake STATUS_CANCELLED, leaving clients waiting + * for the original notify even though the cancellation was accepted. + */ + if (cancelled_notify) + smb2_complete_notify_cancel(cancelled_notify); } else { command_list = &conn->requests; @@ -7996,7 +9793,7 @@ static int smb2_set_flock_flags(struct file_lock *flock, int flags) } static struct ksmbd_lock *smb2_lock_init(struct file_lock *flock, - unsigned int cmd, int flags, + unsigned int cmd, int flags, bool zero_len, struct list_head *lock_list) { struct ksmbd_lock *lock; @@ -8010,8 +9807,7 @@ static struct ksmbd_lock *smb2_lock_init(struct file_lock *flock, lock->start = flock->fl_start; lock->end = flock->fl_end; lock->flags = flags; - if (lock->start == lock->end) - lock->zero_len = 1; + lock->zero_len = zero_len; INIT_LIST_HEAD(&lock->clist); INIT_LIST_HEAD(&lock->flist); INIT_LIST_HEAD(&lock->llist); @@ -8028,6 +9824,19 @@ static void smb2_remove_blocked_lock(void **argv) locks_wake_up(flock); } +static void smb2_free_lock(struct file_lock *flock) +{ + ksmbd_vfs_posix_lock_unblock(flock); + locks_free_lock(flock); +} + +static void smb2_free_blocked_lock(struct file_lock *flock) +{ + ksmbd_vfs_posix_lock_unblock(flock); + locks_wake_up(flock); + locks_free_lock(flock); +} + static inline bool lock_defer_pending(struct file_lock *fl) { /* check pending lock waiters */ @@ -8060,6 +9869,7 @@ int smb2_lock(struct ksmbd_work *work) LIST_HEAD(rollback_list); int prior_lock = 0, bkt; unsigned int id = KSMBD_NO_FID, pid = KSMBD_NO_FID; + bool lock_replayed; WORK_BUFFERS(work, req, rsp); @@ -8088,6 +9898,14 @@ int smb2_lock(struct ksmbd_work *work) goto out2; } + err = smb2_set_request_open(work, fp, &req->hdr, false, false); + if (err) + goto out2; + + lock_replayed = smb2_verify_lock_sequence(work, fp, req); + if (lock_replayed) + goto lock_success; + filp = fp->filp; lock_count = le16_to_cpu(req->LockCount); lock_ele = req->locks; @@ -8114,32 +9932,18 @@ int smb2_lock(struct ksmbd_work *work) lock_start = le64_to_cpu(lock_ele[i].Offset); lock_length = le64_to_cpu(lock_ele[i].Length); - if (lock_start > U64_MAX - lock_length) { + if (lock_start > OFFSET_MAX || + (lock_length && + lock_length - 1 > OFFSET_MAX - lock_start)) { pr_err("Invalid lock range requested\n"); rsp->hdr.Status = STATUS_INVALID_LOCK_RANGE; locks_free_lock(flock); goto out; } - if (lock_start > OFFSET_MAX) - flock->fl_start = OFFSET_MAX; - else - flock->fl_start = lock_start; - - lock_length = le64_to_cpu(lock_ele[i].Length); - if (lock_length > OFFSET_MAX - flock->fl_start) - lock_length = OFFSET_MAX - flock->fl_start; - - flock->fl_end = flock->fl_start + lock_length; - - if (flock->fl_end < flock->fl_start) { - ksmbd_debug(SMB, - "the end offset(%llx) is smaller than the start offset(%llx)\n", - flock->fl_end, flock->fl_start); - rsp->hdr.Status = STATUS_INVALID_LOCK_RANGE; - locks_free_lock(flock); - goto out; - } + flock->fl_start = lock_start; + flock->fl_end = lock_length ? + flock->fl_start + lock_length - 1 : flock->fl_start; /* Check conflict locks in one request */ list_for_each_entry(cmp_lock, &lock_list, llist) { @@ -8155,7 +9959,8 @@ int smb2_lock(struct ksmbd_work *work) } } - smb_lock = smb2_lock_init(flock, cmd, flags, &lock_list); + smb_lock = smb2_lock_init(flock, cmd, flags, !lock_length, + &lock_list); if (!smb_lock) { err = -EINVAL; locks_free_lock(flock); @@ -8164,6 +9969,13 @@ int smb2_lock(struct ksmbd_work *work) } list_for_each_entry_safe(smb_lock, tmp, &lock_list, llist) { + if (lock_count > 1 && + !(le32_to_cpu(lock_ele[0].Flags) & SMB2_LOCKFLAG_UNLOCK) && + !(smb_lock->flags & SMB2_LOCKFLAG_FAIL_IMMEDIATELY)) { + err = -EINVAL; + goto out; + } + if (smb_lock->cmd < 0) { err = -EINVAL; goto out; @@ -8204,14 +10016,14 @@ int smb2_lock(struct ksmbd_work *work) cmp_lock->end == smb_lock->end && !lock_defer_pending(cmp_lock->fl)) { nolock = 0; - list_del(&cmp_lock->flist); - list_del(&cmp_lock->clist); + list_del_init(&cmp_lock->flist); + list_del_init(&cmp_lock->clist); cmp_lock->conn = NULL; spin_unlock(&conn->llist_lock); up_read(&conn_list_lock); ksmbd_conn_put(conn); - locks_free_lock(cmp_lock->fl); + smb2_free_lock(cmp_lock->fl); kfree(cmp_lock); goto out_check_cl; } @@ -8229,7 +10041,7 @@ int smb2_lock(struct ksmbd_work *work) /* check zero byte lock range */ if (cmp_lock->zero_len && !smb_lock->zero_len && cmp_lock->start > smb_lock->start && - cmp_lock->start < smb_lock->end) { + cmp_lock->start <= smb_lock->end) { spin_unlock(&conn->llist_lock); up_read(&conn_list_lock); pr_err("previous lock conflict with zero byte lock range\n"); @@ -8238,17 +10050,15 @@ int smb2_lock(struct ksmbd_work *work) if (smb_lock->zero_len && !cmp_lock->zero_len && smb_lock->start > cmp_lock->start && - smb_lock->start < cmp_lock->end) { + smb_lock->start <= cmp_lock->end) { spin_unlock(&conn->llist_lock); up_read(&conn_list_lock); pr_err("current lock conflict with zero byte lock range\n"); goto out; } - if (((cmp_lock->start <= smb_lock->start && - cmp_lock->end > smb_lock->start) || - (cmp_lock->start < smb_lock->end && - cmp_lock->end >= smb_lock->end)) && + if (cmp_lock->start <= smb_lock->end && + smb_lock->start <= cmp_lock->end && !cmp_lock->zero_len && !smb_lock->zero_len) { spin_unlock(&conn->llist_lock); up_read(&conn_list_lock); @@ -8293,11 +10103,12 @@ skip: ksmbd_debug(SMB, "would have to wait for getting lock\n"); - list_add(&smb_lock->llist, &rollback_list); argv = kmalloc(sizeof(void *), KSMBD_DEFAULT_GFP); if (!argv) { err = -ENOMEM; + smb2_free_blocked_lock(flock); + kfree(smb_lock); goto out; } argv[0] = flock; @@ -8308,8 +10119,11 @@ skip: if (rc) { kfree(argv); err = -ENOMEM; + smb2_free_blocked_lock(flock); + kfree(smb_lock); goto out; } + list_add(&smb_lock->llist, &rollback_list); spin_lock(&fp->f_lock); list_add(&work->fp_entry, &fp->blocked_works); spin_unlock(&fp->f_lock); @@ -8323,35 +10137,32 @@ skip: spin_unlock(&fp->f_lock); list_del(&smb_lock->llist); - release_async_work(work); - - if (work->state == KSMBD_WORK_ACTIVE) - goto retry; - - locks_free_lock(flock); if (work->state == KSMBD_WORK_CANCELLED) { rsp->hdr.Status = STATUS_CANCELLED; kfree(smb_lock); smb2_send_interim_resp(work, STATUS_CANCELLED); + release_async_work(work); + locks_free_lock(flock); work->send_no_response = 1; goto out; } + release_async_work(work); + + if (work->state == KSMBD_WORK_ACTIVE) + goto retry; + + locks_free_lock(flock); + rsp->hdr.Status = STATUS_RANGE_NOT_LOCKED; kfree(smb_lock); - goto out2; + /* rollback_list may still hold earlier grants */ + goto out; } else if (!rc) { list_add(&smb_lock->llist, &rollback_list); - smb_lock->conn = ksmbd_conn_get(work->conn); - spin_lock(&work->conn->llist_lock); - list_add_tail(&smb_lock->clist, - &work->conn->lock_list); - list_add_tail(&smb_lock->flist, - &fp->lock_list); - spin_unlock(&work->conn->llist_lock); ksmbd_debug(SMB, "successful in taking lock\n"); } else { locks_free_lock(flock); @@ -8365,6 +10176,7 @@ skip: if (atomic_read(&fp->f_ci->op_count) > 1) smb_break_all_oplock(work, fp); +lock_success: rsp->StructureSize = cpu_to_le16(4); ksmbd_debug(SMB, "successful in taking lock\n"); rsp->hdr.Status = STATUS_SUCCESS; @@ -8373,6 +10185,23 @@ skip: if (err) goto out; + /* publish only once the whole batch has committed */ + if (!list_empty(&rollback_list)) { + spin_lock(&work->conn->llist_lock); + list_for_each_entry_safe(smb_lock, tmp, &rollback_list, llist) { + list_del_init(&smb_lock->llist); + smb_lock->conn = ksmbd_conn_get(work->conn); + list_add_tail(&smb_lock->clist, + &work->conn->lock_list); + list_add_tail(&smb_lock->flist, + &fp->lock_list); + } + spin_unlock(&work->conn->llist_lock); + } + + if (!lock_replayed) + smb2_update_lock_sequence(work, fp, req); + ksmbd_fd_put(work, fp); return 0; @@ -8400,16 +10229,7 @@ out: } list_del(&smb_lock->llist); - conn = smb_lock->conn; - spin_lock(&conn->llist_lock); - if (!list_empty(&smb_lock->flist)) - list_del(&smb_lock->flist); - list_del(&smb_lock->clist); - smb_lock->conn = NULL; - spin_unlock(&conn->llist_lock); - ksmbd_conn_put(conn); - - locks_free_lock(smb_lock->fl); + smb2_free_lock(smb_lock->fl); if (rlock) locks_free_lock(rlock); kfree(smb_lock); @@ -8461,23 +10281,44 @@ static int fsctl_copychunk(struct ksmbd_work *work, cpu_to_le32(ksmbd_server_side_copy_max_total_size()); chunk_count = le32_to_cpu(ci_req->ChunkCount); - if (chunk_count == 0) + /* + * ChunkCount=0 is the standard SMB2 "query my copy limits" request + * (no data copied) -- but macOS Finder's Cmd+D duplicate sends + * FSCTL_SRV_COPYCHUNK with ChunkCount=0 meaning "copy the whole + * file", relying on the AAPL-negotiated server to do a full copy + * instead. Keep the standard no-op behavior for everyone else. + * + * Gate on the TIME_MACHINE share flag, not just conn->is_aapl: + * that flag alone has ambiguous provenance -- the pre-existing + * narrow UniqueId=0 path can also set it on ordinary, + * non-Time-Machine shares, and this series' stated design keeps + * every AAPL-driven behavior opt-in per share. + */ + if (chunk_count == 0 && + !(work->conn->is_aapl && + test_share_config_flag(work->tcon->share_conf, + KSMBD_SHARE_FLAG_TIME_MACHINE))) goto out; total_size_written = 0; + i = 0; - /* verify the SRV_COPYCHUNK_COPY packet */ - if (chunk_count > ksmbd_server_side_copy_max_chunk_count() || - input_count < struct_size(ci_req, Chunks, chunk_count)) { - rsp->hdr.Status = STATUS_INVALID_PARAMETER; - return -EINVAL; - } + if (chunk_count) { + /* verify the SRV_COPYCHUNK_COPY packet */ + if (chunk_count > ksmbd_server_side_copy_max_chunk_count() || + input_count < struct_size(ci_req, Chunks, chunk_count)) { + rsp->hdr.Status = STATUS_INVALID_PARAMETER; + return -EINVAL; + } - chunks = &ci_req->Chunks[0]; - for (i = 0; i < chunk_count; i++) { - if (le32_to_cpu(chunks[i].Length) == 0 || - le32_to_cpu(chunks[i].Length) > ksmbd_server_side_copy_max_chunk_size()) - break; - total_size_written += le32_to_cpu(chunks[i].Length); + chunks = &ci_req->Chunks[0]; + for (i = 0; i < chunk_count; i++) { + if (le32_to_cpu(chunks[i].Length) == 0 || + le32_to_cpu(chunks[i].Length) > ksmbd_server_side_copy_max_chunk_size()) + break; + total_size_written += le32_to_cpu(chunks[i].Length); + } + } else { + chunks = &ci_req->Chunks[0]; } if (i < chunk_count || @@ -8519,7 +10360,7 @@ static int fsctl_copychunk(struct ksmbd_work *work, if (ret < 0) { if (ret == -EACCES) rsp->hdr.Status = STATUS_ACCESS_DENIED; - if (ret == -EAGAIN) + else if (ret == -EAGAIN) rsp->hdr.Status = STATUS_FILE_LOCK_CONFLICT; else if (ret == -EBADF) rsp->hdr.Status = STATUS_INVALID_HANDLE; @@ -8725,8 +10566,6 @@ static int fsctl_query_allocated_ranges(struct ksmbd_work *work, u64 id, int ret = 0; *out_count = 0; - if (in_count == 0) - return -EINVAL; start = le64_to_cpu(qar_req->file_offset); length = le64_to_cpu(qar_req->length); @@ -8738,11 +10577,28 @@ static int fsctl_query_allocated_ranges(struct ksmbd_work *work, u64 id, if (!fp) return -ENOENT; - ret = ksmbd_vfs_fqar_lseek(fp, start, length, - qar_rsp, in_count, out_count); + if (!(fp->daccess & FILE_READ_DATA_LE)) { + ret = -EACCES; + goto out; + } + + if (!in_count) { + struct file_allocated_range_buffer range; + + ret = ksmbd_vfs_query_allocated_ranges(fp, start, length, + &range, 1, out_count); + if ((!ret || ret == -E2BIG) && *out_count) + ret = -ENOSPC; + *out_count = 0; + } else { + ret = ksmbd_vfs_query_allocated_ranges(fp, start, length, + qar_rsp, in_count, + out_count); + } if (ret && ret != -E2BIG) *out_count = 0; +out: ksmbd_fd_put(work, fp); return ret; } @@ -8809,7 +10665,13 @@ static inline int fsctl_set_sparse(struct ksmbd_work *work, u64 id, if (!fp) return -ENOENT; - if (!(fp->daccess & (FILE_WRITE_DATA_LE | FILE_WRITE_ATTRIBUTES_LE))) { + if (S_ISDIR(file_inode(fp->filp)->i_mode)) { + ret = -EINVAL; + goto out; + } + + if (!(fp->daccess & (FILE_WRITE_DATA_LE | FILE_APPEND_DATA_LE | + FILE_WRITE_ATTRIBUTES_LE))) { ret = -EACCES; goto out; } @@ -8817,23 +10679,34 @@ static inline int fsctl_set_sparse(struct ksmbd_work *work, u64 id, idmap = file_mnt_idmap(fp->filp); old_fattr = fp->f_ci->m_fattr; + if (!sparse->SetSparse && + (old_fattr & FILE_ATTRIBUTE_SPARSE_FILE_LE)) { + ret = ksmbd_vfs_zero_holes(fp); + if (ret) + goto out; + } + if (sparse->SetSparse) fp->f_ci->m_fattr |= FILE_ATTRIBUTE_SPARSE_FILE_LE; else fp->f_ci->m_fattr &= ~FILE_ATTRIBUTE_SPARSE_FILE_LE; - if (fp->f_ci->m_fattr != old_fattr && - test_share_config_flag(work->tcon->share_conf, - KSMBD_SHARE_FLAG_STORE_DOS_ATTRS)) { + if (fp->f_ci->m_fattr != old_fattr) { const struct cred *saved_cred; - struct xattr_dos_attrib da; + struct xattr_dos_attrib da = {0}; ret = ksmbd_vfs_get_dos_attrib_xattr(idmap, fp->filp->f_path.dentry, &da); - if (ret <= 0) - goto out; + if (ret <= 0) { + da.version = 4; + da.itime = fp->itime; + da.create_time = fp->create_time; + da.flags = XATTR_DOSINFO_CREATE_TIME | + XATTR_DOSINFO_ITIME; + } da.attr = le32_to_cpu(fp->f_ci->m_fattr); + da.flags |= XATTR_DOSINFO_ATTRIB; saved_cred = override_creds(fp->filp->f_cred); ret = ksmbd_vfs_set_dos_attrib_xattr(idmap, &fp->filp->f_path, @@ -8877,10 +10750,12 @@ int smb2_ioctl(struct ksmbd_work *work) struct smb2_ioctl_req *req; struct smb2_ioctl_rsp *rsp; unsigned int cnt_code, nbytes = 0, out_buf_len, in_buf_len; - u64 id = KSMBD_NO_FID; + u64 id = KSMBD_NO_FID, pid = KSMBD_NO_FID; struct ksmbd_conn *conn = work->conn; int ret = 0; char *buffer; + bool no_fileid_ioctl = false; + bool chseq_err = false; ksmbd_debug(SMB, "Received smb2 ioctl request\n"); @@ -8893,14 +10768,17 @@ int smb2_ioctl(struct ksmbd_work *work) ksmbd_debug(SMB, "Compound request set FID = %llu\n", work->compound_fid); id = work->compound_fid; + pid = work->compound_pfid; } } else { req = smb_get_msg(work->request_buf); rsp = smb_get_msg(work->response_buf); } - if (!has_file_id(id)) + if (!has_file_id(id)) { id = req->VolatileFileId; + pid = req->PersistentFileId; + } if (req->Flags != cpu_to_le32(SMB2_0_IOCTL_IS_FSCTL)) { ret = -EOPNOTSUPP; @@ -8910,6 +10788,41 @@ int smb2_ioctl(struct ksmbd_work *work) buffer = (char *)req + le32_to_cpu(req->InputOffset); cnt_code = le32_to_cpu(req->CtlCode); + switch (cnt_code) { + case FSCTL_DFS_GET_REFERRALS: + case FSCTL_DFS_GET_REFERRALS_EX: + case FSCTL_QUERY_NETWORK_INTERFACE_INFO: + case FSCTL_VALIDATE_NEGOTIATE_INFO: + case FSCTL_PIPE_WAIT: + case FSCTL_PIPE_TRANSCEIVE: + no_fileid_ioctl = true; + break; + default: + break; + } + + if (!no_fileid_ioctl && has_file_id(id)) { + struct ksmbd_file *fp; + + fp = ksmbd_lookup_fd_slow(work, id, pid); + if (!fp) { + if (cnt_code == FSCTL_DUPLICATE_EXTENTS_TO_FILE) { + rsp->hdr.Status = STATUS_FILE_CLOSED; + goto out2; + } + ret = -ENOENT; + goto out; + } + + ret = smb2_set_request_open(work, fp, &req->hdr, true, false); + ksmbd_fd_put(work, fp); + if (ret) { + rsp->hdr.Status = STATUS_FILE_NOT_AVAILABLE; + chseq_err = true; + goto out; + } + } + ret = smb2_calc_max_out_buf_len(work, offsetof(struct smb2_ioctl_rsp, Buffer), le32_to_cpu(req->MaxOutputResponse)); @@ -8921,6 +10834,30 @@ int smb2_ioctl(struct ksmbd_work *work) in_buf_len = le32_to_cpu(req->InputCount); switch (cnt_code) { + case FSCTL_SRV_ENUM_SNAPS: { + struct srv_snapshot_array *snap_rsp; + struct ksmbd_file *fp; + + if (out_buf_len < sizeof(*snap_rsp)) { + ret = -EINVAL; + goto out; + } + + fp = ksmbd_lookup_fd_fast(work, id); + if (!fp) { + ret = -ENOENT; + goto out; + } + ksmbd_fd_put(work, fp); + + snap_rsp = (struct srv_snapshot_array *)rsp->Buffer; + snap_rsp->NumberOfSnapShots = 0; + snap_rsp->NumberOfSnapShotsReturned = 0; + snap_rsp->SnapShotArraySize = cpu_to_le32(2); + snap_rsp->Reserved = 0; + nbytes = sizeof(*snap_rsp); + break; + } case FSCTL_DFS_GET_REFERRALS: case FSCTL_DFS_GET_REFERRALS_EX: /* Not support DFS yet */ @@ -8994,6 +10931,12 @@ int smb2_ioctl(struct ksmbd_work *work) rsp->hdr.Status = STATUS_FILE_CLOSED; goto out2; } + + if (out_buf_len < sizeof(struct file_object_buf_type1_ioctl_rsp)) { + ksmbd_fd_put(work, fp); + ret = -EINVAL; + goto out; + } ksmbd_fd_put(work, fp); nbytes = sizeof(struct file_object_buf_type1_ioctl_rsp); @@ -9044,6 +10987,12 @@ int smb2_ioctl(struct ksmbd_work *work) rsp->VolatileFileId = SMB2_NO_FID; break; case FSCTL_QUERY_NETWORK_INTERFACE_INFO: + if (req->PersistentFileId != SMB2_NO_FID || + req->VolatileFileId != SMB2_NO_FID) { + ret = -EINVAL; + goto out; + } + ret = fsctl_query_iface_info_ioctl(conn, rsp, out_buf_len); if (ret < 0) goto out; @@ -9072,7 +11021,7 @@ int smb2_ioctl(struct ksmbd_work *work) goto out; } - if (in_buf_len <= sizeof(struct copychunk_ioctl_req)) { + if (in_buf_len < offsetof(struct copychunk_ioctl_req, Chunks)) { ret = -EINVAL; goto out; } @@ -9094,15 +11043,21 @@ int smb2_ioctl(struct ksmbd_work *work) rsp); break; case FSCTL_SET_SPARSE: - if (in_buf_len < sizeof(struct file_sparse)) { + { + struct file_sparse sparse = {0}; + + if (in_buf_len && in_buf_len < sizeof(struct file_sparse)) { ret = -EINVAL; goto out; } - ret = fsctl_set_sparse(work, id, (struct file_sparse *)buffer); + *(u8 *)&sparse = 1; + ret = fsctl_set_sparse(work, id, in_buf_len ? + (struct file_sparse *)buffer : &sparse); if (ret < 0) goto out; break; + } case FSCTL_SET_ZERO_DATA: { struct file_zero_data_information *zero_data; @@ -9147,11 +11102,95 @@ int smb2_ioctl(struct ksmbd_work *work) ret = ksmbd_vfs_zero_data(work, fp, off, len); ksmbd_fd_put(work, fp); - if (ret < 0) + if (ret == -EAGAIN) { + rsp->hdr.Status = STATUS_FILE_LOCK_CONFLICT; + ret = 0; goto out; + } else if (ret < 0) { + goto out; + } } break; } + case FSCTL_FILE_LEVEL_TRIM: + { + struct file_level_trim *trim_req; + struct file_level_trim_output *trim_rsp; + struct ksmbd_file *fp; + u32 i, num_ranges; + + if (!test_tree_conn_flag(work->tcon, KSMBD_TREE_CONN_FLAG_WRITABLE)) { + ksmbd_debug(SMB, + "User does not have write permission\n"); + ret = -EACCES; + goto out; + } + + if (in_buf_len < offsetof(struct file_level_trim, Ranges)) { + ret = -EINVAL; + goto out; + } + + if (out_buf_len < sizeof(struct file_level_trim_output)) { + ret = -EINVAL; + goto out; + } + + trim_req = (struct file_level_trim *)buffer; + num_ranges = le32_to_cpu(trim_req->NumRanges); + if (num_ranges > + (in_buf_len - offsetof(struct file_level_trim, Ranges)) / + sizeof(struct file_level_trim_range)) { + ret = -EINVAL; + goto out; + } + + fp = ksmbd_lookup_fd_fast(work, id); + if (!fp) { + ret = -ENOENT; + goto out; + } + + if (!(fp->daccess & FILE_WRITE_DATA_LE)) { + ksmbd_fd_put(work, fp); + ret = -EACCES; + goto out; + } + + trim_rsp = (struct file_level_trim_output *)&rsp->Buffer[0]; + trim_rsp->NumRangesProcessed = 0; + for (i = 0; i < num_ranges; i++) { + loff_t off = le64_to_cpu(trim_req->Ranges[i].Offset); + loff_t len = le64_to_cpu(trim_req->Ranges[i].Length); + + if (off < 0 || len < 0) { + ret = -EINVAL; + break; + } + + if (!len) { + trim_rsp->NumRangesProcessed = + cpu_to_le32(i + 1); + continue; + } + + ret = ksmbd_vfs_trim_data(work, fp, off, len); + if (ret) + break; + trim_rsp->NumRangesProcessed = cpu_to_le32(i + 1); + } + ksmbd_fd_put(work, fp); + if (ret == -EAGAIN) { + rsp->hdr.Status = STATUS_FILE_LOCK_CONFLICT; + ret = 0; + goto out; + } else if (ret < 0) { + goto out; + } + + nbytes = sizeof(struct file_level_trim_output); + break; + } case FSCTL_QUERY_ALLOCATED_RANGES: if (in_buf_len < sizeof(struct file_allocated_range_buffer)) { ret = -EINVAL; @@ -9177,6 +11216,11 @@ int smb2_ioctl(struct ksmbd_work *work) struct reparse_data_buffer *reparse_ptr; struct ksmbd_file *fp; + if (out_buf_len < sizeof(struct reparse_data_buffer)) { + ret = -EINVAL; + goto out; + } + reparse_ptr = (struct reparse_data_buffer *)&rsp->Buffer[0]; fp = ksmbd_lookup_fd_fast(work, id); if (!fp) { @@ -9209,15 +11253,18 @@ int smb2_ioctl(struct ksmbd_work *work) dup_ext->PersistentFileHandle); if (!fp_in) { pr_err("not found file handle in duplicate extent to file\n"); - ret = -ENOENT; - goto out; + ret = -EBADF; + rsp->hdr.Status = STATUS_INVALID_HANDLE; + goto out2; } fp_out = ksmbd_lookup_fd_fast(work, id); if (!fp_out) { pr_err("not found fp\n"); - ret = -ENOENT; - goto dup_ext_out; + ret = -EBADF; + rsp->hdr.Status = STATUS_FILE_CLOSED; + ksmbd_fd_put(work, fp_in); + goto out2; } if (!test_tree_conn_flag(work->tcon, @@ -9238,21 +11285,32 @@ int smb2_ioctl(struct ksmbd_work *work) src_off = le64_to_cpu(dup_ext->SourceFileOffset); dst_off = le64_to_cpu(dup_ext->TargetFileOffset); length = le64_to_cpu(dup_ext->ByteCount); - /* - * XXX: It is not clear if FSCTL_DUPLICATE_EXTENTS_TO_FILE - * should fall back to vfs_copy_file_range(). This could be - * beneficial when re-exporting nfs/smb mount, but note that - * this can result in partial copy that returns an error status. - * If/when FSCTL_DUPLICATE_EXTENTS_TO_FILE_EX is implemented, - * fall back to vfs_copy_file_range(), should be avoided when - * the flag DUPLICATE_EXTENTS_DATA_EX_SOURCE_ATOMIC is set. - */ - cloned = vfs_clone_file_range(fp_in->filp, src_off, - fp_out->filp, dst_off, length, 0); - if (cloned == -EXDEV || cloned == -EOPNOTSUPP) { + if (src_off < 0 || dst_off < 0 || length < 0 || + src_off + length < src_off || dst_off + length < dst_off) { + ret = -EINVAL; + goto dup_ext_out; + } + if (src_off + length > i_size_read(file_inode(fp_in->filp))) { + ret = -EOPNOTSUPP; + goto dup_ext_out; + } + if (dst_off + length > i_size_read(file_inode(fp_out->filp))) + goto dup_ext_out; + if ((fp_in->f_ci->m_fattr & FILE_ATTRIBUTE_SPARSE_FILE_LE) && + !(fp_out->f_ci->m_fattr & FILE_ATTRIBUTE_SPARSE_FILE_LE)) { ret = -EOPNOTSUPP; goto dup_ext_out; - } else if (cloned != length) { + } + if (file_inode(fp_in->filp) == file_inode(fp_out->filp) && + dst_off + length > src_off && + dst_off < src_off + length) { + ret = -EOPNOTSUPP; + goto dup_ext_out; + } + + cloned = vfs_clone_file_range(fp_in->filp, src_off, + fp_out->filp, dst_off, length, 0); + if (cloned != length) { cloned = vfs_copy_file_range(fp_in->filp, src_off, fp_out->filp, dst_off, length, 0); @@ -9300,7 +11358,7 @@ out: rsp->hdr.Status = STATUS_NOT_SUPPORTED; else if (ret == -ENOSPC) rsp->hdr.Status = STATUS_BUFFER_TOO_SMALL; - else if (ret < 0 || rsp->hdr.Status == 0) + else if (!chseq_err && (ret < 0 || rsp->hdr.Status == 0)) rsp->hdr.Status = STATUS_INVALID_PARAMETER; out2: @@ -9340,6 +11398,14 @@ static void smb20_oplock_break_ack(struct ksmbd_work *work) return; } + ret = smb2_set_request_open(work, fp, &req->hdr, false, false); + if (ret) { + rsp->hdr.Status = STATUS_FILE_CLOSED; + smb2_set_err_rsp(work); + ksmbd_fd_put(work, fp); + return; + } + opinfo = opinfo_get(fp); if (!opinfo) { pr_err("unexpected null oplock_info\n"); @@ -9352,6 +11418,22 @@ static void smb20_oplock_break_ack(struct ksmbd_work *work) if (opinfo->op_state != OPLOCK_ACK_WAIT) { ksmbd_debug(SMB, "unexpected oplock state 0x%x\n", opinfo->op_state); + if (smb3_hdr_replay(&req->hdr) && + opinfo->op_state == OPLOCK_STATE_NONE) { + rsp->StructureSize = cpu_to_le16(24); + rsp->OplockLevel = opinfo->level; + rsp->Reserved = 0; + rsp->Reserved2 = 0; + rsp->VolatileFid = volatile_id; + rsp->PersistentFid = persistent_id; + ret = ksmbd_iov_pin_rsp(work, rsp, + sizeof(struct smb2_oplock_break)); + if (ret) + ksmbd_debug(SMB, + "failed to pin replayed oplock break response: %d\n", + ret); + goto out_no_state_change; + } if (opinfo->level == SMB2_OPLOCK_LEVEL_NONE) status = STATUS_INVALID_OPLOCK_PROTOCOL; else @@ -9418,8 +11500,12 @@ err_out: smb2_set_err_rsp(work); out: - opinfo->op_state = OPLOCK_STATE_NONE; + spin_lock(&opinfo->state_lock); + if (opinfo->op_state != OPLOCK_CLOSING) + opinfo->op_state = OPLOCK_STATE_NONE; + spin_unlock(&opinfo->state_lock); wake_up_interruptible_all(&opinfo->oplock_q); +out_no_state_change: opinfo_put(opinfo); ksmbd_fd_put(work, fp); } @@ -9472,11 +11558,15 @@ static void smb21_lease_break_ack(struct ksmbd_work *work) if (opinfo->op_state == OPLOCK_STATE_NONE) { pr_err("unexpected lease break state 0x%x\n", opinfo->op_state); + if (smb3_hdr_replay(&req->hdr)) + goto replay_rsp; rsp->hdr.Status = STATUS_UNSUCCESSFUL; goto err_out; } if (!atomic_read(&opinfo->breaking_cnt)) { + if (smb3_hdr_replay(&req->hdr)) + goto replay_rsp; rsp->hdr.Status = STATUS_UNSUCCESSFUL; goto err_out; } @@ -9504,13 +11594,29 @@ static void smb21_lease_break_ack(struct ksmbd_work *work) if (ret) goto err_out; - opinfo->op_state = OPLOCK_STATE_NONE; + spin_lock(&opinfo->state_lock); + if (opinfo->op_state != OPLOCK_CLOSING) + opinfo->op_state = OPLOCK_STATE_NONE; + spin_unlock(&opinfo->state_lock); wake_up_interruptible_all(&opinfo->oplock_q); - atomic_dec(&opinfo->breaking_cnt); + atomic_dec_if_positive(&opinfo->breaking_cnt); wake_up_interruptible_all(&opinfo->oplock_brk); opinfo_put(opinfo); return; +replay_rsp: + rsp->StructureSize = cpu_to_le16(36); + rsp->Reserved = 0; + rsp->Flags = 0; + memcpy(rsp->LeaseKey, req->LeaseKey, 16); + rsp->LeaseState = lease->state; + rsp->LeaseDuration = 0; + ret = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_lease_ack)); + if (ret) + goto err_out; + opinfo_put(opinfo); + return; + err_out: smb2_set_err_rsp(work); opinfo_put(opinfo); @@ -9550,6 +11656,139 @@ int smb2_oplock_break(struct ksmbd_work *work) return 0; } +/* + * Cancel handler for a deferred CHANGE_NOTIFY. Races against + * __ksmbd_close_fd()'s notify_pendings drain (vfs_cache.c), which can run + * concurrently on a different connection closing the same handle -- only + * one of the two may claim and free in_work, so both sides check + * list_empty() under fp->f_lock before touching it (list_del_init() + * leaves a node empty, so whichever side removes it first is the owner; + * the loser must not touch in_work again, since the winner may already be + * freeing it). + * + * smb2_cancel() holds conn->request_lock (a spinlock) for the entire + * time it walks conn->async_requests and calls this function -- so this + * runs with preemption disabled and must not sleep or re-acquire that + * same lock. release_async_work() does both (it takes conn->request_lock + * itself, and frees things that can involve sleeping paths), so calling + * it from here would self-deadlock the very thread processing the + * client's CANCEL command. ksmbd_conn_write() can also sleep (it takes + * conn's write mutex). So: do only the non-sleeping, no-relock cleanup + * inline here. smb2_cancel() sends and frees the claimed notify after it + * drops request_lock, preserving response order for a client CANCEL. The + * connection teardown caller has no such post-unlock path, so its wrapper + * defers the send and free to a workqueue. + */ +struct notify_cancel_ctx { + struct work_struct work; + struct ksmbd_work *in_work; +}; + +static void smb2_send_notify_cancelled(struct ksmbd_work *work) +{ + struct smb2_hdr *hdr = smb_get_msg(work->response_buf); + struct ksmbd_conn *conn = work->conn; + struct ksmbd_session *sess; + + sess = ksmbd_session_lookup(conn, le64_to_cpu(hdr->SessionId)); + if (sess) { + work->sess = sess; + if (work->encrypted && sess->enc && conn->ops->encrypt_resp) { + conn->ops->encrypt_resp(work); + } else if (conn->ops->is_sign_req && conn->ops->set_sign_rsp && + conn->ops->is_sign_req(work, + conn->ops->get_cmd_val(work))) { + conn->ops->set_sign_rsp(work); + } + } + + ksmbd_conn_write(work); + if (sess) { + ksmbd_user_session_put(sess); + work->sess = NULL; + } +} + +static void smb2_notify_cancel_deferred(struct work_struct *w) +{ + struct notify_cancel_ctx *ctx = + container_of(w, struct notify_cancel_ctx, work); + struct ksmbd_conn *conn = ctx->in_work->conn; + + smb2_complete_notify_cancel(ctx->in_work); + kfree(ctx); + /* + * The connection teardown waits for r_count before destroying + * connection sessions and their proc entries. + */ + ksmbd_conn_r_count_dec(conn); +} + +static struct ksmbd_work *smb2_notify_cancel_claim(void **argv) +{ + struct ksmbd_work *in_work = (struct ksmbd_work *)argv[0]; + struct ksmbd_file *fp = (struct ksmbd_file *)argv[1]; + bool claimed; + + spin_lock(&fp->f_lock); + claimed = !list_empty(&in_work->notify_entry); + if (claimed) + list_del_init(&in_work->notify_entry); + spin_unlock(&fp->f_lock); + + if (!claimed) + return NULL; + + /* conn->request_lock is held by smb2_cancel() or connection teardown. */ + in_work->cancel_fn = NULL; + kfree(in_work->cancel_argv); + in_work->cancel_argv = NULL; + return in_work; +} + +static void smb2_complete_notify_cancel(struct ksmbd_work *in_work) +{ + struct smb2_hdr *in_hdr = smb_get_msg(in_work->response_buf); + + in_hdr->Status = STATUS_CANCELLED; + smb2_send_notify_cancelled(in_work); + release_async_work(in_work); + ksmbd_free_work_struct(in_work); +} + +static void smb2_notify_cancel_fn(void **argv) +{ + struct ksmbd_work *in_work = smb2_notify_cancel_claim(argv); + struct ksmbd_conn *conn; + struct notify_cancel_ctx *ctx; + + if (!in_work) + return; + conn = in_work->conn; + + ctx = kmalloc(sizeof(*ctx), GFP_ATOMIC); + if (!ctx) { + /* Can't defer the response -- free without sending one. */ + list_del_init(&in_work->async_request_entry); + in_work->asynchronous = false; + if (in_work->async_id) { + ksmbd_release_id(&conn->async_ida, in_work->async_id); + in_work->async_id = 0; + } + ksmbd_free_work_struct(in_work); + return; + } + ctx->in_work = in_work; + INIT_WORK(&ctx->work, smb2_notify_cancel_deferred); + /* + * This deferred work can outlive the connection handler's receive loop. + * Keep teardown from destroying the connection's sessions until the + * deferred response has finished using them. + */ + ksmbd_conn_r_count_inc(conn); + schedule_work(&ctx->work); +} + /** * smb2_notify() - handler for smb2 notify request * @work: smb work containing notify command buffer @@ -9560,6 +11799,9 @@ int smb2_notify(struct ksmbd_work *work) { struct smb2_change_notify_req *req; struct smb2_change_notify_rsp *rsp; + struct ksmbd_work *in_work; + struct smb2_hdr *in_hdr; + struct ksmbd_file *fp; ksmbd_debug(SMB, "Received smb2 notify\n"); @@ -9574,9 +11816,160 @@ int smb2_notify(struct ksmbd_work *work) return -EIO; } - smb2_set_err_rsp(work); - rsp->hdr.Status = STATUS_NOT_IMPLEMENTED; - return -EOPNOTSUPP; + /* + * macOS backupd sends CHANGE_NOTIFY with FileId=FFFF...FFFF (share-root + * sentinel) to watch for changes on the share root without holding an + * open handle. Respond STATUS_PENDING + STATUS_NOTIFY_CLEANUP immediately; + * without this, backupd aborts Time Machine setup on STATUS_FILE_CLOSED. + */ + if (req->VolatileFileId == SMB2_NO_FID && + req->PersistentFileId == SMB2_NO_FID) { + in_work = ksmbd_alloc_work_struct(); + if (!in_work || allocate_interim_rsp_buf(in_work)) { + if (in_work) + ksmbd_free_work_struct(in_work); + rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; + smb2_set_err_rsp(work); + return 0; + } + if (setup_async_work(work, NULL, NULL)) { + ksmbd_free_work_struct(in_work); + rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; + smb2_set_err_rsp(work); + return 0; + } + smb2_send_interim_resp(work, STATUS_PENDING); + in_work->conn = work->conn; + in_hdr = smb_get_msg(in_work->response_buf); + memcpy(in_hdr, ksmbd_resp_buf_next(work), + __SMB2_HEADER_STRUCTURE_SIZE); + in_hdr->Flags |= SMB2_FLAGS_ASYNC_COMMAND; + in_hdr->Id.AsyncId = cpu_to_le64(work->async_id); + smb2_set_err_rsp(in_work); + in_hdr->Status = STATUS_NOTIFY_CLEANUP; + in_work->async_id = work->async_id; + work->async_id = 0; + release_async_work(work); + if (smb2_send_interim_work(in_work, work, false)) + ksmbd_debug(SMB, "failed to send notify cleanup\n"); + ksmbd_free_work_struct(in_work); + work->send_no_response = 1; + return 0; + } + + /* + * KSMBD does not implement a real change-notification backend. + * Genuine SMB2 servers (and macOS smbfs) never complete a + * CHANGE_NOTIFY spontaneously: it is satisfied only by a real + * directory change, or with STATUS_NOTIFY_CLEANUP when the watched + * handle is closed. Completing it early (e.g. on a timer) makes + * Finder treat the cleanup as "directory changed" and re-enumerate + * the directory forever, leaving items unopenable. Returning + * STATUS_NOT_IMPLEMENTED here (like stock ksmbd) makes macOS smbfs + * hard-freeze on unmount, so this must stay deferred. + */ + fp = ksmbd_lookup_fd_slow(work, req->VolatileFileId, req->PersistentFileId); + if (!fp) { + rsp->hdr.Status = STATUS_FILE_CLOSED; + smb2_set_err_rsp(work); + return 0; + } + + in_work = ksmbd_alloc_work_struct(); + if (!in_work || allocate_interim_rsp_buf(in_work)) { + if (in_work) + ksmbd_free_work_struct(in_work); + ksmbd_fd_put(work, fp); + rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; + smb2_set_err_rsp(work); + return 0; + } + /* + * in_work is synthetic (not from the normal request-receiving + * pipeline), so it has no request_buf of its own. It gets registered + * into conn->async_requests below, and smb2_cancel() unconditionally + * computes smb_get_msg(iter->request_buf) for every entry in that + * list while searching for a match -- give it its own small buffer + * (not an alias of response_buf: ksmbd_free_work_struct() kvfree()s + * both separately, so aliasing them would double-free) so that stays + * a harmless read instead of a near-NULL dereference. + */ + in_work->request_buf = kzalloc(MAX_CIFS_SMALL_BUFFER_SIZE, KSMBD_DEFAULT_GFP); + if (!in_work->request_buf) { + ksmbd_free_work_struct(in_work); + ksmbd_fd_put(work, fp); + rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; + smb2_set_err_rsp(work); + return 0; + } + memcpy(smb_get_msg(in_work->request_buf), req, + __SMB2_HEADER_STRUCTURE_SIZE); + + if (setup_async_work(work, NULL, NULL)) { + ksmbd_free_work_struct(in_work); + ksmbd_fd_put(work, fp); + rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; + smb2_set_err_rsp(work); + return 0; + } + + smb2_send_interim_resp(work, STATUS_PENDING); + + /* Keep the async IDA alive until the deferred work is released. */ + in_work->conn = ksmbd_conn_get(work->conn); + in_work->owns_conn_ref = true; + in_work->encrypted = work->encrypted; + in_hdr = smb_get_msg(in_work->response_buf); + memcpy(in_hdr, ksmbd_resp_buf_next(work), __SMB2_HEADER_STRUCTURE_SIZE); + in_hdr->Flags |= SMB2_FLAGS_ASYNC_COMMAND; + in_hdr->Id.AsyncId = cpu_to_le64(work->async_id); + smb2_set_err_rsp(in_work); + in_hdr->Status = STATUS_NOTIFY_CLEANUP; + + /* + * Transfer ownership of the async id to in_work; it stays reserved + * until in_work is freed after the deferred response is sent on + * close, so it can't be reused for an unrelated async response. + */ + in_work->async_id = work->async_id; + work->async_id = 0; + release_async_work(work); + + /* + * work itself is about to be recycled by the normal request-processing + * pipeline, so it can't stay the target of a future CANCEL -- register + * in_work instead, reusing the same async_id, so a client-sent CANCEL + * for this notify actually finds something to cancel instead of + * silently doing nothing until the handle eventually closes. + */ + in_work->asynchronous = true; + in_work->cancel_argv = kmalloc_array(2, sizeof(void *), KSMBD_DEFAULT_GFP); + if (in_work->cancel_argv) { + in_work->cancel_argv[0] = in_work; + in_work->cancel_argv[1] = fp; + in_work->cancel_fn = smb2_notify_cancel_fn; + } + + if (!ksmbd_conn_link_async_request(work->conn, in_work)) { + kfree(in_work->cancel_argv); + in_work->cancel_argv = NULL; + in_work->cancel_fn = NULL; + in_work->asynchronous = false; + ksmbd_fd_put(work, fp); + if (smb2_send_interim_work(in_work, work, false)) + ksmbd_debug(SMB, "failed to send notify cleanup\n"); + ksmbd_free_work_struct(in_work); + work->send_no_response = 1; + return 0; + } + + spin_lock(&fp->f_lock); + list_add_tail(&in_work->notify_entry, &fp->notify_pendings); + spin_unlock(&fp->f_lock); + + ksmbd_fd_put(work, fp); + work->send_no_response = 1; + return 0; } /** @@ -9641,6 +12034,39 @@ int smb2_check_sign_req(struct ksmbd_work *work) } /** + * smb2_get_sign_rsp_iov() - get the iovecs used to sign a response + * @work: work that has the response iovecs + * @hdr: SMB2 header of the response + * @n_vec: set to the number of iovecs to sign + * + * Response data may be in another buffer. In this case, the response uses + * more than one iovec. Find the iovec that starts with @hdr. Sign this + * iovec and all iovecs after it. + * + * Return: The first iovec to sign. + */ +static struct kvec *smb2_get_sign_rsp_iov(struct ksmbd_work *work, + struct smb2_hdr *hdr, int *n_vec) +{ + int i; + + /* + * iov[0] has the RFC1002 message length. It is not part of the SMB2 + * message, so do not sign it. + */ + for (i = 1; i <= work->iov_idx; i++) { + if (work->iov[i].iov_base == hdr) { + *n_vec = work->iov_idx - i + 1; + return &work->iov[i]; + } + } + + WARN_ON_ONCE(1); + *n_vec = 1; + return &work->iov[work->iov_idx]; +} + +/** * smb2_set_sign_rsp() - handler for rsp packet sign processing * @work: smb work containing notify command buffer * @@ -9650,18 +12076,13 @@ void smb2_set_sign_rsp(struct ksmbd_work *work) struct smb2_hdr *hdr; char signature[SMB2_HMACSHA256_SIZE]; struct kvec *iov; - int n_vec = 1; + int n_vec; hdr = ksmbd_resp_buf_curr(work); hdr->Flags |= SMB2_FLAGS_SIGNED; memset(hdr->Signature, 0, SMB2_SIGNATURE_SIZE); - if (hdr->Command == SMB2_READ) { - iov = &work->iov[work->iov_idx - 1]; - n_vec++; - } else { - iov = &work->iov[work->iov_idx]; - } + iov = smb2_get_sign_rsp_iov(work, hdr, &n_vec); ksmbd_sign_smb2_pdu(work->conn, work->sess->sess_key, iov, n_vec, signature); @@ -9744,7 +12165,7 @@ void smb3_set_sign_rsp(struct ksmbd_work *work) char signature[SMB2_CMACAES_SIZE]; struct kvec *iov; u16 command = conn->ops->get_cmd_val(work); - int n_vec = 1; + int n_vec; char *signing_key; hdr = ksmbd_resp_buf_curr(work); @@ -9766,12 +12187,7 @@ void smb3_set_sign_rsp(struct ksmbd_work *work) hdr->Flags |= SMB2_FLAGS_SIGNED; memset(hdr->Signature, 0, SMB2_SIGNATURE_SIZE); - if (hdr->Command == SMB2_READ) { - iov = &work->iov[work->iov_idx - 1]; - n_vec++; - } else { - iov = &work->iov[work->iov_idx]; - } + iov = smb2_get_sign_rsp_iov(work, hdr, &n_vec); ksmbd_sign_smb3_pdu(conn, signing_key, iov, n_vec, signature); memcpy(hdr->Signature, signature, SMB2_SIGNATURE_SIZE); @@ -9869,33 +12285,33 @@ bool smb3_is_transform_hdr(void *buf) int smb3_decrypt_req(struct ksmbd_work *work) { - struct ksmbd_session *sess; char *buf = work->request_buf; unsigned int pdu_length = get_rfc1002_len(buf); struct kvec iov[2]; - int buf_data_size = pdu_length - sizeof(struct smb2_transform_hdr); + unsigned int buf_data_size; struct smb2_transform_hdr *tr_hdr = smb_get_msg(buf); + unsigned int original_msg_size; int rc = 0; - if (pdu_length < sizeof(struct smb2_transform_hdr) || - buf_data_size < sizeof(struct smb2_hdr)) { + if (pdu_length < sizeof(struct smb2_transform_hdr)) { pr_err("Transform message is too small (%u)\n", pdu_length); return -ECONNABORTED; } - if (buf_data_size < le32_to_cpu(tr_hdr->OriginalMessageSize)) { - pr_err("Transform message is broken\n"); + buf_data_size = pdu_length - sizeof(struct smb2_transform_hdr); + original_msg_size = le32_to_cpu(tr_hdr->OriginalMessageSize); + if (buf_data_size < sizeof(struct smb2_compression_hdr) || + original_msg_size < sizeof(struct smb2_compression_hdr)) { + pr_err("Transform message is too small (%u)\n", + pdu_length); return -ECONNABORTED; } - sess = ksmbd_session_lookup_all(work->conn, le64_to_cpu(tr_hdr->SessionId)); - if (!sess) { - pr_err("invalid session id(%llx) in transform header\n", - le64_to_cpu(tr_hdr->SessionId)); + if (buf_data_size < original_msg_size) { + pr_err("Transform message is broken\n"); return -ECONNABORTED; } - ksmbd_user_session_put(sess); iov[0].iov_base = buf; iov[0].iov_len = sizeof(struct smb2_transform_hdr) + 4; @@ -9905,8 +12321,9 @@ int smb3_decrypt_req(struct ksmbd_work *work) if (rc) return rc; - memmove(buf + 4, iov[1].iov_base, buf_data_size); - *(__be32 *)buf = cpu_to_be32(buf_data_size); + /* Drop the AEAD authentication tag from the inner RFC1002 frame. */ + memmove(buf + 4, iov[1].iov_base, original_msg_size); + *(__be32 *)buf = cpu_to_be32(original_msg_size); return rc; } diff --git a/fs/smb/server/smb2pdu.h b/fs/smb/server/smb2pdu.h index aa06c8c905f1..3f08d1ca5a38 100644 --- a/fs/smb/server/smb2pdu.h +++ b/fs/smb/server/smb2pdu.h @@ -66,6 +66,80 @@ struct preauth_integrity_info { /* Apple Defined Contexts */ #define SMB2_CREATE_AAPL "AAPL" +/* + * AAPL SMB2 extension -- kAAPL_SERVER_QUERY create context. + * + * Command code and bitmap values are the existing + * SMB2_CRTCTX_AAPL_* constants in fs/smb/common/smb2pdu.h. + * + * Omitting the model string when reply_bitmap includes + * SMB2_CRTCTX_AAPL_MODEL_INFO causes smbfs.kext to enter a broken + * disconnect path requiring a reboot. + * + * Layout: ccontext(16) + Name[4] + Pad[4] + cmd(4) + reserved(4) + + * reply_bitmap(8) + server_caps(8) + vol_caps(8) + * When MODEL_INFO requested, appended: pad2(4) + model_bytes(4) + UTF-16LE + */ +#define SMB2_CREATE_AAPL_LEN 4 + +/* + * Server capability flags (server_caps field) -- SMB2_CRTCTX_AAPL_UNIX_BASED: + * prevents macOS Windows-compat mode (question-mark icons). + * SMB2_CRTCTX_AAPL_SUPPORTS_OSX_COPYFILE: enables server-side file copy via + * FSCTL_SRV_COPYCHUNK. SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR: inline + * FinderInfo per FIND entry, set when client also advertises the bit; + * format: EaSize=max_access, ShortName[0..7]=rfork_size, + * ShortName[8..23]=FinderInfo(16B), Reserved2=unix_mode. + */ +#define AAPL_SERVER_CAPS_KSMBD (SMB2_CRTCTX_AAPL_UNIX_BASED | \ + SMB2_CRTCTX_AAPL_SUPPORTS_OSX_COPYFILE | \ + SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR) + +/* + * READDIR_ATTR_V2 (SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR_V2, see + * fs/smb/common/smb2pdu.h) extends the same inline-FinderInfo mechanism + * above with a flags field, confirmed byte-identical to V1 otherwise + * against AAPL's actual public client behavior. When a client's own + * client_caps requests V2, the server advertises V2 instead of V1 in + * its own server_caps reply; V1 and V2 are mutually exclusive on the + * wire, not both set together. The wire format's ShortNameLength+Reserved + * (ignored in V1) become a single flags field in V2 -- + * AAPL_READDIR_ATTR_V2_NO_XATTR is the only flag bit currently defined, + * signaling the item has no xattrs/streams so the client can skip a + * separate query. + */ +#define AAPL_READDIR_ATTR_V2_NO_XATTR 0x01 + +/* Model string: up to 31 ASCII chars */ +#define AAPL_MODEL_MAX_CHARS 31 +#define AAPL_MODEL_UTF16_BYTES (AAPL_MODEL_MAX_CHARS * 2) + +/* + * Max AAPL response: header(24) + base data(32) + pad2(4) + model_bytes(4) + * + model(62), 8-byte aligned: ALIGN(126, 8) = 128 bytes. + */ +#define AAPL_RSP_MAX_SIZE 128 + +/* AAPL server query request (client->server) */ +struct aapl_server_query_req { + __le32 cmd; + __le32 reserved; + __le64 req_bitmap; + __le64 client_caps; +} __packed; + +struct create_aapl_rsp { + struct create_context_hdr ccontext; + __u8 Name[4]; + __u8 Pad[4]; + __le32 cmd; + __le32 reserved; + __le64 reply_bitmap; + __le64 server_caps; + __le64 vol_caps; + /* when MODEL_INFO requested: __le32 pad2; __le32 model_bytes; __le16 model[] */ +} __packed; + #define DURABLE_HANDLE_MAX_TIMEOUT 300000 struct create_alloc_size_req { @@ -199,6 +273,13 @@ struct smb2_file_stream_info { char StreamName[]; } __packed; +struct srv_snapshot_array { + __le32 NumberOfSnapShots; + __le32 NumberOfSnapShotsReturned; + __le32 SnapShotArraySize; + __le32 Reserved; +} __packed; + struct smb2_file_standard_info { __le64 AllocationSize; __le64 EndOfFile; @@ -347,6 +428,7 @@ bool smb3_encryption_negotiated(struct ksmbd_conn *conn); /* smb2 misc functions */ int ksmbd_smb2_check_message(struct ksmbd_work *work); +void smb2_complete_request_open(struct ksmbd_work *work); /* smb2 command handlers */ int smb2_handle_negotiate(struct ksmbd_work *work); diff --git a/fs/smb/server/smb_common.c b/fs/smb/server/smb_common.c index 080fbc9eb470..4c2da65510bc 100644 --- a/fs/smb/server/smb_common.c +++ b/fs/smb/server/smb_common.c @@ -164,7 +164,22 @@ int ksmbd_verify_smb_message(struct ksmbd_work *work) hdr = smb_get_msg(work->request_buf); if (*(__le32 *)hdr->Protocol == SMB1_PROTO_NUMBER && hdr->Command == SMB_COM_NEGOTIATE) { - work->conn->outstanding_credits++; + struct ksmbd_conn *conn = work->conn; + + conn->outstanding_credits++; + /* + * A legacy SMB1 multi-protocol negotiate occupies sequence + * number 0 but does not pass through + * ksmbd_smb2_check_message(). Consume it here so that, after + * the connection is upgraded to SMB2, the command sequence + * window can advance instead of staying pinned at 0. + */ + spin_lock(&conn->credits_lock); + if (conn->seq_low == 0) { + __clear_bit(0, conn->seq_bitmap); + conn->seq_low = 1; + } + spin_unlock(&conn->credits_lock); return 0; } diff --git a/fs/smb/server/smb_common.h b/fs/smb/server/smb_common.h index b090b56743c4..7b9c5cfcb63b 100644 --- a/fs/smb/server/smb_common.h +++ b/fs/smb/server/smb_common.h @@ -135,7 +135,7 @@ struct file_id_both_directory_info { struct smb_version_ops { u16 (*get_cmd_val)(struct ksmbd_work *swork); - void (*inc_reqs)(unsigned int cmd); + void (*inc_reqs)(unsigned int cmd, __le32 status); int (*init_rsp_hdr)(struct ksmbd_work *swork); void (*set_rsp_status)(struct ksmbd_work *swork, __le32 err); int (*allocate_rsp_buf)(struct ksmbd_work *work); diff --git a/fs/smb/server/smbacl.c b/fs/smb/server/smbacl.c index c13f07a09ab8..8ad2e5a5cca8 100644 --- a/fs/smb/server/smbacl.c +++ b/fs/smb/server/smbacl.c @@ -27,6 +27,9 @@ static const struct smb_sid creator_owner = { /* security id for everyone/world system group */ static const struct smb_sid creator_group = { 1, 1, {0, 0, 0, 0, 0, 3}, {cpu_to_le32(1)} }; +/* security id for owner rights */ +static const struct smb_sid sid_owner_rights = { + 1, 1, {0, 0, 0, 0, 0, 3}, {cpu_to_le32(4)} }; /* security id for everyone/world system group */ static const struct smb_sid sid_everyone = { @@ -1432,7 +1435,8 @@ bool smb_inherit_flags(int flags, bool is_dir) } int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, - __le32 *pdaccess, int uid) + __le32 *pdaccess, __le32 raw_daccess, int uid, + bool strict) { struct mnt_idmap *idmap = mnt_idmap(path->mnt); struct smb_ntsd *pntsd = NULL; @@ -1442,14 +1446,17 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, unsigned int dacl_offset; size_t dacl_struct_end; struct smb_sid sid; - int granted = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE); + int requested = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE); + int granted = requested; struct smb_ace *ace; int i, found = 0; - unsigned int access_bits = 0; + unsigned int access_bits = 0, denied = 0; struct smb_ace *others_ace = NULL; struct posix_acl_entry *pa_entry; unsigned int sid_type = SIDOWNER; unsigned short ace_size; + bool is_owner, owner_rights = false; + vfsuid_t vfsuid; ksmbd_debug(SMB, "check permission using windows acl\n"); pntsd_size = ksmbd_vfs_get_sd_xattr(conn, idmap, @@ -1479,12 +1486,15 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, goto err_out; } - if (*pdaccess & FILE_MAXIMAL_ACCESS_LE) { - granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES | - DELETE; + if (!uid) + sid_type = SIDUNIX_USER; + id_to_sid(uid, sid_type, &sid); + vfsuid = i_uid_into_vfsuid(idmap, d_inode(path->dentry)); + is_owner = uid == from_kuid(&init_user_ns, vfsuid_into_kuid(vfsuid)); + if (*pdaccess & FILE_MAXIMAL_ACCESS_LE) { ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl)); - aces_size = acl_size - sizeof(struct smb_acl); + aces_size = pdacl_size - sizeof(struct smb_acl); for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) { if (aces_size < offsetof(struct smb_ace, sid) + CIFS_SID_BASE_SIZE) @@ -1495,17 +1505,53 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, CIFS_SID_BASE_SIZE) break; aces_size -= ace_size; - granted |= le32_to_cpu(ace->access_req); + + if (ace->sid.num_subauth > SID_MAX_SUB_AUTHORITIES || + ace_size < offsetof(struct smb_ace, sid) + + CIFS_SID_BASE_SIZE + + sizeof(__le32) * ace->sid.num_subauth) + break; + + if (!compare_sids(&sid_owner_rights, &ace->sid)) { + owner_rights = true; + if (!is_owner) + goto next_ace; + } + + if (ace->flags & INHERIT_ONLY_ACE || + (compare_sids(&sid, &ace->sid) && + compare_sids(&sid_unix_NFS_mode, &ace->sid) && + compare_sids(&sid_everyone, &ace->sid) && + compare_sids(&sid_authusers, &ace->sid) && + compare_sids(&sid_owner_rights, &ace->sid))) + goto next_ace; + + switch (ace->type) { + case ACCESS_ALLOWED_ACE_TYPE: + access_bits |= le32_to_cpu(ace->access_req); + break; + case ACCESS_DENIED_ACE_TYPE: + case ACCESS_DENIED_CALLBACK_ACE_TYPE: + denied |= ~access_bits & + le32_to_cpu(ace->access_req); + break; + } +next_ace: ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size)); } + if (is_owner && !owner_rights) + access_bits |= READ_CONTROL | WRITE_DAC | + FILE_READ_ATTRIBUTES | DELETE; + access_bits &= ~denied; + if ((raw_daccess & FILE_GENERIC_EXECUTE_LE) && + S_ISREG(d_inode(path->dentry)->i_mode) && + (access_bits & GENERIC_READ_FLAGS) == GENERIC_READ_FLAGS) + access_bits |= FILE_EXECUTE; + granted = requested | access_bits; } - if (!uid) - sid_type = SIDUNIX_USER; - id_to_sid(uid, sid_type, &sid); - ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl)); - aces_size = acl_size - sizeof(struct smb_acl); + aces_size = pdacl_size - sizeof(struct smb_acl); for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) { if (aces_size < offsetof(struct smb_ace, sid) + CIFS_SID_BASE_SIZE) @@ -1527,25 +1573,16 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, found = 1; break; } - if (!compare_sids(&sid_everyone, &ace->sid)) + if (!compare_sids(&sid_everyone, &ace->sid) || + !compare_sids(&sid_authusers, &ace->sid)) others_ace = ace; ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size)); } - if (*pdaccess & FILE_MAXIMAL_ACCESS_LE && found) { - granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES | - DELETE; - - granted |= le32_to_cpu(ace->access_req); - - if (!pdacl->num_aces) - granted = GENERIC_ALL_FLAGS; - } - if (IS_ENABLED(CONFIG_FS_POSIX_ACL)) { posix_acls = get_inode_acl(d_inode(path->dentry), ACL_TYPE_ACCESS); - if (!IS_ERR_OR_NULL(posix_acls) && !found) { + if (!IS_ERR_OR_NULL(posix_acls) && !found && !others_ace) { unsigned int id = -1; pa_entry = posix_acls->a_entries; @@ -1583,19 +1620,27 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, } } - switch (ace->type) { - case ACCESS_ALLOWED_ACE_TYPE: - access_bits = le32_to_cpu(ace->access_req); - break; - case ACCESS_DENIED_ACE_TYPE: - case ACCESS_DENIED_CALLBACK_ACE_TYPE: - access_bits = le32_to_cpu(~ace->access_req); - break; + if (!(*pdaccess & FILE_MAXIMAL_ACCESS_LE)) { + switch (ace->type) { + case ACCESS_ALLOWED_ACE_TYPE: + access_bits = le32_to_cpu(ace->access_req); + break; + case ACCESS_DENIED_ACE_TYPE: + case ACCESS_DENIED_CALLBACK_ACE_TYPE: + access_bits = le32_to_cpu(~ace->access_req); + break; + } } check_access_bits: - if (granted & - ~(access_bits | FILE_READ_ATTRIBUTES | READ_CONTROL | WRITE_DAC | DELETE)) { + if (strict) { + access_bits &= granted; + } else { + access_bits |= FILE_READ_ATTRIBUTES | READ_CONTROL | + WRITE_DAC | DELETE; + } + + if (granted & ~access_bits) { ksmbd_debug(SMB, "Access denied with winACL, granted : %x, access_req : %x\n", granted, le32_to_cpu(ace->access_req)); rc = -EACCES; diff --git a/fs/smb/server/smbacl.h b/fs/smb/server/smbacl.h index ab21ba2cd4df..01810c16cc04 100644 --- a/fs/smb/server/smbacl.h +++ b/fs/smb/server/smbacl.h @@ -95,7 +95,8 @@ bool smb_inherit_flags(int flags, bool is_dir); int smb_inherit_dacl(struct ksmbd_conn *conn, const struct path *path, unsigned int uid, unsigned int gid); int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, - __le32 *pdaccess, int uid); + __le32 *pdaccess, __le32 raw_daccess, int uid, + bool strict); int set_info_sec(struct ksmbd_conn *conn, struct ksmbd_tree_connect *tcon, const struct path *path, struct smb_ntsd *pntsd, int ntsd_len, bool type_check, bool get_write); diff --git a/fs/smb/server/stats.h b/fs/smb/server/stats.h index b60c30c69077..bc864efa0d46 100644 --- a/fs/smb/server/stats.h +++ b/fs/smb/server/stats.h @@ -9,12 +9,24 @@ #ifndef __KSMBD_STATS_H__ #define __KSMBD_STATS_H__ +#include "../common/smb2status.h" + #define KSMBD_COUNTER_MAX_REQS 19 enum { KSMBD_COUNTER_SESSIONS = 0, KSMBD_COUNTER_TREE_CONNS, KSMBD_COUNTER_REQUESTS, + KSMBD_COUNTER_STATUS_SUCCESS, + KSMBD_COUNTER_STATUS_INFORMATIONAL, + KSMBD_COUNTER_STATUS_WARNING, + KSMBD_COUNTER_STATUS_ERROR, + KSMBD_COUNTER_ERROR_ACCESS_DENIED, + KSMBD_COUNTER_ERROR_NOT_FOUND, + KSMBD_COUNTER_ERROR_INVALID_PARAMETER, + KSMBD_COUNTER_ERROR_SHARING_VIOLATION, + KSMBD_COUNTER_ERROR_NOT_SUPPORTED, + KSMBD_COUNTER_ERROR_OTHER, KSMBD_COUNTER_READ_BYTES, KSMBD_COUNTER_WRITE_BYTES, KSMBD_COUNTER_FIRST_REQ, @@ -50,10 +62,49 @@ static inline void ksmbd_counter_sub(int type, s64 value) percpu_counter_sub(&ksmbd_counters.counters[type], value); } -static inline void ksmbd_counter_inc_reqs(unsigned int cmd) +static inline void ksmbd_counter_inc_reqs(unsigned int cmd, __le32 status) { - if (cmd < KSMBD_COUNTER_MAX_REQS) + unsigned int severity = le32_to_cpu(status) >> 30; + int type; + + switch (severity) { + case 0: + type = KSMBD_COUNTER_STATUS_SUCCESS; + break; + case 1: + type = KSMBD_COUNTER_STATUS_INFORMATIONAL; + break; + case 2: + type = KSMBD_COUNTER_STATUS_WARNING; + break; + default: + type = KSMBD_COUNTER_STATUS_ERROR; + break; + } + percpu_counter_inc(&ksmbd_counters.counters[type]); + + if (severity == 3) { + if (status == STATUS_ACCESS_DENIED) + type = KSMBD_COUNTER_ERROR_ACCESS_DENIED; + else if (status == STATUS_OBJECT_NAME_NOT_FOUND || + status == STATUS_NO_SUCH_FILE) + type = KSMBD_COUNTER_ERROR_NOT_FOUND; + else if (status == STATUS_INVALID_PARAMETER) + type = KSMBD_COUNTER_ERROR_INVALID_PARAMETER; + else if (status == STATUS_SHARING_VIOLATION) + type = KSMBD_COUNTER_ERROR_SHARING_VIOLATION; + else if (status == STATUS_NOT_SUPPORTED || + status == STATUS_NOT_IMPLEMENTED) + type = KSMBD_COUNTER_ERROR_NOT_SUPPORTED; + else + type = KSMBD_COUNTER_ERROR_OTHER; + percpu_counter_inc(&ksmbd_counters.counters[type]); + } + + if (cmd < KSMBD_COUNTER_MAX_REQS) { + percpu_counter_inc(&ksmbd_counters.counters[KSMBD_COUNTER_REQUESTS]); percpu_counter_inc(&ksmbd_counters.counters[KSMBD_COUNTER_FIRST_REQ + cmd]); + } } static inline s64 ksmbd_counter_sum(int type) @@ -66,7 +117,7 @@ static inline void ksmbd_counter_inc(int type) {} static inline void ksmbd_counter_dec(int type) {} static inline void ksmbd_counter_add(int type, s64 value) {} static inline void ksmbd_counter_sub(int type, s64 value) {} -static inline void ksmbd_counter_inc_reqs(unsigned int cmd) {} +static inline void ksmbd_counter_inc_reqs(unsigned int cmd, __le32 status) {} static inline s64 ksmbd_counter_sum(int type) { return 0; } #endif diff --git a/fs/smb/server/transport_ipc.c b/fs/smb/server/transport_ipc.c index 0c581b9624d3..4b0b572a3e1b 100644 --- a/fs/smb/server/transport_ipc.c +++ b/fs/smb/server/transport_ipc.c @@ -322,6 +322,15 @@ static int ipc_server_config_on_startup(struct ksmbd_startup_request *req) goto out; } server_conf.share_fake_fscaps = req->share_fake_fscaps; + + /* AAPL model string for Finder icon */ + if (req->aapl_model[0]) + strscpy(server_conf.aapl_model, req->aapl_model, + sizeof(server_conf.aapl_model)); + else + strscpy(server_conf.aapl_model, "Xserve", + sizeof(server_conf.aapl_model)); + ksmbd_init_domain(req->sub_auth); if (req->smb2_max_read) @@ -497,6 +506,9 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry) { struct ksmbd_rpc_command *resp = entry->response; + if (entry->msg_sz < sizeof(struct ksmbd_rpc_command)) + return -EINVAL; + if (check_add_overflow(sizeof(struct ksmbd_rpc_command), resp->payload_sz, &msg_sz)) return -EINVAL; @@ -506,6 +518,9 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry) { struct ksmbd_spnego_authen_response *resp = entry->response; + if (entry->msg_sz < sizeof(struct ksmbd_spnego_authen_response)) + return -EINVAL; + msg_sz = sizeof(struct ksmbd_spnego_authen_response) + resp->session_key_len + resp->spnego_blob_len; break; @@ -514,6 +529,9 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry) { struct ksmbd_share_config_response *resp = entry->response; + if (entry->msg_sz < sizeof(struct ksmbd_share_config_response)) + return -EINVAL; + if (resp->payload_sz) { if (resp->payload_sz < resp->veto_list_sz) return -EINVAL; @@ -528,6 +546,9 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry) { struct ksmbd_login_response_ext *resp = entry->response; + if (entry->msg_sz < sizeof(struct ksmbd_login_response_ext)) + return -EINVAL; + if (resp->ngroups) { if (resp->ngroups < 0 || resp->ngroups > NGROUPS_MAX) { @@ -658,7 +679,7 @@ ksmbd_ipc_spnego_authen_request(const char *spnego_blob, int blob_len) return NULL; msg = ipc_msg_alloc(sizeof(struct ksmbd_spnego_authen_request) + - blob_len + 1); + blob_len); if (!msg) return NULL; @@ -839,7 +860,7 @@ struct ksmbd_rpc_command *ksmbd_rpc_write(struct ksmbd_session *sess, int handle if (payload_sz > KSMBD_IPC_MAX_PAYLOAD) return NULL; - msg = ipc_msg_alloc(sizeof(struct ksmbd_rpc_command) + payload_sz + 1); + msg = ipc_msg_alloc(sizeof(struct ksmbd_rpc_command) + payload_sz); if (!msg) return NULL; @@ -898,7 +919,7 @@ struct ksmbd_rpc_command *ksmbd_rpc_ioctl(struct ksmbd_session *sess, int handle if (payload_sz > KSMBD_IPC_MAX_PAYLOAD) return NULL; - msg = ipc_msg_alloc(sizeof(struct ksmbd_rpc_command) + payload_sz + 1); + msg = ipc_msg_alloc(sizeof(struct ksmbd_rpc_command) + payload_sz); if (!msg) return NULL; diff --git a/fs/smb/server/transport_rdma.c b/fs/smb/server/transport_rdma.c index b6d63ff8a8a3..ee28a4d1cc86 100644 --- a/fs/smb/server/transport_rdma.c +++ b/fs/smb/server/transport_rdma.c @@ -76,6 +76,8 @@ static int smb_direct_max_receive_size = 1364; static int smb_direct_max_read_write_size = SMBD_DEFAULT_IOSIZE; +static bool smb_direct_enabled; + static struct smb_direct_listener { int port; @@ -239,17 +241,18 @@ static int smb_direct_read(struct ksmbd_transport *t, char *buf, } static int smb_direct_writev(struct ksmbd_transport *t, - struct kvec *iov, int niovs, int buflen, - bool need_invalidate, unsigned int remote_key) + const struct ksmbd_transport_write *tx) { struct smb_direct_transport *st = SMBD_TRANS(t); struct smbdirect_socket *sc = st->socket; struct iov_iter iter; - iov_iter_kvec(&iter, ITER_SOURCE, iov, niovs, buflen); + iov_iter_kvec(&iter, ITER_SOURCE, tx->iov, tx->iov_cnt, + tx->size); return smbdirect_connection_send_iter(sc, &iter, 0, - need_invalidate, remote_key); + tx->need_invalidate_rkey, + tx->remote_key); } static int smb_direct_rdma_write(struct ksmbd_transport *t, @@ -511,18 +514,26 @@ int ksmbd_rdma_init(void) ksmbd_debug(RDMA, "iWarp RDMA listener. socket=%p\n", smb_direct_iw_listener.socket); + WRITE_ONCE(smb_direct_enabled, true); return 0; err: + WRITE_ONCE(smb_direct_enabled, false); ksmbd_rdma_stop_listening(); return ret; } void ksmbd_rdma_stop_listening(void) { + WRITE_ONCE(smb_direct_enabled, false); smb_direct_listener_destroy(&smb_direct_ib_listener); smb_direct_listener_destroy(&smb_direct_iw_listener); } +bool ksmbd_rdma_enabled(void) +{ + return READ_ONCE(smb_direct_enabled); +} + bool ksmbd_rdma_capable_netdev(struct net_device *netdev) { u8 node_type = smbdirect_netdev_rdma_capable_node_type(netdev); diff --git a/fs/smb/server/transport_rdma.h b/fs/smb/server/transport_rdma.h index 8b78917a1795..23247713b5c3 100644 --- a/fs/smb/server/transport_rdma.h +++ b/fs/smb/server/transport_rdma.h @@ -14,12 +14,14 @@ #ifdef CONFIG_SMB_SERVER_SMBDIRECT int ksmbd_rdma_init(void); void ksmbd_rdma_stop_listening(void); +bool ksmbd_rdma_enabled(void); bool ksmbd_rdma_capable_netdev(struct net_device *netdev); void init_smbd_max_io_size(unsigned int sz); unsigned int get_smbd_max_read_write_size(struct ksmbd_transport *kt); #else static inline int ksmbd_rdma_init(void) { return 0; } static inline void ksmbd_rdma_stop_listening(void) { } +static inline bool ksmbd_rdma_enabled(void) { return false; } static inline bool ksmbd_rdma_capable_netdev(struct net_device *netdev) { return false; } static inline void init_smbd_max_io_size(unsigned int sz) { } static inline unsigned int get_smbd_max_read_write_size(struct ksmbd_transport *kt) { return 0; } diff --git a/fs/smb/server/transport_tcp.c b/fs/smb/server/transport_tcp.c index 13b711ea575d..832e93084605 100644 --- a/fs/smb/server/transport_tcp.c +++ b/fs/smb/server/transport_tcp.c @@ -292,6 +292,12 @@ skip_max_ip_conns_limit: ksmbd_debug(CONN, "connect success: accepted new connection\n"); client_sk->sk->sk_rcvtimeo = KSMBD_TCP_RECV_TIMEOUT; client_sk->sk->sk_sndtimeo = KSMBD_TCP_SEND_TIMEOUT; + /* + * Detect peers that disappear without sending a FIN or RST. + * Otherwise the connection handler can retry receive timeouts + * indefinitely and keep the connection in conn_list. + */ + sock_set_keepalive(client_sk->sk); ksmbd_tcp_new_connection(client_sk); } @@ -417,14 +423,15 @@ static int ksmbd_tcp_read(struct ksmbd_transport *t, char *buf, return ksmbd_tcp_readv(TCP_TRANS(t), &iov, 1, to_read, max_retries); } -static int ksmbd_tcp_writev(struct ksmbd_transport *t, struct kvec *iov, - int nvecs, int size, bool need_invalidate, - unsigned int remote_key) - +static int ksmbd_tcp_writev(struct ksmbd_transport *t, + const struct ksmbd_transport_write *tx) { - struct msghdr smb_msg = {.msg_flags = MSG_NOSIGNAL}; + struct msghdr smb_msg = { + .msg_flags = MSG_NOSIGNAL | tx->msg_flags, + }; - return kernel_sendmsg(TCP_TRANS(t)->sock, &smb_msg, iov, nvecs, size); + return kernel_sendmsg(TCP_TRANS(t)->sock, &smb_msg, tx->iov, + tx->iov_cnt, tx->size); } static void ksmbd_tcp_disconnect(struct ksmbd_transport *t) @@ -434,6 +441,11 @@ static void ksmbd_tcp_disconnect(struct ksmbd_transport *t) atomic_dec(&active_num_conn); } +static void ksmbd_tcp_shutdown(struct ksmbd_transport *t) +{ + kernel_sock_shutdown(TCP_TRANS(t)->sock, SHUT_RDWR); +} + static void tcp_destroy_socket(struct socket *ksmbd_socket) { int ret; @@ -511,6 +523,12 @@ static int create_socket(struct interface *iface) goto out_error; } + /* + * Accepted sockets inherit the listener's net reference. Keep TCP + * timers alive after a kernel socket is released. + */ + sk_net_refcnt_upgrade(ksmbd_socket->sk); + ret = kernel_listen(ksmbd_socket, KSMBD_SOCKET_BACKLOG); if (ret) { pr_err("Port listen() error: %d\n", ret); @@ -680,5 +698,6 @@ static const struct ksmbd_transport_ops ksmbd_tcp_transport_ops = { .read = ksmbd_tcp_read, .writev = ksmbd_tcp_writev, .disconnect = ksmbd_tcp_disconnect, + .shutdown = ksmbd_tcp_shutdown, .free_transport = ksmbd_tcp_free_transport, }; diff --git a/fs/smb/server/vfs.c b/fs/smb/server/vfs.c index e493725c3fe3..d2b524f79cbe 100644 --- a/fs/smb/server/vfs.c +++ b/fs/smb/server/vfs.c @@ -17,10 +17,10 @@ #include <linux/fsnotify.h> #include <linux/dcache.h> #include <linux/slab.h> +#include <linux/sizes.h> #include <linux/vmalloc.h> #include <linux/sched/xacct.h> #include <linux/crc32c.h> -#include <linux/splice.h> #include <linux/fileattr.h> #include "glob.h" @@ -299,9 +299,6 @@ static int check_lock_range(struct file *filp, loff_t start, loff_t end, struct file_lock_context *ctx = locks_inode_context(file_inode(filp)); int error = 0; - if (start == end) - return 0; - if (!ctx || list_empty_careful(&ctx->flc_posix)) return 0; @@ -347,7 +344,7 @@ int ksmbd_vfs_read(struct ksmbd_work *work, struct ksmbd_file *fp, size_t count, ssize_t nbytes = 0; struct inode *inode = file_inode(filp); - if (S_ISDIR(inode->i_mode)) + if (S_ISDIR(inode->i_mode) && !ksmbd_stream_fd(fp)) return -EISDIR; if (unlikely(count == 0)) @@ -476,7 +473,8 @@ int ksmbd_vfs_write(struct ksmbd_work *work, struct ksmbd_file *fp, if (work->conn->connection_type) { if (!(fp->daccess & (FILE_WRITE_DATA_LE | FILE_APPEND_DATA_LE)) || - S_ISDIR(file_inode(fp->filp)->i_mode)) { + (S_ISDIR(file_inode(fp->filp)->i_mode) && + !ksmbd_stream_fd(fp))) { pr_err("no right to write(%pD)\n", fp->filp); err = -EACCES; goto out; @@ -663,15 +661,35 @@ out1: return err; } -int ksmbd_vfs_rename(struct ksmbd_work *work, const struct path *old_path, - char *newname, int flags) +int ksmbd_vfs_check_rename_share(struct ksmbd_work *work, + const struct path *old_path) { + struct ksmbd_file *parent_fp; + int err = 0; + + parent_fp = ksmbd_lookup_fd_inode(old_path->dentry->d_parent); + if (!parent_fp) + return 0; + + if ((parent_fp->daccess & FILE_DELETE_LE) || + (!parent_fp->attrib_only && + !(parent_fp->saccess & FILE_SHARE_DELETE_LE))) { + ksmbd_debug(VFS, "parent dir blocks delete sharing\n"); + err = -ESHARE; + } + ksmbd_fd_put(work, parent_fp); + return err; +} + +int ksmbd_vfs_rename(struct ksmbd_work *work, struct ksmbd_file *old_fp, + char *newname, int flags) +{ + const struct path *old_path = &old_fp->filp->f_path; struct dentry *old_child = old_path->dentry; struct path new_path; struct qstr new_last; struct renamedata rd; struct ksmbd_share_config *share_conf = work->tcon->share_conf; - struct ksmbd_file *parent_fp; int err, lookup_flags = LOOKUP_NO_SYMLINKS; if (ksmbd_override_fsids(work)) @@ -703,25 +721,28 @@ retry: if (err) goto out_drop_write; - if (!work->tcon->posix_extensions && d_is_dir(old_child) && - ksmbd_has_open_files(old_child)) { + if (d_is_dir(old_child) && ksmbd_has_nonposix_open_child(old_fp)) { err = -EACCES; goto out3; } - parent_fp = ksmbd_lookup_fd_inode(old_child->d_parent); - if (parent_fp) { - if ((parent_fp->daccess & FILE_DELETE_LE) || - (!parent_fp->attrib_only && - !(parent_fp->saccess & FILE_SHARE_DELETE_LE))) { - pr_err("parent dir blocks delete sharing\n"); - err = -ESHARE; - ksmbd_fd_put(work, parent_fp); - goto out3; - } - ksmbd_fd_put(work, parent_fp); + /* + * See MS-FSA 2.1.5.15.12. + * An overwrite rename must fail with STATUS_ACCESS_DENIED if the + * existing target still has a non-POSIX open. + */ + if (!(flags & (RENAME_NOREPLACE | RENAME_EXCHANGE)) && + d_inode(rd.new_dentry) && + d_inode(rd.new_dentry) != d_inode(old_child) && + ksmbd_has_other_nonposix_open(rd.new_dentry)) { + err = -EACCES; + goto out3; } + err = ksmbd_vfs_check_rename_share(work, old_path); + if (err) + goto out3; + if (d_is_symlink(rd.new_dentry)) { err = -EACCES; goto out3; @@ -930,54 +951,205 @@ int ksmbd_vfs_zero_data(struct ksmbd_work *work, struct ksmbd_file *fp, loff_t off, loff_t len) { const struct cred *saved_cred; + loff_t pos = off, size; + char *zero_buf = NULL; int err; smb_break_all_levII_oplock(work, fp, 1); + if (!work->tcon->posix_extensions) { + loff_t size = i_size_read(file_inode(fp->filp)); + + if (off < size) { + err = check_lock_range(fp->filp, off, + min(off + len, size) - 1, + WRITE); + if (err) + return -EAGAIN; + } + } + saved_cred = override_creds(fp->filp->f_cred); - if (fp->f_ci->m_fattr & FILE_ATTRIBUTE_SPARSE_FILE_LE) + if (fp->f_ci->m_fattr & FILE_ATTRIBUTE_SPARSE_FILE_LE) { err = vfs_fallocate(fp->filp, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, off, len); - else - err = vfs_fallocate(fp->filp, - FALLOC_FL_ZERO_RANGE | FALLOC_FL_KEEP_SIZE, - off, len); + } else { + size = i_size_read(file_inode(fp->filp)); + if (off >= size) { + err = 0; + goto out; + } + + len = min(len, size - off); + zero_buf = kvzalloc(SZ_64K, GFP_KERNEL); + if (!zero_buf) { + err = -ENOMEM; + goto out; + } + + while (len) { + ssize_t written; + size_t count = min_t(loff_t, len, SZ_64K); + + written = kernel_write(fp->filp, zero_buf, count, &pos); + if (written < 0) { + err = written; + goto out; + } + if (!written) { + err = -EIO; + goto out; + } + len -= written; + } + err = 0; + } +out: revert_creds(saved_cred); + kvfree(zero_buf); return err; } -int ksmbd_vfs_fqar_lseek(struct ksmbd_file *fp, loff_t start, loff_t length, - struct file_allocated_range_buffer *ranges, - unsigned int in_count, unsigned int *out_count) +int ksmbd_vfs_zero_holes(struct ksmbd_file *fp) { struct file *f = fp->filp; - struct inode *inode = file_inode(fp->filp); - loff_t maxbytes = (u64)inode->i_sb->s_maxbytes, end; - loff_t extent_start, extent_end; - int ret = 0; + const struct cred *saved_cred; + loff_t size, pos = 0; + char *zero_buf; + int err; - if (start > maxbytes) - return -EFBIG; + err = file_write_and_wait(f); + if (err) + return err; - if (!in_count) + size = i_size_read(file_inode(f)); + if (!size) return 0; /* - * Shrink request scope to what the fs can actually handle. + * FALLOC_FL_ZERO_RANGE may leave unwritten extents, which SEEK_DATA + * reports as holes. Write zeroes into each hole so that clearing the + * sparse attribute leaves the file fully allocated. */ - if (length > maxbytes || (maxbytes - length) < start) - length = maxbytes - start; + zero_buf = kvzalloc(SZ_64K, GFP_KERNEL); + if (!zero_buf) + return -ENOMEM; + + saved_cred = override_creds(f->f_cred); + while (pos < size) { + loff_t data, hole; + + hole = vfs_llseek(f, pos, SEEK_HOLE); + if (hole == -ENXIO || hole >= size) + break; + if (hole < 0) { + err = hole; + goto out; + } + + data = vfs_llseek(f, hole, SEEK_DATA); + if (data == -ENXIO) { + data = size; + } else if (data < 0) { + err = data; + goto out; + } + data = min(data, size); + if (data <= hole) { + err = -EIO; + goto out; + } + + pos = hole; + while (pos < data) { + ssize_t written; + size_t count = min_t(loff_t, data - pos, SZ_64K); + + written = kernel_write(f, zero_buf, count, &pos); + if (written < 0) { + err = written; + goto out; + } + if (!written) { + err = -EIO; + goto out; + } + } + } + err = file_write_and_wait(f); +out: + revert_creds(saved_cred); + kvfree(zero_buf); + return err; +} + +int ksmbd_vfs_trim_data(struct ksmbd_work *work, struct ksmbd_file *fp, + loff_t off, loff_t len) +{ + const struct cred *saved_cred; + int err; + + smb_break_all_levII_oplock(work, fp, 1); + if (!work->tcon->posix_extensions) { + loff_t size = i_size_read(file_inode(fp->filp)); + + if (off < size) { + err = check_lock_range(fp->filp, off, + min(off + len, size) - 1, + WRITE); + if (err) + return -EAGAIN; + } + } + + saved_cred = override_creds(fp->filp->f_cred); + err = vfs_fallocate(fp->filp, + FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, + off, len); + revert_creds(saved_cred); + return err; +} - if (start + length > inode->i_size) - length = inode->i_size - start; +int ksmbd_vfs_query_allocated_ranges(struct ksmbd_file *fp, loff_t start, + loff_t length, + struct file_allocated_range_buffer *ranges, + unsigned int in_count, + unsigned int *out_count) +{ + struct file *f = fp->filp; + struct inode *inode = file_inode(fp->filp); + loff_t maxbytes = inode->i_sb->s_maxbytes, size; + loff_t extent_start, extent_end, end; + int ret = 0; *out_count = 0; + if (start < 0 || length < 0) + return -EINVAL; + if (start > maxbytes) + return -EFBIG; + if (!in_count) + return 0; + if (length > maxbytes || maxbytes - length < start) + length = maxbytes - start; + size = i_size_read(inode); + if (!length || start >= size) + return 0; + if (length > size - start) + length = size - start; + end = start + length; + if ((fp->f_ci->m_fattr & FILE_ATTRIBUTE_SPARSE_FILE_LE) && + start < end) { + ret = file_write_and_wait_range(f, start, end - 1); + if (ret) + return ret; + } + while (start < end && *out_count < in_count) { extent_start = vfs_llseek(f, start, SEEK_DATA); if (extent_start < 0) { if (extent_start != -ENXIO) - ret = (int)extent_start; + ret = extent_start; break; } @@ -987,7 +1159,7 @@ int ksmbd_vfs_fqar_lseek(struct ksmbd_file *fp, loff_t start, loff_t length, extent_end = vfs_llseek(f, extent_start, SEEK_HOLE); if (extent_end < 0) { if (extent_end != -ENXIO) - ret = (int)extent_end; + ret = extent_end; break; } else if (extent_start >= extent_end) { break; @@ -996,10 +1168,12 @@ int ksmbd_vfs_fqar_lseek(struct ksmbd_file *fp, loff_t start, loff_t length, ranges[*out_count].file_offset = cpu_to_le64(extent_start); ranges[(*out_count)++].length = cpu_to_le64(min(extent_end, end) - extent_start); - start = extent_end; } + if (!ret && start < end && *out_count == in_count) + ret = -E2BIG; + return ret; } @@ -1689,6 +1863,35 @@ int ksmbd_vfs_fill_dentry_attrs(struct ksmbd_work *work, } } + /* + * Only pay for this when it'll actually be used: AAPL + * READDIR_ATTR_V2's flags field (AAPL_READDIR_ATTR_V2_NO_XATTR) is + * the only consumer. XATTR_NAME_STREAM ("user.DosStream.") is a + * reliable, distinct prefix for genuine ADS/stream xattrs -- unlike + * DOSATTRIB or ACL xattrs, which live under different prefixes, so + * this can't false-positive into telling Finder a file has no extra + * data when it actually does. + */ + ksmbd_kstat->has_ads_stream = false; + if (work->conn->aapl_readdir_attr_v2) { + char *xattr_list = NULL, *name; + ssize_t xattr_list_len; + + xattr_list_len = ksmbd_vfs_listxattr(dentry, &xattr_list); + if (xattr_list_len > 0) { + for (name = xattr_list; + name - xattr_list < xattr_list_len; + name += strlen(name) + 1) { + if (!strncmp(name, XATTR_NAME_STREAM, + XATTR_NAME_STREAM_LEN)) { + ksmbd_kstat->has_ads_stream = true; + break; + } + } + } + kvfree(xattr_list); + } + return 0; } @@ -1739,6 +1942,71 @@ int ksmbd_vfs_xattr_stream_name(char *stream_name, char **xattr_stream_name, return 0; } +static ssize_t ksmbd_vfs_copy_file_range_buffered(struct ksmbd_work *work, + struct ksmbd_file *src_fp, + struct ksmbd_file *dst_fp, + loff_t src_off, + loff_t dst_off, size_t len) +{ + size_t buf_size = min_t(size_t, len, SZ_1M); + size_t copied = 0; + char *buf; + ssize_t ret = 0; + + buf = kvmalloc(buf_size, KSMBD_DEFAULT_GFP); + if (!buf) + return -ENOMEM; + + while (copied < len) { + size_t chunk_size = min(buf_size, len - copied); + size_t done = 0; + loff_t src_pos, dst_pos; + + if (dst_off > src_off) { + src_pos = src_off + len - copied - chunk_size; + dst_pos = dst_off + len - copied - chunk_size; + } else { + src_pos = src_off + copied; + dst_pos = dst_off + copied; + } + + while (done < chunk_size) { + loff_t pos = src_pos + done; + + ret = ksmbd_vfs_read(work, src_fp, chunk_size - done, + &pos, buf + done); + if (ret <= 0) { + if (!ret) + ret = -EIO; + goto out; + } + done += ret; + } + + done = 0; + while (done < chunk_size) { + loff_t pos = dst_pos + done; + ssize_t written = 0; + + ret = ksmbd_vfs_write(work, dst_fp, buf + done, + chunk_size - done, &pos, false, + &written); + if (ret < 0) + goto out; + if (!written) { + ret = -EIO; + goto out; + } + done += written; + } + copied += chunk_size; + } + ret = copied; +out: + kvfree(buf); + return ret; +} + int ksmbd_vfs_copy_file_ranges(struct ksmbd_work *work, struct ksmbd_file *src_fp, struct ksmbd_file *dst_fp, @@ -1766,9 +2034,6 @@ int ksmbd_vfs_copy_file_ranges(struct ksmbd_work *work, return -EACCES; } - if (ksmbd_stream_fd(src_fp) || ksmbd_stream_fd(dst_fp)) - return -EBADF; - smb_break_all_levII_oplock(work, dst_fp, 1); if (!work->tcon->posix_extensions) { @@ -1786,35 +2051,138 @@ int ksmbd_vfs_copy_file_ranges(struct ksmbd_work *work, } } - src_file_size = i_size_read(file_inode(src_fp->filp)); + if (ksmbd_stream_fd(src_fp)) { + const struct cred *saved_cred; + + saved_cred = override_creds(src_fp->filp->f_cred); + src_file_size = ksmbd_vfs_casexattr_len( + file_mnt_idmap(src_fp->filp), + src_fp->filp->f_path.dentry, + src_fp->stream.name, src_fp->stream.size); + revert_creds(saved_cred); + if (src_file_size < 0) + return src_file_size; + } else { + src_file_size = i_size_read(file_inode(src_fp->filp)); + } + + /* + * macOS Finder's Cmd+D duplicate sends FSCTL_SRV_COPYCHUNK with + * ChunkCount=0 meaning "copy the whole file/stream", not the + * standard SMB2 "query my copy limits, no data" semantics -- + * fsctl_copychunk() only reaches here with chunk_count == 0 for + * AAPL-negotiated connections, so this doesn't affect compliant + * non-AAPL clients. Without this, the destination stays at its + * just-created 0 bytes / empty stream: the for loop below is a + * no-op when chunk_count is 0, since it never has an iteration to + * treat as "copy everything". + */ + if (chunk_count == 0 && work->conn->is_aapl) { + loff_t off = 0; + + while (off < src_file_size) { + size_t remaining = src_file_size - off; + ssize_t copied; + + /* Same source/destination offset here: an in-place, + * same-inode copy at matching offsets is a degenerate + * no-op range, not a real overlap, but vfs_copy_file_range + * still doesn't support streams -- route those (and the + * same-inode case defensively) through the buffered path. + */ + if (ksmbd_stream_fd(src_fp) || ksmbd_stream_fd(dst_fp) || + file_inode(src_fp->filp) == file_inode(dst_fp->filp)) { + copied = ksmbd_vfs_copy_file_range_buffered(work, src_fp, dst_fp, + off, off, remaining); + } else { + copied = vfs_copy_file_range(src_fp->filp, off, + dst_fp->filp, off, + remaining, 0); + if (copied == -EOPNOTSUPP || copied == -EXDEV) + copied = vfs_copy_file_range(src_fp->filp, off, + dst_fp->filp, off, + remaining, + COPY_FILE_SPLICE); + } + if (copied < 0) + return copied; + if (copied == 0) + break; + off += copied; + } + + /* + * This is a synthesized whole-file copy, not a response to + * any chunk descriptor the client actually sent (it sent + * none -- chunk_count is 0). Report zero chunks/chunk-bytes + * rather than inventing a chunk that doesn't correspond to + * anything in the request; only total_size_written (bytes + * actually copied) is meaningful here. + */ + *chunk_count_written = 0; + *chunk_size_written = 0; + *total_size_written = off; + return 0; + } for (i = 0; i < chunk_count; i++) { + bool stream_len_mismatch = false; + size_t copy_len; + src_off = le64_to_cpu(chunks[i].SourceOffset); dst_off = le64_to_cpu(chunks[i].TargetOffset); len = le32_to_cpu(chunks[i].Length); + copy_len = len; - if (src_off + len > src_file_size) + if (src_off < 0) return -E2BIG; + if (src_off > src_file_size || len > src_file_size - src_off) { + /* + * macOS can reuse the main file's chunk list when copying + * streams, so the requested range can exceed the size of + * the xattr-backed stream. For an AAPL connection, copy the + * available stream data and report the requested length to + * avoid a copy length mismatch. + */ + if (!work->conn->is_aapl || + !ksmbd_stream_fd(src_fp) || + !ksmbd_stream_fd(dst_fp)) + return -E2BIG; + + stream_len_mismatch = true; + if (src_off < src_file_size) + copy_len = src_file_size - src_off; + else + copy_len = 0; + } + /* - * vfs_copy_file_range does not allow overlapped copying - * within the same file. + * vfs_copy_file_range does not support streams or overlapping + * ranges within the same file. */ - if (file_inode(src_fp->filp) == file_inode(dst_fp->filp) && - dst_off + len > src_off && - dst_off < src_off + len) - ret = do_splice_direct(src_fp->filp, &src_off, - dst_fp->filp, &dst_off, - min_t(size_t, len, MAX_RW_COUNT), 0); - else - ret = vfs_copy_file_range(src_fp->filp, src_off, - dst_fp->filp, dst_off, len, 0); - if (ret == -EOPNOTSUPP || ret == -EXDEV) + if (!copy_len) { + ret = 0; + } else if (ksmbd_stream_fd(src_fp) || ksmbd_stream_fd(dst_fp) || + (file_inode(src_fp->filp) == file_inode(dst_fp->filp) && + dst_off + copy_len > src_off && + dst_off < src_off + copy_len)) { + ret = ksmbd_vfs_copy_file_range_buffered(work, src_fp, + dst_fp, src_off, + dst_off, copy_len); + } else { ret = vfs_copy_file_range(src_fp->filp, src_off, - dst_fp->filp, dst_off, len, - COPY_FILE_SPLICE); + dst_fp->filp, dst_off, copy_len, 0); + if (ret == -EOPNOTSUPP || ret == -EXDEV) + ret = vfs_copy_file_range(src_fp->filp, src_off, + dst_fp->filp, dst_off, + copy_len, + COPY_FILE_SPLICE); + } if (ret < 0) return ret; + if (stream_len_mismatch) + ret = len; *chunk_count_written += 1; *total_size_written += ret; @@ -1907,15 +2275,11 @@ void ksmbd_vfs_update_compressed_fattr(struct dentry *dentry, __le32 *fattr) struct file_kattr fa = { .flags_valid = true }; rc = vfs_fileattr_get(dentry, &fa); - if (rc == -ENOIOCTLCMD) - *fattr &= ~FILE_ATTRIBUTE_COMPRESSED_LE; if (rc) return; if (fa.flags & FS_COMPR_FL) *fattr |= FILE_ATTRIBUTE_COMPRESSED_LE; - else - *fattr &= ~FILE_ATTRIBUTE_COMPRESSED_LE; } int ksmbd_vfs_get_compression(struct ksmbd_file *fp, u16 *fmt) @@ -1924,15 +2288,19 @@ int ksmbd_vfs_get_compression(struct ksmbd_file *fp, u16 *fmt) int rc; rc = vfs_fileattr_get(fp->filp->f_path.dentry, &fa); - if (rc == -ENOIOCTLCMD) { - *fmt = COMPRESSION_FORMAT_NONE; + if (rc == -ENOIOCTLCMD || rc == -ENOTTY || rc == -EINVAL || + rc == -EOPNOTSUPP) { + if (fp->f_ci->m_fattr & FILE_ATTRIBUTE_COMPRESSED_LE) + *fmt = COMPRESSION_FORMAT_LZNT1; + else + *fmt = COMPRESSION_FORMAT_NONE; rc = 0; goto out; } if (rc) goto out; - if (fa.flags & FS_COMPR_FL) + if (fp->f_ci->m_fattr & FILE_ATTRIBUTE_COMPRESSED_LE) *fmt = COMPRESSION_FORMAT_LZNT1; else *fmt = COMPRESSION_FORMAT_NONE; @@ -1941,7 +2309,9 @@ out: return rc; } -int ksmbd_vfs_set_compression(struct ksmbd_work *work, struct ksmbd_file *fp, u16 fmt) +static int __ksmbd_vfs_set_compression(struct ksmbd_work *work, + struct ksmbd_file *fp, u16 fmt, + bool check_access) { const struct cred *saved_cred = NULL; struct file_kattr fa; @@ -1951,13 +2321,23 @@ int ksmbd_vfs_set_compression(struct ksmbd_work *work, struct ksmbd_file *fp, u1 __le32 old_fattr; int rc; - if (!(fp->daccess & FILE_WRITE_DATA_LE)) { + if (check_access && !(fp->daccess & FILE_WRITE_DATA_LE)) { rc = -EACCES; goto out; } + if (fmt != COMPRESSION_FORMAT_NONE && + fmt != COMPRESSION_FORMAT_DEFAULT && + fmt != COMPRESSION_FORMAT_LZNT1) { + rc = -EINVAL; + goto out; + } + saved_cred = override_creds(fp->filp->f_cred); rc = vfs_fileattr_get(dentry, &fa); + if (rc == -ENOIOCTLCMD || rc == -ENOTTY || rc == -EINVAL || + rc == -EOPNOTSUPP) + goto update_fattr; if (rc) goto out; @@ -1967,9 +2347,6 @@ int ksmbd_vfs_set_compression(struct ksmbd_work *work, struct ksmbd_file *fp, u1 } else if (fmt == COMPRESSION_FORMAT_DEFAULT || fmt == COMPRESSION_FORMAT_LZNT1) { flags |= FS_COMPR_FL; - } else { - rc = -EINVAL; - goto out; } if (flags != fa.flags) { @@ -1980,28 +2357,34 @@ int ksmbd_vfs_set_compression(struct ksmbd_work *work, struct ksmbd_file *fp, u1 rc = vfs_fileattr_set(idmap, dentry, &fa); mnt_drop_write_file(fp->filp); + if (rc == -ENOIOCTLCMD || rc == -ENOTTY || rc == -EINVAL || + rc == -EOPNOTSUPP) + goto update_fattr; if (rc) goto out; } +update_fattr: old_fattr = fp->f_ci->m_fattr; if (fmt == COMPRESSION_FORMAT_NONE) fp->f_ci->m_fattr &= ~FILE_ATTRIBUTE_COMPRESSED_LE; else fp->f_ci->m_fattr |= FILE_ATTRIBUTE_COMPRESSED_LE; - if (fp->f_ci->m_fattr != old_fattr && - test_share_config_flag(work->tcon->share_conf, - KSMBD_SHARE_FLAG_STORE_DOS_ATTRS)) { - struct xattr_dos_attrib da; + if (fp->f_ci->m_fattr != old_fattr) { + struct xattr_dos_attrib da = {0}; rc = ksmbd_vfs_get_dos_attrib_xattr(idmap, dentry, &da); if (rc <= 0) { - rc = 0; - goto out; + da.version = 4; + da.itime = fp->itime; + da.create_time = fp->create_time; + da.flags = XATTR_DOSINFO_CREATE_TIME | + XATTR_DOSINFO_ITIME; } da.attr = le32_to_cpu(fp->f_ci->m_fattr); + da.flags |= XATTR_DOSINFO_ATTRIB; rc = ksmbd_vfs_set_dos_attrib_xattr(idmap, &fp->filp->f_path, &da, true); @@ -2014,3 +2397,15 @@ out: revert_creds(saved_cred); return rc; } + +int ksmbd_vfs_set_compression(struct ksmbd_work *work, + struct ksmbd_file *fp, u16 fmt) +{ + return __ksmbd_vfs_set_compression(work, fp, fmt, true); +} + +int ksmbd_vfs_set_compression_create(struct ksmbd_work *work, + struct ksmbd_file *fp, u16 fmt) +{ + return __ksmbd_vfs_set_compression(work, fp, fmt, false); +} diff --git a/fs/smb/server/vfs.h b/fs/smb/server/vfs.h index 7b3d2f4fd985..55d099de71f5 100644 --- a/fs/smb/server/vfs.h +++ b/fs/smb/server/vfs.h @@ -70,6 +70,7 @@ struct ksmbd_kstat { struct kstat *kstat; unsigned long long create_time; __le32 file_attributes; + bool has_ads_stream; /* AAPL READDIR_ATTR V2 xattr-presence flag */ }; int ksmbd_vfs_lock_parent(struct dentry *parent, struct dentry *child); @@ -87,8 +88,10 @@ int ksmbd_vfs_remove_file(struct ksmbd_work *work, const struct path *path); int ksmbd_vfs_link(struct ksmbd_work *work, const char *oldname, const char *newname); int ksmbd_vfs_getattr(const struct path *path, struct kstat *stat); -int ksmbd_vfs_rename(struct ksmbd_work *work, const struct path *old_path, - char *newname, int flags); +int ksmbd_vfs_rename(struct ksmbd_work *work, struct ksmbd_file *old_fp, + char *newname, int flags); +int ksmbd_vfs_check_rename_share(struct ksmbd_work *work, + const struct path *old_path); int ksmbd_vfs_truncate(struct ksmbd_work *work, struct ksmbd_file *fp, loff_t size); struct srv_copychunk; @@ -132,10 +135,15 @@ int ksmbd_vfs_empty_dir(struct ksmbd_file *fp); void ksmbd_vfs_set_fadvise(struct file *filp, __le32 option); int ksmbd_vfs_zero_data(struct ksmbd_work *work, struct ksmbd_file *fp, loff_t off, loff_t len); +int ksmbd_vfs_zero_holes(struct ksmbd_file *fp); +int ksmbd_vfs_trim_data(struct ksmbd_work *work, struct ksmbd_file *fp, + loff_t off, loff_t len); struct file_allocated_range_buffer; -int ksmbd_vfs_fqar_lseek(struct ksmbd_file *fp, loff_t start, loff_t length, - struct file_allocated_range_buffer *ranges, - unsigned int in_count, unsigned int *out_count); +int ksmbd_vfs_query_allocated_ranges(struct ksmbd_file *fp, loff_t start, + loff_t length, + struct file_allocated_range_buffer *ranges, + unsigned int in_count, + unsigned int *out_count); int ksmbd_vfs_unlink(struct file *filp); void *ksmbd_vfs_init_kstat(char **p, struct ksmbd_kstat *ksmbd_kstat); int ksmbd_vfs_fill_dentry_attrs(struct ksmbd_work *work, @@ -171,4 +179,6 @@ int ksmbd_vfs_inherit_posix_acl(struct mnt_idmap *idmap, void ksmbd_vfs_update_compressed_fattr(struct dentry *dentry, __le32 *fattr); int ksmbd_vfs_get_compression(struct ksmbd_file *fp, u16 *fmt); int ksmbd_vfs_set_compression(struct ksmbd_work *work, struct ksmbd_file *fp, u16 fmt); +int ksmbd_vfs_set_compression_create(struct ksmbd_work *work, + struct ksmbd_file *fp, u16 fmt); #endif /* __KSMBD_VFS_H__ */ diff --git a/fs/smb/server/vfs_cache.c b/fs/smb/server/vfs_cache.c index a141025581af..81626d204249 100644 --- a/fs/smb/server/vfs_cache.c +++ b/fs/smb/server/vfs_cache.c @@ -66,33 +66,73 @@ static const struct ksmbd_const_name ksmbd_lease_const_names[] = { static const struct ksmbd_const_name ksmbd_oplock_const_names[] = { {SMB2_OPLOCK_LEVEL_NONE, "OPLOCK_NONE"}, {SMB2_OPLOCK_LEVEL_II, "OPLOCK_II"}, - {SMB2_OPLOCK_LEVEL_EXCLUSIVE, "OPLOCK_EXECL"}, + {SMB2_OPLOCK_LEVEL_EXCLUSIVE, "OPLOCK_EXCLUSIVE"}, {SMB2_OPLOCK_LEVEL_BATCH, "OPLOCK_BATCH"}, }; +static const struct ksmbd_const_name ksmbd_file_state_names[] = { + {FP_NEW, "new"}, + {FP_INITED, "open"}, + {FP_CLOSED, "closed"}, +}; + +#define KSMBD_PROC_FILE_DURABLE BIT(0) +#define KSMBD_PROC_FILE_PERSISTENT BIT(1) +#define KSMBD_PROC_FILE_RESILIENT BIT(2) +#define KSMBD_PROC_FILE_DELETE_ON_CLOSE BIT(3) +#define KSMBD_PROC_FILE_STREAM BIT(4) +#define KSMBD_PROC_FILE_POSIX BIT(5) +#define KSMBD_PROC_FILE_ATTRIB_ONLY BIT(6) + +static const struct ksmbd_const_name ksmbd_file_flag_names[] = { + {KSMBD_PROC_FILE_DURABLE, "durable"}, + {KSMBD_PROC_FILE_PERSISTENT, "persistent"}, + {KSMBD_PROC_FILE_RESILIENT, "resilient"}, + {KSMBD_PROC_FILE_DELETE_ON_CLOSE, "delete-on-close"}, + {KSMBD_PROC_FILE_STREAM, "stream"}, + {KSMBD_PROC_FILE_POSIX, "posix"}, + {KSMBD_PROC_FILE_ATTRIB_ONLY, "attrib-only"}, +}; + +static unsigned int ksmbd_proc_file_flags(struct ksmbd_file *fp) +{ + unsigned int flags = 0; + + if (fp->is_durable) + flags |= KSMBD_PROC_FILE_DURABLE; + if (fp->is_persistent) + flags |= KSMBD_PROC_FILE_PERSISTENT; + if (fp->is_resilient) + flags |= KSMBD_PROC_FILE_RESILIENT; + if (fp->coption & FILE_DELETE_ON_CLOSE_LE) + flags |= KSMBD_PROC_FILE_DELETE_ON_CLOSE; + if (fp->stream.name) + flags |= KSMBD_PROC_FILE_STREAM; + if (fp->is_posix_ctxt) + flags |= KSMBD_PROC_FILE_POSIX; + if (fp->attrib_only) + flags |= KSMBD_PROC_FILE_ATTRIB_ONLY; + return flags; +} + static int proc_show_files(struct seq_file *m, void *v) { struct ksmbd_file *fp = NULL; unsigned int id; struct oplock_info *opinfo; - seq_printf(m, "#%-10s %-10s %-10s %-10s %-15s %-10s %-10s %s\n", - "<tree id>", "<pid>", "<vid>", "<refcnt>", - "<oplock>", "<daccess>", "<saccess>", - "<name>"); - read_lock(&global_ft.lock); idr_for_each_entry(global_ft.idr, fp, id) { - seq_printf(m, "%#-10x %#-10llx %#-10llx %#-10x", - fp->tcon ? fp->tcon->id : 0, - fp->persistent_id, - fp->volatile_id, - atomic_read(&fp->refcount)); + seq_printf(m, "tree_id:\t0x%x\n", fp->tcon ? fp->tcon->id : 0); + seq_printf(m, "persistent_id:\t0x%llx\n", fp->persistent_id); + seq_printf(m, "volatile_id:\t0x%llx\n", fp->volatile_id); + seq_printf(m, "refcount:\t%d\n", atomic_read(&fp->refcount)); rcu_read_lock(); opinfo = rcu_dereference(fp->f_opinfo); if (opinfo) { const struct ksmbd_const_name *const_names; + const char *name; int count; unsigned int level; @@ -106,16 +146,32 @@ static int proc_show_files(struct seq_file *m, void *v) level = opinfo->level; } rcu_read_unlock(); - ksmbd_proc_show_const_name(m, " %-15s", - const_names, count, level); + name = ksmbd_proc_const_name(const_names, count, level); + if (name) + seq_printf(m, "oplock:\t%s\n", name); + else + seq_printf(m, "oplock:\t0x%x\n", level); } else { rcu_read_unlock(); - seq_printf(m, " %-15s", " "); + seq_puts(m, "oplock:\tnone\n"); } - seq_printf(m, " %#010x %#010x %s\n", - le32_to_cpu(fp->daccess), - le32_to_cpu(fp->saccess), + seq_printf(m, "state:\t%s\n", + ksmbd_proc_const_name(ksmbd_file_state_names, + ARRAY_SIZE(ksmbd_file_state_names), + fp->f_state)); + seq_printf(m, "durable_timeout:\t%u\n", fp->durable_timeout); + seq_printf(m, "create_options:\t0x%08x\n", + le32_to_cpu(fp->coption)); + seq_printf(m, "desired_access:\t0x%08x\n", + le32_to_cpu(fp->daccess)); + seq_printf(m, "share_access:\t0x%08x\n", + le32_to_cpu(fp->saccess)); + seq_puts(m, "flags:\t"); + ksmbd_proc_show_flag_names(m, ksmbd_file_flag_names, + ARRAY_SIZE(ksmbd_file_flag_names), + ksmbd_proc_file_flags(fp)); + seq_printf(m, "\nname:\t%s\n\n", fp->filp->f_path.dentry->d_name.name); } read_unlock(&global_ft.lock); @@ -124,7 +180,8 @@ static int proc_show_files(struct seq_file *m, void *v) static int create_proc_files(void) { - ksmbd_proc_create("files", proc_show_files, NULL); + if (!ksmbd_proc_create("files", proc_show_files, NULL)) + return -ENOMEM; return 0; } #else @@ -135,6 +192,16 @@ static bool durable_scavenger_running; static DEFINE_MUTEX(durable_scavenger_lock); static wait_queue_head_t dh_wq; +bool ksmbd_durable_scavenger_active(void) +{ + bool active; + + mutex_lock(&durable_scavenger_lock); + active = durable_scavenger_running; + mutex_unlock(&durable_scavenger_lock); + return active; +} + void ksmbd_set_fd_limit(unsigned long limit) { limit = min(limit, get_max_files()); @@ -232,6 +299,12 @@ bool ksmbd_inode_pending_delete(struct ksmbd_file *fp) down_read(&ci->m_lock); ret = (ci->m_flags & S_DEL_PENDING); up_read(&ci->m_lock); + if (ret || !ksmbd_stream_fd(fp)) + return ret; + + spin_lock(&fp->f_lock); + ret = fp->stream_del_pending; + spin_unlock(&fp->f_lock); return ret; } @@ -294,6 +367,40 @@ void ksmbd_fd_set_delete_on_close(struct ksmbd_file *fp, up_write(&ci->m_lock); } +/* + * FileDispositionInformation (SET_INFO) on a stream handle must only + * mark the stream for deletion, not the whole file -- otherwise + * deleting a single alternate data stream (e.g. AFP_AfpInfo) deletes + * the entire file's data along with it. + * + * This is tracked on fp itself (stream_del_pending), not the shared + * ksmbd_inode: the inode-wide S_DEL_ON_CLS_STREAM flag used by + * ksmbd_fd_set_delete_on_close() can't record *which* stream should be + * deleted, so if a different stream handle on the same file closed + * first, it would delete the wrong stream. + */ +void ksmbd_fd_set_delete_pending(struct ksmbd_file *fp) +{ + if (ksmbd_stream_fd(fp)) { + spin_lock(&fp->f_lock); + fp->stream_del_pending = true; + spin_unlock(&fp->f_lock); + } else { + ksmbd_set_inode_pending_delete(fp); + } +} + +void ksmbd_fd_clear_delete_pending(struct ksmbd_file *fp) +{ + if (ksmbd_stream_fd(fp)) { + spin_lock(&fp->f_lock); + fp->stream_del_pending = false; + spin_unlock(&fp->f_lock); + } else { + ksmbd_clear_inode_pending_delete(fp); + } +} + static void ksmbd_inode_hash(struct ksmbd_inode *ci) { struct hlist_head *b = inode_hashtable + @@ -414,6 +521,19 @@ static void __ksmbd_inode_close(struct ksmbd_file *fp) } up_write(&ci->m_lock); + /* + * Per-handle delete-pending from ksmbd_fd_set_delete_pending() + * (FileDispositionInformation on this stream) -- separate from + * the inode-wide flag above, which only ever meant "some + * stream on this file" with no way to say which one. + */ + spin_lock(&fp->f_lock); + if (fp->stream_del_pending) { + fp->stream_del_pending = false; + remove_stream_xattr = true; + } + spin_unlock(&fp->f_lock); + if (remove_stream_xattr) { const struct cred *saved_cred; @@ -497,6 +617,7 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp) { struct file *filp; struct ksmbd_lock *smb_lock, *tmp_lock; + struct ksmbd_work *cn_work; fd_limit_close(); ksmbd_remove_durable_fd(fp); @@ -510,8 +631,9 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp) if (!IS_ERR_OR_NULL(filp)) fput(filp); - /* because the reference count of fp is 0, it is guaranteed that - * there are not accesses to fp->lock_list. + /* + * The zero fp reference count serializes access to fp->lock_list, but + * the VFS may still have blocked requests chained below these locks. */ list_for_each_entry_safe(smb_lock, tmp_lock, &fp->lock_list, flist) { struct ksmbd_conn *conn = smb_lock->conn; @@ -524,12 +646,59 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp) ksmbd_conn_put(conn); } - list_del(&smb_lock->flist); + list_del_init(&smb_lock->flist); + ksmbd_vfs_posix_lock_unblock(smb_lock->fl); locks_free_lock(smb_lock->fl); kfree(smb_lock); } /* + * Complete any CHANGE_NOTIFY left pending on this handle now that + * it is closed. KSMBD never completes CHANGE_NOTIFY spontaneously + * (no real change-notification backend), only on close -- matching + * genuine SMB2/macOS smbfs semantics and avoiding the Finder + * "directory changed, re-enumerate everything" loop. + * + * smb2_notify() on another connection can be adding to + * notify_pendings under fp->f_lock at the same time this handle is + * closed, and a client-sent CANCEL can concurrently be racing to + * claim the same entry via smb2_notify_cancel_fn() (smb2pdu.c). + * Pop one entry at a time under the lock via list_del_init() rather + * than a bulk list_splice_init(): list_del_init() leaves the node + * self-linked ("empty"), which is what the cancel path checks under + * the same lock to tell whether it lost the race -- a bulk splice + * would instead relink every entry into a shared local list, so an + * entry claimed here would still read as "not empty" to a racing + * cancel_fn, and both sides could end up freeing the same work. + * ksmbd_conn_write() can sleep (it takes conn's write mutex), so it + * must not be called while fp->f_lock is held -- release the lock + * before processing each popped entry, then reacquire it for the + * next. + */ + for (;;) { + spin_lock(&fp->f_lock); + if (list_empty(&fp->notify_pendings)) { + spin_unlock(&fp->f_lock); + break; + } + cn_work = list_first_entry(&fp->notify_pendings, + struct ksmbd_work, notify_entry); + list_del_init(&cn_work->notify_entry); + spin_unlock(&fp->f_lock); + + ksmbd_conn_write(cn_work); + /* + * release_async_work() removes cn_work from + * conn->async_requests, frees cancel_argv, and releases+zeroes + * async_id -- all needed before ksmbd_free_work_struct(), which + * only releases async_id itself if still nonzero (i.e. if this + * hadn't already been done). + */ + release_async_work(cn_work); + ksmbd_free_work_struct(cn_work); + } + + /* * Drop fp's strong reference on conn (taken in ksmbd_open_fd() / * ksmbd_reopen_durable_fd()). Durable fps that reached the * scavenger have already had fp->conn cleared by session_fd_check(), @@ -629,6 +798,11 @@ static struct ksmbd_file *ksmbd_fp_get(struct ksmbd_file *fp) return fp; } +struct ksmbd_file *ksmbd_file_get(struct ksmbd_file *fp) +{ + return ksmbd_fp_get(fp); +} + static struct ksmbd_file *__ksmbd_lookup_fd(struct ksmbd_file_table *ft, u64 id) { @@ -822,16 +996,15 @@ bool ksmbd_has_other_active_fd(struct ksmbd_file *fp) return ret; } -static struct ksmbd_file *ksmbd_lookup_fd_app_instance_id(char *app_instance_id) +struct ksmbd_file *ksmbd_lookup_fd_app_instance_id(char *app_instance_id) { struct ksmbd_file *fp = NULL; unsigned int id; - if (!memchr_inv(app_instance_id, 0, SMB2_CREATE_GUID_SIZE)) - return NULL; - read_lock(&global_ft.lock); idr_for_each_entry(global_ft.idr, fp, id) { + if (!fp->has_app_instance_id) + continue; if (!memcmp(fp->app_instance_id, app_instance_id, SMB2_CREATE_GUID_SIZE)) { fp = ksmbd_fp_get(fp); @@ -966,9 +1139,37 @@ struct ksmbd_file *ksmbd_lookup_fd_inode(struct dentry *dentry) return NULL; } -bool ksmbd_has_open_files(struct dentry *dentry) +bool ksmbd_has_other_nonposix_open(struct dentry *dentry) { struct ksmbd_file *fp; + struct inode *inode = d_inode(dentry); + unsigned int id; + bool ret = false; + + if (!inode) + return false; + + read_lock(&global_ft.lock); + idr_for_each_entry(global_ft.idr, fp, id) { + if (READ_ONCE(fp->f_state) != FP_INITED) + continue; + if (inode != file_inode(fp->filp)) + continue; + if (fp->is_posix_ctxt) + continue; + + ret = true; + break; + } + read_unlock(&global_ft.lock); + + return ret; +} + +bool ksmbd_has_nonposix_open_child(struct ksmbd_file *old_fp) +{ + struct dentry *dentry = old_fp->filp->f_path.dentry; + struct ksmbd_file *fp; unsigned int id; bool ret = false; @@ -980,6 +1181,8 @@ bool ksmbd_has_open_files(struct dentry *dentry) continue; if (fp_dentry == dentry) continue; + if (old_fp->is_posix_ctxt && fp->is_posix_ctxt) + continue; if (is_subdir(fp_dentry, dentry)) { ret = true; break; @@ -1050,6 +1253,7 @@ struct ksmbd_file *ksmbd_open_fd(struct ksmbd_work *work, struct file *filp) INIT_LIST_HEAD(&fp->blocked_works); INIT_LIST_HEAD(&fp->node); INIT_LIST_HEAD(&fp->lock_list); + INIT_LIST_HEAD(&fp->notify_pendings); spin_lock_init(&fp->f_lock); mutex_init(&fp->readdir_lock); atomic_set(&fp->refcount, 1); @@ -1446,9 +1650,12 @@ void ksmbd_launch_ksmbd_durable_scavenger(void) server_conf.dh_task = kthread_run(ksmbd_durable_scavenger, (void *)NULL, "ksmbd-durable-scavenger"); - if (IS_ERR(server_conf.dh_task)) + if (IS_ERR(server_conf.dh_task)) { pr_err("cannot start conn thread, err : %ld\n", PTR_ERR(server_conf.dh_task)); + server_conf.dh_task = NULL; + durable_scavenger_running = false; + } mutex_unlock(&durable_scavenger_lock); } @@ -1471,7 +1678,7 @@ void ksmbd_stop_durable_scavenger(void) } /* - * ksmbd_vfs_copy_durable_owner - Copy owner info for durable reconnect + * ksmbd_vfs_set_durable_owner - Store owner info for durable replay/reconnect * @fp: ksmbd file pointer to store owner info * @user: user pointer to copy from * @@ -1480,10 +1687,10 @@ void ksmbd_stop_durable_scavenger(void) * * Return: 0 on success, or negative error code on failure */ -static int ksmbd_vfs_copy_durable_owner(struct ksmbd_file *fp, - struct ksmbd_user *user) +int ksmbd_vfs_set_durable_owner(struct ksmbd_file *fp, + struct ksmbd_user *user) { - char *name; + char *name, *old_name; if (!user) return -EINVAL; @@ -1494,10 +1701,12 @@ static int ksmbd_vfs_copy_durable_owner(struct ksmbd_file *fp, return -ENOMEM; spin_lock(&fp->f_lock); + old_name = fp->owner.name; fp->owner.uid = user->uid; fp->owner.gid = user->gid; fp->owner.name = name; spin_unlock(&fp->f_lock); + kfree(old_name); return 0; } @@ -1552,7 +1761,7 @@ static bool session_fd_check(struct ksmbd_tree_connect *tcon, if (WARN_ON_ONCE(!fp->conn)) return false; - if (ksmbd_vfs_copy_durable_owner(fp, user)) + if (ksmbd_vfs_set_durable_owner(fp, user)) return false; /* @@ -1620,7 +1829,8 @@ void ksmbd_close_session_fds(struct ksmbd_work *work) int ksmbd_init_global_file_table(void) { - create_proc_files(); + if (create_proc_files()) + pr_warn("Unable to create files procfs entry\n"); return ksmbd_init_file_table(&global_ft); } @@ -1673,7 +1883,7 @@ int ksmbd_reopen_durable_fd(struct ksmbd_work *work, struct ksmbd_file *fp) unsigned int old_f_state; write_lock(&global_ft.lock); - if (!fp->is_durable || fp->conn || fp->tcon) { + if ((!fp->is_durable && !fp->is_persistent) || fp->conn || fp->tcon) { write_unlock(&global_ft.lock); pr_err("Invalid durable fd [%p:%p]\n", fp->conn, fp->tcon); return -EBADF; @@ -1722,7 +1932,7 @@ int ksmbd_reopen_durable_fd(struct ksmbd_work *work, struct ksmbd_file *fp) down_write(&ci->m_lock); list_for_each_entry_rcu(op, &ci->m_op_list, op_entry, lockdep_is_held(&ci->m_lock)) { - if (op->conn) + if (op->conn || op->o_fp != fp) continue; op->conn = ksmbd_conn_get(fp->conn); op->sess = work->sess; diff --git a/fs/smb/server/vfs_cache.h b/fs/smb/server/vfs_cache.h index b9e27307a26c..502efb16f05f 100644 --- a/fs/smb/server/vfs_cache.h +++ b/fs/smb/server/vfs_cache.h @@ -82,6 +82,13 @@ struct durable_owner { char *name; }; +#define KSMBD_LOCK_SEQ_ARRAY_SIZE 64 + +struct ksmbd_lock_sequence { + bool valid; + u8 sequence; +}; + struct ksmbd_file { struct file *filp; u64 persistent_id; @@ -101,6 +108,7 @@ struct ksmbd_file { __le32 saccess; __le32 coption; __le32 cdoption; + __le32 create_file_attributes; __u64 create_time; __u64 change_time; __u64 allocation_size; @@ -109,6 +117,7 @@ struct ksmbd_file { bool is_nt_open; bool attrib_only; + bool allocation_size_set; char client_guid[16]; char create_guid[16]; @@ -118,9 +127,18 @@ struct ksmbd_file { struct list_head node; struct list_head blocked_works; struct list_head lock_list; + /* + * Per-handle FileDispositionInformation delete-pending state for a + * stream handle -- separate from ksmbd_inode's inode-wide m_flags, + * which have no way to record which stream on a multi-stream file + * was actually marked for deletion. See ksmbd_fd_set_delete_pending(). + */ + bool stream_del_pending; unsigned int durable_timeout; unsigned int durable_scavenger_timeout; + /* CREATE action returned when this durable handle was established. */ + __le32 create_action; /* if ls is happening on directory, below is valid*/ struct ksmbd_readdir_data readdir_data; @@ -131,10 +149,25 @@ struct ksmbd_file { bool is_durable; bool is_persistent; bool is_resilient; + bool has_app_instance_id; + bool app_instance_version_valid; + u64 app_instance_version_high; + u64 app_instance_version_low; bool durable_reconnect_disabled; + bool durable_replay_consumed; bool is_posix_ctxt; struct durable_owner owner; + __le16 channel_sequence; + unsigned int outstanding_requests; + unsigned int outstanding_pre_requests; + struct ksmbd_lock_sequence lock_seq[KSMBD_LOCK_SEQ_ARRAY_SIZE]; + + /* + * Pending CHANGE_NOTIFY completions for this handle, sent with + * STATUS_NOTIFY_CLEANUP when the handle is closed. + */ + struct list_head notify_pendings; }; static inline void set_ctx_actor(struct dir_context *ctx, @@ -167,6 +200,9 @@ struct ksmbd_file *ksmbd_lookup_fd_fast(struct ksmbd_work *work, u64 id); struct ksmbd_file *ksmbd_lookup_foreign_fd(struct ksmbd_work *work, u64 id); struct ksmbd_file *ksmbd_lookup_fd_slow(struct ksmbd_work *work, u64 id, u64 pid); +int ksmbd_vfs_set_durable_owner(struct ksmbd_file *fp, + struct ksmbd_user *user); +struct ksmbd_file *ksmbd_file_get(struct ksmbd_file *fp); void ksmbd_fd_put(struct ksmbd_work *work, struct ksmbd_file *fp); struct ksmbd_inode *ksmbd_inode_lookup_lock(struct dentry *d); void ksmbd_inode_put(struct ksmbd_inode *ci); @@ -177,14 +213,17 @@ void ksmbd_put_durable_fd(struct ksmbd_file *fp); int ksmbd_invalidate_durable_fd(unsigned long long id); bool ksmbd_has_other_active_fd(struct ksmbd_file *fp); bool ksmbd_has_stream_without_delete_share(struct ksmbd_file *fp); +struct ksmbd_file *ksmbd_lookup_fd_app_instance_id(char *app_instance_id); int ksmbd_close_fd_app_instance_id(char *app_instance_id); struct ksmbd_file *ksmbd_lookup_fd_cguid(char *cguid); struct ksmbd_file *ksmbd_lookup_fd_inode(struct dentry *dentry); -bool ksmbd_has_open_files(struct dentry *dentry); +bool ksmbd_has_other_nonposix_open(struct dentry *dentry); +bool ksmbd_has_nonposix_open_child(struct ksmbd_file *old_fp); unsigned int ksmbd_open_durable_fd(struct ksmbd_file *fp); struct ksmbd_file *ksmbd_open_fd(struct ksmbd_work *work, struct file *filp); void ksmbd_launch_ksmbd_durable_scavenger(void); void ksmbd_stop_durable_scavenger(void); +bool ksmbd_durable_scavenger_active(void); void ksmbd_close_tree_conn_fds(struct ksmbd_work *work); void ksmbd_close_session_fds(struct ksmbd_work *work); int ksmbd_close_inode_fds(struct ksmbd_work *work, struct inode *inode); @@ -214,6 +253,8 @@ void ksmbd_set_inode_pending_delete(struct ksmbd_file *fp); void ksmbd_clear_inode_pending_delete(struct ksmbd_file *fp); void ksmbd_fd_set_delete_on_close(struct ksmbd_file *fp, int file_info); +void ksmbd_fd_set_delete_pending(struct ksmbd_file *fp); +void ksmbd_fd_clear_delete_pending(struct ksmbd_file *fp); int ksmbd_reopen_durable_fd(struct ksmbd_work *work, struct ksmbd_file *fp); int ksmbd_validate_name_reconnect(struct ksmbd_share_config *share, struct ksmbd_file *fp, char *name); diff --git a/fs/smb/smbdirect/accept.c b/fs/smb/smbdirect/accept.c index 529740005838..039c9bfbd8ac 100644 --- a/fs/smb/smbdirect/accept.c +++ b/fs/smb/smbdirect/accept.c @@ -145,11 +145,21 @@ int smbdirect_accept_connect_request(struct smbdirect_socket *sc, rdma_accept_failed: /* - * smbdirect_connection_destroy_qp() calls ib_drain_qp(), - * so that smbdirect_accept_negotiate_recv_done() will - * call smbdirect_connection_put_recv_io() + * The recv_io posted above is now owned by the QP (recv_io was set to + * NULL after a successful post). smbdirect_connection_destroy_qp() + * calls ib_drain_qp(), whose completion + * (smbdirect_accept_negotiate_recv_done) returns the recv_io to the + * free list via smbdirect_connection_put_recv_io(). It therefore MUST + * run BEFORE smbdirect_connection_destroy_mem_pools(): otherwise the + * posted recv_io is still outstanding when kmem_cache_destroy() runs + * ("Slab cache still has objects") and is later freed into an + * already-destroyed mempool (mempool_free_bulk NULL-ptr-deref). */ + smbdirect_connection_destroy_qp(sc); + smbdirect_connection_destroy_mem_pools(sc); + return ret; post_recv_io_failed: + /* post failed: recv_io was not accepted by the QP, still in hand */ if (recv_io) smbdirect_connection_put_recv_io(recv_io); get_recv_io_failed: diff --git a/fs/smb/smbdirect/connection.c b/fs/smb/smbdirect/connection.c index 8adf58097534..fe9953720717 100644 --- a/fs/smb/smbdirect/connection.c +++ b/fs/smb/smbdirect/connection.c @@ -403,12 +403,21 @@ void smbdirect_connection_destroy_qp(struct smbdirect_socket *sc) sc->ib.qp = NULL; rdma_destroy_qp(sc->rdma.cm_id); } + /* + * These CQs were created with ib_alloc_cq_any(), which arms an internal + * completion handler (ib_cq_poll_work for IB_POLL_WORKQUEUE). They MUST be + * torn down with ib_free_cq(), which cancel_work_sync()es that poll work + * before freeing the CQ. ib_destroy_cq() skips that step, so a completion + * posted late by the (software) provider — e.g. rxe posting an RNR error + * from rxe_receiver after rdma_destroy_qp() — re-queues ib_cq_poll_work on + * an already-freed CQ (KASAN slab-use-after-free in ib_cq_poll_work). + */ if (sc->ib.recv_cq) { - ib_destroy_cq(sc->ib.recv_cq); + ib_free_cq(sc->ib.recv_cq); sc->ib.recv_cq = NULL; } if (sc->ib.send_cq) { - ib_destroy_cq(sc->ib.send_cq); + ib_free_cq(sc->ib.send_cq); sc->ib.send_cq = NULL; } if (sc->ib.pd) { diff --git a/fs/smb/smbdirect/socket.c b/fs/smb/smbdirect/socket.c index 39cca7219c4d..bb02df6158b9 100644 --- a/fs/smb/smbdirect/socket.c +++ b/fs/smb/smbdirect/socket.c @@ -305,12 +305,26 @@ void __smbdirect_socket_schedule_cleanup(struct smbdirect_socket *sc, * disconnect all pending and ready sockets * * First we move ready sockets to pending again. + * + * Only a socket that was a listener (listen.backlog != -1) owns a + * populated listen.ready/pending list. Guarding on that also keeps + * lockdep quiet: without it, the listener holds sc->listen.lock while + * the loop recurses into each child psc, which takes psc->listen.lock. + * Those are always different instances of the same lock class -- a + * child never listens, so the nesting is strictly listener -> child + * and cannot really deadlock -- but lockdep only sees the class and + * reports "possible recursive locking". A child has empty listen + * lists and nothing to do here, so skipping it loses nothing, and a + * pending child stays on its listener's list for the free path + * (smbdirect_socket_destroy) to reap. */ - spin_lock_irqsave(&sc->listen.lock, flags); - list_splice_init(&sc->listen.ready, &sc->listen.pending); - list_for_each_entry_safe(psc, tsc, &sc->listen.pending, accept.list) - smbdirect_socket_schedule_cleanup(psc, sc->first_error); - spin_unlock_irqrestore(&sc->listen.lock, flags); + if (sc->listen.backlog != -1) { /* was a listener */ + spin_lock_irqsave(&sc->listen.lock, flags); + list_splice_init(&sc->listen.ready, &sc->listen.pending); + list_for_each_entry_safe(psc, tsc, &sc->listen.pending, accept.list) + smbdirect_socket_schedule_cleanup(psc, sc->first_error); + spin_unlock_irqrestore(&sc->listen.lock, flags); + } switch (sc->status) { case SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED: @@ -405,12 +419,20 @@ static void smbdirect_socket_cleanup_work(struct work_struct *work) * disconnect all pending and ready sockets * * First we move ready sockets to pending again. + * + * Guarded on listen.backlog != -1 for the same reason as in + * __smbdirect_socket_schedule_cleanup(): only a listener owns a + * populated listen list, and skipping the block for a child avoids + * nesting psc->listen.lock under a listener's listen.lock (different + * instances of one class -- harmless, but lockdep cannot tell). */ - spin_lock_irqsave(&sc->listen.lock, flags); - list_splice_init(&sc->listen.ready, &sc->listen.pending); - list_for_each_entry_safe(psc, tsc, &sc->listen.pending, accept.list) - smbdirect_socket_schedule_cleanup(psc, sc->first_error); - spin_unlock_irqrestore(&sc->listen.lock, flags); + if (sc->listen.backlog != -1) { /* was a listener */ + spin_lock_irqsave(&sc->listen.lock, flags); + list_splice_init(&sc->listen.ready, &sc->listen.pending); + list_for_each_entry_safe(psc, tsc, &sc->listen.pending, accept.list) + smbdirect_socket_schedule_cleanup(psc, sc->first_error); + spin_unlock_irqrestore(&sc->listen.lock, flags); + } switch (sc->status) { case SMBDIRECT_SOCKET_NEGOTIATE_NEEDED: @@ -473,6 +495,7 @@ static void smbdirect_socket_destroy(struct smbdirect_socket *sc) struct smbdirect_recv_io *recv_io; struct smbdirect_recv_io *recv_tmp; LIST_HEAD(all_list); + LIST_HEAD(pending_list); unsigned long flags; smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, @@ -530,24 +553,29 @@ static void smbdirect_socket_destroy(struct smbdirect_socket *sc) * disconnect all pending and ready sockets * * We move ready sockets to pending again. + * + * Capture them here -- rdma_lock_handler(sc->rdma.cm_id) is held above, + * so a concurrent CM CONNECT_REQUEST cannot add more; sc->listen.lock + * below only protects the list splice itself -- but DEFER releasing + * them until the listener's cm_id is destroyed: + * + * - smbdirect_socket_release() -> smbdirect_socket_destroy() takes the + * child's own rdma_lock_handler() lock (&id_priv->handler_mutex). + * The listener's and the child's cm_id are always different + * instances, so the nesting cannot really deadlock, but lockdep only + * sees one lock class and reports "possible recursive locking". + * + * - rdma_destroy_id() of a child before the listener's own + * rdma_destroy_id() below lets _cma_cancel_listens() walk the freed + * child id_priv (KASAN slab-use-after-free in __mutex_lock()). + * + * The children are independent sockets whose teardown does not need + * the listener's handler lock. */ spin_lock_irqsave(&sc->listen.lock, flags); - list_splice_tail_init(&sc->listen.ready, &all_list); - list_splice_tail_init(&sc->listen.pending, &all_list); + list_splice_tail_init(&sc->listen.ready, &pending_list); + list_splice_tail_init(&sc->listen.pending, &pending_list); spin_unlock_irqrestore(&sc->listen.lock, flags); - psockets = list_count_nodes(&all_list); - if (sc->listen.backlog != -1) /* was a listener */ - smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, - "release %zu pending sockets\n", psockets); - list_for_each_entry_safe(psc, tsc, &all_list, accept.list) { - list_del_init(&psc->accept.list); - psc->accept.listener = NULL; - smbdirect_socket_release(psc); - } - if (sc->listen.backlog != -1) /* was a listener */ - smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, - "released %zu pending sockets\n", psockets); - INIT_LIST_HEAD(&all_list); /* It's not possible for upper layer to get to reassembly */ if (sc->listen.backlog == -1) /* was not a listener */ @@ -577,6 +605,26 @@ static void smbdirect_socket_destroy(struct smbdirect_socket *sc) sc->rdma.cm_id = NULL; } + /* + * The listener's rdma_lock_handler() lock is dropped and its cm_id is + * destroyed, so it is safe to release the child sockets captured + * above: each release recurses into smbdirect_socket_destroy() and + * takes that child's own handler_mutex without nesting it under the + * listener's, and _cma_cancel_listens() can no longer reach them. + */ + psockets = list_count_nodes(&pending_list); + if (sc->listen.backlog != -1) /* was a listener */ + smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, + "release %zu pending sockets\n", psockets); + list_for_each_entry_safe(psc, tsc, &pending_list, accept.list) { + list_del_init(&psc->accept.list); + psc->accept.listener = NULL; + smbdirect_socket_release(psc); + } + if (sc->listen.backlog != -1) /* was a listener */ + smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, + "released %zu pending sockets\n", psockets); + if (sc->listen.backlog == -1) /* was not a listener */ smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, "destroying mem pools\n"); diff --git a/fs/ubifs/auth.c b/fs/ubifs/auth.c index a4a0158f712d..1f770795ee70 100644 --- a/fs/ubifs/auth.c +++ b/fs/ubifs/auth.c @@ -217,7 +217,7 @@ int ubifs_sb_verify_signature(struct ubifs_info *c, signode = snod->node; - if (le32_to_cpu(signode->len) > snod->len + sizeof(struct ubifs_sig_node)) { + if (le32_to_cpu(signode->len) > snod->len - sizeof(struct ubifs_sig_node)) { ubifs_err(c, "invalid signature len %d", le32_to_cpu(signode->len)); err = -EINVAL; goto out_destroy; diff --git a/fs/ubifs/debug.h b/fs/ubifs/debug.h index d425861e6b82..48cd64eb2d01 100644 --- a/fs/ubifs/debug.h +++ b/fs/ubifs/debug.h @@ -26,7 +26,7 @@ typedef int (*dbg_znode_callback)(struct ubifs_info *c, #define UBIFS_DFS_DIR_LEN (3 + 1 + 2 + 3 + 1) /** - * ubifs_debug_info - per-FS debugging information. + * struct ubifs_debug_info - per-FS debugging information. * @old_zroot: old index root - used by 'dbg_check_old_index()' * @old_zroot_level: old index root level - used by 'dbg_check_old_index()' * @old_zroot_sqnum: old index root sqnum - used by 'dbg_check_old_index()' @@ -119,7 +119,7 @@ struct ubifs_debug_info { }; /** - * ubifs_global_debug_info - global (not per-FS) UBIFS debugging information. + * struct ubifs_global_debug_info - global (not per-FS) UBIFS debugging information. * * @chk_gen: if general extra checks are enabled * @chk_index: if index xtra checks are enabled diff --git a/fs/ubifs/key.h b/fs/ubifs/key.h index 8142d9d6fe5d..71497c1b4753 100644 --- a/fs/ubifs/key.h +++ b/fs/ubifs/key.h @@ -33,7 +33,7 @@ /** * key_mask_hash - mask a valid hash value. - * @val: value to be masked + * @hash: value to be masked * * We use hash values as offset in directories, so values %0 and %1 are * reserved for "." and "..". %2 is reserved for "end of readdir" marker. This diff --git a/fs/ubifs/ubifs.h b/fs/ubifs/ubifs.h index b62a154c7bd4..00db0d19a85e 100644 --- a/fs/ubifs/ubifs.h +++ b/fs/ubifs/ubifs.h @@ -738,6 +738,7 @@ struct ubifs_jhead { * struct ubifs_zbranch - key/coordinate/length branch stored in znodes. * @key: key * @znode: znode address in memory + * @leaf: leaf node * @lnum: LEB number of the target node (indexing node or data node) * @offs: target node offset within @lnum * @len: target node length @@ -801,7 +802,7 @@ struct ubifs_znode { * @gc_seq: GC sequence number to detect races with GC * @cnt: number of data nodes for bulk read * @blk_cnt: number of data blocks including holes - * @oef: end of file reached + * @eof: end of file reached */ struct bu_info { union ubifs_key key; @@ -985,7 +986,7 @@ struct ubifs_budg_info { }; /** - * ubifs_stats_info - per-FS statistics information. + * struct ubifs_stats_info - per-FS statistics information. * @magic_errors: number of bad magic numbers (will be reset with a new mount). * @node_errors: number of bad nodes (will be reset with a new mount). * @crc_errors: number of bad crcs (will be reset with a new mount). @@ -1051,6 +1052,7 @@ struct ubifs_debug_info; * @rw_incompat: the media is not R/W compatible * @assert_action: action to take when a ubifs_assert() fails * @authenticated: flag indigating the FS is mounted in authenticated mode + * @superblock_need_write: superblock node needs to be written * * @tnc_mutex: protects the Tree Node Cache (TNC), @zroot, @cnext, @enext, and * @calc_idx_sz @@ -1588,8 +1590,9 @@ int ubifs_prepare_auth_node(struct ubifs_info *c, void *node, * @expected: first hash * @got: second hash * - * Compare two hashes @expected and @got. Returns 0 when they are equal, a - * negative error code otherwise. + * Compare two hashes @expected and @got. + * + * Returns: 0 when they are equal, a negative error code otherwise. */ static inline int ubifs_check_hash(const struct ubifs_info *c, const u8 *expected, const u8 *got) @@ -1603,8 +1606,9 @@ static inline int ubifs_check_hash(const struct ubifs_info *c, * @expected: first HMAC * @got: second HMAC * - * Compare two hashes @expected and @got. Returns 0 when they are equal, a - * negative error code otherwise. + * Compare two hashes @expected and @got. + * + * Returns: 0 when they are equal, a negative error code otherwise. */ static inline int ubifs_check_hmac(const struct ubifs_info *c, const u8 *expected, const u8 *got) @@ -1644,7 +1648,7 @@ static inline void ubifs_exit_authentication(struct ubifs_info *c) * @c: UBIFS file-system description object * @br: branch to get the hash from * - * This returns a pointer to the hash of a branch. Since the key already is a + * Returns: a pointer to the hash of a branch. Since the key already is a * dynamically sized object we cannot use a struct member here. */ static inline u8 *ubifs_branch_hash(struct ubifs_info *c, @@ -1694,7 +1698,7 @@ static inline int ubifs_node_verify_hmac(const struct ubifs_info *c, * ubifs_auth_node_sz - returns the size of an authentication node * @c: UBIFS file-system description object * - * This function returns the size of an authentication node which can + * Returns: the size of an authentication node which can * be 0 for unauthenticated filesystems or the real size of an auth node * authentication is enabled. */ diff --git a/fs/xfs/Makefile b/fs/xfs/Makefile index 9f7133e02576..399a207f2d0e 100644 --- a/fs/xfs/Makefile +++ b/fs/xfs/Makefile @@ -91,6 +91,7 @@ xfs-y += xfs_aops.o \ xfs_healthmon.o \ xfs_icache.o \ xfs_ioctl.o \ + xfs_ioend.o \ xfs_iomap.o \ xfs_iops.o \ xfs_inode.o \ diff --git a/fs/xfs/libxfs/xfs_rtgroup.h b/fs/xfs/libxfs/xfs_rtgroup.h index c0b9f9f2c413..fca2eb74908c 100644 --- a/fs/xfs/libxfs/xfs_rtgroup.h +++ b/fs/xfs/libxfs/xfs_rtgroup.h @@ -359,7 +359,11 @@ static inline int xfs_initialize_rtgroups(struct xfs_mount *mp, # define xfs_rtgroup_unlock(rtg, gf) ((void)0) # define xfs_rtgroup_trans_join(tp, rtg, gf) ((void)0) # define xfs_update_rtsb(bp, sb_bp) ((void)0) -# define xfs_log_rtsb(tp, sb_bp) (NULL) +static inline struct xfs_buf *xfs_log_rtsb(struct xfs_trans *tp, + const struct xfs_buf *sb_bp) +{ + return NULL; +} # define xfs_rtgroup_get_geometry(rtg, rgeo) (-EOPNOTSUPP) #endif /* CONFIG_XFS_RT */ diff --git a/fs/xfs/libxfs/xfs_sb.c b/fs/xfs/libxfs/xfs_sb.c index 75f2a021ee6d..f0341adbb879 100644 --- a/fs/xfs/libxfs/xfs_sb.c +++ b/fs/xfs/libxfs/xfs_sb.c @@ -1470,36 +1470,33 @@ xfs_sync_sb_buf( bool update_rtsb) { struct xfs_trans *tp; - struct xfs_buf *bp; - struct xfs_buf *rtsb_bp = NULL; int error; error = xfs_trans_alloc(mp, &M_RES(mp)->tr_sb, 0, 0, 0, &tp); if (error) return error; - bp = xfs_trans_getsb(tp); xfs_log_sb(tp); - xfs_trans_bhold(tp, bp); - if (update_rtsb) { - rtsb_bp = xfs_log_rtsb(tp, bp); - if (rtsb_bp) - xfs_trans_bhold(tp, rtsb_bp); - } + if (update_rtsb) + xfs_log_rtsb(tp, xfs_trans_getsb(tp)); xfs_trans_set_sync(tp); error = xfs_trans_commit(tp); if (error) - goto out; - /* - * write out the sb buffer to get the changes to disk - */ - error = xfs_bwrite(bp); - if (!error && rtsb_bp) - error = xfs_bwrite(rtsb_bp); -out: - if (rtsb_bp) - xfs_buf_relse(rtsb_bp); - xfs_buf_relse(bp); + return error; + + /* Re-acquire and write the sb and rtsb to disk. */ + xfs_buf_lock(mp->m_sb_bp); + error = xfs_bwrite(mp->m_sb_bp); + xfs_buf_unlock(mp->m_sb_bp); + if (error) + return error; + + if (update_rtsb && mp->m_rtsb_bp) { + xfs_buf_lock(mp->m_rtsb_bp); + error = xfs_bwrite(mp->m_rtsb_bp); + xfs_buf_unlock(mp->m_rtsb_bp); + } + return error; } diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 74a6089abadf..8b6119776fb3 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -20,6 +20,7 @@ #include "xfs_errortag.h" #include "xfs_error.h" #include "xfs_icache.h" +#include "xfs_ioend.h" #include "xfs_zone_alloc.h" #include "xfs_rtgroup.h" #include <linux/bio-integrity.h> @@ -37,15 +38,6 @@ XFS_WPC(struct iomap_writepage_ctx *ctx) } /* - * Fast and loose check if this write could update the on-disk inode size. - */ -static inline bool xfs_ioend_is_append(struct iomap_ioend *ioend) -{ - return ioend->io_offset + ioend->io_size > - XFS_I(ioend->io_inode)->i_disk_size; -} - -/* * Update on-disk file size now that data has been written to disk. */ int @@ -80,175 +72,6 @@ xfs_setfilesize( return xfs_trans_commit(tp); } -static void -xfs_ioend_put_open_zones( - struct iomap_ioend *ioend) -{ - struct iomap_ioend *tmp; - - /* - * Put the open zone for all ioends merged into this one (if any). - */ - list_for_each_entry(tmp, &ioend->io_list, io_list) - xfs_open_zone_put(tmp->io_private); - - /* - * The main ioend might not have an open zone if the submission failed - * before xfs_zone_alloc_and_submit got called. - */ - if (ioend->io_private) - xfs_open_zone_put(ioend->io_private); -} - -/* - * IO write completion. - */ -STATIC void -xfs_end_ioend_write( - struct iomap_ioend *ioend) -{ - struct xfs_inode *ip = XFS_I(ioend->io_inode); - struct xfs_mount *mp = ip->i_mount; - bool is_zoned = xfs_is_zoned_inode(ip); - xfs_off_t offset = ioend->io_offset; - size_t size = ioend->io_size; - unsigned int nofs_flag; - int error; - - /* - * We can allocate memory here while doing writeback on behalf of - * memory reclaim. To avoid memory allocation deadlocks set the - * task-wide nofs context for the following operations. - */ - nofs_flag = memalloc_nofs_save(); - - /* - * Just clean up the in-memory structures if the fs has been shut down. - */ - if (xfs_is_shutdown(mp)) { - error = -EIO; - goto done; - } - - /* - * Clean up all COW blocks and underlying data fork delalloc blocks on - * I/O error. The delalloc punch is required because this ioend was - * mapped to blocks in the COW fork and the associated pages are no - * longer dirty. If we don't remove delalloc blocks here, they become - * stale and can corrupt free space accounting on unmount. - */ - error = blk_status_to_errno(ioend->io_bio.bi_status); - if (unlikely(error)) { - /* - * Zoned writes update the in-core open zone accounting before - * I/O submission. A failed write leaves that state - * inconsistent, so shut down the filesystem instead of letting - * later writers wait forever for open zone space to become - * available. - */ - if (is_zoned) { - xfs_force_shutdown(mp, SHUTDOWN_META_IO_ERROR); - goto done; - } - if (ioend->io_flags & IOMAP_IOEND_SHARED) { - ASSERT(!is_zoned); - xfs_reflink_cancel_cow_range(ip, offset, size, true); - xfs_bmap_punch_delalloc_range(ip, XFS_DATA_FORK, offset, - offset + size, NULL); - } - goto done; - } - - /* - * Success: commit the COW or unwritten blocks if needed. - */ - if (is_zoned) - error = xfs_zoned_end_io(ip, offset, size, ioend->io_sector, - ioend->io_private, NULLFSBLOCK); - else if (ioend->io_flags & IOMAP_IOEND_SHARED) - error = xfs_reflink_end_cow(ip, offset, size); - else if (ioend->io_flags & IOMAP_IOEND_UNWRITTEN) - error = xfs_iomap_write_unwritten(ip, offset, size, false); - - if (!error && - !(ioend->io_flags & IOMAP_IOEND_DIRECT) && - xfs_ioend_is_append(ioend)) - error = xfs_setfilesize(ip, offset, size); -done: - if (is_zoned) - xfs_ioend_put_open_zones(ioend); - iomap_finish_ioends(ioend, error); - memalloc_nofs_restore(nofs_flag); -} - -/* - * Finish all pending IO completions that require transactional modifications. - * - * We try to merge physical and logically contiguous ioends before completion to - * minimise the number of transactions we need to perform during IO completion. - * Both unwritten extent conversion and COW remapping need to iterate and modify - * one physical extent at a time, so we gain nothing by merging physically - * discontiguous extents here. - * - * The ioend chain length that we can be processing here is largely unbound in - * length and we may have to perform significant amounts of work on each ioend - * to complete it. Hence we have to be careful about holding the CPU for too - * long in this loop. - */ -void -xfs_end_io( - struct work_struct *work) -{ - struct xfs_inode *ip = - container_of(work, struct xfs_inode, i_ioend_work); - struct iomap_ioend *ioend; - struct list_head tmp; - unsigned long flags; - - spin_lock_irqsave(&ip->i_ioend_lock, flags); - list_replace_init(&ip->i_ioend_list, &tmp); - spin_unlock_irqrestore(&ip->i_ioend_lock, flags); - - iomap_sort_ioends(&tmp); - while ((ioend = list_first_entry_or_null(&tmp, struct iomap_ioend, - io_list))) { - list_del_init(&ioend->io_list); - iomap_ioend_try_merge(ioend, &tmp); - if (bio_op(&ioend->io_bio) == REQ_OP_READ) - iomap_finish_ioends(ioend, - blk_status_to_errno(ioend->io_bio.bi_status)); - else - xfs_end_ioend_write(ioend); - cond_resched(); - } -} - -void -xfs_end_bio( - struct bio *bio) -{ - struct iomap_ioend *ioend = iomap_ioend_from_bio(bio); - struct xfs_inode *ip = XFS_I(ioend->io_inode); - struct xfs_mount *mp = ip->i_mount; - unsigned long flags; - - /* - * For Appends record the actually written block number and set the - * boundary flag if needed. - */ - if (IS_ENABLED(CONFIG_XFS_RT) && bio_is_zone_append(bio)) { - ioend->io_sector = bio->bi_iter.bi_sector; - xfs_mark_rtg_boundary(ioend); - } - - spin_lock_irqsave(&ip->i_ioend_lock, flags); - if (list_empty(&ip->i_ioend_list)) - WARN_ON_ONCE(!queue_work(mp->m_unwritten_workqueue, - &ip->i_ioend_work)); - list_add_tail(&ioend->io_list, &ip->i_ioend_list); - spin_unlock_irqrestore(&ip->i_ioend_lock, flags); -} - /* * We cannot cancel the ioend directly on error. We may have already set other * pages under writeback and hence we have to run I/O completion to mark the @@ -631,13 +454,8 @@ xfs_zoned_map_blocks( XFS_BMAPI_REMAP); xfs_iunlock(ip, XFS_ILOCK_EXCL); - wpc->iomap.type = IOMAP_MAPPED; - wpc->iomap.flags = IOMAP_F_DIRTY; - wpc->iomap.bdev = mp->m_rtdev_targp->bt_bdev; - wpc->iomap.offset = offset; - wpc->iomap.length = XFS_FSB_TO_B(mp, count_fsb); - wpc->iomap.flags = IOMAP_F_ANON_WRITE; - + xfs_iomap_set_anon_write(ip, &wpc->iomap, offset, + XFS_FSB_TO_B(mp, count_fsb)); trace_xfs_zoned_map_blocks(ip, offset, wpc->iomap.length); return 0; } diff --git a/fs/xfs/xfs_aops.h b/fs/xfs/xfs_aops.h index 5a7a0f1a0b49..d5ae5c9d4c26 100644 --- a/fs/xfs/xfs_aops.h +++ b/fs/xfs/xfs_aops.h @@ -10,6 +10,5 @@ extern const struct address_space_operations xfs_address_space_operations; extern const struct address_space_operations xfs_dax_aops; int xfs_setfilesize(struct xfs_inode *ip, xfs_off_t offset, size_t size); -void xfs_end_bio(struct bio *bio); #endif /* __XFS_AOPS_H__ */ diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 7bff07e31cbd..426a67b813a7 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -25,7 +25,7 @@ #include "xfs_iomap.h" #include "xfs_reflink.h" #include "xfs_file.h" -#include "xfs_aops.h" +#include "xfs_ioend.h" #include "xfs_zone_alloc.h" #include "xfs_error.h" #include "xfs_errortag.h" diff --git a/fs/xfs/xfs_fsmap.c b/fs/xfs/xfs_fsmap.c index b6a3bc9f143c..041bb2105ec6 100644 --- a/fs/xfs/xfs_fsmap.c +++ b/fs/xfs/xfs_fsmap.c @@ -1174,8 +1174,7 @@ xfs_getfsmap( if (!xfs_getfsmap_check_keys(&head->fmh_keys[0], &head->fmh_keys[1])) return -EINVAL; - use_rmap = xfs_has_rmapbt(mp) && - has_capability_noaudit(current, CAP_SYS_ADMIN); + use_rmap = xfs_has_rmapbt(mp) && capable_noaudit(CAP_SYS_ADMIN); head->fmh_entries = 0; /* Set up our device handlers. */ diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index 9d8dd30bd927..a857b8aa255c 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -82,24 +82,20 @@ static inline xa_mark_t ici_tag_to_mark(unsigned int tag) /* * Allocate and initialise an xfs_inode. + * + * This can happen in context of already dirtied transactions, so the memory + * allocations must not fail. */ struct xfs_inode * xfs_inode_alloc( struct xfs_mount *mp, xfs_ino_t ino) { + gfp_t gfp = GFP_KERNEL | __GFP_NOFAIL; struct xfs_inode *ip; - /* - * XXX: If this didn't occur in transactions, we could drop GFP_NOFAIL - * and return NULL here on ENOMEM. - */ - ip = alloc_inode_sb(mp->m_super, xfs_inode_cache, GFP_KERNEL | __GFP_NOFAIL); - - if (inode_init_always(mp->m_super, VFS_I(ip))) { - kmem_cache_free(xfs_inode_cache, ip); - return NULL; - } + ip = alloc_inode_sb(mp->m_super, xfs_inode_cache, gfp); + inode_init_always_gfp(mp->m_super, VFS_I(ip), gfp); VFS_I(ip)->i_ino = ino; /* VFS doesn't initialise i_mode! */ diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c index 1b53701bebea..96ca3e480cb9 100644 --- a/fs/xfs/xfs_ioctl.c +++ b/fs/xfs/xfs_ioctl.c @@ -647,7 +647,7 @@ xfs_ioctl_setattr_get_trans( goto out_error; error = xfs_trans_alloc_ichange(ip, NULL, NULL, pdqp, - has_capability_noaudit(current, CAP_FOWNER), &tp); + capable_noaudit(CAP_FOWNER), &tp); if (error) goto out_error; diff --git a/fs/xfs/xfs_ioend.c b/fs/xfs/xfs_ioend.c new file mode 100644 index 000000000000..40695d18dac0 --- /dev/null +++ b/fs/xfs/xfs_ioend.c @@ -0,0 +1,184 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2016-2025 Christoph Hellwig. + * All Rights Reserved. + */ +#include "xfs_platform.h" +#include "xfs_shared.h" +#include "xfs_format.h" +#include "xfs_log_format.h" +#include "xfs_trans_resv.h" +#include "xfs_mount.h" +#include "xfs_inode.h" +#include "xfs_iomap.h" +#include "xfs_trace.h" +#include "xfs_bmap_util.h" +#include "xfs_reflink.h" +#include "xfs_zone_alloc.h" +#include "xfs_ioend.h" + +static void +xfs_ioend_put_open_zones( + struct iomap_ioend *ioend) +{ + struct iomap_ioend *tmp; + + /* + * Put the open zone for all ioends merged into this one (if any). + */ + list_for_each_entry(tmp, &ioend->io_list, io_list) + xfs_open_zone_put(tmp->io_private); + + /* + * The main ioend might not have an open zone if the submission failed + * before xfs_zone_alloc_and_submit got called. + */ + if (ioend->io_private) + xfs_open_zone_put(ioend->io_private); +} + +static void +xfs_end_ioend_write( + struct iomap_ioend *ioend) +{ + struct xfs_inode *ip = XFS_I(ioend->io_inode); + struct xfs_mount *mp = ip->i_mount; + bool is_zoned = xfs_is_zoned_inode(ip); + xfs_off_t offset = ioend->io_offset; + size_t size = ioend->io_size; + unsigned int nofs_flag; + int error; + + /* + * We can allocate memory here while doing writeback on behalf of + * memory reclaim. To avoid memory allocation deadlocks set the + * task-wide nofs context for the following operations. + */ + nofs_flag = memalloc_nofs_save(); + + /* + * Just clean up the in-memory structures if the fs has been shut down. + */ + if (xfs_is_shutdown(mp)) { + error = -EIO; + goto done; + } + + /* + * Clean up all COW blocks and underlying data fork delalloc blocks on + * I/O error. The delalloc punch is required because this ioend was + * mapped to blocks in the COW fork and the associated pages are no + * longer dirty. If we don't remove delalloc blocks here, they become + * stale and can corrupt free space accounting on unmount. + */ + error = blk_status_to_errno(ioend->io_bio.bi_status); + if (unlikely(error)) { + /* + * Zoned writes update the in-core open zone accounting before + * I/O submission. A failed write leaves that state + * inconsistent, so shut down the filesystem instead of letting + * later writers wait forever for open zone space to become + * available. + */ + if (is_zoned) { + xfs_force_shutdown(mp, SHUTDOWN_META_IO_ERROR); + goto done; + } + if (ioend->io_flags & IOMAP_IOEND_SHARED) { + ASSERT(!is_zoned); + xfs_reflink_cancel_cow_range(ip, offset, size, true); + xfs_bmap_punch_delalloc_range(ip, XFS_DATA_FORK, offset, + offset + size, NULL); + } + goto done; + } + + /* + * Success: commit the COW or unwritten blocks if needed. + */ + if (is_zoned) + error = xfs_zoned_end_io(ip, offset, size, ioend->io_sector, + ioend->io_private, NULLFSBLOCK); + else if (ioend->io_flags & IOMAP_IOEND_SHARED) + error = xfs_reflink_end_cow(ip, offset, size); + else if (ioend->io_flags & IOMAP_IOEND_UNWRITTEN) + error = xfs_iomap_write_unwritten(ip, offset, size, false); + + if (!error && + !(ioend->io_flags & IOMAP_IOEND_DIRECT) && + xfs_ioend_is_append(ioend)) + error = xfs_setfilesize(ip, offset, size); +done: + if (is_zoned) + xfs_ioend_put_open_zones(ioend); + iomap_finish_ioends(ioend, error); + memalloc_nofs_restore(nofs_flag); +} + +/* + * Finish all pending IO completions that require transactional modifications. + * + * We try to merge physical and logically contiguous ioends before completion to + * minimise the number of transactions we need to perform during IO completion. + * Both unwritten extent conversion and COW remapping need to iterate and modify + * one physical extent at a time, so we gain nothing by merging physically + * discontiguous extents here. + * + * The ioend chain length that we can be processing here is largely unbound in + * length and we may have to perform significant amounts of work on each ioend + * to complete it. Hence we have to be careful about holding the CPU for too + * long in this loop. + */ +void +xfs_end_io( + struct work_struct *work) +{ + struct xfs_inode *ip = + container_of(work, struct xfs_inode, i_ioend_work); + struct iomap_ioend *ioend; + struct list_head tmp; + unsigned long flags; + + spin_lock_irqsave(&ip->i_ioend_lock, flags); + list_replace_init(&ip->i_ioend_list, &tmp); + spin_unlock_irqrestore(&ip->i_ioend_lock, flags); + + iomap_sort_ioends(&tmp); + while ((ioend = list_first_entry_or_null(&tmp, struct iomap_ioend, + io_list))) { + list_del_init(&ioend->io_list); + iomap_ioend_try_merge(ioend, &tmp); + if (bio_op(&ioend->io_bio) == REQ_OP_READ) + iomap_finish_ioends(ioend, + blk_status_to_errno(ioend->io_bio.bi_status)); + else + xfs_end_ioend_write(ioend); + cond_resched(); + } +} + +void +xfs_end_bio( + struct bio *bio) +{ + struct iomap_ioend *ioend = iomap_ioend_from_bio(bio); + struct xfs_inode *ip = XFS_I(ioend->io_inode); + struct xfs_mount *mp = ip->i_mount; + unsigned long flags; + + /* + * For Appends record the actually written block number and set the + * boundary flag if needed. + */ + if (IS_ENABLED(CONFIG_XFS_RT) && bio_is_zone_append(bio)) { + ioend->io_sector = bio->bi_iter.bi_sector; + xfs_mark_rtg_boundary(ioend); + } + + spin_lock_irqsave(&ip->i_ioend_lock, flags); + if (list_empty(&ip->i_ioend_list)) + WARN_ON_ONCE(!queue_work(mp->m_unwritten_workqueue, + &ip->i_ioend_work)); + list_add_tail(&ioend->io_list, &ip->i_ioend_list); + spin_unlock_irqrestore(&ip->i_ioend_lock, flags); +} diff --git a/fs/xfs/xfs_ioend.h b/fs/xfs/xfs_ioend.h new file mode 100644 index 000000000000..525865767fca --- /dev/null +++ b/fs/xfs/xfs_ioend.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __XFS_IOEND_H +#define __XFS_IOEND_H + +/* + * Fast and loose check if this write could update the on-disk inode size. + */ +static inline bool xfs_ioend_is_append(struct iomap_ioend *ioend) +{ + return ioend->io_offset + ioend->io_size > + XFS_I(ioend->io_inode)->i_disk_size; +} + +void xfs_end_bio(struct bio *bio); + +#endif /* __XFS_IOEND_H */ diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c index 71c45be8c652..7c6238fed61e 100644 --- a/fs/xfs/xfs_iomap.c +++ b/fs/xfs/xfs_iomap.c @@ -1083,12 +1083,7 @@ xfs_zoned_direct_write_iomap_begin( return error; } - iomap->type = IOMAP_MAPPED; - iomap->flags = IOMAP_F_DIRTY; - iomap->bdev = ip->i_mount->m_rtdev_targp->bt_bdev; - iomap->offset = offset; - iomap->length = length; - iomap->flags = IOMAP_F_ANON_WRITE; + xfs_iomap_set_anon_write(ip, iomap, offset, length); return 0; } diff --git a/fs/xfs/xfs_iomap.h b/fs/xfs/xfs_iomap.h index cffcec532ea6..c906c62d46f3 100644 --- a/fs/xfs/xfs_iomap.h +++ b/fs/xfs/xfs_iomap.h @@ -29,6 +29,20 @@ int xfs_zero_range(struct xfs_inode *ip, loff_t pos, loff_t len, int xfs_truncate_page(struct xfs_inode *ip, loff_t pos, struct xfs_zone_alloc_ctx *ac, bool *did_zero); +static inline void +xfs_iomap_set_anon_write( + struct xfs_inode *ip, + struct iomap *iomap, + loff_t offset, + loff_t length) +{ + iomap->type = IOMAP_MAPPED; + iomap->bdev = ip->i_mount->m_rtdev_targp->bt_bdev; + iomap->offset = offset; + iomap->length = length; + iomap->flags = IOMAP_F_ANON_WRITE | IOMAP_F_DIRTY; +} + static inline xfs_filblks_t xfs_aligned_fsb_count( xfs_fileoff_t offset_fsb, diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c index 4a3299abf774..d1306e723899 100644 --- a/fs/xfs/xfs_iops.c +++ b/fs/xfs/xfs_iops.c @@ -834,7 +834,7 @@ xfs_setattr_nonsize( } error = xfs_trans_alloc_ichange(ip, udqp, gdqp, NULL, - has_capability_noaudit(current, CAP_FOWNER), &tp); + capable_noaudit(CAP_FOWNER), &tp); if (error) goto out_dqrele; diff --git a/fs/xfs/xfs_platform.h b/fs/xfs/xfs_platform.h index 59a33c60e0ca..5d542e95fe44 100644 --- a/fs/xfs/xfs_platform.h +++ b/fs/xfs/xfs_platform.h @@ -289,15 +289,4 @@ int xfs_rw_bdev(struct block_device *bdev, sector_t sector, unsigned int count, # define PTR_FMT "%p" #endif -/* - * Helper for IO routines to grab backing pages from allocated kernel memory. - */ -static inline struct page * -kmem_to_page(void *addr) -{ - if (is_vmalloc_addr(addr)) - return vmalloc_to_page(addr); - return virt_to_page(addr); -} - #endif /* _XFS_PLATFORM_H */ diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index 4b2eeb7783f7..b24db75eaedc 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -445,7 +445,7 @@ xfs_shutdown_devices( blkdev_issue_flush(mp->m_logdev_targp->bt_bdev); invalidate_bdev(mp->m_logdev_targp->bt_bdev); } - if (mp->m_rtdev_targp) { + if (mp->m_rtdev_targp && mp->m_rtdev_targp != mp->m_ddev_targp) { blkdev_issue_flush(mp->m_rtdev_targp->bt_bdev); invalidate_bdev(mp->m_rtdev_targp->bt_bdev); } diff --git a/fs/xfs/xfs_trans_buf.c b/fs/xfs/xfs_trans_buf.c index 1e025848811a..a5d25b703dfc 100644 --- a/fs/xfs/xfs_trans_buf.c +++ b/fs/xfs/xfs_trans_buf.c @@ -521,7 +521,8 @@ xfs_trans_log_buf( { struct xfs_buf_log_item *bip = bp->b_log_item; - ASSERT(first <= last && last < BBTOB(bp->b_length)); + ASSERT(first <= last); + ASSERT(last < BBTOB(bp->b_length)); ASSERT(!(bip->bli_flags & XFS_BLI_ORDERED)); xfs_trans_dirty_buf(tp, bp); diff --git a/fs/xfs/xfs_zone_alloc.c b/fs/xfs/xfs_zone_alloc.c index 7d13fa7ab30a..bdbb60cc5d5b 100644 --- a/fs/xfs/xfs_zone_alloc.c +++ b/fs/xfs/xfs_zone_alloc.c @@ -793,17 +793,35 @@ xfs_get_cached_zone( rcu_read_lock(); oz = VFS_I(ip)->i_private; - if (oz) { - /* - * GC only steals open zones at mount time, so no GC zones - * should end up in the cache. - */ - ASSERT(!oz->oz_is_gc); - if (!atomic_inc_not_zero(&oz->oz_ref)) + if (!oz) + goto out_unlock; + + /* + * GC only steals open zones at mount time, so no GC zones should end up + * in the cache. + */ + ASSERT(!oz->oz_is_gc); + + /* + * Drop the old cached open zone if it is full. + */ + if (oz->oz_allocated == rtg_blocks(oz->oz_rtg)) { + spin_lock(&ip->i_flags_lock); + oz = VFS_I(ip)->i_private; + if (oz && oz->oz_allocated == rtg_blocks(oz->oz_rtg)) { + VFS_I(ip)->i_private = NULL; + spin_unlock(&ip->i_flags_lock); + xfs_open_zone_put(oz); oz = NULL; + goto out_unlock; + } + spin_unlock(&ip->i_flags_lock); } - rcu_read_unlock(); + if (!atomic_inc_not_zero(&oz->oz_ref)) + oz = NULL; +out_unlock: + rcu_read_unlock(); return oz; } @@ -818,18 +836,41 @@ xfs_get_cached_zone( * that were every written to, but significantly simplifies the cached zone * lookup. Because the open_zone is clearly marked as full when all data * in the underlying RTG was written, the caching is always safe. + * + * Called with a reference on @oz held. And returns two references on the + * returned zone: one for the caller and one for pinning the zone in + * inode->i_private. */ -static void +static struct xfs_open_zone * xfs_set_cached_zone( struct xfs_inode *ip, struct xfs_open_zone *oz) { struct xfs_open_zone *old_oz; + /* + * If the open zone cached in the inode still has free space, use that + * instead of the new open zone just selected. This can happen when + * multiple threads race to perform zone selection for an inode. + * io_uring worker threads seem to be good way to trigger this. + * + * We need to grab an extra reference to this open zone as the caller + * owns a reference in addition to the i_private pointer. + */ + spin_lock(&ip->i_flags_lock); + old_oz = VFS_I(ip)->i_private; + if (old_oz && old_oz->oz_allocated < rtg_blocks(old_oz->oz_rtg) && + atomic_inc_not_zero(&old_oz->oz_ref)) { + spin_unlock(&ip->i_flags_lock); + xfs_open_zone_put(oz); + return old_oz; + } + VFS_I(ip)->i_private = oz; atomic_inc(&oz->oz_ref); - old_oz = xchg(&VFS_I(ip)->i_private, oz); + spin_unlock(&ip->i_flags_lock); if (old_oz) xfs_open_zone_put(old_oz); + return oz; } static void @@ -873,14 +914,13 @@ xfs_zone_alloc_and_submit( * the inode is still associated with a zone and use that if so. */ if (!*oz) +select_zone: *oz = xfs_get_cached_zone(ip); - if (!*oz) { -select_zone: *oz = xfs_select_zone(mp, write_hint, pack_tight); if (!*oz) goto out_error; - xfs_set_cached_zone(ip, *oz); + *oz = xfs_set_cached_zone(ip, *oz); } alloc_len = xfs_zone_alloc_blocks(*oz, XFS_B_TO_FSB(mp, ioend->io_size), diff --git a/fs/xfs/xfs_zone_gc.c b/fs/xfs/xfs_zone_gc.c index d0b85179a3d2..5fdcf98a2133 100644 --- a/fs/xfs/xfs_zone_gc.c +++ b/fs/xfs/xfs_zone_gc.c @@ -869,6 +869,11 @@ xfs_zone_gc_write_chunk( WRITE_ONCE(chunk->state, XFS_GC_BIO_NEW); list_move_tail(&chunk->entry, &data->writing); + /* + * If we run on top of stacked block device, the read I/O might have + * reset bi_bdev, restore it to the one we want. + */ + bio_set_dev(&chunk->bio, mp->m_rtdev_targp->bt_bdev); bio_reuse(&chunk->bio, REQ_OP_WRITE); while ((split_chunk = xfs_zone_gc_split_write(data, chunk))) xfs_zone_gc_submit_write(data, split_chunk); diff --git a/include/linux/capability.h b/include/linux/capability.h index 37db92b3d6f8..f8532d92fcad 100644 --- a/include/linux/capability.h +++ b/include/linux/capability.h @@ -145,6 +145,7 @@ extern bool has_capability_noaudit(struct task_struct *t, int cap); extern bool has_ns_capability_noaudit(struct task_struct *t, struct user_namespace *ns, int cap); extern bool capable(int cap); +bool capable_noaudit(int cap); extern bool ns_capable(struct user_namespace *ns, int cap); extern bool ns_capable_noaudit(struct user_namespace *ns, int cap); extern bool ns_capable_setid(struct user_namespace *ns, int cap); @@ -167,6 +168,10 @@ static inline bool capable(int cap) { return true; } +static inline bool capable_noaudit(int cap) +{ + return true; +} static inline bool ns_capable(struct user_namespace *ns, int cap) { return true; diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index 331dcbefe72f..42801f0b6456 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -91,6 +91,15 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, struct io_br_sel *sel, unsigned int issue_flags); +int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq, + void (*release)(void *), unsigned int index, + unsigned int issue_flags); +int io_buffer_register_bvec(struct io_uring_cmd *cmd, const struct bio_vec *bvs, + unsigned int nr_bvecs, void (*release)(void *), + void *priv, u8 dir, unsigned int index, + unsigned int issue_flags); +int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index, + unsigned int issue_flags); #else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, @@ -133,6 +142,29 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, { return true; } +static inline int io_buffer_register_request(struct io_uring_cmd *cmd, + struct request *rq, + void (*release)(void *), + unsigned int index, + unsigned int issue_flags) +{ + return -EOPNOTSUPP; +} +static inline int io_buffer_register_bvec(struct io_uring_cmd *cmd, + const struct bio_vec *bvs, + unsigned int nr_bvecs, + void (*release)(void *), void *priv, + u8 dir, unsigned int index, + unsigned int issue_flags) +{ + return -EOPNOTSUPP; +} +static inline int io_buffer_unregister(struct io_uring_cmd *cmd, + unsigned int index, + unsigned int issue_flags) +{ + return -EOPNOTSUPP; +} #endif static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req) @@ -182,10 +214,4 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret, return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true); } -int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq, - void (*release)(void *), unsigned int index, - unsigned int issue_flags); -int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index, - unsigned int issue_flags); - #endif /* _LINUX_IO_URING_CMD_H */ diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 6b9e9c669bb3..39629ee77b91 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -53,6 +53,11 @@ enum io_uring_cmd_flags { IO_URING_F_COMPAT = (1 << 12), }; +enum { + IO_BUF_DEST = 1 << ITER_DEST, + IO_BUF_SOURCE = 1 << ITER_SOURCE, +}; + struct iou_loop_params; struct io_wq_work_node { diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h index 44e5e9fa12e1..1a3981c26b23 100644 --- a/include/linux/nfs4.h +++ b/include/linux/nfs4.h @@ -390,6 +390,14 @@ enum { }; /* + * Symbol name and value are from draft-ietf-nfsv4-uncacheable-files + * Section 7. "XDR for Uncacheable Attribute" + */ +enum { + FATTR4_UNCACHEABLE_FILE_DATA = 87, +}; + +/* * The following internal definitions enable processing the above * attribute bits within 32-bit word boundaries. */ @@ -475,6 +483,7 @@ enum { #define FATTR4_WORD2_ACL_TRUEFORM_SCOPE BIT(FATTR4_ACL_TRUEFORM_SCOPE - 64) #define FATTR4_WORD2_POSIX_DEFAULT_ACL BIT(FATTR4_POSIX_DEFAULT_ACL - 64) #define FATTR4_WORD2_POSIX_ACCESS_ACL BIT(FATTR4_POSIX_ACCESS_ACL - 64) +#define FATTR4_WORD2_UNCACHEABLE_FILE_DATA BIT(FATTR4_UNCACHEABLE_FILE_DATA - 64) /* MDS threshold bitmap bits */ #define THRESHOLD_RD (1UL << 0) diff --git a/include/linux/nfs_fs.h b/include/linux/nfs_fs.h index 764056498eba..b85a73ae7919 100644 --- a/include/linux/nfs_fs.h +++ b/include/linux/nfs_fs.h @@ -110,6 +110,7 @@ struct nfs_open_context { #define NFS_CONTEXT_UNLOCK (3) #define NFS_CONTEXT_FILE_OPEN (4) #define NFS_CONTEXT_WRITE_SYNC (5) +#define NFS_CONTEXT_O_DIRECT (6) struct nfs4_threshold *mdsthreshold; struct list_head list; @@ -162,6 +163,8 @@ struct nfs_inode { struct timespec64 btime; + bool uncacheable_file_data : 1; + /* * read_cache_jiffies is when we started read-caching this inode. * attrtimeo is for how long the cached information is assumed @@ -319,6 +322,7 @@ struct nfs4_copy_state { #define NFS_INO_INVALID_NLINK BIT(16) /* cached nlinks is invalid */ #define NFS_INO_INVALID_MODE BIT(17) /* cached mode is invalid */ #define NFS_INO_INVALID_BTIME BIT(18) /* cached btime is invalid */ +#define NFS_INO_INVALID_UNCACHEABLE_FILE_DATA BIT(19) /* cached uncacheable_file_data is invalid */ #define NFS_INO_INVALID_ATTR (NFS_INO_INVALID_CHANGE \ | NFS_INO_INVALID_CTIME \ diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h index 11c5b31cfc7d..2e1987ac403d 100644 --- a/include/linux/nfs_xdr.h +++ b/include/linux/nfs_xdr.h @@ -17,6 +17,9 @@ #define NFS_BITMASK_SZ 3 +/* aux_flags in nfs_fattr */ +#define NFS_AUX_UNCACHEABLE_FILE_DATA BIT(0) + struct nfs4_string { unsigned int len; char *data; @@ -68,6 +71,7 @@ struct nfs_fattr { struct timespec64 mtime; struct timespec64 ctime; struct timespec64 btime; + __u32 aux_flags; /* NFSv4 auxiliary flags bitfield */ __u64 change_attr; /* NFSv4 change attribute */ __u64 pre_change_attr;/* pre-op NFSv4 change attribute */ __u64 pre_size; /* pre_op_attr.size */ @@ -108,6 +112,7 @@ struct nfs_fattr { #define NFS_ATTR_FATTR_GROUP_NAME BIT_ULL(24) #define NFS_ATTR_FATTR_V4_SECURITY_LABEL BIT_ULL(25) #define NFS_ATTR_FATTR_BTIME BIT_ULL(26) +#define NFS_ATTR_FATTR_UNCACHEABLE_FILE_DATA BIT_ULL(27) #define NFS_ATTR_FATTR (NFS_ATTR_FATTR_TYPE \ | NFS_ATTR_FATTR_MODE \ @@ -129,7 +134,8 @@ struct nfs_fattr { #define NFS_ATTR_FATTR_V4 (NFS_ATTR_FATTR \ | NFS_ATTR_FATTR_SPACE_USED \ | NFS_ATTR_FATTR_BTIME \ - | NFS_ATTR_FATTR_V4_SECURITY_LABEL) + | NFS_ATTR_FATTR_V4_SECURITY_LABEL \ + | NFS_ATTR_FATTR_UNCACHEABLE_FILE_DATA) /* * Maximal number of supported layout drivers. diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index a283766a192a..c8a36423cb34 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -661,6 +661,7 @@ extern void workqueue_set_min_active(struct workqueue_struct *wq, int min_active); extern struct work_struct *current_work(void); extern bool current_is_workqueue_rescuer(void); +extern bool current_is_workqueue_mem_reclaim(void); extern bool workqueue_congested(int cpu, struct workqueue_struct *wq); extern unsigned int work_busy(struct work_struct *work); extern __printf(1, 2) void set_worker_desc(const char *fmt, ...); diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index 270c1a2c24c4..1dd9fc5afc46 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -2114,6 +2114,14 @@ DEFINE_EVENT(f2fs_zip_end, f2fs_decompress_pages_end, ); #ifdef CONFIG_F2FS_IOSTAT +/* + * Number of read folio order buckets emitted by the f2fs_iostat tracepoint. + * TP_printk() cannot loop, so the field count is fixed here and must be >= + * the largest possible NR_PAGE_ORDERS (14 on arm64 with 64K pages). The + * BUILD_BUG_ON() in f2fs_update_read_folio_count() enforces this. + */ +#define F2FS_IOSTAT_RD_FOLIO_ORDERS 16 + TRACE_EVENT(f2fs_iostat, TP_PROTO(struct f2fs_sb_info *sbi, unsigned long long *iostat, @@ -2151,7 +2159,7 @@ TRACE_EVENT(f2fs_iostat, __field(unsigned long long, fs_mrio) __field(unsigned long long, fs_discard) __field(unsigned long long, fs_reset_zone) - __array(unsigned long long, read_folio_count, 11) + __array(unsigned long long, read_folio_count, F2FS_IOSTAT_RD_FOLIO_ORDERS) ), TP_fast_assign( @@ -2186,7 +2194,8 @@ TRACE_EVENT(f2fs_iostat, __entry->fs_reset_zone = iostat[FS_ZONE_RESET_IO]; memset(__entry->read_folio_count, 0, sizeof(__entry->read_folio_count)); memcpy(__entry->read_folio_count, read_folio_count, - sizeof(unsigned long long) * min_t(int, NR_PAGE_ORDERS, 11)); + sizeof(unsigned long long) * + min_t(int, NR_PAGE_ORDERS, F2FS_IOSTAT_RD_FOLIO_ORDERS)); ), TP_printk("dev = (%d,%d), " @@ -2201,7 +2210,8 @@ TRACE_EVENT(f2fs_iostat, "fs [data=%llu, (gc_data=%llu, cdata=%llu), " "node=%llu, meta=%llu], " "read_folio_count [0=%llu, 1=%llu, 2=%llu, 3=%llu, 4=%llu, " - "5=%llu, 6=%llu, 7=%llu, 8=%llu, 9=%llu, 10=%llu]", + "5=%llu, 6=%llu, 7=%llu, 8=%llu, 9=%llu, 10=%llu, 11=%llu, " + "12=%llu, 13=%llu, 14=%llu, 15=%llu]", show_dev(__entry->dev), __entry->app_wio, __entry->app_dio, __entry->app_bio, __entry->app_mio, __entry->app_bcdio, __entry->app_mcdio, __entry->fs_dio, __entry->fs_cdio, @@ -2218,7 +2228,9 @@ TRACE_EVENT(f2fs_iostat, __entry->read_folio_count[4], __entry->read_folio_count[5], __entry->read_folio_count[6], __entry->read_folio_count[7], __entry->read_folio_count[8], __entry->read_folio_count[9], - __entry->read_folio_count[10]) + __entry->read_folio_count[10], __entry->read_folio_count[11], + __entry->read_folio_count[12], __entry->read_folio_count[13], + __entry->read_folio_count[14], __entry->read_folio_count[15]) ); #ifndef __F2FS_IOSTAT_LATENCY_TYPE diff --git a/include/uapi/linux/f2fs.h b/include/uapi/linux/f2fs.h index 795e26258355..4409ada2fecb 100644 --- a/include/uapi/linux/f2fs.h +++ b/include/uapi/linux/f2fs.h @@ -45,6 +45,9 @@ #define F2FS_IOC_START_ATOMIC_REPLACE _IO(F2FS_IOCTL_MAGIC, 25) #define F2FS_IOC_GET_DEV_ALIAS_FILE _IOR(F2FS_IOCTL_MAGIC, 26, __u32) #define F2FS_IOC_IO_PRIO _IOW(F2FS_IOCTL_MAGIC, 27, __u32) +#define F2FS_IOC_RESERVE_DEV_ALIAS _IO(F2FS_IOCTL_MAGIC, 28) +#define F2FS_IOC_RELEASE_DEV_ALIAS _IO(F2FS_IOCTL_MAGIC, 29) +#define F2FS_IOC_GET_DEV_ALIAS_STATUS _IOR(F2FS_IOCTL_MAGIC, 30, __u32) /* * should be same as XFS_IOC_GOINGDOWN. @@ -70,6 +73,10 @@ enum { F2FS_IOPRIO_MAX, }; +/* for F2FS_IOC_GET_DEV_ALIAS_STATUS */ +#define F2FS_DEV_ALIAS_STATUS_RELEASED 0 +#define F2FS_DEV_ALIAS_STATUS_RESERVED 1 + struct f2fs_gc_range { __u32 sync; __u64 start; diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h index c13e1f9a2f12..7435e09c87fe 100644 --- a/include/uapi/linux/fuse.h +++ b/include/uapi/linux/fuse.h @@ -240,6 +240,14 @@ * - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE + * + * 7.46 + * - add FUSE_IO_URING_CMD_ADD_QUEUE + * - add FUSE_HAS_IO_URING_BUFPOOL + * - add fuse_uring_cmd_req bufpool struct + * - add bufpool offset field to fuse_uring_ent_in_out struct + * - add FUSE_URING_ZERO_COPY, FUSE_URING_ENT_ZERO_COPY, and + * FOPEN_IO_URING_ZERO_COPY flag */ #ifndef _LINUX_FUSE_H @@ -275,7 +283,7 @@ #define FUSE_KERNEL_VERSION 7 /** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 45 +#define FUSE_KERNEL_MINOR_VERSION 46 /** The node ID of the root inode */ #define FUSE_ROOT_ID 1 @@ -383,6 +391,12 @@ struct fuse_file_lock { * FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE) * FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode * FOPEN_PASSTHROUGH: passthrough read/write io for this open file + * FOPEN_IO_URING_ZERO_COPY: use io-uring zero-copy for reads/writes on this + * open file. Honored only when the serving io-uring + * queue was set up for zero-copy + * (FUSE_URING_ZERO_COPY) and the request carries page + * payload. Otherwise reads/writes fall back to + * copying. */ #define FOPEN_DIRECT_IO (1 << 0) #define FOPEN_KEEP_CACHE (1 << 1) @@ -392,6 +406,7 @@ struct fuse_file_lock { #define FOPEN_NOFLUSH (1 << 5) #define FOPEN_PARALLEL_DIRECT_WRITES (1 << 6) #define FOPEN_PASSTHROUGH (1 << 7) +#define FOPEN_IO_URING_ZERO_COPY (1 << 8) /** * INIT request/reply flags @@ -448,6 +463,7 @@ struct fuse_file_lock { * FUSE_OVER_IO_URING: Indicate that client supports io-uring * FUSE_REQUEST_TIMEOUT: kernel supports timing out requests. * init_out.request_timeout contains the timeout (in secs) + * FUSE_HAS_IO_URING_BUFPOOL: kernel supports io-uring buffer pools */ #define FUSE_ASYNC_READ (1 << 0) #define FUSE_POSIX_LOCKS (1 << 1) @@ -495,6 +511,7 @@ struct fuse_file_lock { #define FUSE_ALLOW_IDMAP (1ULL << 40) #define FUSE_OVER_IO_URING (1ULL << 41) #define FUSE_REQUEST_TIMEOUT (1ULL << 42) +#define FUSE_HAS_IO_URING_BUFPOOL (1ULL << 43) /** * CUSE INIT request/reply flags @@ -1251,6 +1268,13 @@ struct fuse_supp_groups { #define FUSE_URING_IN_OUT_HEADER_SZ 128 #define FUSE_URING_OP_IN_OUT_SZ 128 +/** + * fuse_uring_ent_in_out flags + * + * FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied + */ +#define FUSE_URING_ENT_ZERO_COPY (1 << 0) + /* Used as part of the fuse_uring_req_header */ struct fuse_uring_ent_in_out { uint64_t flags; @@ -1263,7 +1287,9 @@ struct fuse_uring_ent_in_out { /* size of user payload buffer */ uint32_t payload_sz; - uint32_t padding; + + /* Offset into the bufpool, if bufpools are used */ + uint32_t offset; uint64_t reserved; }; @@ -1292,8 +1318,22 @@ enum fuse_uring_cmd { /* commit fuse request result and fetch next request */ FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2, + + /* add a queue */ + FUSE_IO_URING_CMD_ADD_QUEUE = 3, + + /* add a bufpool to a queue */ + FUSE_IO_URING_CMD_ADD_BUFPOOL = 4, }; +/* + * fuse_uring_cmd_req flags for FUSE_IO_URING_CMD_ADD_QUEUE + * + * FUSE_URING_ZERO_COPY is only supported for queues with bufpools on privileged + * servers + */ +#define FUSE_URING_ZERO_COPY (1 << 0) + /** * In the 80B command area of the SQE. */ @@ -1306,6 +1346,25 @@ struct fuse_uring_cmd_req { /* queue the command is for (queue index) */ uint16_t qid; uint8_t padding[6]; + + union { + struct { + /* base address of bufpool */ + uint64_t uaddr; + uint32_t len; + uint32_t reserved; + } bufpool; + + /* + * Index of this entry's slot in the server's io_uring + * registered buffer table, where the kernel registers the + * request's pages for zero-copy. Set for + * FUSE_IO_URING_CMD_REGISTER cmds only, and only on queues + * created with FUSE_URING_ZERO_COPY. On a non-zero-copy queue + * this must be 0 + */ + uint16_t ent_zero_copy_buf_index; + }; }; #endif /* _LINUX_FUSE_H */ diff --git a/include/uapi/mtd/ubi-user.h b/include/uapi/mtd/ubi-user.h index aa872a41ffb9..3538e11b5175 100644 --- a/include/uapi/mtd/ubi-user.h +++ b/include/uapi/mtd/ubi-user.h @@ -289,6 +289,13 @@ enum { * If @disable_fm is not zero, ubi doesn't create new fastmap even the module * param 'fm_autoconvert' is set, and existed old fastmap will be destroyed * after doing full scanning. + * + * The @wl_threshold defines the maximum difference between the highest and the + * lowest erase counter value of eraseblocks of this UBI device. When this + * threshold is exceeded, UBI starts performing wear leveling by means of + * moving data from eraseblock with low erase counter to eraseblocks with high + * erase counter. If @wl_threshold is zero, the default kernel value of + * %CONFIG_MTD_UBI_WL_THRESHOLD is used. The accepted range is 2-65536. */ struct ubi_attach_req { __s32 ubi_num; @@ -297,7 +304,8 @@ struct ubi_attach_req { __s16 max_beb_per1024; __s8 disable_fm; __s8 need_resv_pool; - __s8 padding[8]; + __s32 wl_threshold; + __s8 padding[4]; }; /* diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 76f049e29aa2..61053421d809 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3236,7 +3236,7 @@ static int __init io_uring_init(void) io_uring_optable_init(); /* imu->dir is u8 */ - BUILD_BUG_ON((IO_IMU_DEST | IO_IMU_SOURCE) > U8_MAX); + BUILD_BUG_ON((IO_BUF_DEST | IO_BUF_SOURCE) > U8_MAX); /* * Allow user copy in the per-command field, which starts after the diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 174f74cbbf60..51b46e624ddd 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -912,7 +912,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, imu->release = io_release_ubuf; imu->priv = imu; imu->flags = 0; - imu->dir = IO_IMU_DEST | IO_IMU_SOURCE; + imu->dir = IO_BUF_DEST | IO_BUF_SOURCE; if (coalesced) imu->folio_shift = data.folio_shift; refcount_set(&imu->refs, 1); @@ -1015,71 +1015,124 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, return ret; } -int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq, - void (*release)(void *), unsigned int index, - unsigned int issue_flags) +static struct io_mapped_ubuf *io_kernel_buffer_init(struct io_ring_ctx *ctx, + unsigned int nr_bvecs, + unsigned int total_bytes, + u8 dir, + void (*release)(void *), + void *priv, + unsigned int index) { - struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx; struct io_rsrc_data *data = &ctx->buf_table; - struct req_iterator rq_iter; struct io_mapped_ubuf *imu; struct io_rsrc_node *node; - struct bio_vec bv; - unsigned int nr_bvecs = 0; - int ret = 0; - io_ring_submit_lock(ctx, issue_flags); - if (index >= data->nr) { - ret = -EINVAL; - goto unlock; - } + if (index >= data->nr) + return ERR_PTR(-EINVAL); index = array_index_nospec(index, data->nr); - if (data->nodes[index]) { - ret = -EBUSY; - goto unlock; - } + if (data->nodes[index]) + return ERR_PTR(-EBUSY); node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); - if (!node) { - ret = -ENOMEM; - goto unlock; - } + if (!node) + return ERR_PTR(-ENOMEM); - /* - * blk_rq_nr_phys_segments() may overestimate the number of bvecs - * but avoids needing to iterate over the bvecs - */ - imu = io_alloc_imu(ctx, blk_rq_nr_phys_segments(rq)); + imu = io_alloc_imu(ctx, nr_bvecs); if (!imu) { io_cache_free(&ctx->node_cache, node); - ret = -ENOMEM; - goto unlock; + return ERR_PTR(-ENOMEM); } imu->ubuf = 0; - imu->len = blk_rq_bytes(rq); + imu->len = total_bytes; imu->folio_shift = PAGE_SHIFT; + imu->nr_bvecs = nr_bvecs; refcount_set(&imu->refs, 1); imu->release = release; - imu->priv = rq; + imu->priv = priv; + imu->dir = dir; imu->flags = IO_REGBUF_F_KBUF; - imu->dir = 1 << rq_data_dir(rq); + node->buf = imu; + data->nodes[index] = node; + + return imu; +} + +int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq, + void (*release)(void *), unsigned int index, + unsigned int issue_flags) +{ + struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx; + struct req_iterator rq_iter; + struct io_mapped_ubuf *imu; + struct bio_vec bv; + /* + * blk_rq_nr_phys_segments() may overestimate the number of bvecs + * but avoids needing to iterate over the bvecs + */ + unsigned int nr_bvecs = blk_rq_nr_phys_segments(rq); + unsigned int total_bytes = blk_rq_bytes(rq); + int ret = 0; + + io_ring_submit_lock(ctx, issue_flags); + + imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes, + 1 << rq_data_dir(rq), release, rq, index); + if (IS_ERR(imu)) { + ret = PTR_ERR(imu); + goto unlock; + } + + nr_bvecs = 0; rq_for_each_bvec(bv, rq, rq_iter) imu->bvec[nr_bvecs++] = bv; imu->nr_bvecs = nr_bvecs; - node->buf = imu; - data->nodes[index] = node; +unlock: + io_ring_submit_unlock(ctx, issue_flags); + return ret; +} +EXPORT_SYMBOL_GPL(io_buffer_register_request); + +/* + * bvs is copied internally. caller may free it on return. + */ +int io_buffer_register_bvec(struct io_uring_cmd *cmd, const struct bio_vec *bvs, + unsigned int nr_bvecs, void (*release)(void *), + void *priv, u8 dir, unsigned int index, + unsigned int issue_flags) +{ + struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx; + struct io_mapped_ubuf *imu; + struct bio_vec *bvec; + unsigned int i, total_bytes = 0; + int ret = 0; + + for (i = 0; i < nr_bvecs; i++) + total_bytes += bvs[i].bv_len; + + io_ring_submit_lock(ctx, issue_flags); + imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes, dir, release, + priv, index); + if (IS_ERR(imu)) { + ret = PTR_ERR(imu); + goto unlock; + } + + bvec = imu->bvec; + for (i = 0; i < nr_bvecs; i++) + bvec[i] = bvs[i]; + unlock: io_ring_submit_unlock(ctx, issue_flags); return ret; } EXPORT_SYMBOL_GPL(io_buffer_register_bvec); -int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index, - unsigned int issue_flags) +int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index, + unsigned int issue_flags) { struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx; struct io_rsrc_data *data = &ctx->buf_table; @@ -1109,7 +1162,7 @@ unlock: io_ring_submit_unlock(ctx, issue_flags); return ret; } -EXPORT_SYMBOL_GPL(io_buffer_unregister_bvec); +EXPORT_SYMBOL_GPL(io_buffer_unregister); static int validate_fixed_range(u64 buf_addr, size_t len, const struct io_mapped_ubuf *imu) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index eacfdb70f203..9ef88383b363 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -26,11 +26,6 @@ struct io_rsrc_node { }; enum { - IO_IMU_DEST = 1 << ITER_DEST, - IO_IMU_SOURCE = 1 << ITER_SOURCE, -}; - -enum { IO_REGBUF_F_KBUF = 1, }; diff --git a/kernel/capability.c b/kernel/capability.c index 829f49ae07b9..90e6ab62f6db 100644 --- a/kernel/capability.c +++ b/kernel/capability.c @@ -326,7 +326,6 @@ bool has_capability_noaudit(struct task_struct *t, int cap) { return has_ns_capability_noaudit(t, &init_user_ns, cap); } -EXPORT_SYMBOL(has_capability_noaudit); static bool ns_capable_common(struct user_namespace *ns, int cap, @@ -416,6 +415,24 @@ bool capable(int cap) return ns_capable(&init_user_ns, cap); } EXPORT_SYMBOL(capable); + +/** + * capable_noaudit - Determine if the current task has a superior + * capability in effect by checking the process's effective + * capabilities (unaudited). + * @cap: The capability to be tested for + * + * This is the same as capable(), except it uses CAP_OPT_NOAUDIT as to prevent + * issuing spurious audit messages. + * + * This sets PF_SUPERPRIV on the task if the capability is available on the + * assumption that it's about to be used. + */ +bool capable_noaudit(int cap) +{ + return ns_capable_noaudit(&init_user_ns, cap); +} +EXPORT_SYMBOL(capable_noaudit); #endif /* CONFIG_MULTIUSER */ /** diff --git a/kernel/workqueue.c b/kernel/workqueue.c index bfeef512f6dd..3c034cbc5bb3 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -6294,6 +6294,30 @@ bool current_is_workqueue_rescuer(void) } /** + * current_is_workqueue_mem_reclaim - is %current a %WQ_MEM_RECLAIM worker? + * + * Determine whether %current is a workqueue worker executing on a workqueue + * created with %WQ_MEM_RECLAIM. This mirrors the condition that + * check_flush_dependency() warns on: flushing (or otherwise waiting on) a + * !WQ_MEM_RECLAIM workqueue from such a context breaks the forward-progress + * guarantee and can deadlock. Callers that may recurse into such a flush -- + * e.g. NFS LOCALIO submitting into a stacked filesystem that flushes its own + * !WQ_MEM_RECLAIM workqueue -- can use this to decide whether they must defer + * the work to a !WQ_MEM_RECLAIM workqueue rather than run it inline. + * + * Return: %true if %current is a %WQ_MEM_RECLAIM worker. %false otherwise. + */ +bool current_is_workqueue_mem_reclaim(void) +{ + struct worker *worker = current_wq_worker(); + + return worker && + ((worker->current_pwq->wq->flags & + (WQ_MEM_RECLAIM | __WQ_LEGACY)) == WQ_MEM_RECLAIM); +} +EXPORT_SYMBOL_GPL(current_is_workqueue_mem_reclaim); + +/** * workqueue_congested - test whether a workqueue is congested * @cpu: CPU in question * @wq: target workqueue diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c index efa26899bc7d..6cedc824cf82 100644 --- a/net/sunrpc/clnt.c +++ b/net/sunrpc/clnt.c @@ -96,7 +96,10 @@ static void rpc_unregister_client(struct rpc_clnt *clnt) static void __rpc_clnt_remove_pipedir(struct rpc_clnt *clnt) { - rpc_remove_client_dir(clnt); + if (clnt->pipefs_sb) { + rpc_remove_client_dir(clnt); + clnt->pipefs_sb = NULL; + } } static void rpc_clnt_remove_pipedir(struct rpc_clnt *clnt) @@ -177,19 +180,28 @@ static int rpc_clnt_skip_event(struct rpc_clnt *clnt, unsigned long event) } static int __rpc_clnt_handle_event(struct rpc_clnt *clnt, unsigned long event, - struct super_block *sb) + struct super_block *sb) { + int err = 0; + switch (event) { case RPC_PIPEFS_MOUNT: - return rpc_setup_pipedir_sb(sb, clnt); + clnt->pipefs_sb = sb; + err = rpc_setup_pipedir_sb(sb, clnt); + if (err) + clnt->pipefs_sb = NULL; + break; case RPC_PIPEFS_UMOUNT: - __rpc_clnt_remove_pipedir(clnt); + if (clnt->pipefs_sb == sb) { + __rpc_clnt_remove_pipedir(clnt); + clnt->pipefs_sb = NULL; + } break; default: printk(KERN_ERR "%s: unknown event: %ld\n", __func__, event); return -ENOTSUPP; } - return 0; + return err; } static int __rpc_pipefs_event(struct rpc_clnt *clnt, unsigned long event, diff --git a/net/sunrpc/rpcb_clnt.c b/net/sunrpc/rpcb_clnt.c index 6aa372188c86..4c0b7fefee4e 100644 --- a/net/sunrpc/rpcb_clnt.c +++ b/net/sunrpc/rpcb_clnt.c @@ -490,6 +490,8 @@ static int rpcb_register_inet4(struct sunrpc_net *sn, int result; map->r_addr = rpc_sockaddr2uaddr(sap, GFP_KERNEL); + if (!map->r_addr) + return -ENOMEM; msg->rpc_proc = &rpcb_procedures4[RPCBPROC_UNSET]; if (port != 0) { @@ -516,6 +518,8 @@ static int rpcb_register_inet6(struct sunrpc_net *sn, int result; map->r_addr = rpc_sockaddr2uaddr(sap, GFP_KERNEL); + if (!map->r_addr) + return -ENOMEM; msg->rpc_proc = &rpcb_procedures4[RPCBPROC_UNSET]; if (port != 0) { diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index 359407aae03e..7f60723fa64d 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -1202,9 +1202,9 @@ static void xs_save_old_callbacks(struct sock_xprt *transport, struct sock *sk) static void xs_restore_old_callbacks(struct sock_xprt *transport, struct sock *sk) { - sk->sk_data_ready = transport->old_data_ready; + WRITE_ONCE(sk->sk_data_ready, transport->old_data_ready); sk->sk_state_change = transport->old_state_change; - sk->sk_write_space = transport->old_write_space; + WRITE_ONCE(sk->sk_write_space, transport->old_write_space); sk->sk_error_report = transport->old_error_report; } @@ -1664,6 +1664,7 @@ static void xs_udp_do_set_buffer_size(struct rpc_xprt *xprt) { struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); struct sock *sk = transport->inet; + void (*write_space)(struct sock *sock); if (transport->rcvsize) { sk->sk_userlocks |= SOCK_RCVBUF_LOCK; @@ -1672,7 +1673,8 @@ static void xs_udp_do_set_buffer_size(struct rpc_xprt *xprt) if (transport->sndsize) { sk->sk_userlocks |= SOCK_SNDBUF_LOCK; sk->sk_sndbuf = transport->sndsize * xprt->max_reqs * 2; - sk->sk_write_space(sk); + write_space = READ_ONCE(sk->sk_write_space); + write_space(sk); } } @@ -1988,8 +1990,8 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt, xs_save_old_callbacks(transport, sk); sk->sk_user_data = xprt; - sk->sk_data_ready = xs_data_ready; - sk->sk_write_space = xs_udp_write_space; + WRITE_ONCE(sk->sk_data_ready, xs_data_ready); + WRITE_ONCE(sk->sk_write_space, xs_udp_write_space); sk->sk_state_change = xs_local_state_change; sk->sk_error_report = xs_error_report; sk->sk_use_task_frag = false; @@ -2191,8 +2193,8 @@ static void xs_udp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock) xs_save_old_callbacks(transport, sk); sk->sk_user_data = xprt; - sk->sk_data_ready = xs_data_ready; - sk->sk_write_space = xs_udp_write_space; + WRITE_ONCE(sk->sk_data_ready, xs_data_ready); + WRITE_ONCE(sk->sk_write_space, xs_udp_write_space); sk->sk_use_task_frag = false; xprt_set_connected(xprt); @@ -2378,9 +2380,9 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock) xs_save_old_callbacks(transport, sk); sk->sk_user_data = xprt; - sk->sk_data_ready = xs_data_ready; + WRITE_ONCE(sk->sk_data_ready, xs_data_ready); sk->sk_state_change = xs_tcp_state_change; - sk->sk_write_space = xs_tcp_write_space; + WRITE_ONCE(sk->sk_write_space, xs_tcp_write_space); sk->sk_error_report = xs_error_report; sk->sk_use_task_frag = false; @@ -2648,7 +2650,17 @@ static int xs_tls_handshake_sync(struct rpc_xprt *lower_xprt, struct xprtsec_par rc = wait_for_completion_interruptible_timeout(&lower_transport->handshake_done, XS_TLS_HANDSHAKE_TO); if (rc <= 0) { - tls_handshake_cancel(sk); + if (!tls_handshake_cancel(sk)) { + /* + * Cancellation lost to handshake_complete(): the + * callback still owns its xprt reference and is in + * flight. Wait for it to finish before returning. + */ + wait_for_completion(&lower_transport->handshake_done); + if (rc == 0) + rc = -ETIMEDOUT; + goto out; + } if (rc == 0) rc = -ETIMEDOUT; goto out_put_xprt; |