| Age | Commit message (Collapse) | Author |
|---|
|
Pull nfs client updates from Anna Schumaker:
"New Features:
- Modernize the direct I/O path
- Size the LAYOUTGET reply buffer for wide flexfiles layouts
- Flexfiles device notifications and caching for wide striped layouts
- isolate mTLS client credentials by network namespace
Bugfixes:
- Various localio, flexfiles, xprtrdma, and TLS fixes
- Merged bugfixes from Trond that he was unable to send during the 7.3-rc cycle
Other Cleanups and Improvements:
- Split up pNFS block layout and SCSI layout support"
* tag 'nfs-for-7.4-1' of git://git.linux-nfs.org/projects/anna/linux-nfs: (68 commits)
nfs/localio: pin UUIDs during global invalidation
NFSv4/flexfiles: fail I/O to the data server once a lock is lost
pNFS/flexfiles: don't report an interrupted DS connect as a DS error
pNFS/flexfiles: make pg_init retry loops killable
nfsv4/pnfs: use the MDS authflavor for data server connections
xprtrdma: trim head iovec when the payload arrives via a Write chunk
nfs: set gencount on protocol-specific remove
xprtrdma: prevent disconnect feedback during endpoint teardown
NFS: add a key type that reveals the namespace .nfs keyring serial number
NFS: name the namespace .nfs keyring in the x509 handshake
SUNRPC: pass a keyring serial number to the TLS handshake
NFS: allocate the .nfs keyring per network namespace
NFS: name the init_nfs_fs() error labels
NFS: return DENIED in decode_lock_denied if we cannot decode owner
pNFS/flexfiles: don't reset to MDS for v4 error when previous layout forbid it
pNFS/flexfiles: don't read through MDS when previous layout forbid it
pNFS/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS over the mdsthreshold hint
NFS/localio: report the stability a DIO WRITE actually has
NFS/localio: detect a short read or write before the iterator has moved
NFS/localio: fix nfs_local_dio_misaligned tracepoint
...
|
|
The handshake upcall links the keyring named in ta_keyring into
tlshd's process keyring before tlshd reads the client certificate and
private key. That link is how tlshd gains possession of keys that
grant no user read permission. xprtsock never fills the field, so an
x509 handshake can present only keys that tlshd's own credentials can
read.
Add a keyring serial number to struct xprtsec_parms and pass it to
the x509 handshake, so an RPC client can name the keyring that holds
its credentials.
Signed-off-by: Chuck Lever <cel@kernel.org>
Signed-off-by: Anna Schumaker <anna.schumaker@hammerspace.com>
|
|
READDIR is the last NFSv2 procedure whose reply is still produced by
a hand-written encoder. Its entry list is the value-follows form of
RFC 1094's "entry *entries" -- each entry prefixed by TRUE, the
sequence closed by FALSE -- streamed straight from the directory as
it is read rather than from a materialized array. The generated
encoder could not drive that: the union encoder writes the reply
status word before the entry list, yet a streamed read does not learn
the status until the directory has been read, and the aggregate loop
was bounded by a count the begin hook had to supply up front.
The preceding patches remove both obstacles. The optional-data
aggregate encoder now pulls entries until the hook signals end, so no
count is needed, and the streaming directory reader yields one entry
at a time during encode. Mark readdirok's entry list as an aggregate
and regenerate; the struct definitions stay the verbatim RFC 1094
types, and the lone "pragma aggregate readdirok entries" selects the
hook-driven codec.
Split the read across the two phases so the generated encoder owns
the framing. nfsd_proc_readdir() opens the directory, where an open
error becomes the reply status the union encoder then writes, and the
encode hooks stream the entries: each pull reserves a placeholder
cookie and back-patches the preceding entry's cookie once the
following entry's resume offset is known; the end hook back-patches
the final cookie and reports eof. nfssvc_release_readdirres() closes
the reader and releases the file handle. The decode hooks are stubs
-- a server never decodes a READDIR result -- present only to satisfy
the generated decoder's linkage.
Retire struct nfsd_readdirres for an embedded struct readdirres so
the RPC dispatch layer can cast the response to its xdrgen type.
The directory's svc_fh moves into that wrapper and struct
readdirargs_wrapper goes away. svc_process() zeroes the result buffer
before decode but leaves the argument buffer as the previous request
left it, and ->pc_release runs even when decoding fails. A file
handle kept in the argument wrapper would reach fh_put() on that path
uninitialized.
The generated encoder reserves reply space through the xdr_stream,
which does not advance rq_next_page as it moves onto the next page.
svcrdma retains only the pages below rq_next_page across Send
completion, so the end hook sets rq_next_page past the last page the
reply uses, as nfsd4_encode_operation() does.
A host error during the streamed read cannot retract entries already
on the wire, so it ends the list as a truncated, non-eof reply rather
than an error status; an error opening the directory is still
reported as the reply status. The bytes on the wire are otherwise
unchanged.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260916162859.2051-30-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
A SYMLINK request's target pathname can extend from the head of the
receive buffer into its page data; the decoder reassembles it from a
head segment and a page via svc_fill_symlink_pathname(). Represent
the pathname in place rather than materializing it inline: the
"pages" directive expresses it as a struct xdr_buf and decodes it
with svcxdr_decode_opaque_payload(), which subsegments the source
stream instead of pulling the string inline through the bounded xdr
scratch buffer.
Annotate the to member of symlinkargs with "pragma pages" and
replace the NFSPROC_SYMLINK entry in nfsd_procedures2 with the
generated nfs_svc_decode_symlinkargs and nfs_svc_encode_nfsstat. A
wrapper structure bridges the generated symlinkargs type and the
svc_fh and iattr representations the NFSD VFS layer still uses.
nfsd_proc_symlink() copies the pathname out of the xdr_buf with
read_bytes_from_xdr_buf(), so a target that lands in the receive
buffer's tail, or straddles its pages and tail, is handled the same
way as one in the head.
The pc_argzero field is now set to zero for the NFSv2 SYMLINK
procedure. The generated decoder initializes every argument in the
argp->xdrgen field, making the early defensive memset unnecessary.
The hand-coded nfsd_proc_symlink() accepts a target of any length
from the decoder and answers one longer than NFS_MAXPATHLEN with
NFSERR_NAMETOOLONG. The generated decoder enforces the path typedef's
bound itself, so such a request now fails decoding and nfsd_dispatch()
returns RPC GARBAGE_ARGS instead. The Linux NFS client sends targets
up to PAGE_SIZE on an NFSv2 mount, and in that case it reports EIO to
the application rather than ENAMETOOLONG.
nfssvc_decode_symlinkargs() no longer has any callers, so it is
removed.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260916162859.2051-23-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
The NFSv2 WRITE argument decoder could not be generated because
the write payload must be directed into the server's page cache
and decoded in place; a materializing decoder would copy every
write. The "pages" directive now expresses this: it represents
the payload as a struct xdr_buf and decodes it with
svcxdr_decode_opaque_payload(), which subsegments the source
stream in place rather than copying it.
Annotate the data member of writeargs with "pragma pages" and
replace the NFSPROC_WRITE entry in nfsd_procedures2 with the
generated nfs_svc_decode_writeargs and nfs_svc_encode_attrstat. A
wrapper structure bridges the generated writeargs type and the
svc_fh the NFSD VFS layer still uses; the result reuses the
attrstat wrapper from the GETATTR conversion.
The pc_argzero field is now set to zero for the NFSv2 WRITE
procedure. The generated decoder initializes every argument in
the argp->xdrgen field, making the early defensive memset
unnecessary.
This refactor replaces the use of svcxdr_encode_fattr(), so the
reference to the file handle can be released directly by
nfsd_proc_write(). A separate ->pc_release callback is thus no
longer needed. This makes it straightforward to confirm that the
file handle is properly released during every error flow.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260916162859.2051-17-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
Replace the hand-written NFSv2 READ codec pair with the
xdrgen-generated functions. The NFSPROC_READ entry in the
nfsd_procedures2 array now dispatches nfs_svc_decode_readargs and
nfs_svc_encode_readres. A wrapper structure bridges the generated
xdrgen readargs type and the legacy svc_fh representation the NFSD
VFS layer still uses.
The file's data does not reside at the data pointer of the
readresok data member: nfsd_read() deposits it directly in the
pages of the Reply buffer, so the generic encoder that copies an
opaque from the member's data pointer cannot encode it. Mark the
member with the "pragma pages" directive so the generated encoder
calls svcxdr_encode_opaque_payload(), which encodes the length
prefix and inserts the payload pages into the stream by reference.
The page anchor that the result structure carried duplicated
rq_res.pages -- svc_process() points rq_res.pages at the first
Reply page before dispatch, and the readargs decoder consumes no
pages -- so the helper locates the payload through the Reply buffer
itself, and the wrapper collapses to the generated readres type.
The pc_argzero field is now set to zero for the NFSv2 READ
procedure. The xdrgen decoders are trusted to initialize all
arguments in the argp->xdrgen field, making the early defensive
memset unnecessary. The remaining argp fields are cleared as
needed.
Because the generated encoder no longer needs the svc_fh at encode
time, nfsd_proc_read() releases the file handle directly and no
separate ->pc_release callback is needed. This makes it
straightforward to confirm that the file handle is properly
released during every error flow.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260916162859.2051-15-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
Replace the hand-written NFSv2 READLINK codec pair with the
xdrgen-generated functions. The NFSPROC_READLINK entry in the
nfsd_procedures2 array now dispatches nfs_svc_decode_fhandle and
nfs_svc_encode_readlinkres. A wrapper structure bridges the
generated xdrgen fhandle type and the legacy svc_fh representation
the NFSD VFS layer still uses.
The link target does not reside at the data pointer of the
readlinkres union arm: nfsd_readlink() deposits it directly in a
page of the Reply buffer, so the generic encoder that copies an
opaque from the member's data pointer cannot encode it. Mark the
arm with the "pragma pages" directive so the generated encoder
calls svcxdr_encode_opaque_payload(), which encodes the length
prefix and inserts the payload page into the stream by reference.
The page anchor that the result structure carried duplicated
rq_res.pages -- svc_process() points rq_res.pages at the first
Reply page before dispatch, and the fhandle argument decoder
consumes no pages -- so the helper locates the payload through the
Reply buffer itself, and the wrapper collapses to the generated
readlinkres type.
The pc_argzero field is now set to zero for the NFSv2 READLINK
procedure. The xdrgen decoders are trusted to initialize all
arguments in the argp->xdrgen field, making the early defensive
memset unnecessary. The remaining argp fields are cleared as
needed.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260916162859.2051-14-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
Replace the NFSPROC_GETATTR entry in the nfsd_procedures2 array
with an entry that dispatches the xdrgen-generated
nfs_svc_decode_fhandle and nfs_svc_encode_attrstat. Wrapper
structures bridge the generated xdrgen types and the legacy
svc_fh and kstat representations the NFSD VFS layer still uses. A
few helper functions are introduced which will be used here and
in subsequent patches.
The pc_argzero field is now set to zero for the NFSv2 GETATTR
procedure. The xdrgen decoders are trusted to initialize all
arguments in the argp->xdrgen field, making the early defensive
memset unnecessary. The remaining argp fields are cleared as
needed.
This refactor replaces the use of svcxdr_encode_fattr(), so the
reference to the file handle can be released directly by
nfsd_proc_getattr(). A separate ->pc_release callback is thus no
longer needed. This makes it straightforward to confirm that the
file handle is properly released during every error flow.
The new nfsd_mode_to_ftype() utility no longer maps socket and
FIFO devices to non-standard values: NFSOCK=6 which is outside
the ftype enum, and NFCHR for FIFOs which is semantically wrong.
The Linux NFS client implementation does not seem to use or
depend on these non-standard values. The new utility function
returns NFNON for both, which is the RFC-compliant behavior.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260916162859.2051-10-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
NFSD's NFSv2 XDR codecs are hand-maintained; they are being converted
to xdrgen-generated code one RPC procedure at a time. The first step
is to generate and commit the xdrgen C code based on nfs2.x. The
generated code can be refreshed any time nfs2.x is modified by
running "(cd fs/nfsd && make xdrgen)".
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260916162859.2051-4-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
svc_bind() creates the local rpcbind client, and svc_register() then
makes one synchronous call for each program and version. Both run under
the caller's mutex. A caller that registers from userland needs neither.
Add sv_no_rpcbind to struct svc_serv, and fix up the code to honor it.
No caller sets the flag yet, so behaviour does not change.
Assisted-by: LLM
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260915-nfsd-norpcb-v3-1-3c60d49ade02@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
svc_rdma_recv_cid_init() initializes a Receive Queue completion ID,
but its kernel-doc heading names svc_rdma_send_cid_init(). The latter is
a separate helper immediately below it, so kernel-doc cannot match this
comment to the receive helper.
Fix the copied function name.
Fixes: 28ee0ec8948a ("svcrdma: De-duplicate completion ID initialization helpers")
Assisted-by: LLM
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Link: https://patch.msgid.link/20260909064343.90437-1-kmehltretter@gmail.com
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
The optional-data aggregate encoder loops cursor.count times, so the
begin hook must report the element count before the first element is
written. A producer that reads its elements while the reply is
encoded -- NFSv2 READDIR streaming directory entries straight into the
send buffer -- does not know that count in advance, and would have to
materialize the whole list just to satisfy the loop bound.
An optional-data list carries no count on the wire, so nothing forces
one on the encoder. Drive it from the item hook instead: a false
return ends the list, and the framing writes the terminator.
The counted-array form and its hook contract are unchanged. No
in-tree spec marks an optional-data member yet, so no generated code
changes here.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260908134234.512312-11-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
A generated variable-length array codec stages the wire array
as a C array and walks it. When the in-kernel object is not that
array, building that array is wasted memory traffic. The NFS_ACLv2
secattr aclent<> list is one such case: it is a Solaris wire form of
the kernel's posix_acl. The hand-rolled codec that performs that
transform forces its containing types to keep external linkage
(pragma public), and that linkage collides at link time with the
same generated types in a sibling program.
Add a per-member "pragma aggregate <struct> <member>" directive. For
a marked member the generated code owns only the wire framing and
calls hand-written begin, item, and end hooks that handle one
element at a time. Interior codec signatures do not change.
The framing-owned loop follows xdr_encode_array2(), whose only
remaining user is fs/nfs_common/nfsacl.c, and seq_file's
start/show/stop, where a guaranteed stop() releases what start()
took. Those APIs take an ops vtable because one loop serves
consumers chosen at run time. Here the consumer is known at
generation time, so the hooks bind by name at link time, as the
ASN.1 compiler's actions do. A missing hook is a link error, the
emitted prototypes enforce the element type, and no indirect call
is added per element.
The item hooks return only success or failure, with no early "no
more elements" return, so that the same hook contract can later
frame the RFC 4506 optional-data idiom, where the application
pushes entries into generated framing.
The front end resolves each marker against the specification and
reports one it cannot honor against its own source position, rather
than degrading silently to the array-walking codec or surfacing as a
traceback from whichever emitter reached it. Absent the marker the
generator emits byte-identical output.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260908134234.512312-9-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
The pages directive so far drives only the encoder. A bulk argument
such as the payload of an NFS WRITE arrives in the pages of the
server's Receive buffer, but the generated opaque decoder copies it
into a contiguous data pointer. The copy wastes memory bandwidth and
forces the containing procedure decoder to be hand-rolled.
Let the directive drive the decoder as well. Whether a marked member
is decoded in place or encoded by reference follows from whether its
containing type is reachable from an RPC procedure argument. Record
that set during parsing, then close it over nested aggregates and
typedef aliases before the directives are checked. A decoded
page-resident member is represented as a struct xdr_buf so that the
payload can be handed on without a copy. A marked union arm of an
argument type is rejected, since the union generator still emits
only the copying decoder. A marked type reachable from both an
argument and a result is rejected as well: the two representations
differ, and a type carries only one.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260908134234.512312-8-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
A bulk argument payload such as the content of an NFS WRITE request
arrives in the pages of the server's Receive buffer. Copying it into
a contiguous data pointer wastes memory bandwidth and defeats a
transport capable of direct data placement, so a server decodes such
an item in place: it reads the length prefix and captures the payload
octets by reference in an xdr_buf that later stages hand to the VFS.
svcxdr_encode_opaque_payload() already provides the encode-side
counterpart for a page-resident result payload. Add the decode-side
helper so a generated decoder can express the same in-place handling
for a page-resident argument.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260908134234.512312-7-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
The encoder functions that xdrgen emits handle a variable-length
opaque with xdr_stream_encode_opaque(), which copies the opaque
content into the RPC buffer. On the server, an NFS READ payload
already resides in the pages of rqstp->rq_res, placed there by an
underlying file system read. Copying that payload into the RPC
Reply buffer adds a data copy on every READ and prevents a
transport capable of direct data placement from conveying the
payload separately from the rest of the Reply.
Add a helper that encodes a page-resident opaque by reference: it
encodes the length, inserts the payload pages into the XDR stream,
then marks the byte range as a result payload.
Generated code calls this helper, so its parameters are limited to
the xdr_stream and the payload length. It recovers the svc_rqst
with svcxdr_rqst(), and takes the reply's page array from the
stream's buffer.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260908134234.512312-4-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
A generated XDR codec function receives only an xdr_stream, but
some codecs need the RPC transaction the stream belongs to: to
reach the procedure's argument and result structures, or to mark
a page-resident byte range as a result payload. A client stream
already records its transaction in the rqst field; a server
stream records nothing, so each generated codec that needs the
svc_rqst would have to store it in xdrgen_ctx itself, and every
such site would decide on its own what the pointer means.
Store the svc_rqst in xdrgen_ctx from svcxdr_init_decode() and
svcxdr_init_encode() instead, and read it back through
svcxdr_rqst() so that no codec open-codes the cast from void *.
A codec running on a stream either of those initialized reaches
the transaction, and through it rq_argp and rq_resp. Every other
initializer leaves xdrgen_ctx NULL, so svcxdr_rqst() returns NULL
there and the guarantee is scoped to those two entry points; a
codec that needs the svc_rqst names them in its Context: line.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260908134234.512312-3-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
A generated XDR codec function receives only an xdr_stream, yet
some codecs need state that the stream does not carry. Passing
that state as an additional argument changes the signature of
every function on the path from the program entry point down to
the one that consumes it, and for generated code it makes each
function's argument list depend on an analysis of the entire
specification.
Add xdrgen_ctx to struct xdr_stream, a pointer that the layer
initializing the stream sets aside for generated codecs to find,
following the precedent of seq_file->private and
netlink_callback->data. Every xdr_stream initializer clears it, so
a codec never reads a residual value left in a stream that was
allocated on the stack or taken from a transport's free pool.
Acked-by: Jeff Layton <jlayton@kernel.org>
Reviewed-by: NeilBrown <neil@brown.name>
Link: https://patch.msgid.link/20260908134234.512312-2-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
A caller that creates many listeners in one operation calls svc_register()
once for each of them. Every call waits for the local rpcbind on its own,
so a rpcbind that never answers costs the caller one timeout per listener.
The caller has no way to learn that the first call already failed.
Split rpcb_register_call() failures by whether rpcbind answered:
- answered: a FALSE reply, or an RPC-level rejection that
rpc_decode_header() derives from the reply. -EACCES (FALSE reply or
AUTH_ERROR), -EPROTONOSUPPORT, -EPFNOSUPPORT, -EOPNOTSUPP.
- never sent: -ENOMEM, -EMSGSIZE, -ERESTARTSYS.
- no answer: everything else, i.e. transport errors.
Any no-answer error gets represented by -EIO, which is already what the
RPC layer reports for most of it: rpc_check_timeout() returns -EIO for a
soft timeout without RPC_TASK_TIMEOUT, and call_status() documents -EIO
as "shutdown or soft timeout".
Keep a count of the number of rpcbind failures in the serv. Later
patches will use that to watch for hard rpcbind failures, and alter
their behavior accordingly.
Assisted-by: LLM
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260901-nfsd-nl-hang-v5-6-a540d86668b0@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
The xpt_reserved counter exists for UDP socket-buffer back-pressure.
svc_udp_has_wspace() is the only has_wspace implementation that
consults it, so on TCP and RDMA the counter is maintained and never
read. svc_handle_xprt() adds to it once per RPC. svc_reserve()
shrinks it again on each call from svc_process_common(), from
svc_xprt_release(), and from each proc function that calls
svc_reserve_auth(). Every shrinking call also runs
svc_xprt_resource_released(), which issues an smp_mb() and can
enqueue the transport.
Add an xcl_flags field to svc_xprt_class and set
SVC_XPRT_FLAG_WSPACE_RESERVE on the UDP class. Gate the xpt_reserved
accounting on that flag.
After the change, svc_reserve() no longer calls
svc_xprt_resource_released() on TCP and RDMA. Two paths still cover
that enqueue. svc_xprt_release() reaches the helper through
svc_xprt_release_slot(), and svc_xprt_received() enqueues a transport
whose XPT_DATA remains set.
Link: https://patch.msgid.link/20260828135036.796842-4-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
Pull RDMA updates from Jason Gunthorpe:
"About the normal size, still a lot of AI bug fixes and so on, but some
interesting new functionality too:
- Assorted locking, bounds-checking, cleanup, and error-path fixes
across UCMA/CMA, bng_re, bnxt_re, cxgb4, EFA, ERDMA, HFI1, HNS,
ionic, iRDMA, mlx4/mlx5, RXE, SIW, SRP/SRPT, and iSER target.
- netlink report for max # of supported resources
- get_zeroed_page()/etc removal
- Robust udata for ionic
- Allow unique RDMA device names per network namespace
- Completion counters and v2 admit queue support for EFA
- UC QP support for MANA
- Completion timestamps for ionic
- Harden uverbs data validation and resource lifetime handling,
fixing several core use-after-free conditions.
- bnxt_re toggle-page ownership and lifetime bug fixes
- dmabuf SRQ support for mlx5"
* tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma: (160 commits)
RDMA/ucma: Allow path records to exactly fit the output buffer
RDMA/uverbs: Guard legacy bundles without method_elm
RDMA/efa: Add support for 128B admin v2 SQ entry
RDMA/efa: Generalize the admin SQ
RDMA/efa: Decouple admin command payload from admin header
RDMA/rxe: Fix OOB in free_rd_atomic_resources()
RDMA/cma: Fix WARNING in res_to_rt
RDMA/cxgb4: Free debugfs on registration failure
RDMA/cxgb4: Cancel reg_work before freeing device on remove
RDMA/ucma: Lock the handler in ucma_set_ib_path()
RDMA/ucma: Lock the handler in ucma_write_cm_event()
RDMA/erdma: restrict the driver to little-endian systems
RDMA/ionic: Embed counter driver data in rdma_counter allocation
RDMA/ionic: Cap eq_count to the eth driver's interrupt vector budget
RDMA/siw: Fix use-after-free in siw_accept()
IB/isert: post the full-feature receive buffers after session registration
IB/isert: delay the final Login Response until the session is registered
RDMA/srp: fix heap information leak on a truncated SRP_CRED_REQ
RDMA/erdma: Hold QP references for AE and CM processing
RDMA/erdma: Hold CQ references when processing EQ events
...
|
|
Now that svc_seq_show() and the nfsd netlink stats handler both use
the per-netns svc_stat vs_count arrays, the global per-version
vs_count percpu counters are no longer read by anything. Remove the
vs_count field from struct svc_version and all the associated
DEFINE_PER_CPU_ALIGNED arrays and initializers across nfsd, lockd,
and the NFS client callback service.
Assisted-by: LLM
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260717-exportd-netlink-v7-4-b7ce17b83b60@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
The existing per-procedure call counts live in global
svc_version->vs_count[] arrays which are not network-namespace-aware.
Add per-netns equivalents in struct svc_stat so the upcoming netlink
stats interface can return namespace-scoped statistics.
Add a vs_count pointer array to struct svc_stat, along with
svc_stat_alloc_counts() and svc_stat_free_counts() helpers to manage
per-version percpu call count arrays.
Increment the per-net counter alongside the global one in
svc_generic_init_request(). Call the alloc/free helpers from
nfsd_net_init() and nfsd_net_exit().
Assisted-by: LLM
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260717-exportd-netlink-v7-1-b7ce17b83b60@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
Commit ae78eb497868 ("xdrgen: Implement short (16-bit) integer
types") taught the generator to emit XDR_short and
XDR_unsigned_short in the computed maxsize macros and added the
matching encode and decode primitives to _builtins.h, but it left
the two width macros themselves undefined in _defs.h.
Define XDR_short and XDR_unsigned_short, each one XDR unit wide, to
match the width the generator's maxsize table assigns them.
Fixes: ae78eb497868 ("xdrgen: Implement short (16-bit) integer types")
Link: https://patch.msgid.link/20260712193122.116845-5-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
The generated server-side decoder and encoder for a void procedure
argument or result are named after the RPC program (for example,
nfs_svc_decode_void). xdrgen derives that prefix from the program
name alone, not the version, so two versions of one program built
into the same module emit the identical symbol. NFSv2 and NFSv3
both declare program NFS_PROGRAM; once both are converted, fs/nfsd
fails to link with multiple definitions of nfs_svc_decode_void and
nfs_svc_encode_void.
A void handler carries no program- or version-specific behavior:
each merely forwards to xdrgen_decode_void() or xdrgen_encode_void().
Define one shared pair, xdrgen_svc_decode_void() and
xdrgen_svc_encode_void(), in the xdrgen builtins, and stop the
program generator from emitting a per-program void handler.
lockd is the one in-tree consumer that already emits per-program
void handlers, so regenerate the NLMv3 and NLMv4 XDR code to drop
nlm_svc_{decode,encode}_void() and nlm4_svc_{decode,encode}_void()
and point both procedure tables at the shared handlers. The shared
handlers are identical to the generated ones they replace, so no
wire behavior changes.
Only the server (svc) handlers are affected. The client-side void
stubs remain static and per-program, so they do not collide.
Link: https://patch.msgid.link/20260712193122.116845-3-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
Clean up.
The declaration templates for structs, pointers, and typedefs each
begin with a blank line, which keeps successive declarations and the
include block above them visually separated. The enum declaration
template omits that blank line. trim_blocks collapses the template's
lone comment line to nothing, so the omission stayed invisible as
long as every generated header happened to lead with a non-enum
declaration.
Fixes: 4329010ad9c3 ("xdrgen: Address some checkpatch whitespace complaints")
Link: https://patch.msgid.link/20260712193122.116845-2-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
Now that the pool mode is always pernode, svc_serv.sv_nrpools is
redundant with sv_is_pooled: an unpooled service always has a single
pool, and a pooled service has svc_pool_map.npools pools (which is one on
a single-node host). sv_nrpools cannot distinguish an unpooled service
from a pooled service that happens to have one pool, so it is sv_nrpools,
not sv_is_pooled, that carries no unique information.
Replace the cached field with a svc_serv_nrpools() helper that derives
the count from sv_is_pooled and the pool map, and convert all readers to
it. svc_pool_map is file-local to svc.c, so export the helper for the
svc_xprt.c and nfsd callers.
Reading svc_pool_map.npools without svc_pool_map_mutex is safe: the
mutex protects only svc_pool_map.count, and npools is already read
locklessly in svc_pool_for_cpu().
A pooled service holds a map reference for its whole lifetime, so npools
is stable while any reader could observe it. The hot path
(svc_pool_for_cpu()) already dereferences svc_pool_map for to_pool, and
npools shares that cacheline, so there is no new locking or coherence
cost.
__svc_create() keeps using its local npools argument for the sv_pools[]
allocation, since sv_is_pooled is not set until svc_create_pooled() has
returned from it.
Doing this also removes a modulus operation from svc_pool_for_cpu(),
which should make for more efficient RPC queueing.
Assisted-by: Claude:claude-opus-4-8
Suggested-by: NeilBrown <neilb@ownmail.net>
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260706-sunrpc-pool-mode-v5-5-6c4ee7cd89aa@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
The checked-in NLMv4 xdrgen output predates the addition of enum
value validation to generated decoders. As a result the decoders for
fsh4_mode, fsh4_access, and nlm4_stats still accept any 32-bit value,
while the current generator rejects values outside the enumeration.
Resync the generated files with the in-tree xdrgen by regenerating
from the unchanged nlm4.x specification.
This is a plain regeneration with no specification change; it also
refreshes the recorded specification modification time to show that
all existing enum decoders have picked up the xdrgen tool fix.
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260630155638.874492-1-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
RFC8881bis adds some new flags to GET_DIR_DELEGATION that later patches
will consume. In particular, Linux nfsd can't easily provide info about
directory cookies and ordering. The new flags allow it to omit that
information.
There is some risk here -- RFC8881bis is still a working group document,
and has been for years. The changes to directory delegations have been
stable for the last year or so however, so the hope is that those parts
won't change (much).
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260616-dir-deleg-v7-3-6cbc7eac0ade@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
Add the necessary bits to nfs4_1.x and remove the duplicate definitions
from nfs4.h and the uapi nfs4 header. Regenerate the xdr files.
Note that regenerating these files caused conflicts with the definitions
of NFS4_VERIFIER_SIZE and NFS4_FHSIZE in include/uapi/linux/nfs4.h.
These constants are defined by the RFC, and are not part of the kernel
API. They have been removed. Userspace consumers who require those
constants should plan to get them from more authoritative sources.
The nfsstat4 enum defined in the .x is fed to the xdrgen-generated wire
encoder and decoder, which treat every enumerated value as legal on the
wire. Do not carry the NFS4ERR_FIRST_FREE sentinel (which is not a
protocol error code) into the .x; keeping it would make 10097 a value
that could leak onto the wire. Instead base nfsd's internal error codes
(NFSERR_EOF and friends) at an impossible nfsstat4 value, as lockd does
for its nlm__int__* status codes.
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260616-dir-deleg-v7-2-6cbc7eac0ade@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
A pooled RPC service sizes its threads dynamically, growing and
shrinking each pool between its minimum and maximum bounds as load
varies. The count of running threads therefore reflects recent
demand, not the service's capacity. A consumer that sizes a data
structure against the concurrency the service can sustain -- NFSD's
NFSv4 session slot tables, for one -- needs that stable ceiling, and
computing it means summing sp_nrthrmax across every pool.
Add svc_serv_maxthreads() so the summation, and its dependence
on the layout of struct svc_serv and struct svc_pool, stays within
sunrpc. The read is lock-free: pool maxima change only when a service
is reconfigured, a path callers already serialize against startup and
shutdown, so a racing reader observes at worst a transient value. This
is acceptable for the sizing heuristics that will consume it.
nfsd_nrthreads() already sums sp_nrthrmax across pools by hand; convert
it to svc_serv_maxthreads(), giving the new export an in-tree consumer
and removing a copy of the dependence on svc_serv internals.
Reviewed-by: NeilBrown <neil@brown.name>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Reviewed-by: Benjamin Coddington <bcodding@hammerspace.com>
Link: https://patch.msgid.link/20260610-nfsd-slot-growth-clamp-v1-1-7b966700df0b@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
|
|
A backchannel receive can complete a request while the NFS callback
service is being torn down. xprt_complete_bc_request() removes the
request from bc_pa_list, drops bc_alloc_count, marks the request in use,
and then asks xprt_enqueue_bc_request() to hand it to the callback
service.
If teardown has already cleared xprt->bc_serv, xprt_enqueue_bc_request()
currently returns without enqueueing or freeing the committed request.
The xprt_get() taken on entry is leaked as well. If the producer wins
the race before bc_serv is cleared, it can also enqueue onto sv_cb_list
after nfs_callback_down() has stopped the callback threads, leaving the
request linked to a svc_serv that is about to be freed.
Close the producer side before callback threads are stopped. Add
xprt_svc_shutdown_bc() to clear xprt->bc_serv under bc_pa_lock, and call
it on callback shutdown and callback-start failure before stopping the
service threads. Requests that lose the NULL transition in
xprt_enqueue_bc_request() are released through the normal backchannel
free path after balancing bc_slot_count. Finally, drain any remaining
sv_cb_list requests after the callback threads have stopped and before
svc_destroy() frees the service.
Fixes: 441244d4273a ("SUNRPC: cleanup common code in backchannel request")
Fixes: 9e9fdd0ad0fb ("NFSv4.1: protect destroying and nullifying bc_serv structure")
Cc: stable@vger.kernel.org
Signed-off-by: Chris Mason <clm@meta.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260528-tier2-v1-6-d026a1415e0b@oracle.com
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
Read chunk position and length validation is currently scattered
across three consumer functions: svc_rdma_read_data_item(),
svc_rdma_read_multiple_chunks(), and svc_rdma_read_call_chunk().
Each independently guards against the same class of unsigned
arithmetic underflow from untrusted wire values. Any new consumer
of the parsed Read chunk list must replicate these checks or risk
re-introducing the defects fixed by earlier patches in this series.
Add pcl_check_read_chunk_positions() to consolidate position and
length validation into a single post-decode pass, called from
svc_rdma_xdr_decode_req() after all three chunk lists have been
parsed and the inline body length is known. The pass verifies
three properties:
- Each Read chunk's inline-body offset (its unreduced-stream
position minus the cumulative length of preceding Read chunks)
falls within the inline body length, or within the Call chunk
length for interleaved reads.
- Adjacent Read chunk positions do not overlap: cumulative read
bytes at each transition do not exceed the next position.
- Each chunk length does not exceed the receive context's page
budget.
Malformed frames are rejected before reaching any consumer. The
existing consumer-side guards remain as defense in depth.
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260526-rpc-kernel-bugs-v1-6-e251306ccca9@oracle.com
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
When a parsed chunk list contains a chunk whose ch_segcount is zero,
pcl_for_each_segment computes its inclusive upper bound as
&chunk->ch_segments[ch_segcount - 1]. ch_segcount is u32, so the
subtraction wraps to 0xFFFFFFFF and the bound lands far past the
ch_segments flex array. The loop body then walks unrelated memory at
sizeof(struct svc_rdma_segment) stride until it faults.
A zero-segcount chunk is reachable from the wire:
xdr_check_write_chunk() only rejects segcount values greater than
rc_maxpages, and pcl_alloc_write() links a freshly allocated chunk
onto rc_write_pcl/rc_reply_pcl before its segment-fill loop runs,
so a Write or Reply chunk advertising zero segments leaves
ch_segcount == 0 on the list. When the transport has negotiated
Send-With-Invalidate, svc_rdma_get_inv_rkey() iterates all four
PCLs with pcl_for_each_segment and dereferences segment->rs_handle
on each iteration, turning the underflow into an out-of-bounds read
and a general protection fault.
xdr_check_write_list / xdr_check_reply_chunk
pcl_alloc_write()
chunk = pcl_alloc_chunk(...) /* ch_segcount = 0 */
list_add_tail(&chunk->ch_list, &pcl->cl_chunks)
/* fill loop iterates zero times for wire segcount 0 */
svc_rdma_get_inv_rkey()
pcl_for_each_chunk(rc_write_pcl)
pcl_for_each_segment(segment, chunk)
pos <= &ch_segments[0u - 1u] /* 0xFFFFFFFF */
segment->rs_handle /* OOB read -> GPF */
Fix by switching the macro to a half-open upper bound that uses
ch_segcount directly. For ch_segcount == 0 the loop start equals the
loop end and the body is skipped; for ch_segcount > 0 the iteration
range is unchanged. All six existing call sites in
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c and
net/sunrpc/xprtrdma/svc_rdma_rw.c remain correct under the new bound,
so no caller changes are needed.
Fixes: 78147ca8b4a9 ("svcrdma: Add a "parsed chunk list" data structure")
Cc: stable@vger.kernel.org
Assisted-by: kres (claude-opus-4-7)
Signed-off-by: Chris Mason <clm@meta.com>
Acked-by: Jeff Layton <jlayton@kernel.org>
Link: https://patch.msgid.link/20260526-rpc-kernel-bugs-v1-4-e251306ccca9@oracle.com
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
New HW no longer implements a separate class for latency-sensitive QPs
and advertises this by a new cap bit.
Signed-off-by: Leon Romanovsky <leon@kernel.org>
|
|
The capability counter fields in struct ib_device_attr are declared
as signed int, but these values are inherently non-negative. Drivers
maintain their cached caps as u32 and assign them directly into these
int fields; if a cap exceeds INT_MAX the implicit narrowing yields a
negative value visible to the IB core.
Change the signed int capability fields to u32 to match the
underlying nature of the data. Also update consumers across the IB
core, ULPs, NVMe-oF target, RDS, and NFS/RDMA so the new u32 values
are not forced back through signed int or u8 via min()/min_t() or
narrowing local variables.
The nvmet-rdma consumer of max_srq clamps it against
ib_device.num_comp_vectors, which stays a signed int, so that site
uses min_t() instead of min() to handle the signed/unsigned mismatch.
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Erni Sri Satya Vennela <ernis@linux.microsoft.com>
Link: https://patch.msgid.link/20260709055211.2498307-1-ernis@linux.microsoft.com
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Acked-by: Stefan Metzmacher <metze@samba.org> # smbdirect
Signed-off-by: Leon Romanovsky <leon@kernel.org>
|
|
The TLS connect path has a use-after-free: nothing pins the
upper rpc_clnt across the delayed connect_worker. xs_connect()
stores task->tk_client in sock_xprt::clnt as a raw pointer
and queues the worker; for TLS-secured transports that worker
is xs_tcp_tls_setup_socket(), which reads several fields out
of the saved pointer (cl_timeout, cl_program, cl_prog,
cl_vers, cl_cred, cl_stats) to construct the args for the
inner handshake rpc_clnt.
The xprt does not reference the rpc_clnt; the rpc_clnt
references the xprt. xs_destroy() does cancel the
connect_worker, but it runs only when the xprt's refcount
drops to zero, which cannot happen until the rpc_clnt
releases its cl_xprt reference in rpc_free_client_work().
When a TLS handshake fails fatally (for example, an mTLS
mount whose client cert does not match the server), the
connecting task is woken with -EACCES and exits, the mount
caller invokes rpc_shutdown_client(), and the upper rpc_clnt
is freed before the queued connect_worker fires.
xs_tcp_tls_setup_socket() then dereferences the freed clnt,
producing the refcount_t underflow Michael Nemanov reported.
Take a reference on the upper rpc_clnt in xs_connect() for
TLS transports via a new rpc_hold_client() helper, and drop
it in the connect_worker's exit path with rpc_release_client().
The xprt_lock_connect() / xprt_unlock_connect() pairing
already serialises xs_connect() with xs_tcp_tls_setup_socket(),
so the take and release are balanced one-for-one.
The non-TLS connect worker (xs_tcp_setup_socket) never reads
sock_xprt::clnt, so leave that path alone and avoid the
clnt-holds-xprt-holds-clnt cycle that would otherwise prevent
xprt destruction.
Reported-by: Michael Nemanov <michael.nemanov@vastdata.com>
Closes: https://lore.kernel.org/linux-nfs/40e3d522-dfcf-4fc1-9c55-b5e81f1536d5@vastdata.com/
Fixes: 75eb6af7acdf ("SUNRPC: Add a TCP-with-TLS RPC transport class")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Tested-by: Michael Nemanov <michael.nemanov@vastdata.com>
Reviewed-by: Michael Nemanov <michael.nemanov@vastdata.com>
Signed-off-by: Anna Schumaker <anna.schumaker@hammerspace.com>
|
|
Threads parked in svc_rdma_sq_wait() on sc_sq_ticket_wait or
sc_send_wait can hang indefinitely in TASK_UNINTERRUPTIBLE state
across transport teardown, pinning svc_xprt references and
blocking svc_rdma_free().
The close path sets XPT_CLOSE before invoking xpo_detach and both
wait_event predicates include an XPT_CLOSE term, but the
predicates are re-evaluated only on wakeup. sc_sq_ticket_wait has
no completion-driven wake path; it is advanced solely by the
chained ticket handoff inside svc_rdma_sq_wait() itself. Without
an explicit wake at close, parked threads never observe
XPT_CLOSE, hold their svc_xprt_get reference forever, and
svc_rdma_free() blocks on xpt_ref dropping to zero.
Two close entry points reach this transport. Local teardown runs
svc_rdma_detach() from svc_handle_xprt() -> svc_delete_xprt() ->
xpo_detach() on a worker thread. A remote disconnect arrives at
svc_rdma_cma_handler(), which calls svc_xprt_deferred_close():
that sets XPT_CLOSE and enqueues the transport but does not
access either RDMA waitqueue, so a worker already parked in
svc_rdma_sq_wait() never re-evaluates its predicate. With every
worker parked on this transport, no thread is available to run
the local teardown either, and the wake site there is
unreachable.
Introduce svc_rdma_xprt_deferred_close(), a thin svcrdma wrapper
that calls svc_xprt_deferred_close() and then wakes both
sc_sq_ticket_wait and sc_send_wait. Convert the svcrdma producers
that called svc_xprt_deferred_close() directly:
svc_rdma_cma_handler(), qp_event_handler(),
svc_rdma_post_send_err(), svc_rdma_wc_send(), the sendto drop
path, the rw completion error paths, and the recvfrom flush and
read-list error paths.
Wake both waitqueues from svc_rdma_detach() as well. The
synchronous svc_xprt_close() path (backchannel ENOTCONN, device
removal via svc_rdma_xprt_done) reaches detach without flowing
through svc_xprt_deferred_close() and therefore does not invoke
the new helper.
Fixes: ccc89b9d1ed2 ("svcrdma: Add fair queuing for Send Queue access")
Cc: stable@vger.kernel.org
Assisted-by: kres (claude-opus-4-7)
Signed-off-by: Chris Mason <clm@meta.com>
[ cel: add svc_rdma_xprt_deferred_close() to complete the fix ]
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
xdr_buf_to_bvec() returns a slot count even when the caller's bvec
budget is exhausted partway through the xdr_buf. Callers feed that
count into iov_iter_bvec() and continue as if the conversion had
succeeded, silently sending or writing fewer bytes than the data
length declares. For an NFS WRITE the server reports the truncated
transfer to the client as full success.
The overflow represents an internal invariant violation: a higher
layer reserved a bvec budget too small for the xdr_buf it then
asked the encoder to convert. That is a server-side fault, not a
media I/O failure and not a malformed client argument.
Change xdr_buf_to_bvec() to return a signed int and have the
overflow label return -ESERVERFAULT. Update the three callers to
detect the negative return and fail the request: nfsd_vfs_write()
folds the error into host_err, which nfserrno() translates to
nfserr_serverfault for the WRITE reply; svc_udp_sendto() and
svc_tcp_sendmsg() propagate the error out of the send path.
Reported-by: Chris Mason <clm@meta.com>
Fixes: 2eb2b9358181 ("SUNRPC: Convert svc_tcp_sendmsg to use bio_vecs directly")
Cc: stable@vger.kernel.org
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
In order to generate source code to encode and decode NLMv3 protocol
elements, include a copy of the RPC language description of NLMv3
for xdrgen to process. The language description is derived from the
Open Group's XNFS specification:
https://pubs.opengroup.org/onlinepubs/9629799/chap10.htm#tagcjh_11_03
The C code committed here was generated from the new nlm3.x file
using tools/net/sunrpc/xdrgen/xdrgen.
The goals of replacing hand-written XDR functions with ones that
are tool-generated are to improve memory safety and make XDR
encoding and decoding less brittle to maintain. Parts of the
NFSv4 protocol are still being extended actively. Tool-generated
XDR code reduces the time it takes to get a working implementation
of new protocol elements.
The xdrgen utility derives both the type definitions and the
encode/decode functions directly from protocol specifications,
using names and symbols familiar to anyone who knows those specs.
Unlike hand-written code that can inadvertently diverge from the
specification, xdrgen guarantees that the generated code matches
the specification exactly.
We would eventually like xdrgen to generate Rust code as well,
making the conversion of the kernel's NFS stacks to use Rust just
a little easier for us.
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
Send completion currently queues a work item to an unbound
workqueue for each completed send context. Under load, the
Send Completion handlers contend for the shared workqueue
pool lock.
Replace the workqueue with a per-transport lock-free list
(llist). The Send completion handler appends the send_ctxt
to sc_send_release_list and does no further teardown. The
nfsd thread drains the list in xpo_release_ctxt between
RPCs, performing DMA unmapping, chunk I/O resource release,
and page release in a batch.
This eliminates both the workqueue pool lock and the DMA
unmap cost from the Send completion path. DMA unmapping can
be expensive when an IOMMU is present in strict mode, as
each unmap triggers a synchronous hardware IOTLB
invalidation. Moving it to the nfsd thread, where that
latency is harmless, avoids penalizing completion handler
throughput.
The nfsd threads absorb the release cost at a point where
the client is no longer waiting on a reply, and natural
batching amortizes the overhead when completions arrive
faster than RPCs complete.
A self-enqueue backstops drain on a quiescing transport.
When svc_rdma_send_ctxt_put() observes that its llist_add()
transitions sc_send_release_list from empty to non-empty,
it sets XPT_DATA and calls svc_xprt_enqueue() so that
svc_xprt_ready() schedules an nfsd thread. The thread
enters svc_rdma_recvfrom(), finds no pending receive,
clears XPT_DATA, and returns 0; svc_xprt_release() then
runs xpo_release_ctxt and drains the list. Under steady
load the foreground drain keeps the list non-empty between
adds and no enqueue fires; only the trailing edge of a
burst pays for a wakeup. Without this path, a Send
completion arriving after the last xpo_release_ctxt on an
idle connection would leave the send_ctxt's DMA mappings
and reply pages pinned until the next RPC, send-context
exhaustion, or transport close.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
Each RDMA Send completion triggers a cascade of work items on the
svcrdma_wq unbound workqueue:
ib_cq_poll_work (on ib_comp_wq, per-CPU)
-> svc_rdma_send_ctxt_put -> queue_work [work item 1]
-> svc_rdma_write_info_free -> queue_work [work item 2]
Every transition through queue_work contends on the unbound
pool's spinlock. Profiling an 8KB NFSv3 read/write workload
over RDMA shows about 4% of total CPU cycles spent on this
lock, with the cascading re-queue of write_info release
contributing roughly 1%.
The initial queue_work in svc_rdma_send_ctxt_put is needed to
move release work off the CQ completion context (which runs on
a per-CPU bound workqueue). However, once executing on
svcrdma_wq, there is no need to re-queue for each write_info
structure. svc_rdma_reply_chunk_release already calls
svc_rdma_cc_release inline from the same svcrdma_wq context,
and svc_rdma_recv_ctxt_put does the same from nfsd thread
context.
Release write chunk resources inline in
svc_rdma_write_info_free, removing the intermediate
svc_rdma_write_info_free_async work item and the wi_work
field from struct svc_rdma_write_info.
Reviewed-by: Mike Snitzer <snitzer@kernel.org>
Tested-by: Jonathan Flynn <jonathan.flynn@hammerspace.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
The migration to crypto/krb5 eliminated the per-enctype
function dispatch and direct crypto API usage, leaving
behind a number of orphaned definitions.
Remove the following from gss_krb5.h:
- GSS_KRB5_K5CLENGTH, used only by removed key derivation
- KG_TOK_MIC_MSG and KG_TOK_WRAP_MSG (Kerberos v1 token
types; v1 support was dropped earlier)
- KG2_TOK_INITIAL and KG2_TOK_RESPONSE (context
establishment token types; no remaining users)
- KG2_RESP_FLAG_ERROR and KG2_RESP_FLAG_DELEG_OK
- enum sgn_alg and enum seal_alg (v1 algorithm constants)
- All CKSUMTYPE_* definitions, now duplicated by
KRB5_CKSUMTYPE_* in <crypto/krb5.h>
- The KG_ error constants from gssapi_err_krb5.h, which
have no remaining users
- The ENCTYPE_* constant block, replaced by KRB5_ENCTYPE_*
from <crypto/krb5.h>
- KG_USAGE_SEAL/SIGN/SEQ (3DES usage constants)
- KEY_USAGE_SEED_CHECKSUM/ENCRYPTION/INTEGRITY, duplicated
by <crypto/krb5.h>
- #include <crypto/skcipher.h>, no longer needed
Remove the cksum[] field from struct krb5_ctx in
gss_krb5_internal.h; no code reads or writes it after the
key derivation removal.
Switch gss_krb5_enctypes[] in gss_krb5_mech.c to the
canonical KRB5_ENCTYPE_* names from <crypto/krb5.h>.
Remove stale #include directives:
- <crypto/skcipher.h> from gss_krb5_wrap.c
- <linux/random.h> and <linux/crypto.h> from
gss_krb5_seal.c
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Acked-by: Anna Schumaker <anna.schumaker@hammerspace.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
With all per-message crypto operations routed through crypto/krb5,
a substantial body of code in rpcsec_gss_krb5 has no remaining
callers. The internal key derivation functions (krb5_derive_key_v2,
krb5_kdf_hmac_sha2, krb5_kdf_feedback_cmac) and the low-level
crypto primitives (krb5_encrypt, gss_krb5_checksum, krb5_cbc_cts_
encrypt/decrypt, krb5_etm_checksum) are unreachable because their
only call sites were the per-enctype function pointers removed in
previous patches. Delete gss_krb5_keys.c entirely and strip the
dead functions from gss_krb5_crypto.c.
The KUnit test suite in gss_krb5_test.c exercised exactly these
internal functions: RFC 3961 n-fold, RFC 3962 key derivation,
RFC 6803 Camellia key derivation, and RFC 8009 AES-SHA2 key
derivation, plus encryption self-tests that drove the now-removed
encrypt routines. The corresponding test coverage is provided by
the crypto/krb5 selftests in crypto/krb5/selftest.c. Remove the
test file, the RPCSEC_GSS_KRB5_KUNIT_TEST Kconfig symbol, the
.kunitconfig, and all VISIBLE_IF_KUNIT / EXPORT_SYMBOL_IF_KUNIT
annotations.
xdr_process_buf() walked xdr_buf segments through a per-segment
callback and existed solely for the crypto routines in
gss_krb5_crypto.c. With that file removed, xdr_process_buf()
has no remaining callers. Its successor, xdr_buf_to_sg(),
populates a scatterlist directly from an xdr_buf byte range
and was introduced earlier in this series.
With every consumer of struct gss_krb5_enctype removed, replace
its remaining uses with the equivalent fields from struct
krb5_enctype (key_len). Remove struct gss_krb5_enctype, the
supported_gss_krb5_enctypes[] table, gss_krb5_lookup_enctype(),
and the gk5e pointer from krb5_ctx.
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Acked-by: Anna Schumaker <anna.schumaker@hammerspace.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
The crypto/krb5 library accepts data in scatterlist form, but
the GSS-API layer presents RPC payloads as struct xdr_buf.
Bridge that gap with a pair of helper functions:
xdr_buf_to_sg() - populate a caller-supplied scatterlist
array from a byte range
xdr_buf_to_sg_alloc() - populate a caller-supplied inline
scatterlist, chaining to a heap-
allocated overflow for large payloads
The inline array (typically stack-allocated at eight entries)
covers the common case of small RPCs with no heap allocation
on the encrypt/decrypt path. Only buffers spanning many pages
incur a kmalloc for the chained extension.
The segment-walking logic follows the same head, page array,
tail traversal as xdr_process_buf(), but populates a
scatterlist directly rather than invoking a per-segment
callback. sg_next() traversal makes the walker safe for
chained scatterlists. Once subsequent patches reroute all
per-message crypto operations through crypto/krb5,
xdr_process_buf() loses its last callers and is removed.
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Acked-by: Anna Schumaker <anna.schumaker@hammerspace.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
The auth.unix.ip and auth.unix.gid caches live in the sunrpc module,
so they cannot use the nfsd generic netlink family. Create a new
"sunrpc" generic netlink family with its own "exportd" multicast
group to support cache upcall notifications for sunrpc-resident
caches.
Define a YAML spec (sunrpc_cache.yaml) with a cache-type enum
(ip_map, unix_gid), a cache-notify multicast event, and the
corresponding uapi header.
Implement sunrpc_cache_notify() in cache.c, which checks for
listeners on the exportd multicast group, builds and sends a
SUNRPC_CMD_CACHE_NOTIFY message with the cache-type attribute.
Register/unregister the sunrpc_nl_family in init_sunrpc() and
cleanup_sunrpc().
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
Add sunrpc_cache_requests_count() and sunrpc_cache_requests_snapshot()
to allow callers to count and snapshot the pending upcall request list
without exposing struct cache_request outside of cache.c.
Both functions skip entries that no longer have CACHE_PENDING set.
The snapshot function takes a cache_get() reference on each item so the
caller can safely use them after the queue_lock is released.
These will be used by the nfsd generic netlink dumpit handler for
svc_export upcall requests.
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
A later patch will be changing the kernel to send a netlink notification
when there is a pending cache_request. Add a new cache_notify operation
to struct cache_detail for this purpose.
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
This function doesn't have anything to do with a timeout. The only
difference is that it warns if there are no listeners. Rename it to
sunrpc_cache_upcall_warn().
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
|
Since it will soon also send an upcall via netlink, if configured.
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
|
