NFSv4/flexfiles: only cancel I/O to a failed mirror instance

When an error causes the flexfiles driver to return a layout,
ff_layout_cancel_io() kills every in-flight RPC for the layout
segment, across all mirror instances.  Cancelled requests that had
already been transmitted to a healthy data server cannot be un-sent:
they complete on the data server after the client has sent its
LAYOUTRETURN, and the metadata server then observes writes to a file
for which no write layout is outstanding.

RFC 8881 Section 20.3.4 recommends that the client wait for the
response from in-process or in-flight READ, WRITE, or COMMIT
operations before returning the layout, and the machinery for that
wait already exists: the LAYOUTRETURN is deferred until every request
drops its layout segment reference, and requests that have not yet
been transmitted exit at RPC prepare time once the segment has been
invalidated.  Cancellation is only needed to avoid waiting forever on
a device that will never answer.

Pass the failed instance's device ID when marking the layout for
return, so that ff_layout_cancel_io() cancels only I/O directed at
the device we have given up on.  In-flight I/O to the remaining
healthy instances drains normally -- typically within a round trip --
before the LAYOUTRETURN is sent.  If a spared instance turns out to
be unresponsive, its requests fail with their own device error, and
the resulting layout return cancels its I/O in turn.

Layout recalls with clora_changed set, bulk returns, and layout
revocations continue to cancel I/O to every device, as do error
paths where no single failed device can be identified.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
Reviewed-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Signed-off-by: Anna Schumaker <anna.schumaker@hammerspace.com>

2 files changed, 8 insertions, 4 deletions

diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index d27e0adb2709..94cc324b591f 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -1594,7 +1594,8 @@ static void ff_layout_io_track_ds_error(struct pnfs_layout_segment *lseg,
 		fallthrough;
 	default:
 		pnfs_error_mark_layout_for_return(lseg->pls_layout->plh_inode,
-						  lseg, NULL);
+						  lseg,
+						  &mirror->dss[dss_id].devid);
 	}
 
 out:
@@ -2256,7 +2257,8 @@ out_failed:
 		 * FF_FLAGS_NO_IO_THRU_MDS: force fresh LAYOUTGET,
 		 * never fall through to MDS I/O.
 		 */
-		pnfs_error_mark_layout_for_return(hdr->inode, lseg, NULL);
+		pnfs_error_mark_layout_for_return(hdr->inode, lseg,
+						  &mirror->dss[dss_id].devid);
 		return PNFS_TRY_AGAIN;
 	}
 	trace_pnfs_mds_fallback_read_pagelist(hdr->inode,
@@ -2359,7 +2361,8 @@ out_failed:
 		 * FF_FLAGS_NO_IO_THRU_MDS: force fresh LAYOUTGET,
 		 * never fall through to MDS I/O.
 		 */
-		pnfs_error_mark_layout_for_return(hdr->inode, lseg, NULL);
+		pnfs_error_mark_layout_for_return(hdr->inode, lseg,
+						  &mirror->dss[dss_id].devid);
 		return PNFS_TRY_AGAIN;
 	}
 	trace_pnfs_mds_fallback_write_pagelist(hdr->inode,
diff --git a/fs/nfs/flexfilelayout/flexfilelayoutdev.c b/fs/nfs/flexfilelayout/flexfilelayoutdev.c
index 7bb0f2094e9d..6165c41fcf62 100644
--- a/fs/nfs/flexfilelayout/flexfilelayoutdev.c
+++ b/fs/nfs/flexfilelayout/flexfilelayoutdev.c
@@ -453,7 +453,8 @@ noconnect:
 				 opnum, GFP_NOIO);
 	ff_layout_send_layouterror(lseg);
 	if (opnum != OP_READ || !ff_layout_has_available_ds(lseg))
-		pnfs_error_mark_layout_for_return(ino, lseg, NULL);
+		pnfs_error_mark_layout_for_return(ino, lseg,
+						  &mirror->dss[dss_id].devid);
 	ds = ERR_PTR(status);
 out:
 	return ds;