xprtrdma: prevent disconnect feedback during endpoint teardown

xprt_autoclose() clears XPRT_CLOSE_WAIT before invoking the transport's
->close() operation. For RPC/RDMA, ->close() calls
rpcrdma_xprt_disconnect(), which disconnects the RDMA CM endpoint and
drains its QP.

Since commit 31e62d25b5b8 ("xprtrdma: Simplify RPC wake-ups on
connect"), RDMA_CM_EVENT_DISCONNECTED requests a SUNRPC transport
disconnect. Today that path, along with failed completion handling,
reaches xprt_force_disconnect() through rpcrdma_force_disconnect().

Commit c487eb7d8e41 ("xprtrdma: Clean up disconnect") later added
re_force_disconnect to ensure that multiple failure indications request
only one SUNRPC disconnect. However, the gate remains open when SUNRPC
itself begins endpoint teardown.

If re_force_disconnect is still clear, either a CM disconnect event or a
completion error can set XPRT_CLOSE_WAIT again while the current close
is in progress. When the transport send lock is subsequently released,
xprt_clear_locked() observes XPRT_CLOSE_WAIT and queues another
autoclose.

Requesting a disconnect this way is necessary when a connection is lost
unexpectedly, but not when the event was generated by
rpcrdma_xprt_disconnect() itself. In that case SUNRPC is already
committed to closing the transport.

Mark re_force_disconnect before initiating local endpoint teardown so
that CM events and completion errors caused by that teardown cannot
request another close. An unexpected error that wins the race before
local endpoint teardown begins can still request a disconnect normally.

Use a fully ordered exchange so that the gate is globally closed before
rdma_disconnect() can transition the QP to error and generate the
callbacks that the gate suppresses.

In 30-second traces of a sustained recovery loop, without this change
there were 7,693 xprt_force_disconnect() calls, 6,267 SUNRPC autocloses,
and 172,892 receive-side IB_WC_WR_FLUSH_ERR completions. With this
change, the same tracepoints recorded 3,512 xprt_force_disconnect()
calls, 3,212 autocloses, and 131,287 receive-side flush completions.
Thus QP teardown continues to produce flushed completions, but closing
the disconnect gate before teardown substantially reduces their
feedback into the SUNRPC close path.

Fixes: 31e62d25b5b8 ("xprtrdma: Simplify RPC wake-ups on connect")
Cc: stable@vger.kernel.org
Signed-off-by: Tim Menninger <tmenninger@everpuredata.com>
Reviewed-by: Chuck Lever <cel@kernel.org>
Signed-off-by: Anna Schumaker <anna.schumaker@hammerspace.com>

0 files changed, 0 insertions, 0 deletions