| Age | Commit message (Collapse) | Author |
|---|
|
A child of a listener only left the listen.pending or listen.ready
list via smbdirect_socket_accept() or when the listener was
destroyed. So every connection that failed before it was accepted
(negotiate timeout, peer disconnect, invalid negotiate request, ...)
stayed on the list and counted against the backlog until the listener
was destroyed. After about backlog (10 for ksmbd) such connections
smbdirect_listen_connect_request() rejected every new connection with
-EBUSY, so an unauthenticated peer was able to permanently stop
ksmbd from accepting SMB-Direct connections.
Now a failed child moves itself to a new listen.orphaned list at the
end of smbdirect_socket_cleanup_work(), after the disconnect was
started, and queues listen.purge_orphaned_work, which releases it
with smbdirect_socket_release(), without holding the listener's
rdma_lock_handler() lock, in the same way smbdirect_socket_destroy()
does for the remaining children. Orphaned sockets still count against
the backlog until they are released.
sc->accept.listener is now only changed under listener->listen.lock
and whoever clears it is responsible for releasing the socket:
smbdirect_socket_accept(), purge_orphaned_work or the destroy of the
listener.
In order to make that work:
- a struct smbdirect_socket that is a listener is now freed via
kfree_rcu(), so that a child can dereference sc->accept.listener
under rcu_read_lock(), even if the listener is released concurrently.
- smbdirect_listen_connect_request() adds nsc to the listen.pending
list and sets nsc->accept.listener before it calls
smbdirect_accept_connect_request(), and always returns 0 so the
rdma_cm core keeps the connection id and nsc owns it. On a non-zero
return it schedules nsc's teardown with
smbdirect_socket_schedule_cleanup(), which orphans it off the
listener so that purge_orphaned_work releases it; nsc destroys its
own connection id with rdma_destroy_id() in that teardown. The
teardown has to be deferred because the rdma_cm core holds nsc's
id_priv->handler_mutex across the handler, so nsc->rdma.cm_id must
not be destroyed synchronously from here. That same handler_mutex
(and the listener's) also guarantees nsc can't go away under us,
so setting the listener before the call is safe and lets a failure
orphan nsc.
- smbdirect_accept_negotiate_recv_work() no longer moves a socket to
the ready list if it already failed or no longer belongs to the
listener. Every accepting socket has a listener, so if
sc->accept.listener is already cleared, it no longer sends a
negotiate response, as the socket is about to be released.
- smbdirect_socket_destroy() of the listener disables
purge_orphaned_work and also releases the orphaned sockets.
Fixes: dc691b91ad16 ("smb: smbdirect: introduce smbdirect_socket_{listen,accept}()")
Reported-by: Lee Seong Hyeon <ihopenre@gmail.com>
Closes: https://lore.kernel.org/linux-cifs/CANTrAmxL5sWU8Sn29JAGZvSq5PBB2OA+VKA0MHsZJMesagtfzA@mail.gmail.com/
Reported-by: Sashiko <sashiko-bot+sashiko@kernel.org>
Closes: https://sashiko.dev/#/patchset/cover.1791224972.git.metze%40samba.org
Closes: https://sashiko.dev/#/patchset/cover.1791229120.git.metze%40samba.org
Cc: Namjae Jeon <linkinjeon@kernel.org>
Cc: Paulo Alcantara <pc@manguebit.org>
Cc: Tom Talpey <tom@talpey.com>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>
|
|
smbdirect_socket_destroy_sync()
smbdirect_socket_destroy_sync() waited for RDMA_CM_EVENT_DISCONNECTED
after smbdirect_socket_cleanup_work() called rdma_disconnect(). That
can take very long, e.g. if the peer just disappeared, until the
rdma cm gives up, or forever if the event already happened, but the
status was overwritten afterwards.
This is not needed: smbdirect_socket_destroy() drains the qp under
rdma_lock_handler(), destroys it and calls rdma_destroy_id(), which
only waits for a currently running event handler and makes sure no
further events are delivered. The rest of the disconnect protocol
(DREQ/DREP and timewait for IB, abrupt close for iWarp) is handled
by the rdma core asynchronously. drivers/nvme/host/rdma.c also just
calls rdma_disconnect() and ib_drain_qp() before rdma_destroy_id().
So smbdirect_socket_destroy() now also accepts
SMBDIRECT_SOCKET_DISCONNECTING and changes the status to
SMBDIRECT_SOCKET_DISCONNECTED itself under rdma_lock_handler().
We could still get RDMA_CM_EVENT_DISCONNECTED in the small
windows between rdma_unlock_handler() and rdma_destroy_id(),
but in that case smbdirect_connection_rdma_event_handler()
is basically a no-op.
Fixes: 422a2436697d ("smb: smbdirect: introduce smbdirect_socket_destroy[_sync]()")
Cc: Namjae Jeon <linkinjeon@kernel.org>
Cc: Paulo Alcantara <pc@manguebit.org>
Cc: Tom Talpey <tom@talpey.com>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Cc: linux-rdma@vger.kernel.org
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>
|
|
smbdirect_socket_destroy() releases the listener's pending/ready child
sockets while still holding the listener's handler lock, the
&id_priv->handler_mutex taken via rdma_lock_handler(), not
sc->listen.lock, and before the listener's own rdma_destroy_id().
That ordering has one real consequence and one cosmetic one.
The real one: smbdirect_socket_release() drops the child's last
reference, which destroys the child's cm_id. Doing that before the
listener's rdma_destroy_id() lets _cma_cancel_listens(), running from
the listener's _destroy_id(), walk an already freed child id_priv,
which KASAN catches as a slab-use-after-free during listener shutdown:
[ 4758.909130] BUG: KASAN: slab-use-after-free in __mutex_lock+0x1469/0x1560
[ 4758.911450] Read of size 1 at addr ffff88821c381db4 by task ksmbd.control/1652
[ 4758.913262] Call Trace:
[ 4758.913267] <TASK>
[ 4758.913299] __mutex_lock+0x1469/0x1560
[ 4758.913408] _cma_cancel_listens+0x312/0x3b0
[ 4758.913413] _destroy_id+0x363/0xee0
[ 4758.913417] smbdirect_socket_destroy_sync+0x17d5/0x2440
[ 4758.913443] smbdirect_socket_release+0x124/0x230
[ 4758.913451] ksmbd_rdma_stop_listening+0x9f/0x190
[ 4758.913457] ksmbd_conn_transport_destroy+0x65/0x3c0
[ 4758.913463] kill_server_store+0x1fb/0x2b0
[ 4758.913501] kernfs_fop_write_iter+0x349/0x4d0
[ 4758.913507] vfs_write+0x5e7/0xc70
[ 4758.913528] ksys_write+0x12a/0x210
[ 4758.913541] do_syscall_64+0x135/0x460
[ 4758.913555] entry_SYSCALL_64_after_hwframe+0x77/0x7f
The cosmetic one: releasing a child recurses into
smbdirect_socket_destroy(), which takes the child's own
rdma_lock_handler() lock nested under the listener's. The listener's
and the child's cm_id are always different instances, so this cannot
deadlock for real; the CM core itself nests a new connection id's
handler_mutex under the listening id's in cma_ib_req_handler(). But
lockdep only sees one lock class, reports possible recursive locking,
and then disables itself, hiding real locking bugs for the rest of the
run:
[ 2424.579653] WARNING: possible recursive locking detected
[ 2424.581180] 7.1.0-next-20260623+ #89 Not tainted
[ 2424.582548] --------------------------------------------
[ 2424.584500] ksmbd.control/8854 is trying to acquire lock:
[ 2424.586817] ffff888102303c20 (&id_priv->handler_mutex){+.+.}-{4:4}, at: smbdirect_socket_destroy_sync+0xc39/0x2440
[ 2424.590590]
[ 2424.590590] but task is already holding lock:
[ 2424.591601] ffff888102046c20 (&id_priv->handler_mutex){+.+.}-{4:4}, at: smbdirect_socket_destroy_sync+0xc39/0x2440
[ 2424.594178]
[ 2424.594178] other info that might help us debug this:
[ 2424.596634] Possible unsafe locking scenario:
[ 2424.596634]
[ 2424.598841] CPU0
[ 2424.599765] ----
[ 2424.600695] lock(&id_priv->handler_mutex);
[ 2424.601836] lock(&id_priv->handler_mutex);
[ 2424.602590]
[ 2424.602590] *** DEADLOCK ***
[ 2424.602590]
[ 2424.604512] May be due to missing lock nesting notation
Splice the pending/ready children onto a local list under the
listener's listen.lock, while the handler lock is held so a concurrent
CM CONNECT_REQUEST cannot add more, but defer the actual
smbdirect_socket_release() calls until after the listener's cm_id has
been destroyed and its handler lock dropped. The children are
independent sockets whose teardown needs neither the listener's
handler lock nor its cm_id.
Found with ksmbdzzer [2], a KSMBD fuzzer that drives libFuzzer with a
kcov-dataflow [1] coverage vector: it folds each instrumented
comparison/argument's runtime operand value together with its PC (the
default arm mixes them as pc⊕val) so that a new operand value at a known
site counts as new coverage.
[1] https://lwn.net/Articles/1077606/
[2] https://github.com/yskzalloc/kcov-dataflow
Fixes: dc691b91ad16 ("smb: smbdirect: introduce smbdirect_socket_{listen,accept}()")
Signed-off-by: Yunseong Kim <yunseong.kim@est.tech>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>
|
|
__smbdirect_socket_schedule_cleanup() and smbdirect_socket_cleanup_work()
take sc->listen.lock and walk the listener's pending list, recursing
into smbdirect_socket_schedule_cleanup() for each child, and every
child takes its own listen.lock inside that walk.
This cannot deadlock for real: the nesting is strictly listener into
child, a child never listens, so the outer and the inner lock are
always different instances. lockdep only tracks lock classes, though;
it sees the same class acquired twice and reports a possible recursive
locking deadlock. This change therefore works around a lockdep
limitation rather than fixing a real hang, but the report is still
worth avoiding: lockdep disables itself after the first splat and then
hides real locking bugs for the rest of the run.
Only a socket that was a listener owns a populated listen.ready/pending
list; a child has empty lists and nothing to do in these blocks. Guard
both of them with sc->listen.backlog != -1, the "was a listener" marker
that smbdirect_socket_destroy() already uses: listen.backlog leaves its
initial -1 exactly once, when smbdirect_socket_listen() succeeds. The
alternative !sc->accept.listener test reads as "not a listener" while
meaning the opposite, and it is also true for an accepted child, whose
accept.listener has been cleared on hand-over. With the guard the walk
only runs for a listener and never nests a child's listen.lock under
it; a pending child stays on its listener's list for the free path
(smbdirect_socket_destroy) to reap.
[ 741.705044] WARNING: possible recursive locking detected
[ 741.705403] 7.1.0-next-20260623+ #75 Not tainted
[ 741.705695] --------------------------------------------
[ 741.706022] ksmbd.control/18502 is trying to acquire lock:
[ 741.706379] ffff888108d612f8 (&sc->listen.lock){....}-{3:3}, at: __smbdirect_socket_schedule_cleanup+0x719/0xd70
[ 741.707008]
[ 741.707008] but task is already holding lock:
[ 741.707396] ffff8881087642f8 (&sc->listen.lock){....}-{3:3}, at: __smbdirect_socket_schedule_cleanup+0x719/0xd70
[ 741.708025]
[ 741.708025] other info that might help us debug this:
[ 741.708448] Possible unsafe locking scenario:
[ 741.708448]
[ 741.708845] CPU0
[ 741.709016] ----
[ 741.709186] lock(&sc->listen.lock);
[ 741.709453] lock(&sc->listen.lock);
[ 741.709705]
[ 741.709705] *** DEADLOCK ***
[ 741.709705]
[ 741.710095] May be due to missing lock nesting notation
[ 741.710095]
[ 741.710663] 6 locks held by ksmbd.control/18502:
[ 741.710975] #0: ffff888109e51420 (sb_writers#7){.+.+}-{0:0}, at: vfs_write+0x1e7/0xc70
[ 741.711561] #1: ffff888126ec3880 (&of->mutex){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x1be/0x4d0
[ 741.712147] #2: ffff888102af17b0 (kn->active#45){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x205/0x4d0
[ 741.712803] #3: ffffffff85ad1e00 (ctrl_lock){+.+.}-{4:4}, at: kill_server_store+0x1e0/0x2b0
[ 741.713381] #4: ffffffff85ad41a0 (init_lock){+.+.}-{4:4}, at: ksmbd_conn_transport_destroy+0x5b/0x3c0
[ 741.713995] #5: ffff8881087642f8 (&sc->listen.lock){....}-{3:3}, at: __smbdirect_socket_schedule_cleanup+0x719/0xd70
[ 741.714736]
[ 741.714736] stack backtrace:
[ 741.715038] CPU: 4 UID: 0 PID: 18502 Comm: ksmbd.control Not tainted 7.1.0-next-20260623+ #75 PREEMPT(lazy)
[ 741.715043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014
[ 741.715046] Call Trace:
[ 741.715049] <TASK>
[ 741.715052] dump_stack_lvl+0x77/0xa0
[ 741.715058] print_deadlock_bug+0x279/0x290
[ 741.715065] __lock_acquire+0x272a/0x2e30
[ 741.715070] ? stack_trace_save+0xae/0x100
[ 741.715075] ? smb_direct_logging_vaprintf+0x1a0/0x230
[ 741.715079] ? __pfx_smb_direct_logging_vaprintf+0x10/0x10
[ 741.715082] ? __timer_delete+0x58/0x320
[ 741.715087] lock_acquire+0xd3/0x270
[ 741.715091] ? __smbdirect_socket_schedule_cleanup+0x719/0xd70
[ 741.715095] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 741.715099] _raw_spin_lock_irqsave+0x42/0x60
[ 741.715105] ? __smbdirect_socket_schedule_cleanup+0x719/0xd70
Note the two addresses above: ffff888108d612f8 is the child's lock,
ffff8881087642f8 the listener's, always distinct objects.
Found with ksmbdzzer [2], a KSMBD fuzzer that drives libFuzzer with a
kcov-dataflow [1] coverage vector: it folds each instrumented
comparison/argument's runtime operand value together with its PC (the
default arm mixes them as pc⊕val) so that a new operand value at a known
site counts as new coverage.
[1] https://lwn.net/Articles/1077606/
[2] https://github.com/yskzalloc/kcov-dataflow
Fixes: dc691b91ad16 ("smb: smbdirect: introduce smbdirect_socket_{listen,accept}()")
Signed-off-by: Yunseong Kim <yunseong.kim@est.tech>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>
|
|
This is a better solution than
EXPORT_SYMBOL_FOR_MODULES(__sym, "cifs,ksmbd") as it makes
it possible to rebuild smbdirect.ko against a
running kernel and then load the existing cifs.ko and ksmbd.ko
from the running kernel.
Suggested-by: Christoph Hellwig <hch@infradead.org>
Link: https://lore.kernel.org/linux-cifs/aehrPuY60VMcYGU8@infradead.org/
Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: Namjae Jeon <linkinjeon@kernel.org>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Steve French <stfrench@microsoft.com>
|
|
This also removes the smbdirect_ prefix from the files.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Link: https://lore.kernel.org/linux-cifs/CAHk-=whmue3PVi88K0UZLZO0at22QhQZ-yu+qO2TOKyZpGqecw@mail.gmail.com/
Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Acked-by: Namjae Jeon <linkinjeon@kernel.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Steve French <stfrench@microsoft.com>
|
