xsk: reclaim invalid Tx descriptors in ZC batch path

The zero-copy Tx batch parser stops when it encounters an invalid
descriptor. If this happens after one or more continuation descriptors,
the Tx consumer can be advanced past fragments that are neither submitted
to the driver nor returned to userspace through the completion ring.

A similar problem occurs when a packet exceeds xdp_zc_max_segs. The
descriptors consumed up to the limit are released without completion, and
the remaining continuation descriptors can subsequently be interpreted
as the beginning of another packet.

Parse Tx batches in packet units and distinguish descriptors belonging to
complete valid packets from descriptors consumed while draining an
invalid or oversized packet. Return the former to the driver and append
the latter to the CQ address area so userspace can reclaim their UMEM
frames.

Treat a standalone invalid descriptor as a one-descriptor reclaim-only
packet. Advancing the Tx-ring consumer releases the ring slot, but does
not by itself return ownership of the referenced UMEM frame to userspace.

Once draining starts, continue until the packet's end-of-packet
descriptor is consumed. Preserve the drain state on the socket when EOP
has not yet been supplied, so draining can continue during a later call.
Leave incomplete but otherwise valid packets on the Tx ring.

Shared-UMEM pools using multi-buffer Tx also need packet-framed parsing.
Walk their Tx sockets one packet at a time, preserving the existing
per-socket fairness scheme, instead of using the legacy one-descriptor
fallback. Keep that fallback for shared pools that do not use
multi-buffer Tx. Since the drain state is maintained per socket and both
the singular and shared paths can resume an interrupted drain, changing
the socket list from singular to shared requires no special bind-time
transition.

CQ entries are positional, and drivers may complete only part of the Tx
work returned by xsk_tx_peek_release_desc_batch(). Therefore, reclaim-only
entries cannot be published immediately when earlier driver-visible
descriptors are still outstanding.

Track the number of driver-visible CQ entries preceding the reclaim
entries. Let xsk_tx_completed() publish partial hardware Tx completions,
and publish the reclaim entries only after every earlier Tx descriptor
has completed. Complete a reclaim-only batch immediately when there is no
driver-visible work in front of it, and prevent another Tx batch from
being appended while reclaim entries remain pending.

Also cap batch processing by the size of the pool's temporary descriptor
array, as Tx rings belonging to sockets sharing a UMEM may have different
sizes.

This ensures that every invalid Tx descriptor consumed by the ZC batch
path is either submitted to the driver as part of a valid packet or
returned to userspace without violating CQ completion ordering.

Reviewed-by: Jason Xing <kernelxing@tencent.com>
Signed-off-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Fixes: cf24f5a5feea ("xsk: add support for AF_XDP multi-buffer on Tx path")
Link: https://patch.msgid.link/20260719135609.147823-5-maciej.fijalkowski@intel.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

1 files changed, 30 insertions, 24 deletions

diff --git a/Documentation/networking/af_xdp.rst b/Documentation/networking/af_xdp.rst
index 50d92084a49c..cc3f0d16b28f 100644
--- a/Documentation/networking/af_xdp.rst
+++ b/Documentation/networking/af_xdp.rst
@@ -43,12 +43,13 @@ UMEM also has two rings: the FILL ring and the COMPLETION ring. The
 FILL ring is used by the application to send down addr for the kernel
 to fill in with RX packet data. References to these frames will then
 appear in the RX ring once each packet has been received. The
-COMPLETION ring, on the other hand, contains frame addr that the
-kernel has transmitted completely and can now be used again by user
-space, for either TX or RX. Thus, the frame addrs appearing in the
-COMPLETION ring are addrs that were previously transmitted using the
-TX ring. In summary, the RX and FILL rings are used for the RX path
-and the TX and COMPLETION rings are used for the TX path.
+COMPLETION ring, on the other hand, contains frame addresses from Tx
+descriptors that the kernel has finished processing and that can now be
+used again by user space, for either Tx or Rx. This includes frames whose
+transmission has completed as well as frames referenced by invalid Tx
+descriptors rejected by the kernel. A completion therefore returns
+ownership of a frame to user space, but does not by itself guarantee that
+the packet was successfully transmitted.
 
 The socket is then finally bound with a bind() call to a device and a
 specific queue id on that device, and it is not until bind is
@@ -169,14 +170,15 @@ chunks mode, then the incoming addr will be left untouched.
 UMEM Completion Ring
 ~~~~~~~~~~~~~~~~~~~~
 
-The COMPLETION Ring is used transfer ownership of UMEM frames from
+The COMPLETION Ring is used to transfer ownership of UMEM frames from
 kernel-space to user-space. Just like the FILL ring, UMEM indices are
-used.
-
-Frames passed from the kernel to user-space are frames that has been
-sent (TX ring) and can be used by user-space again.
-
-The user application consumes UMEM addrs from this ring.
+used. Frames passed from the kernel to user-space are frames referenced
+by Tx descriptors that the kernel has finished processing and can be
+used by user-space again. This includes both frames whose transmission
+has completed and frames referenced by invalid Tx descriptors that were
+rejected and reclaimed by the kernel. A completion entry does not
+guarantee successful packet transmission. The user application consumes
+UMEM addrs from this ring.
 
 
 RX Ring
@@ -504,21 +506,25 @@ will be treated as an invalid descriptor.
 These are the semantics for producing packets onto AF_XDP Tx ring
 consisting of multiple frames:
 
-* When an invalid descriptor is found, all the other
-  descriptors/frames of this packet are marked as invalid and not
-  completed. The next descriptor is treated as the start of a new
-  packet, even if this was not the intent (because we cannot guess
-  the intent). As before, if your program is producing invalid
-  descriptors you have a bug that must be fixed.
+* When an invalid descriptor is found, the complete packet is treated as
+  invalid. The kernel consumes descriptors through the descriptor marking
+  the end of the packet and returns all their frame addresses through the
+  COMPLETION ring. A standalone invalid descriptor is treated as a
+  one-descriptor invalid packet. The descriptor following the end of the
+  invalid packet is treated as the start of a new packet. As before, if
+  your program is producing invalid descriptors you have a bug that must
+  be fixed. Rejected descriptors are reported in the ``tx_invalid_descs``
+  statistic.
 
 * Zero length descriptors are treated as invalid descriptors.
 
 * For copy mode, the maximum supported number of frames in a packet is
-  equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all
-  descriptors accumulated so far are dropped and treated as
-  invalid. To produce an application that will work on any system
-  regardless of this config setting, limit the number of frags to 18,
-  as the minimum value of the config is 17.
+  equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all descriptors
+  through the end of the oversized packet are consumed, treated as invalid,
+  and their frame addresses are returned through the COMPLETION ring. To
+  produce an application that will work on any system regardless of this
+  config setting, limit the number of frags to 18, as the minimum value of
+  the config is 17.
 
 * For zero-copy mode, the limit is up to what the NIC HW
   supports. Usually at least five on the NICs we have checked. We