| Age | Commit message (Collapse) | Author |
|---|
|
The exporter records an importer when it answers NPA-REQ, so it can send
NPA-REVOKE as soon as the BO is freed, before the importer has finished
building the dma-buf for that handle. The revoke handler assumes a fully
imported node: it dereferences imp_xa_node->dmabuf, which is still NULL
until the import completes, and drops the xarray reference the importing
thread still relies on. The importer then links the node and marks it
READY regardless, so the node can be freed while still on the per-remote
list.
Only tear down a node that is READY. A PENDING node is still being
imported and nothing has been handed to user-space yet, so only mark it
for teardown and send NPA-RELEASE. The importer checks for teardown under
the xarray lock before linking the node and marking it READY, and unwinds
otherwise. As NPA-REVOKE always follows NPA-RSP, a revoke that finds the
node NOT_READY is stale, and one that finds it in teardown hits a node
that is already being released, so both are ignored.
Fixes: 7cc82cd90d35 ("drm/amdgpu: Implement mechanism to revoke exported memory")
Assisted-by: LLM
Signed-off-by: David Carlier <devnexen@gmail.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
On the A+A VF the GPU reaches its HBM over the system memory path at a
system physical address, so page table entries covering VRAM have to
carry AMDGPU_PTE_SYSTEM and the addresses they hold have to be physical
rather than MC addresses.
Set the SYSTEM bit in the PDB0 entries and in amdgpu_ttm_tt_pde_flags(),
and convert the VRAM address in amdgpu_gmc_get_pde_for_bo() with
amdgpu_gmc_vram_mc2pa().
NPA mappings still require VSCT=0011 (PTE.S=0). Clear SYSTEM in
gmc_v12_1_get_npa_flags() and amdgpu_ualink_peer_remote_init() so NPA
PTEs do not inherit the VRAM SYSTEM bit.
v2:
- Use (sriov_vf || passthrough) and amdgpu_virt_vram_is_spm() for
PDB0/TTM SYSTEM bits and mc2pa.
v3:
- Rely on amdgpu_virt_vram_is_spm() without redundant VF/passthrough
checks.
v4:
- Explicitly clear AMDGPU_PTE_SYSTEM for NPA (VSCT=0011).
Signed-off-by: Zhigang Luo <zhigang.luo@amd.com>
Signed-off-by: Fangsheng Huang <Fangsheng.Huang@amd.com>
Signed-off-by: YuBiao Wang <YuBiao.Wang@amd.com>
Reviewed-by: Philip Yang <philip.yang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Use amdgpu_vm_bo_move() for updating the NPA BO memory accounting stats
when the NPA BO is moved out from NPA domain into an empty placement.
Without this, when the app finishes, it spits out a warning:
"VM memory stats for proc .. task .. is non-zero when fini"
This was only affecting the memory accounting stats, we were not leaking
any memory here.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Acked-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
The export cleanup worker checks amdgpu_ualink_check_conn_ready() once
when building npa_release_bitmap, but not again in the send loop. Cleanup
workers run concurrently on the unbounded NPA WQ (one per exported
handle), so when a peer's egress dies many workers have already seeded
npa_release_bitmap from a connection that was still ESTABLISHED and all
issue an LSDMA copy to the dead peer, flooding the log with repeated
NPA-REVOKE send failures.
Re-check the connection immediately before each send, as the tlb
shootdown, force_retry_rpcs and unmap_all_npa_addr loops already do, so
once the first worker marks a peer NOT_READY the rest skip it. With the
re-check in place, seed npa_release_bitmap with a single bitmap_copy() of
importers_bitmap instead of a separate build loop; peers that fail the
re-check or the send are dropped from the bitmap in the send loop.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
amdgpu_ualink_export_handle() tests robj->ualink_handle_lo and stores
the generated handle into it without any lock; xa_lock only covers
handle generation and the xarray insert.
Two threads can therefore both observe a zero handle for the same BO and
both allocate an exporter node, take an amdgpu_bo_ref() and insert into
exp_xa, with the last store to ualink_handle_lo winning. Teardown only
erases the handle recorded in the BO, so the losing node keeps its BO
reference and pinned VRAM until device teardown.
Each racing thread still generates its own handle and inserts its own
node, but the handle is published to the BO under the BO reservation
lock after re-checking that ualink_handle_lo is still zero. Exactly one
thread makes that transition; the others find a non-zero value, roll
back their node (erase from exp_xa, drop the DMABuf and BO references,
free it) and return -EAGAIN so the caller retries and picks up the
winning handle. This keeps a single exporter node per BO.
Fixes: c67cb19811ec ("drm/amdgpu: Implement UALink handle export")
Reported-by: Sashiko-bot <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260914054604.12850-1-kmehltretter@gmail.com?part=1
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
amdgpu_ualink_export_handle() takes an explicit amdgpu_bo_ref() before
building the exporter node, and the out: label only drops the reference
from drm_gem_object_lookup(). The xarray insert failure path unrefs the
BO, but the drm_gem_prime_handle_to_dmabuf() failure path does not.
Userspace can hit that path at will, for example by exhausting its file
descriptor table so the dmabuf export returns -EMFILE, leaking a BO
reference and its pinned VRAM on every attempt.
Fixes: c67cb19811ec ("drm/amdgpu: Implement UALink handle export")
Reported-by: Sashiko-bot <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260914054604.12850-1-kmehltretter@gmail.com?part=1
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
amdgpu_gem_ualink_handle_ioctl() dereferences adev->ualink.info before
validating it, but that structure is only allocated when the UALink IP
block is present, which is currently added only for GFX 12.1 only. The
ioctl is registered for every amdgpu device with DRM_RENDER_ALLOW, so
on any other ASIC a render node user can oops the kernel with a single
call.
Return -EOPNOTSUPP when the UALink software state is absent.
Fixes: 15d1c306d829 ("drm/amdgpu: Add ioctl infra for exporting/importing UALink handles")
Reported-by: Sashiko-bot <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260914054604.12850-1-kmehltretter@gmail.com?part=1
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
When one side of UALink nodes reboots, its ring wptr resets to 0 while
the peer keeps a stale value. Read the remote wptr and adopt it as the
local wptr and rptr to resync, before the first message is sent. Either
side can notice first, so do this both when setting up the connection
and when receiving the HELLO from the peer.
On the sdma path, read the wptr in a fence callback because the IB buffer
holding the result is recycled once the fence signals, so reading it
after dma_fence_wait_timeout() is too late. The lsdma path copies
synchronously, so it reads the wptr directly.
Take peer->lock in the callers (amdgpu_ualink_remote_interrupt and
amdgpu_ualink_remote_shootdown) instead of in amdgpu_ualink_send_command,
so the ring->ready check and the send are atomic against a concurrent
ring reset.
Move the rptr refresh and ring-full check out of amdgpu_ualink_send_command()
into amdgpu_ualink_ring_check_space(), so it runs under the caller's
peer->lock before the job is allocated.
On a failed NPA-REQ send, reset the connection like the NPA-RSP timeout path
already does, so the next attempt resyncs the ring wptr.
On connection reset, amdgpu_ualink_reset_peer_rings() only clears
ring->ready, so the writeback address is re-registered on the next send.
v10:
- Fix the resync missed on the receiving side: after a reboot the peer
may send HELLO before we set up the connection, so resync the ring
wptr in amdgpu_ualink_process_hello_msg() too.
v9:
- Fix the unnecessary connection reset after an app exits: the peer is
idle here but still busy handling NPA revokes on the remote, so the
HELLO reset a healthy connection. Only send it if the wptr changed.
- Resync both rings together under peer->lock, read both remote wptr
before updating either so a failed read changes nothing.
v8:
- Fix regression with concurrent import ioctls: do the idle check and
ring reset in amdgpu_ualink_setup_connection() under conn_state->lock,
and return -EAGAIN for IN_PROGRESS/PENDING first.
- Reset the peer rings only, the imp/exp xarray cleanup is not needed to
resync wptr.
v7:
- With v6 reboot one node of vPOD test passed on minirack.
- Use the per-remote imported and exported handle lists for the idle
check instead of walking the xarrays.
- Rename amdgpu_ualink_resync_peer_rings() to
amdgpu_ualink_reset_peer_rings(), it only clears ring->ready.
v6:
- Check idleness per remote accelerator instead of globally, and reset
the stale connection at the import ioctl so the peer re-runs the
HELLO handshake after a reboot.
v5:
- Resync the ring pointers at the import ioctl when the connection is
completely idle (no exported or imported BOs), so the first command
from a new application after a remote reboot/reset succeeds.
- Put the sdma wptr read fence callback on the stack instead of
allocating it; it is cancelled on timeout so it never outlives the
call.
v4:
- Cancel the sdma fence callback on the wait timeout/failure path so it
cannot write into the caller's stack after the function returns, and
free the callback state in one place.
v3:
- Resync both rings at connection setup instead of on the send path.
- Set the writeback rptr too, the stale value disagrees with the
resynced wptr until the remote FW writes it back.
v2:
- Match the FW change: resync wptr only on the first send or after a
timeout, instead of checking the remote periodically.
Suggested-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
amdgpu_generate_ualink_handle() continues when handle_lo is zero. For
accelerator ID 0, the random value can become zero after the low 10 bits
are cleared. This path reaches while (!unique) without assigning unique.
Clang's -Wconditional-uninitialized reports the read. A reduced
MemorySanitizer test reports use-of-uninitialized-value at the loop
condition.
If the uninitialized value ends the loop, handle 0 is inserted into
exp_xa while robj->ualink_handle_lo still says that the BO has no handle.
A later export can create another entry for the same BO.
Initialize unique to false so the zero-handle path repeats the loop.
Fixes: c67cb19811ec ("drm/amdgpu: Implement UALink handle export")
Assisted-by: LLM
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Commit 0a6da5dcfe42 ("drm/amdgpu: Improve ualink state transitions")
introduced an access to info->accel_state before checking whether info
is NULL.
Move the access after the NULL check to avoid a potential NULL pointer
dereference.
Fixes: 0a6da5dcfe42 ("drm/amdgpu: Improve ualink state transitions")
Reported-by: Dan Carpenter <error27@gmail.com>
Cc: Lijo Lazar <lijo.lazar@amd.com>
Cc: Felix Kuehling <felix.kuehling@amd.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
There is an include loop between amdgpu.h and amdgpu_sdma.h which
prevents adding any interesting static inline helpers to the latter.
Break it by moving the type definitions to a new amdgpu_sdma_types.h
header.
This is required so static inline helpers which need to dereference
struct amdgpu_device can be added to amdgpu_sdma.h.
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Timur Kristóf <timur.kristof@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
When a HELLO arrived on an already ESTABLISHED connection,
amdgpu_ualink_process_hello_msg() set the state to PENDING before
calling amdgpu_ualink_handle_connection_reset(). That function only
runs the imp/exp XA cleanup when it observes the state as ESTABLISHED,
so the premature change skipped the cleanup and left stale NPA
mappings and pinned BOs from the previous connection.
Leave the state as ESTABLISHED and let
amdgpu_ualink_handle_connection_reset() perform the transition, so the
cleanup runs.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Reviewed-by : Philip Yang <philip.yang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
If sending TLB shootdown to a remote GPU fails, reset the connection and
mark the connection as NOT_READY.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
On the LSDMA path amdgpu_ualink_send_command() ignored the return of
amdgpu_lsdma_copy_mem() and polled the remote completion for the full
timeout even when the copy failed. With a peer down, every NPA-REVOKE
send in amdgpu_ualink_exp_cleanup_worker() wasted ~2s, and the peer
stayed ESTABLISHED so each subsequent handle cleanup timed out again,
piling up past the 120s hung-task threshold.
Return immediately when an LSDMA copy fails, and mark the connection
NOT_READY on a failed revoke send so other cleanups skip the unreachable
peer via amdgpu_ualink_check_conn_ready().
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Add the ioctl infrastructure to support exporting and importing BOs
to facilitate NPA based memory sharing across GPUs in a rack scale
setup.
Proposed userspace:
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L971
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L1036
v2: Move the ioctl wire-up to the end of the series.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Horatio Zhang <hongkun.zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
amdgpu_ualink_map_npa_to_dmabuf() created the NPA import BO in the
CPU/SYSTEM domain and then moved it to NPA. SYSTEM is use_tt=true, so TTM
attaches a host-page ttm_tt that the null move to NPA leaves behind. On
dma-buf export, ttm_bo_populate() then fills that ttm_tt with system
pages, spiking host RAM by the entire NPA window size.
Create the BO directly in the NPA domain (use_tt=false) so no ttm_tt is
ever attached, and relocate it to the exact remote window.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
When two threads import the same UALINK handle concurrently, the first
xa_insert() succeeds and the second gets -EBUSY. Convert -EBUSY to
-EAGAIN so user-space retries instead of seeing a spurious error.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Mirror the read-only ualink info attributes on each secondary compute
partition as a per-partition ualink node, so a partition-scoped consumer
(e.g. a container that only sees its partition's device node) can read
the ualink identity and state. Partition 0 shares the primary device,
which already exposes that node, so it is skipped. An inactive partition
device won't be having any attributes listed under ualink node.
The per-partition attributes are served by thin wrappers that delegate
to the existing device-level info show functions. A reference on the
info kobject is held for the node's lifetime so it cannot be freed
while a partition still uses it.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Assisted-by: Claude (claude-opus-4.7)
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Deactivate accelerator from vpod during hw_fini sequence.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Hawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Move more IP version specific related things to ualink version file.
Use soc v1.0 IH client id defintion and define mpnht interrupt source
id in ivsrcid header.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Hawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
amdgpu_ualink_manager_stop() only called xa_destroy() on the exporter,
importer and handle-invalid xarrays, which frees the internal tree but
not the node objects. Entries still present at teardown thus leaked the
nodes, left NPA addresses mapped and kept exported BOs pinned.
Free the entries in all three xarrays explicitly, without exchanging any
NPA protocol messages (the F/W is already halted by
amdgpu_ualink_sw_fini()): mark connections down so the drained workers
and teardown take the message-free path, drain in-flight exporter
cleanup work to empty handle_invalid_xa, then free the remaining
importer and exporter entries before the NPA allocator is torn down.
Also factor the shared importer-node release into
amdgpu_ualink_release_imp_xa_node(), drop/drain stale remote interrupts
in the IRQ handler when the accelerator is not ACTIVE, and refuse outbound
messages in amdgpu_ualink_send_command() when not ACTIVE so an in-flight
cleanup worker cannot emit onto torn-down peer/SDMA state or block
waiting for a response that will never arrive.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
An already-ACTIVE accelerator with an unchanged vpod_id skipped
re-activation, so growing or shrinking a live vPod left staying GPUs
with links/GART built for the old member set. Detect membership changes
(owned snapshot vs firmware read-back) and bounce affected ACTIVE peers
once the integrity check passes. Firmware requires a full metadata
reload while halted, so a full deactivate/activate is needed rather than
per-peer deltas.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Add UALINK:-tagged dev_info/dev_warn logging along the vpod commit ->
update_accel_state -> integrity -> activate path so failing and -EAGAIN
peer-wait cases are visible without dynamic debug. Also fix a wrong
print arg (vpod->id -> vpod->addr_mode) in the invalid addr mode
message.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
remote->active_accel_bits aliased info->vpod.active_accel_bits, which
psp_ual_query_info() refreshes on every commit. Teardown
(peer_remote_fini / gart unmap) then iterated the new set instead of the
one sw_init mapped, orphaning drm_mm nodes in the shared GTT manager.
Make active_accel_bits an owned snapshot copied at sw_init time. Drop the
now-redundant not-initialised guard in metadata_npa_unmapping(), since the
earlier remote->ring_bo NULL-check already covers that case.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Philip Yang <philip.yang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
For updating the vpod configuration, use similar logic in both sideband
and inband paths.
In inband path, force the vpod id as invalid when ppod configuration is
done. Remove the redundant query call after station configuration as it
station configuration doesn't affect vpod configuration.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Keep state transitions under lock. Validate the ppod/vpod config or both
based on the state passed by ASP. When a config update is received, if
the GPU is already active on a vpod, local vpod gpu integrity check is
skipped to keep minimal disruption. A gpu removed from the vpod will get
the new vpod id as 0. A GPU is not expected to transition directly from a
valid/nonzero vpod id to another valid vpod id. It needs to be removed
from the existing vpod first.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Add handlers for ualink notifications from ASP. Pause/Resume
notifications are not handled currently.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Use __check_local_vpod_integrity and __check_vpod_info for vpod
validation checks.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Add explicit PPOD/VPOD states for clarity. Assign the state accordingly
as received from ASP during initialization.
UALink accel state transitions:
UNCONFIGURED -> PPOD_CONFIGURED -> VPOD_CONFIGURED -> READY -> ACTIVE
- PPOD: sysfs ppod commit/ ASP {UAL_CFG_PPOD}
- VPOD: sysfs vpod commit/ ASP {UAL_CFG_VPOD} | {UAL_CFG_STATION}
- READY: local vpod integrity + activate_accelerator()
- ACTIVE: all local gpus belonging to one vpod are configured and ready.
deactivate_accelerator(): READY -> PPOD_CONFIGURED
invalid vpod: -> ERROR
Query ualink info during late init phase. Add a local vpod integrity
check before activating devices in the vpod. Move the devices to active
state once integrity check passes.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Message handshake with NHT firmware is IP specific. Separate that into
IP specific file. Move ip block v1.0 definition also the version
specific file.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Manage ualink lifecycle as a separate IP block. For now, use GC version
to add specific ualink ip block.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Acked-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Separate out sw_init and hw_init part of ualink. ualink init errors
are ignored temporarily. Keep ualink sysfs init together with sysfs init
sequence of other IPs.
Signed-off-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
With latest POR, remote importers aren't guaranteed to support
instruction replay-on-fault. Therefore, we cannot support unpinned
exports or unpinned page-tables in the exporter mappings into NPA space.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
amdgpu_ualink_map_npa_addr() already uses vm->last_update as the fence
output for amdgpu_vm_update_range(), but the two unmap functions used a
local fence variable instead. This is incorrect because
amdgpu_vm_update_pdes() always commits into vm->last_update regardless
of the fence passed to update_range(). As a result the PDE update job
(submitted after the range update) would not wait before the TLB flush.
As a result, switch both amdgpu_ualink_unmap_npa_addr() and
amdgpu_ualink_unmap_all_npa_addr() to use vm->last_update fence.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Initialize fence to NULL in UALink NPA unmap to make sure we
don't end with a garbage value which then gets incorrectly
dereferenced inside amdgpu_vm_tlb_flush().
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Hawking Zhang <Hawking.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
The UALink soft ring overflows under bursts of interrupts from remote
GPUs. Size it at 16x the default soft ring to absorb the burst.
While here, read the source accel_id and context dwords before the
delegate path and log them, so the originating GPU is recorded for every
interrupt.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
At export time the driver code always builds a dma_buf for the BO
which is then shared for local imports.
During cleanup, if there are no importers, which is true for local imports,
we never free the dma_buf thus causing the memory leak.
Fix by dropping the dma_buf reference in the no-importers branch as well.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
The UALink exp_xa, imp_xa and handle_invalid_xa xarrays are only ever
accessed from process context: ioctls or workqueues. None of them run
in softirq or hardirq context, so initializing the Xarray with BH flag
is unnecessary.
Drop XA_FLAGS_LOCK_BH and use the default plain-spinlock locking
(xa_init() and the plain xa_* accessors).
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Initialize the list head at node creation so list_del_init() is safe
on a node that has not yet been added to imp_handles_list.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Processing of NPA-RELEASE messages can run concurrently with
sending of NPA-REVOKEs to all the importers. This can lead to
a race condition where the interrupt handler can signal npa_done
as soon as npa_release_bitmap becomes empty even though the cleanup
worker was still sending NPA-REVOKEs to all the importers. This would
prevent the cleanup worker thread to wait for all NPA-RELEASE messages
to come before freeing the NPA addresses.
Fix this by setting the importer_bitmap for all importers before sending
any NPA-REVOKE messages.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Always check the return value of amdgpu_ualink_reserve_npa_vm_and_bos.
It's not expected to fail when used in non-interruptible mode. But if it
fails anyway it calls drm_exec_fini. The caller must not call
amdgpu_ualink_unreserve_npa_vm_and_bos in this case to avoid a
double-free.
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
This is a temporary patch to always flush TLBs when mapping NPA
addresses.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Remote TLB shootdown is not working currently and hangs the driver.
As a result, temporarily prevent sending the Remote TLB shootdown
messages to remote GPUs for now.
Additionally, also temporarily set PTE.X = 0 for imported memory.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Introduce a debugfs-controlled bitmap, amdgpu_ualink_drop_msg_bitmap,
where each bit corresponds to a value from
enum AMDGPU_UALINK_PROTOCOL_MESSAGES (e.g. NPA-REQ, NPA-RSP, NPA-FAIL,
NPA-REVOKE, NPA-RELEASE).
When a bit is set, the next incoming message of that type is dropped
in amdgpu_ualink_process_irq() and the bit is atomically cleared via
test_and_clear_bit(), so subsequent incoming messages are processed
normally. This allows exercising the UALink connection reset and
recovery paths (NPA-RSP timeout on the importer, retransmit/teardown
on the exporter, etc.) by injecting a single packet loss from
userspace, e.g.:
# drop one NPA-REQ
echo 0x08 > /sys/kernel/debug/dri/0/amdgpu_ualink_drop_msg_bitmap
# drop one NPA-RSP
echo 0x10 > /sys/kernel/debug/dri/0/amdgpu_ualink_drop_msg_bitmap
The bitmap lives in struct amdgpu_ualink_mgr and is exposed via
debugfs_create_ulong() next to the existing amdgpu_ualink_test entry.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
When the remote GPU is part of the local_accels array (same physical
pod), bypass the HELLO/NPA-REQ messaging protocol and directly look up
the exported BO in the peer's exporter xarray. Export it as a DMA-buf
and return the fd to user-space without creating an importer xarray
entry.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Replace the stub implementation of TLB shootdown message with
the actual implementation.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Add interrupt handling to process the NPA remote interrupts.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
This patch adds connection reset handling. There can be
two case which signal connection reset:
1. Receiving a HELLO message from a remote GPU, when the
connection state is already setup, signals the remote
GPU underwent a reset.
2. If no response received for a NPA-REQ/NPA_REVOKE
message.
In either of the two case, we cleanup all exported and
imported ualink handles exchanged with the remote GPU.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Add implementation for cleaning up ualink handles imported
on the importer GPU and the corresponding xarray entries.
This includes cleaning up the NPA BO dmabuf and freeing the
NPA BO.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Add implementation for freeing up exporter GPU's UALink
handles and the corresponding xarray entries. Before freeing,
the exporter GPU would send the NPA-REVOKE message to all the
importers and wait for the NPA-RELEASE messages to come back
before freeing the xarray entry.
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
