summaryrefslogtreecommitdiff
path: root/drivers/gpu/drm/xe/xe_bo.c
AgeCommit message (Collapse)Author
2025-09-10drm/xe: Convert pinned suspend eviction for exhaustive evictionThomas Hellström
Pinned suspend eviction and preparation for eviction validates system memory for eviction buffers. Do that under a validation exclusive lock to avoid interfering with other processes validating system graphics memory. v2: - Avoid gotos from within xe_validation_guard(). - Adapt to signature change of xe_validation_guard(). Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250908101246.65025-14-thomas.hellstrom@linux.intel.com
2025-09-10drm/xe: Rework instances of variants of xe_bo_create_locked()Thomas Hellström
A common pattern is to create a locked bo, pin it without mapping and then unlock it. Add a function to do that, which internally uses xe_validation_guard(). With that we can remove xe_bo_create_locked_range() and add exhaustive eviction to stolen, pf_provision_vf_lmem and psmi_alloc_object. v4: - New patch after reorganization. v5: - Replace DRM_XE_GEM_CPU_CACHING_WB with 0. (CI) Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250908101246.65025-13-thomas.hellstrom@linux.intel.com
2025-09-10drm/xe: Convert xe_bo_create_pin_map() for exhaustive evictionThomas Hellström
Introduce an xe_bo_create_pin_map_novm() function that does not take the drm_exec paramenter to simplify the conversion of many callsites. For the rest, ensure that the same drm_exec context that was used for locking the vm is passed down to validation. Use xe_validation_guard() where appropriate. v2: - Avoid gotos from within xe_validation_guard(). (Matt Brost) - Break out the change to pf_provision_vf_lmem8 to a separate patch. - Adapt to signature change of xe_validation_guard(). Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250908101246.65025-12-thomas.hellstrom@linux.intel.com
2025-09-10drm/xe: Convert xe_bo_create_pin_map_at() for exhaustive evictionThomas Hellström
Most users of xe_bo_create_pin_map_at() and xe_bo_create_pin_map_at_aligned() are not using the vm parameter, and that simplifies conversion. Introduce an xe_bo_create_pin_map_at_novm() function and make the _aligned() version static. Use xe_validation_guard() for conversion. v2: - Adapt to signature change of xe_validation_guard(). (Matt Brost) - Fix up documentation. v4: - Postpone the change to i915_gem_stolen_insert_node_in_range() to a later patch. Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250908101246.65025-11-thomas.hellstrom@linux.intel.com
2025-09-10drm/xe: Rename ___xe_bo_create_locked()Thomas Hellström
Don't start external function names with underscores. Rename to xe_bo_init_locked(). Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250908101246.65025-10-thomas.hellstrom@linux.intel.com
2025-09-10drm/xe: Convert the CPU fault handler for exhaustive evictionThomas Hellström
The CPU fault handler may populate bos and migrate, and in doing so might interfere with other tasks validating. Rework the CPU fault handler completely into a fastpath and a slowpath. The fastpath trylocks only the validation lock in read-mode. If that fails, there's a fallback to the slowpath, where we do a full validation transaction. This mandates open-coding of bo locking, bo idling and bo populating, but we still call into TTM for fault finalizing. v2: - Rework the CPU fault handler to actually take part in the exhaustive eviction scheme (Matthew Brost). v3: - Don't return anything but VM_FAULT_RETRY if we've dropped the mmap_lock. Not even if a signal is pending. - Rebase on gpu_madvise() and split out fault migration. - Wait for idle after migration. - Check whether the resource manager uses tts to determine whether to map the tt or iomem. - Add a number of asserts. - Allow passing a ttm_operation_ctx to xe_bo_migrate() so that it's possible to try non-blocking migration. - Don't fall through to TTM on migration / population error Instead remove the gfp_retry_mayfail in mode 2 where we must succeed. (Matthew Brost) v5: - Don't allow faulting in the imported bo case (Matthew Brost) Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthews Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250908101246.65025-7-thomas.hellstrom@linux.intel.com
2025-09-10drm/xe: Convert xe_bo_create_user() for exhaustive evictionThomas Hellström
Use the xe_validation_guard() to convert xe_bo_create_user() for exhaustive eviction. v2: - Adapt to argument changes of xe_validation_guard() Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> #v1 Link: https://lore.kernel.org/r/20250908101246.65025-4-thomas.hellstrom@linux.intel.com
2025-09-10drm/xe: Pass down drm_exec context to validationThomas Hellström
We want all validation (potential backing store allocation) to be part of a drm_exec transaction. Therefore add a drm_exec pointer argument to xe_bo_validate() and ___xe_bo_create_locked(). Upcoming patches will deal with making all (or nearly all) calls to these functions part of a drm_exec transaction. In the meantime, define special values of the drm_exec pointer: XE_VALIDATION_UNIMPLEMENTED: Implementation of the drm_exec transaction has not been done yet. XE_VALIDATION_UNSUPPORTED: Some Middle-layers (dma-buf) doesn't allow the drm_exec context to be passed down to map_attachment where validation takes place. XE_VALIDATION_OPT_OUT: May be used only for kunit tests where exhaustive eviction isn't crucial and the ROI of converting those is very small. For XE_VALIDATION_UNIMPLEMENTED and XE_VALIDATION_OPT_OUT there is also a lockdep check that a drm_exec transaction can indeed start at the location where the macro is expanded. This is to encourage developers to take this into consideration early in the code development process. v2: - Fix xe_vm_set_validation_exec() imbalance. Add an assert that hopefully catches future instances of this (Matt Brost) v3: - Extend to psmi_alloc_object Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> #v3 Link: https://lore.kernel.org/r/20250908101246.65025-2-thomas.hellstrom@linux.intel.com
2025-09-09drm/xe: Attempt to bring bos back to VRAM after evictionThomas Hellström
VRAM+TT bos that are evicted from VRAM to TT may remain in TT also after a revalidation following eviction or suspend. This manifests itself as applications becoming sluggish after buffer objects get evicted or after a resume from suspend or hibernation. If the bo supports placement in both VRAM and TT, and we are on DGFX, mark the TT placement as fallback. This means that it is tried only after VRAM + eviction. This flaw has probably been present since the xe module was upstreamed but use a Fixes: commit below where backporting is likely to be simple. For earlier versions we need to open- code the fallback algorithm in the driver. v2: - Remove check for dgfx. (Matthew Auld) - Update the xe_dma_buf kunit test for the new strategy (CI) - Allow dma-buf to pin in current placement (CI) - Make xe_bo_validate() for pinned bos a NOP. Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/5995 Fixes: a78a8da51b36 ("drm/ttm: replace busy placement with flags v6") Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Auld <matthew.auld@intel.com> Cc: <stable@vger.kernel.org> # v6.9+ Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Auld <matthew.auld@intel.com> Link: https://lore.kernel.org/r/20250904160715.2613-2-thomas.hellstrom@linux.intel.com (cherry picked from commit cb3d7b3b46b799c96b54f8e8fe36794a55a77f0b) Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
2025-09-09drm/xe/vf: Use single check when calling VF CCS functionsMichal Wajdeczko
All xe_sriov_vf_ccs() functions but init() expect to be called when initialization was successful and CCS handling is ready. Update IS_VF_CCS_READY macro and use it as single entry guard. Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Link: https://lore.kernel.org/r/20250908123025.747-5-michal.wajdeczko@intel.com
2025-09-05drm/xe: Attempt to bring bos back to VRAM after evictionThomas Hellström
VRAM+TT bos that are evicted from VRAM to TT may remain in TT also after a revalidation following eviction or suspend. This manifests itself as applications becoming sluggish after buffer objects get evicted or after a resume from suspend or hibernation. If the bo supports placement in both VRAM and TT, and we are on DGFX, mark the TT placement as fallback. This means that it is tried only after VRAM + eviction. This flaw has probably been present since the xe module was upstreamed but use a Fixes: commit below where backporting is likely to be simple. For earlier versions we need to open- code the fallback algorithm in the driver. v2: - Remove check for dgfx. (Matthew Auld) - Update the xe_dma_buf kunit test for the new strategy (CI) - Allow dma-buf to pin in current placement (CI) - Make xe_bo_validate() for pinned bos a NOP. Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/5995 Fixes: a78a8da51b36 ("drm/ttm: replace busy placement with flags v6") Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Auld <matthew.auld@intel.com> Cc: <stable@vger.kernel.org> # v6.9+ Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Auld <matthew.auld@intel.com> Link: https://lore.kernel.org/r/20250904160715.2613-2-thomas.hellstrom@linux.intel.com
2025-09-05drm/xe: improve dma-resv handling for backup objectMatthew Auld
Since the dma-resv is shared we don't need to reserve and add a fence slot fence twice, plus no need to loop through the dependencies. Signed-off-by: Matthew Auld <matthew.auld@intel.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Jonathan Cavitt <jonathan.cavitt@intel.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Link: https://lore.kernel.org/r/20250829164715.720735-2-matthew.auld@intel.com
2025-09-02drm/xe: Fix incorrect migration of backed-up object to VRAMThomas Hellström
If an object is backed up to shmem it is incorrectly identified as not having valid data by the move code. This means moving to VRAM skips the -EMULTIHOP step and the bo is cleared. This causes all sorts of weird behaviour on DGFX if an already evicted object is targeted by the shrinker. Fix this by using ttm_tt_is_swapped() to identify backed-up objects. Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/5996 Fixes: 00c8efc3180f ("drm/xe: Add a shrinker for xe bos") Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Auld <matthew.auld@intel.com> Cc: <stable@vger.kernel.org> # v6.15+ Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Auld <matthew.auld@intel.com> Link: https://lore.kernel.org/r/20250828134837.5709-1-thomas.hellstrom@linux.intel.com (cherry picked from commit 1047bd82794a1eab64d643f196d09171ce983f44) Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
2025-08-29drm/xe: Fix incorrect migration of backed-up object to VRAMThomas Hellström
If an object is backed up to shmem it is incorrectly identified as not having valid data by the move code. This means moving to VRAM skips the -EMULTIHOP step and the bo is cleared. This causes all sorts of weird behaviour on DGFX if an already evicted object is targeted by the shrinker. Fix this by using ttm_tt_is_swapped() to identify backed-up objects. Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/5996 Fixes: 00c8efc3180f ("drm/xe: Add a shrinker for xe bos") Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Auld <matthew.auld@intel.com> Cc: <stable@vger.kernel.org> # v6.15+ Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Auld <matthew.auld@intel.com> Link: https://lore.kernel.org/r/20250828134837.5709-1-thomas.hellstrom@linux.intel.com
2025-08-26drm/xe: Don't trigger rebind on initial dma-buf validationMatthew Brost
On the first validate of an imported dma-buf (initial bind), the device has no GPU mappings, so a rebind is unnecessary. Rebinding here is harmful in multi-GPU setups and for VMs using preempt-fence mode, as it would evict in-flight GPU work. v2: - Drop dma_buf_validated, check for XE_PL_SYSTEM (Thomas) Fixes: dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs") Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Link: https://lore.kernel.org/r/20250825152841.3837378-1-matthew.brost@intel.com (cherry picked from commit ffdf968762e4fb3cdae54e811ec3525e67440a60) Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
2025-08-26drm/xe/vm: Don't pin the vm_resv during validationThomas Hellström
The pinning has the odd side-effect that unlocking *any* resv during validation triggers an "unlocking pinned lock" warning. Cc: Matthew Brost <matthew.brost@intel.com> Fixes: 5cc3325584c4 ("drm/xe: Rework eviction rejection of bound external bos") Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250821143045.106005-2-thomas.hellstrom@linux.intel.com (cherry picked from commit 0a51bf3e54dd8b77e6f1febbbb66def0660862d2) Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
2025-08-26drm/xe/bo: Update atomic_access attribute on madviseHimal Prasad Ghimiray
Update the bo_atomic_access based on user-provided input and determine the migration to smem during a CPU fault v2 (Matthew Brost) - Avoid cpu unmapping if bo is already in smem - check atomics on smem too for ioctl - Add comments v3 - Avoid migration in prefetch v4 (Matthew Brost) - make sanity check function bool - add assert for smem placement - fix doc v5 (Matthew Brost) - NACK atomic fault with DRM_XE_ATOMIC_CPU Cc: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250821173104.3030148-16-himal.prasad.ghimiray@intel.com Signed-off-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
2025-08-25drm/xe: Don't trigger rebind on initial dma-buf validationMatthew Brost
On the first validate of an imported dma-buf (initial bind), the device has no GPU mappings, so a rebind is unnecessary. Rebinding here is harmful in multi-GPU setups and for VMs using preempt-fence mode, as it would evict in-flight GPU work. v2: - Drop dma_buf_validated, check for XE_PL_SYSTEM (Thomas) Fixes: dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs") Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Link: https://lore.kernel.org/r/20250825152841.3837378-1-matthew.brost@intel.com
2025-08-25drm/xe/vm: Don't pin the vm_resv during validationThomas Hellström
The pinning has the odd side-effect that unlocking *any* resv during validation triggers an "unlocking pinned lock" warning. Cc: Matthew Brost <matthew.brost@intel.com> Fixes: 9d5558649f68 ("drm/xe: Rework eviction rejection of bound external bos") Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250821143045.106005-2-thomas.hellstrom@linux.intel.com
2025-08-06drm/xe/svm: Migrate folios when possibleFrancois Dugast
The DMA mapping can now correspond to a folio (order > 0), so move the iterator by the number of pages in the folio in order to migrate all pages at once. This requires forcing contiguous memory for SVM BOs, which greatly simplifies the code and enables 2MB device page support, allowing a major performance improvement. Negative effects like extra eviction are unlikely as SVM BOs have a maximal size of 2MB. v2: - Improve commit message (Matthew Brost) - Fix increment, chunk, assert match (Matthew Brost) Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250805140028.599361-7-francois.dugast@intel.com Signed-off-by: Francois Dugast <francois.dugast@intel.com>
2025-07-31drm/xe: Fix oops in xe_gem_fault when running core_hotunplug test.Maarten Lankhorst
I saw an oops in xe_gem_fault when running the xe-fast-feedback testlist against the realtime kernel without debug options enabled. The panic happens after core_hotunplug unbind-rebind finishes. Presumably what happens is that a process mmaps, unlocks because of the FAULT_FLAG_RETRY_NOWAIT logic, has no process memory left, causing ttm_bo_vm_dummy_page() to return VM_FAULT_NOPAGE, since there was nothing left to populate, and then oopses in "mem_type_is_vram(tbo->resource->mem_type)" because tbo->resource is NULL. It's convoluted, but fits the data and explains the oops after the test exits. Reviewed-by: Matthew Auld <matthew.auld@intel.com> Link: https://lore.kernel.org/r/20250715152057.23254-2-dev@lankhorst.se Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
2025-07-23drm/xe/vf: Attach and detach CCS copy commands with BOSatyanarayana K V P
Attach CCS read/write copy commands to BO for old and new mem types as NULL -> tt or system -> tt. Detach the CCS read/write copy commands from BO while deleting ttm bo from xe_ttm_bo_delete_mem_notify(). Signed-off-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Michał Winiarski <michal.winiarski@intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Signed-off-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250722120506.6483-3-satyanarayana.k.v.p@intel.com
2025-07-16drm/xe: Move struct xe_vram_region to a dedicated headerPiotr Piórkowski
Let's move the xe_vram_region structure to a new header dedicated to VRAM to improve modularity and avoid unnecessary dependencies when only VRAM-related structures are needed. v2: Fix build if CONFIG_DRM_XE_DEVMEM_MIRROR is enabled v3: Fix build if CONFIG_DRM_XE_DISPLAY is enabled v4: Move helper to get tile dpagemap to xe_svm.c Signed-off-by: Piotr Piórkowski <piotr.piorkowski@intel.com> Suggested-by: Jani Nikula <jani.nikula@intel.com> Reviewed-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> # rev3 Acked-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250714184818.89201-4-piotr.piorkowski@intel.com Signed-off-by: Lucas De Marchi <lucas.demarchi@intel.com>
2025-07-14drm/xe: Remove unused functionsDr. David Alan Gilbert
xe_bo_create_from_data() last use was removed in 2023 by commit 0e1a47fcabc8 ("drm/xe: Add a helper for DRM device-lifetime BO create") xe_rtp_match_first_gslice_fused_off() last use was removed in 2023 by commit 4e124151fcfc ("drm/xe/dg2: Drop pre-production workarounds") Remove them, and xe_dss_mask_empty whose last use was by xe_rtp_match_first_gslice_fused_off(). (Xe has a bunch ofother symbols that have been added but not used, given how new it is, I've left those, as opposed to these that had the code that used them removed). Reviewed-by: Lucas De Marchi <lucas.demarchi@intel.com> Signed-off-by: Dr. David Alan Gilbert <linux@treblig.org> Link: https://lore.kernel.org/r/20250713152531.219326-1-linux@treblig.org Signed-off-by: Lucas De Marchi <lucas.demarchi@intel.com>
2025-07-10drm/xe/bo: add GPU memory trace pointsJuston Li
Add TRACE_GPU_MEM tracepoints for tracking global GPU memory usage. These are required by VSR on Android 12+ for reporting GPU driver memory allocations. v5: - Drop process_mem tracking - Set the gpu_id field to dev->primary->index (Lucas, Tvrtko) - Formatting cleanup under 80 columns v3: - Use now configurable CONFIG_TRACE_GPU_MEM instead of adding a per-driver Kconfig (Lucas) v2: - Use u64 as preferred by checkpatch (Tvrtko) - Fix errors in comments/Kconfig description (Tvrtko) - drop redundant "CONFIG" in Kconfig Signed-off-by: Juston Li <justonli@chromium.org> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com> Reviewed-by: Lucas De Marchi <lucas.demarchi@intel.com> Link: https://lore.kernel.org/r/20250709192313.479336-2-lucas.demarchi@intel.com Signed-off-by: Lucas De Marchi <lucas.demarchi@intel.com>
2025-06-27drm/xe: Drop bo->sizeMatthew Brost
bo->size is redundant because the base GEM object already has a size field with the same value. Drop bo->size and use the base GEM object’s size instead. While at it, introduce xe_bo_size() to abstract the BO size. v2: - Fix typo in kernel doc (Ashutosh) - Fix kunit (CI) - Fix line wrap (Checkpatch) v3: - Fix sriov build (CI) v4: - Fix display build (CI) Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Ashutosh Dixit <ashutosh.dixit@intel.com> Link: https://lore.kernel.org/r/20250625144128.2827577-1-matthew.brost@intel.com
2025-06-09drm/xe: Add xe_ggtt_might_lockMaarten Lankhorst
Another requirement of hiding more of struct xe_ggtt. Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250505121924.921544-3-dev@lankhorst.se Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
2025-06-06drm/xe: don't store the xe device pointer inside xe_ttm_ttDave Airlie
This device pointer is nearly always available without storing an extra copy for each tt in the system. Just noticed this while reading over the xe shrinker code. Signed-off-by: Dave Airlie <airlied@redhat.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Signed-off-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250605062103.1234620-1-airlied@gmail.com
2025-06-02drm/xe/pxp: Use the correct define in the set_property_funcs arrayDaniele Ceraolo Spurio
The define of the extension type was accidentally used instead of the one of the property itself. They're both zero, so no functional issue, but we should use the correct define for code correctness. Fixes: 41a97c4a1294 ("drm/xe/pxp/uapi: Add API to mark a BO as using PXP") Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> Cc: John Harrison <John.C.Harrison@Intel.com> Reviewed-by: John Harrison <John.C.Harrison@Intel.com> Link: https://lore.kernel.org/r/20250522225401.3953243-6-daniele.ceraolospurio@intel.com
2025-06-02drm/xe: Rework eviction rejection of bound external bosThomas Hellström
For preempt_fence mode VM's we're rejecting eviction of shared bos during VM_BIND. However, since we do this in the move() callback, we're getting an eviction failure warning from TTM. The TTM callback intended for these things is eviction_valuable(). However, the latter doesn't pass in the struct ttm_operation_ctx needed to determine whether the caller needs this. Instead, attach the needed information to the vm under the vm->resv, until we've been able to update TTM to provide the needed information. And add sufficient lockdep checks to prevent misuse and races. v2: - Fix a copy-paste error in xe_vm_clear_validating() v3: - Fix kerneldoc errors. Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Fixes: 0af944f0e308 ("drm/xe: Reject BO eviction if BO is bound to current VM") Reviewed-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20250528164105.234718-1-thomas.hellstrom@linux.intel.com
2025-05-29drm/xe/bo: optimise CCS case for WB pagesMatthew Auld
Dealing with CCS state is significant on LNL+, where we end up clearing the compression state on every page alloc using the blitter for user buffers, including also saving and restoring it when moving between domains, plus we need to alloc extra pages to hold the raw CCS state for the save step. However all compression PAT modes, on platforms like LNL, also require coh_none, meaning that only WC memory can use compression in the first place. With this we can be sneaky and completely ignore CCS for WB buffers, which is likely the common case anyway. This would then skip all blitter moves/clears between sys <-> tt and then also means we can drop the extra CCS pages. This should be safe since there is no way to interact with the compression state (potentially uncleared) without using a PAT enabled index (which is rejected at bind), including if trying to be malicious and copy the raw CCS state from userpace, which should give back all zeroes if the src surface (indirect) is lacking compressed PAT index. Signed-off-by: Matthew Auld <matthew.auld@intel.com> Cc: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Reviewed-by: José Roberto de Souza <jose.souza@intel.com> Reviewed-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Link: https://lore.kernel.org/r/20250516153810.223530-2-matthew.auld@intel.com
2025-05-07drm/xe: Use copy_from_user() instead of __copy_from_user()Harish Chegondi
copy_from_user() has more checks and is more safer than __copy_from_user() Suggested-by: Kees Cook <kees@kernel.org> Signed-off-by: Harish Chegondi <harish.chegondi@intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Ashutosh Dixit <ashutosh.dixit@intel.com> Signed-off-by: Ashutosh Dixit <ashutosh.dixit@intel.com> Link: https://lore.kernel.org/r/acabf20aa8621c7bc8de09b1bffb8d14b5376484.1746126614.git.harish.chegondi@intel.com
2025-04-28Merge drm/drm-next into drm-xe-nextThomas Hellström
Additional backmerge to avoid excessive diffstats when sending PR. Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
2025-04-26Merge tag 'drm-xe-next-2025-04-17' of ↵Dave Airlie
https://gitlab.freedesktop.org/drm/xe/kernel into drm-next Core Changes: Fix drm_gpusvm kernel-doc (Lucas) Driver Changes: - Release guc ids before cancelling work (Tejas) - Remove a duplicated pc_start_call (Rodrigo) - Fix an incorrect assert in previous userptr fixes (Thomas) - Remove gen11 assertions and prefixes (Lucas) - Drop sentinels from arg to xe_rtp_process_to_src (Lucas) - Temporarily disable D3Cold on BMG (Rodrigo) - Fix MOCS debugfs LNCF readout (Tvrtko) - Some ring flush cleanups (Tvrtko) - Use unsigned int for alignment in fb pinning code (Tvrtko) - Retry and wait longer for GuC PC start (Rodrigo) - Recognize 3DSTATE_COARSE_PIXEL in LRC dumps (Matt Roper) - Remove reduntant check in xe_vm_create_ioctl() (Xin) - A bunch of SRIOV updates (Michal) - Add stats for SVM page-faults (Francois) - Fix an UAF (Harish) - Expose fan speed (Raag) - Fix exporting xe buffer objects multiple times (Tomasz) - Apply a workaround (Vinay) - Simplify pinned bo iteration (Thomas) - Remove an incorrect "static" keywork (Lucas) - Add support for separate firmware files on each GT (Lucas) - Survivability handling fixes (Lucas) - Allow to inject error in early probe (Lucas) - Fix unmet direct dependencies warning (Yue Haibing) - More error injection during probe (Francois) - Coding style fix (Maarten) - Additional stats support (Riana) - Add fault injection for xe_oa_alloc_regs (Nakshrtra) - Add a BMG PCI ID (Matt Roper) - Some SVM fixes and preliminary SVM multi-device work (Thomas) - Switch the migrate code from drm managed to dev managed (Aradhya) - Fix an out-of-bounds shift when invalidating TLB (Thomas) - Ensure fixed_slice_mode gets set after ccs_mode change (Niranjana) - Use local fence in error path of xe_migrate_clear (Matthew Brost) - More Workarounds (Julia) - Define sysfs_ops on all directories (Tejas) - Set power state to D3Cold during s2idle/s3 (Badal) - Devcoredump output fix (John) - Avoid plain 64-bit division (Arnd Bergmann) - Reword a debug message (John) - Don't print a hwconfig error message when forcing execlists (Stuart) - Restore an error code to avoid a smatch warning (Rodrigo) - Invalidate L3 read-only cachelines for geometry streams too (Kenneth) - Make PPHWSP size explicit in xe_gt_lrc_size() (Gustavo) - Add GT frequency events (Vinay) - Fix xe_pt_stage_bind_walk kerneldoc (Thomas) - Add a workaround (Aradhya) - Rework pinned save/restore (Matthew Auld, Matthew Brost) - Allow non-contig VRAM kernel BO (Matthew Auld) - Support non-contig VRAM provisioning for SRIOV (Matthew Auld) - Allow scratch-pages for unmapped parts of page-faulting VMs. (Oak) - Ensure XE_BO_FLAG_CPU_ADDR_MIRROR had a unique value (Matt Roper) - Fix taking an invalid lock on wedge (Lucas) - Configs and documentation for survivability mode (Riana) - Remove an unused macro (Shuicheng) - Work around a page-fault full error (Matt Brost) - Enable a SRIOV workaround (John) - Bump the recommended GuC version (John) - Allow to drop VRAM resizing (Lucas) - Don't expose privileged debugfs files if VF (Michal) - Don't show GGTT/LMEM debugfs files under media GT (Michal) - Adjust ring-buffer emission for maximum possible size (Tvrtko) - Fix notifier vs folio lock deadlock (Matthew Auld) - Stop relying on placement for dma-buf unmap Matthew Auld) Signed-off-by: Dave Airlie <airlied@redhat.com> From: Thomas Hellstrom <thomas.hellstrom@linux.intel.com> Link: https://lore.kernel.org/r/aADWaEFKVmxSnDLo@fedora
2025-04-24drm/xe: Update xe_ttm_access_memory to use GPU for non-visible accessMatthew Brost
Add migrate layer functions to access VRAM and update xe_ttm_access_memory to use for non-visible access and large (more than 16k) BO access. 8G devcoreump on BMG observed 3 minute CPU copy time vs. 3s GPU copy time. v4: - Fix non-page aligned accesses - Add support for small / unaligned access - Update commit message indicating migrate used for large accesses (Auld) - Fix warning in xe_res_cursor for non-zero offset v5: - Fix 32 bit build (CI) v6: - Rebase and use SVM migration copy functions v7: - Fix build error (CI) v8: - Remove ifdef around VRAM copy functions (CI) - Use break statement in dma unmmaping (Jonathan) - Use if/else rather than goto (Jonathan) - Use single return point (Jonathan) Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Jonathan Cavitt <jonathan.cavitt@intel.com> Link: https://lore.kernel.org/r/20250423171725.597955-3-matthew.brost@intel.com
2025-04-24drm/ttm/xe: drop unused force_alloc flagDave Airlie
This flag used to be used in the old memory tracking code, that code got migrated into the vmwgfx driver[1], and then got removed from the tree[2], but this piece got left behind. [1] f07069da6b4c ("drm/ttm: move memory accounting into vmwgfx v4") [2] 8aadeb8ad874 ("drm/vmwgfx: Remove the dedicated memory accounting") Cleanup the dead code. Reviewed-by: Matthew Brost <matthew.brost@intel.com> Signed-off-by: Dave Airlie <airlied@redhat.com>
2025-04-23drm/xe: handle pinned memory in PM notifierMatthew Auld
Userspace is still alive and kicking at this point so actually moving pinned stuff here is tricky. However, we can instead pre-allocate the backup storage upfront from the notifier, such that we scoop up as much as we can, and then leave the final .suspend() to do the actual copy (or allocate anything that we missed). That way the bulk of our allocations will hopefully be done outside the more restrictive .suspend(). We do need to be extra careful though, since the pinned handling can now race with PM notifier, like something becoming unpinned after we prepare it from the notifier. v2 (Thomas): - Fix kernel doc and drop the pin as soon as we are done with the restore, instead of deferring to later. Suggested-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Signed-off-by: Matthew Auld <matthew.auld@intel.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Link: https://lore.kernel.org/r/20250416150913.434369-8-matthew.auld@intel.com
2025-04-23drm/xe: share bo dma-resv with backup objectMatthew Auld
We end up needing to grab both locks together anyway and keep them held until we complete the copy or add the fence. Plus the backup_obj is short lived and tied to the parent object, so seems reasonable to share the same dma-resv. This will simplify the locking here, and in follow up patches. v2: - Hold reference to the parent bo to be sure the shared dma-resv can't go out of scope too soon. (Thomas) Signed-off-by: Matthew Auld <matthew.auld@intel.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Link: https://lore.kernel.org/r/20250416150913.434369-7-matthew.auld@intel.com
2025-04-04drm/xe: allow non-contig VRAM kernel BOMatthew Auld
If the kernel bo doesn't care about vmap(), either directly or indirectly with save/restore then we don't need to force contig for such buffers. Signed-off-by: Matthew Auld <matthew.auld@intel.com> Cc: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Link: https://lore.kernel.org/r/20250403102440.266113-15-matthew.auld@intel.com
2025-04-04drm/xe: unconditionally apply PINNED for pin_map()Matthew Auld
Some users apply PINNED and some don't when using pin_map(). The pin in pin_map() should imply PINNED so just unconditionally apply it and clean up all users. Signed-off-by: Matthew Auld <matthew.auld@intel.com> Cc: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Link: https://lore.kernel.org/r/20250403102440.266113-14-matthew.auld@intel.com
2025-04-04drm/xe: add XE_BO_FLAG_PINNED_LATE_RESTOREMatthew Auld
With the idea of having more pinned objects using the blitter engine where possible, during suspend/resume, mark the pinned objects which can be done during the late phase once submission/migration has been setup. Start out simple with lrc and page-tables from userspace. v2: - s/early_restore/late_restore; early restore was way too bold with too many places being impacted at once. v3: - Split late vs early into separate lists, to align with newly added apply-to-pinned infra. v4: - Rebase. v5: - Make sure we restore the late phase kernel_bo_present in igpu. Signed-off-by: Matthew Auld <matthew.auld@intel.com> Cc: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Link: https://lore.kernel.org/r/20250403102440.266113-13-matthew.auld@intel.com
2025-04-04drm/xe: Add XE_BO_FLAG_PINNED_NORESTOREMatthew Brost
Not all BOs need to be restored on resume / d3cold exit, add XE_BO_FLAG_PINNED_NO_RESTORE which skips restoring of BOs rather just allocates VRAM for the BO. This should slightly speedup resume / d3cold exit flows. Marking GuC ADS, GuC CT, GuC log, GuC PC, and SA as NORESTORE. v2: - s/WONTNEED/NORESTORE (Vivi) - Rebase on newly added g2g and backup object flow Signed-off-by: Matthew Brost <matthew.brost@intel.com> Signed-off-by: Matthew Auld <matthew.auld@intel.com> Cc: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com> Link: https://lore.kernel.org/r/20250403102440.266113-11-matthew.auld@intel.com
2025-04-04drm/xe: use backup object for pinned save/restoreMatthew Auld
Currently we move pinned objects, relying on the fact that the lpfn/fpfn will force the placement to occupy the same pages when restoring. However this then limits all such pinned objects to be contig underneath. In addition it is likely a little fragile moving pinned objects in the first place. Rather than moving such objects rather copy the page contents to a secondary system memory object, that way the VRAM pages never move and remain pinned. This also opens the door for eventually having non-contig pinned objects that can also be saved/restored using blitter. v2: - Make sure to drop the fence ref. - Handle NULL bo->migrate. v3: - Ensure we add the copy fence to the BOs, otherwise backup_obj can be freed before pipelined copy finishes. v4: - Rebase on newly added apply-to-pinned infra. Link: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/1182 Signed-off-by: Matthew Auld <matthew.auld@intel.com> Cc: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Satyanarayana K V P <satyanarayana.k.v.p@intel.com> Link: https://lore.kernel.org/r/20250403102440.266113-10-matthew.auld@intel.com
2025-03-27drm/xe: Make the PT code handle placement per PTE rather than per vma / rangeThomas Hellström
With SVM, ranges forwarded to the PT code for binding can, mostly due to races when migrating, point to both VRAM and system / foreign device memory. Make the PT code able to handle that by checking, for each PTE set up, whether it points to local VRAM or to system memory. v2: - Fix system memory GPU atomic access. v3: - Avoid the UAPI change. It needs more thought. Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com> Link: https://lore.kernel.org/r/20250326080551.40201-6-thomas.hellstrom@linux.intel.com
2025-03-27drm/xe/bo: Add a bo remove callbackThomas Hellström
On device unbind, migrate exported bos, including pagemap bos to system. This allows importers to take proper action without disruption. In particular, SVM clients on remote devices may continue as if nothing happened, and can chose a different placement. The evict_flags() placement is chosen in such a way that bos that aren't exported are purged. For pinned bos, we unmap DMA, but their pages are not freed yet since we can't be 100% sure they are not accessed. All pinned external bos (not just the VRAM ones) are put on the pinned.external list with this patch. But this only affects the xe_bo_pci_dev_remove_pinned() function since !VRAM bos are ignored by the suspend / resume functionality. As a follow-up we could look at removing the suspend / resume iteration over pinned external bos since we currently don't allow pinning external bos in VRAM, and other external bos don't need any special treatment at suspend / resume. v2: - Address review comments. (Matthew Auld). v3: - Don't introduce an external_evicted list (Matthew Auld) - Add a discussion around suspend / resume behaviour to the commit message. - Formatting fixes. v4: - Move dma-unmaps of pinned kernel bos to a dev managed callback to give subsystems using these bos a chance to clean them up. (Matthew Auld) Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Auld <matthew.auld@intel.com> Link: https://lore.kernel.org/r/20250326080551.40201-4-thomas.hellstrom@linux.intel.com
2025-03-11Merge tag 'drm-xe-next-2025-03-07' of ↵Dave Airlie
https://gitlab.freedesktop.org/drm/xe/kernel into drm-next UAPI Changes: - Expose per-engine activity via perf pmu (Riana, Lucas, Umesh) - Add support for EU stall sampling (Harish, Ashutosh) - Allow userspace to provide low latency hint for submission (Tejas) - GPU SVM and Xe SVM implementation (Matthew Brost) Cross-subsystem Changes: - devres handling for component drivers (Lucas) - Backmege drm-next to allow cross dependent change with i915 - GPU SVM and Xe SVM implementation (Matthew Brost) Core Changes: Driver Changes: - Fixes to userptr and missing validations (Matthew Auld, Thomas Hellström, Matthew Brost) - devcoredump typos and error handling improvement (Shuicheng) - Allow oa_exponent value of 0 (Umesh) - Finish moving device probe to devm (Lucas) - Fix race between submission restart and scheduled being freed (Tejas) - Fix counter overflows in gt_stats (Francois) - Refactor and add missing workarounds and tunings for pre-Xe2 platforms (Aradhya, Tvrtko) - Fix PXP locks interaction with exec queues being killed (Daniele) - Eliminate TIMESTAMP_OVERRIDE from xe (Matt Roper) - Change xe_gen_wa_oob to allow building on MacOS (Daniel Gomez) - New workarounds for Panther Lake (Tejas) - Fix VF resume errors (Satyanarayana) - Fix workaround infra skipping some workarounds dependent on engine initialization (Tvrtko) - Improve per-IP descriptors (Gustavo) - Add more error injections to probe sequence (Francois) Signed-off-by: Dave Airlie <airlied@redhat.com> From: Lucas De Marchi <lucas.demarchi@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/ilc5jvtyaoyi6woyhght5a6sw5jcluiojjueorcyxbynrcpcjp@mw2mi6rd6a7l
2025-03-07Merge tag 'drm-misc-next-2025-03-06' of ↵Dave Airlie
https://gitlab.freedesktop.org/drm/misc/kernel into drm-next drm-misc-next for v6.15: Cross-subsystem Changes: base: - component: Provide helper to query bound status fbdev: - fbtft: Remove access to page->index Core Changes: - Fix usage of logging macros in several places gem: - Add test function for imported dma-bufs and use it in core and helpers - Avoid struct drm_gem_object.import_attach tests: - Fix lockdep warnings ttm: - Add helpers for TTM shrinker Driver Changes: adp: - Add support for Apple Touch Bar displays on M1/M2 amdxdna: - Fix interrupt handling appletbdrm: - Add support for Apple Touch Bar displays on x86 bridge: - synopsys: Add HDMI audio support - ti-sn65dsi83: Support negative DE polarity ipu-v3: - Remove unused code nouveau: - Avoid multiple -Wflex-array-member-not-at-end warnings panthor: - Fix CS_STATUS_ defines - Improve locking rockchip: - analogix_dp: Add eDP support - lvds: Improve logging - vop2: Improve HDMI mode handling; Add support for RK3576 - Fix shutdown - Support rk3562-mali xe: - Use TTM shrinker Signed-off-by: Dave Airlie <airlied@redhat.com> From: Thomas Zimmermann <tzimmermann@suse.de> Link: https://patchwork.freedesktop.org/patch/msgid/20250306130700.GA485504@linux.fritz.box
2025-03-06drm/xe: Basic SVM BO evictionMatthew Brost
Wire xe_bo_move to GPU SVM migration via new helper xe_svm_bo_evict. v2: - Use xe_svm_bo_evict - Drop bo->range v3: - Kernel doc (Thomas) v4: - Add missing xe_bo.c code v5: - Add XE_BO_FLAG_CPU_ADDR_MIRROR flag in this patch (Thomas) - Add message on eviction failure v6: - Only compile if CONFIG_DRM_GPUSVM selected (CI, Lucas) Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-29-matthew.brost@intel.com
2025-03-06drm/xe/bo: Introduce xe_bo_put_asyncThomas Hellström
Introduce xe_bo_put_async to put a bo where the context is such that the bo destructor can't run due to lockdep problems or atomic context. If the put is the final put, freeing will be done from a work item. v5: - Kerenl doc for xe_bo_put_async (Thomas) v7: - Fix kernel doc (CI) Signed-off-by: Matthew Brost <matthew.brost@intel.com> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com> Tested-by: Gwan-gyeong Mun <gwan-gyeong.mun@intel.com> Reviewed-by: Gwan-gyeong Mun <gwan-gyeong.mun@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-6-matthew.brost@intel.com
2025-03-06drm/xe: Retry BO allocationMatthew Brost
TTM doesn't support fair eviction via WW locking, this mitigated in by using retry loops in exec and preempt rebind worker. Extend this retry loop to BO allocation. Once TTM supports fair eviction this patch can be reverted. v4: - Keep line break (Stuart) Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Gwan-gyeong Mun <gwan-gyeong.mun@intel.com> Reviewed-by: Stuart Summers <stuart.summers@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-2-matthew.brost@intel.com