| Age | Commit message (Collapse) | Author |
|---|
|
BarUser should not use 'gpu as a name for its lifetime; stick to a
generic 'a until we need pin-init selfref for this struct.
Reviewed-by: Alexandre Courbot <acourbot@nvidia.com>
Link: https://patch.msgid.link/20260926124405.604707-5-dakr@kernel.org
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
|
|
There is no need for a reference count here. BarUserAccess can just
borrow from BarUser.
Reviewed-by: Alexandre Courbot <acourbot@nvidia.com>
Link: https://patch.msgid.link/20260926124405.604707-4-dakr@kernel.org
[ Update stale comment in run_self_test(). - Danilo ]
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
|
|
Remove the unnecessary Arc around BarUser and get rid of
Result<impl PinInit<T>> in favor of impl PinInit<T, Error>.
Reviewed-by: Alexandre Courbot <acourbot@nvidia.com>
Link: https://patch.msgid.link/20260926124405.604707-3-dakr@kernel.org
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
|
|
Make Bar1 an alias for a borrowed pci::Bar in order to match the Bar0
type alias.
Reviewed-by: Alexandre Courbot <acourbot@nvidia.com>
Link: https://patch.msgid.link/20260926124405.604707-2-dakr@kernel.org
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
|
|
Add self-tests for BAR1 access during driver probe when
CONFIG_NOVA_CORE_SELFTESTS is enabled (default disabled). This results in
testing the Vmm, GPU buddy allocator and BAR1 region all of which should
function correctly for the tests to pass.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: use existing self-test runner and CONFIG_NOVA_CORE_SELFTESTS]
[ecourtney: update for mutable GpuMm, PRAMIN views and VramAddress raw API]
[ecourtney: restore the conditional dead_code expect]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-16-8dd5d4225d2e@nvidia.com
[acourbot: use vertical import style.]
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Add the BAR1 user interface for CPU access to GPU virtual memory through
the BAR1 aperture.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: map BAR1 in NovaCore, borrow it in Gpu and BarUser, no Devres]
[ecourtney: update for the VramAddress raw API and gsp_resources chipset]
[ecourtney: drop the owned GpuMm, pass it mutably through map and release]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-15-8dd5d4225d2e@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Add the page table mapping and unmapping API to the Virtual Memory
Manager, implementing a two-phase prepare/execute model suitable for
use both inside and outside the DMA fence signalling critical path.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: pass mutable GpuMm, replace window guards with scoped borrows]
[ecourtney: use current raw address helpers, drop stale expects]
[ecourtney: zero page table pages through one PRAMIN view]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-13-8dd5d4225d2e@nvidia.com
[acourbot: use vertical import style.]
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Add virtual address range tracking to the VMM using a maple tree
allocator. This enables contiguous virtual address range allocation
for mappings.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-12-8dd5d4225d2e@nvidia.com
[acourbot: use vertical import style.]
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Add the Virtual Memory Manager (VMM) infrastructure for GPU address
space management. Each Vmm instance manages a single address space
identified by its Page Directory Base (PDB) address, used for Channel,
BAR1 and BAR2 mappings.
Mapping APIs and virtual address range tracking are added in later
commits.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: pass mutable GpuMm to mapping lookup]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-11-8dd5d4225d2e@nvidia.com
[acourbot: use vertical import style.]
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Add the page table walker implementation that traverses the page table
hierarchy for both MMU v2 (5-level) and MMU v3 (6-level) to resolve
virtual addresses to physical addresses or find PTE locations.
Currently only v2 has been tested (nova-core currently boots pre-hopper)
with some initial preparatory work done for v3.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: take mutable GpuMm and Pramin, not a bound device and window]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-10-8dd5d4225d2e@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Introduce `MmuConfig`, the trait that ties the entry-operation traits
(`PteOps`, `PdeOps`, `DualPdeOps`) together with the version-specific
constants and helpers.
`MmuV2` and `MmuV3` are zero-sized marker structs that implement
`MmuConfig` for Turing/Ampere/Ada and Hopper/Blackwell respectively.
Dispatch is fully resolved at compile time through these markers, so
version-specific code is selected without runtime overhead and without
wrapper enums.
This enables version-agnostic page-table operations while keeping
version-specific implementation details encapsulated in the `ver2` and
`ver3` modules.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-9-8dd5d4225d2e@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Add page table entry and directory structures for MMU version 3 used by
Hopper and later GPUs. The `Pte`, `Pde`, and `DualPde` types each
implement the `PteOps`, `PdeOps`, and `DualPdeOps` traits introduced
earlier in the series, providing the version-agnostic API used by the
forthcoming page-table walker and mapper.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: update for the VramAddress raw API, apply rustfmt]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-8-8dd5d4225d2e@nvidia.com
[acourbot: use vertical import style.]
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Add page table entry and directory structures for MMU version 2 used by
Hopper and later GPUs. The `Pte`, `Pde`, and `DualPde` types each
implement the `PteOps`, `PdeOps`, and `DualPdeOps` traits introduced
earlier in the series, providing the version-agnostic API used by the
forthcoming page-table walker and mapper.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-7-8dd5d4225d2e@nvidia.com
[acourbot: use vertical import style.]
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Introduce a trait for 128-bit Dual Page Directory Entries. The
`read()`/`write()` helpers issue two 64-bit accesses through a
`Pramin` to load/store the 128-bit value.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: replace PraminWindow accesses with typed MMIO views]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-6-8dd5d4225d2e@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Introduce a trait for GPU Page Directory Entries
(PDEs). Default `read()`/`write()` helpers via a `Pramin` are
provided.
The forthcoming MMU v2, v3 PDE structs will each implement `PdeOps`,
allowing the later page-table walker and mapper to call PDE operations.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: replace PraminWindow accesses with typed MMIO views]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-5-8dd5d4225d2e@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Introduce a trait for GPU Page Table Entries (PTEs). New
`read()`/`write()` helpers are provided that go through a
`Pramin`).
The forthcoming MMU v2, v3 PTE structs will each implement `PteOps`,
allowing the later page-table walker and mapper to call PTE operations.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: replace PraminWindow accesses with typed MMIO views]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-4-8dd5d4225d2e@nvidia.com
[acourbot: use vertical import style.]
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Add common page table types shared between MMU v2 and v3. These types
are hardware-agnostic and used by both MMU versions.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-3-8dd5d4225d2e@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Extend GpuMm with the remaining two memory-management components:
- Buddy allocator for VRAM allocation.
- TLB manager for translation buffer operations.
PRAMIN was added in an earlier commit; this completes the centralized
ownership model with accessor methods for each component.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: rebase GpuMm for borrowed BAR0, pin Tlb, write regs directly]
[ecourtney: size the buddy from the first usable FB region, drop dev_info]
[ecourtney: update for the VramAddress raw API and typed register base]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260909-mmrebase-v1-2-8dd5d4225d2e@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Add self-tests for the PRAMIN aperture mechanism to verify correct
operation during GPU probe. The tests validate various alignment
requirements and corner cases.
The tests are default disabled and behind CONFIG_NOVA_CORE_SELFTESTS.
When enabled, tests run after GSP boot during probe.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: convert the tests to window_at(), macros, and the new types]
[ecourtney: cfg-gate the tests and expect(dead_code), not a runtime no-op]
[ecourtney: run the self-tests on all architectures, drop the chipset arg]
[ecourtney: test within a usable FB region, skip when none is large enough]
[ecourtney: report failures without failing probe, start banner at dev_dbg]
[ecourtney: removed the mm-specific Kconfig option]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260827-pramin-split-v3-10-24b24d7afc52@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
PRAMIN apertures are a crucial mechanism for direct CPU read/write to
VRAM. Add a `Pramin` manager whose `window_at()` returns a typed MMIO
view of VRAM through the 1 MiB PRAMIN aperture in BAR0, validating the
view against the VRAM region and repositioning the window as needed for
the accessed address.
A view borrows `Pramin` mutably, so the window cannot move while
the view is in use, and it inserts an ordering point on Drop.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: split the registers and HAL into the two preceding patches]
[ecourtney: rebase w.r.t. Bar0 lifetime changes and register projections]
[ecourtney: drop the window guard and mutex, use &mut self]
[ecourtney: position at init to avoid reads, reposition in window_offset]
[ecourtney: return typed MMIO views instead of read/write accessors]
[ecourtney: insert an ordering read when a view drops]
[ecourtney: declare the window location, drop the doc examples]
[ecourtney: add the copyright header, doc and naming cleanups]
[ecourtney: the pramin module is mm-internal]
Co-developed-by: Eliot Courtney <ecourtney@nvidia.com>
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260827-pramin-split-v3-6-24b24d7afc52@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
Positioning the PRAMIN window requires writing an architecture-specific
register: `NV_PBUS_BAR0_WINDOW` on Turing, Ampere and Ada, and
`NV_XAL_EP_BAR0_WINDOW` with a different field width on Hopper and on
Blackwell.
A `MmHal` trait with one implementation per hardware family hides the
register choice from the rest of the mm code, matching the layout of
the driver's other HALs.
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260827-pramin-split-v3-4-24b24d7afc52@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
|
PRAMIN apertures are a crucial mechanism for direct CPU read/write to
VRAM. Add the BAR0 window registers that position the PRAMIN aperture
on all supported GPU architectures: Turing, Ampere, Ada (via
`NV_PBUS_BAR0_WINDOW`), Hopper (via `gh100::NV_XAL_EP_BAR0_WINDOW`),
and Blackwell (via `gb100::NV_XAL_EP_BAR0_WINDOW`).
Hopper/Blackwell window-base registers are based on Eliot Courtney's
offlist reference patch.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[ecourtney: split the registers out of the PRAMIN patch into mm/regs.rs]
[ecourtney: drop the register read path]
[ecourtney: reword the message for the split, narrow visibility to mm]
[ecourtney: plain base fields, as the bitfield cast+shift patch is dropped]
[ecourtney: rename the target to VidMem, fix derives, redo the target docs]
[ecourtney: annotate the register base types]
Signed-off-by: Eliot Courtney <ecourtney@nvidia.com>
Acked-by: Danilo Krummrich <dakr@kernel.org>
Link: https://patch.msgid.link/20260827-pramin-split-v3-3-24b24d7afc52@nvidia.com
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
|
