| Age | Commit message (Collapse) | Author |
|---|
|
* kvm-arm64/cca-7.4:
: \
: Preliminary changes that pave the way for CCA in a future revision
: of Linux, introducing some required abstractions (vcpu ops, S2 ops)
: and making some concepts common with pKVM.
:
: Patches courtesy of Suzuki Poulose.
: /
KVM: arm64: Prevent unsupported vcpu features for VM types
KVM: arm64: Mandate VGIC v3 for pKVM VMs and Realms
KVM: arm64: Abstract out memory abort handling
KVM: arm64: Use a local kvm pointer in kvm_handle_guest_abort()
KVM: arm64: Add VM specific callback for S2 MMU operations
KVM: arm64: Consolidate stage2 unmap range into kvm_stage2_unmap_range
KVM: arm64: Add vcpu load/put call backs for flavors
KVM: arm64: Refactor the vcpu_load to allow for VM specific callbacks
KVM: arm64: Don't call vcpu_set_pauth_traps for pKVM host
KVM: arm64: Track the type of VM in kvm_arch
KVM: arm64: Avoid including linux/kvm_host.h in kvm_pgtable.h
KVM: arm64: Include kvm_emulate.h in kvm/arm_psci.h
KVM: arm64: protected VM: Handle user writes to CNTVCT_EL0/CNTPCT_EL0
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
* kvm-arm64/misc-7.4:
: \
: Misc KVM/arm64 fixes for 7.4:
:
: - Correctly report SMCCC_VERSION and SMCCC_ARCH_FEATURES as implemented
:
: - Don't register a page table dumper for pKVM VMs, as S2 is not
: accessible when in protected mode
:
: - Drop PAGE_SIZE alignment requirement for hyp trace event sections
:
: - Tighten PMUv3 being disabled for the guest even if vcpus were created
: with one.
:
: - Fix timer emulation when the HW doesn't support offsetting of the
: counter.
: /
KVM: arm64: selftests: Test a timer set past the counter's wrap
KVM: arm64: nv: Read a guest hypervisor's CNTV_CVAL_EL0 from memory on x1e
KVM: arm64: timers: Compute an offset-applied CVAL from the current count
KVM: arm64: Check ID_AA64DFR0_EL1.PMUVer in pmu_visibility()
KVM: arm64: Handle ID_AA64DFR0_EL1.PMUVer == NI in __kvm_pmu_event_mask()
KVM: arm64: Remove PAGE_SIZE alignment for hyp event ELF sections
KVM: arm64: Disable stage-2 ptdump of pKVM
KVM: arm64: selftests: Check the mandatory SMCCC_ARCH_FEATURES queries
KVM: arm64: Report SMCCC_VERSION and SMCCC_ARCH_FEATURES as implemented
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Fix a potential build error (like below, when asm/kvm_emulate.h gets
included after the kvm/arm_psci.h) by including the missing header file
in kvm/arm_psci.h:
./include/kvm/arm_psci.h: In function ‘kvm_psci_version’:
./include/kvm/arm_psci.h:29:13: error: implicit declaration of function
‘vcpu_has_feature’; did you mean ‘cpu_have_feature’? [-Werror=implicit-function-declaration]
29 | if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_PSCI_0_2)) {
| ^~~~~~~~~~~~~~~~
| cpu_have_feature
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Jonathan Cameron <jonathan.cameron@oss.qualcomm.com>
Tested-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Link: https://patch.msgid.link/20261007073537.2454351-4-suzuki.poulose@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
When the hardware won't apply a timer's offset, KVM programs CVAL +
offset and lets it compare that against the raw counter. The compare is
unsigned (IsTimerConditionMet, DDI0487M_c J1.4.3.29):
condition_met = (UInt(PhysicalCountInt() - offset) -
UInt(compare_value)) >= 0;
so adding the offset to both sides holds only while both sums wrap or
neither does. A guest whose counter is ahead of the host's by more than
CVAL has an expired timer that never fires, and one behind it can have
a far-future timer fire at once.
On a CNTPOFF_EL2 host the first case livelocks: each entry reloads the
guest's CVAL, the timer fires, and the exit rewrites it as CVAL + offset
before the host can take the interrupt. Without CNTPOFF_EL2 the timer
hangs until the vCPU is next loaded, which is what Zenghui reported on
a Kunpeng920.
Program the value from the current count instead: CVAL + offset while
the deadline is ahead, 0 once the timer has expired for the guest, ~0
past the counter's wrap. KVM recomputes it at load, at exit and on a
trapped CVAL write, and an expired timer stays expired in between. The
value can't be inverted, so with an offset timer_save_state() keeps the
memory copy, and kvm_hyp_handle_timer() returns that copy for a guest
hypervisor's read of its physical CVAL rather than subtracting the
offset back out.
Fixes: c605ee245097 ("KVM: arm64: timers: Allow physical offset without CNTPOFF_EL2")
Fixes: 9404673293b0 ("KVM: arm64: timers: Correctly handle TGE flip with CNTPOFF_EL2")
Fixes: 0bc9a9e85fcf ("KVM: arm64: Work around x1e's CNTVOFF_EL2 bogosity")
Reported-by: Zenghui Yu <yuzenghui@huawei.com>
Closes: https://lore.kernel.org/r/460258be-0102-e922-c342-4e87cd94b9e5@huawei.com
Cc: stable@vger.kernel.org
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20261007074343.324269-2-fuad.tabba@linux.dev
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
As part of saving and restoring the state of a GICv5-based system,
userspace must save and restore the IRS MMIO registers. These include
important information such as the guest IST configuration, and KVM
must present consistent state to the guest after migration.
Introduce KVM_DEV_ARM_VGIC_GRP_IRS_REGS and provide accessors to read
and write the virtual IRS register state. This is modelled on the
GICv3 ITS register interface, as the migration requirements are
broadly the same.
Reuse the guest MMIO handlers where userspace and guest accesses have
the same semantics. Add userspace-specific handling where restoring a
register image must not trigger the operation associated with a guest
MMIO write.
Validate restored ID register fields against the capabilities KVM and
the host can support. Restore the emulated IST configuration without
allocating or freeing a host IST, and report operation status
registers as idle. Accept writes to IRS_SPI_CFGR, IRS_IIDR, and
IRS_AIDR without changing their state, allowing userspace to replay
the values it previously read.
Restoring IRS_IST_BASER.Valid recreates the guest-visible
configuration, but the host LPI IST cannot be allocated until
userspace supplies its contents. Track this as a pending LPI IST
restore and reject guest entry until userspace restores the IST
through KVM_DEV_ARM_VGIC_GRP_IST.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-33-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Only a subset of the possible PPIs are exposed to a guest when running
with a vGICv5. First of all, only the architected PPIs are considered
by KVM. Secondly, only a set of those is exposed to a guest: those
corresponding to devices that KVM emulates, such as the timers and
PMU, and the GICv5 SW_PPI.
The finalisation of exposed PPIs happens on first vCPU run, as this is
the first time when the full set of exposed devices is known. At this
stage a mask is calculated, and this mask is applied both to hide
non-exposed PPI state from the guest and to reduce overhead when
iterating over the PPIs.
While preparing userspace access to the GICv5 system registers, it
became apparent that restoring the GICv5 PPI registers can result in a
mismatch between the state supplied by userspace and the state KVM
intends to expose. Userspace can provide Enable, Active, and Pending
state for PPIs that KVM has chosen to hide from the guest.
Userspace must restore PPI state before any vCPU runs. The userspace
access path added subsequently enforces this ordering. Rework
vgic_v5_finalize_ppi_state() to calculate the mask of exposed PPIs and
clear any state belonging to non-exposed PPIs. This ensures that only
the state KVM intends to expose is visible to the guest.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-31-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
There are times, such as with GICv5 SPIs and LPIs, where the hardware
itself manages parts of the interrupt lifecycle. This means that
pending state can be directly communicated to the hardware instead of
being represented only in the VGIC shadow state.
In order to accommodate cases where the hardware handles pending state
directly, add a new set_pending_state() function pointer to
irq_ops. The intent is for this to be used after the VGIC shadow
pending state has changed, allowing the backend to mirror the updated
state into hardware.
This new function is plumbed into kvm_vgic_inject_irq(), and is only
called if irq_ops are provided and this function pointer is explicitly
set. In the general case, this has no effect.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-27-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Now that we have an emulated IRS, it needs to be registered, which
ensures that guest accesses to the MMIO regions handled by the device
are handled appropriately in KVM. Therefore, as part of
vgic_map_resources, the GICv5 IRS IODEV is registered. If the address
for the IRS is not provided, bail out reporting an error - this is not
a supported config.
As part of this change, expose setting the address of the emulated IRS
via KVM_VGIC_V5_ADDR_TYPE_IRS to userspace. Also allow userspace to
set the number of SPIs handled by the emulated GICv5 implementation,
using a GICv5-specific SPI count rather than the legacy total
interrupt count.
Limit the configurable range to 32 through 1024 SPIs, in multiples of
32. KVM keeps one struct vgic_irq per SPI in a physically contiguous
allocation, and allowing the full 16-bit KVM_IRQ_LINE SPI namespace
would make that allocation exceed KMALLOC_MAX_SIZE on common arm64
configurations.
The default routing has one IRQCHIP route per configured SPI, so size
the common IRQ routing table for the largest GICv5 configuration. The
model-specific routing validation retains the 988-pin limit for GICv2
and GICv3.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-23-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
In order to properly support GICv5-based VMs in KVM, emulate the
CONFIG_FRAME for a virtual IRS. This emulation needs to handle guest
accesses to the MMIO region and mimic the behaviour of a real IRS.
Introduce an IODEV for the GICv5 IRS and an associated initialisation
function that sets up the SPIs and initial IRS state. The MMIO
emulation allows the guest to query the IRS_IDx registers, manipulate
SPIs, configure ISTs, and so forth.
Allow 32-bit accesses to the 64-bit IRS registers in addition to
64-bit accesses. Reconstruct IRS_IST_BASER for reads and merge partial
writes so that updating either word preserves the other half.
The emulation tracks selector state across MMIO accesses. For example,
a guest writes IRS_PE_SELR to select a PE by IAFFID. This is the VPE
ID for a VM, but the guest does not know this. If the guest reads
IRS_PE_STATUSR, KVM checks whether that IAFFID selects a valid VPE and
sets the V bit accordingly. IRS_PE_CR0 is accepted as write-ignored
because KVM does not support 1-of-N routing.
The same selector and status register model is exposed for SPIs.
Track the state of IRS_CR0.IRSEN and introduce KVM_REQ_RELOAD_GICv5 to
reload the GICv5 context of running vCPUs when it changes. Only make
the request when the enable state changes, avoiding unnecessary IPIs
for writes that leave it unchanged.
The LPI IST requires KVM to perform actions on behalf of the guest.
Treat changes to IRS_IST_BASER.Valid as the lifetime of the guest's
IST. On an Invalid-to-Valid transition, validate IRS_IST_CFGR,
allocate a shadow host IST, and assign it to the physical IRS through
the VMTE. On a Valid-to-Invalid transition, invalidate and free the
host IST. Ignore guest address changes while the BASER remains valid,
and prevent changes to IRS_IST_CFGR during that time.
As far as the guest is concerned, the IST memory it provided is being
used by the hardware, but the physical IRS uses the host shadow IST
instead.
This change provides the core IRS IODEV and MMIO emulation, but does
not plumb the device into the rest of KVM yet. The CoreSight
identification registers are added separately.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-21-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
In order to support proper VMs (that support more than just PPIs) for
GICv5, it is important to emulate the GICv5 IRS too. The IRS includes
an MMIO interface which is used to interact with and configure the
IRS.
As part of providing the emulated IRS MMIO interface in KVM, extend
enum iodev_type to include a GICv5 IRS device, and extend the MMIO
code to handle reads and writes to that type of IO device. This will
allow the creation of a GICv5 IRS IO Device in KVM.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-19-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
In order to properly emulate the operation of the IRS from KVM, we
require storage for the MMIO register state. This change introduces
struct vgic_v5_irs, and adds a pointer to it to the struct vgic_dist.
This new data structure contains the storage for IRS MMIO state that
is required for emulating the MMIO interface in KVM. This provides
persistent storage, and a way to track data across MMIO writes, e.g.,
selecting an SPI and updating the configuration of it is two MMIO
writes.
Note that only a pointer to the data structure is added to struct
vgic_dist as this new structure is very large, and hence it makes
sense to dynamically allocate it and just provide a pointer to
retrieve it in struct vgic_dist.
In addition to adding a structure to store the MMIO state for the IRS,
we add the base address in GPA space to struct vgic_v5_irs.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-18-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
GICv5 introduces the concept of VPE residency - a VPE can be either
resident or non-resident. When the VPE is resident, the IRS is allowed
to select interrupts that target that VPE (or the VM) as the HPPI
(Highest Priority Pending Interrupt). As the IRS handles both SPIs and
LPIs, these will only be picked as the IRS's HPPI when a VPE is
resident.
A GICv5 VPE is made resident by writing ICH_CONTEXTR_EL2 with
ICH_CONTEXTR_EL2.V set, together with valid VM and VPE IDs. This
informs the IRS that a specific VPE is running, and that it can begin
HPPI selection for that VPE. Making a VPE non-resident (by making the
ICH_CONTEXTR_EL2 invalid) informs the IRS that the VPE is no longer
running, and it stops HPPI selection for it.
This change introduces two new hyp calls - one to make a VPE resident
and its counterpart to make a VPE non-resident. As part of making a
VPE resident, the resulting ICH_CONTEXTR_EL2.F bit is checked to catch
residency faults. Such a fault indicates a broken VM/VPE setup, so
warn and mark the VM dead.
Both of these new hypercalls are explicitly no-ops with pKVM as we
currently don't support the combination of GICv5 and pKVM.
Furthermore, this change extends vgic_v5_load() and vgic_v5_put() to
make the VPEs resident and non-resident, respectively. Hence, the VPE
is considered resident for the entire load-to-put interval.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-16-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
A GICv5 VM needs a VM table entry before it can use SPIs and LPIs,
which are backed by the host IRS. The VM table itself is created at
probe time, but each VM still needs to claim and populate one VMTE
before it can use those interrupts.
VPE doorbells are allocated from the host LPI irq domain. Without that
domain, KVM cannot issue IRS commands or receive doorbell wakeups.
Fail the GICv5 KVM probe if the host driver did not create an LPI
domain.
Allocate a VM ID during vgic_v5_init(). The VM ID is also the index
into the VM table, so allocating it selects the VMTE slot that will be
used for the lifetime of the VM.
Create a per-VM VPE doorbell irq domain, allocate one doorbell
interrupt per vCPU, request the interrupts, and keep the doorbell IRQ
number in the vCPU's GICv5 state. The doorbell handler marks the VPE
doorbell as fired, raises KVM_REQ_IRQ_PENDING, and kicks the target
vCPU so that KVM can re-evaluate pending interrupt state.
The doorbell domain indexes its interrupts using the dense vcpu_idx,
while GICv5 uses the userspace-provided vcpu_id as the VPE ID. Store a
backpointer to the VM and use it to resolve the vCPU before issuing
VPE-specific IRS commands. This preserves the userspace VPE ID when
vCPU IDs are sparse.
With the VM ID and doorbells in place, initialise the VMTE backing
state, including the VM descriptor, VPE table, and preallocated VPED
storage. The doorbells have to exist before making the VMTE valid, as
they provide the IRQ-side conduit used by the IRS commands. Make the
VMTE valid via the IRS, then populate the VPETE for each vCPU.
Add vgic_v5_teardown() to unwind the state in the reverse order. Make
the VMTE invalid, clear the per-vCPU VPETEs, release the VMTE backing
state, free the doorbell IRQs and irq domain, and finally release the
VM ID so that the VMTE slot can be reused by a later VM. If
invalidating the VMTE, clearing a VPETE, or releasing the VMTE fails,
still free the software doorbells and domain, but keep the VM ID
allocated so that the VMTE slot is not reused while hardware-visible
state may remain.
On init failure, call the same teardown path so that partially created
state is unwound consistently.
As part of resetting vCPUs, mark them as valid in the VM's VPE table.
This informs the IRS that a specific VPE may be made resident. Without
this, the IRS will treat the VPE as invalid.
Also introduce vgic_v5_send_command(), a wrapper around the VPE
doorbells. It takes a struct kvm_vcpu pointer and the command to run,
and invokes the function bound to that command through the vCPU's
doorbell.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-15-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
A GICv5 host with FEAT_GCIE_LEGACY can expose either a native vGICv5
or a vGICv3 device. These models do not necessarily have the same vCPU
limit: the native GICv5 limit is probed from the IRS VPE capacity,
while the GICv3 limit remains the fixed KVM vGICv3 limit.
Keep the IRS-derived limit separately for vGICv5 creation. The
pre-VGIC KVM_CAP_MAX_VCPUS value continues to expose the largest limit
among the still-selectable models, and kvm_vgic_create() clamps the VM
to the limit of the VGIC model userspace actually selected.
Userspace can create vCPUs before creating a VGIC. Hence, checking
only the number of existing vCPUs after reducing the limit is not
sufficient. For example, a single vCPU with ID 500 passes the GICv2
count limit even though GICv2 can represent only eight target
CPUs. The GICv2 code subsequently uses the vCPU ID in target and SGI
source masks, leading to shifts beyond the width of the operand.
GICv5 has a similar requirement because the userspace-provided vcpu_id
is used as the index into the VPET. This means that the vcpu_id also
represents the IAFFID for a VPE, and therefore is visible to the
guest.
After selecting the model-specific limit, validate every existing vCPU
ID against it. KVM_CREATE_VCPU already enforces the same limit for
vCPUs created after the VGIC, making the result independent of
creation order.
Link: https://lore.kernel.org/r/20260807133051.15D381F000E9@smtp.kernel.org
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-13-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
GICv5 uses a set of in-memory tables to track and manage VM state.
These must be allocated by the hypervisor and provided to the IRS.
The VMT (Virtual Machine Table) is a linear or two-level table
comprising VMT Entries (VMTEs). Each VMTE describes the state for a
single VM. This state includes things such as the SPI and LPI IST
configuration (coming in a future commit), an implementation-defined
VM Descriptor, and a VPE Table (VPET).
The VPET contains one entry per possible VPE ID belonging to a VM. It
is used to mark a VPE as valid and provide the address of an
implementation-defined VPE Descriptor (VPED), which the hardware uses
to track and manage VPE state.
Allocate each VM's VPEDs as a single dense array indexed by vcpu_idx,
while the VPET remains indexed by the userspace-provided vcpu_id. This
keeps VPED storage proportional to the number of vCPUs even when their
IDs are sparse.
The VMT and VPET are shared with the IRS. On systems with a
non-coherent IRS, cache maintenance operates at cache-line
granularity, while multiple entries can occupy the same cache line.
Use a common lock for CPU accesses to these tables and IRS command
processing so that writing back one entry cannot overwrite an IRS
update to a neighbouring entry.
The implementation-defined VMD and VPED storage is also visible to the
IRS. Round these allocations up to whole cache lines to prevent cache
maintenance from corrupting unrelated slab objects. Initialise the
storage before publishing its addresses to the IRS.
This commit adds support for allocating the VMT and its descriptor
backing state, and for managing VMTEs. The VMTEs can be initialised or
released for reuse. VM IDs are allocated with an IDA, while an XArray
tracks the host-side allocations associated with populated VMTEs.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-10-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
VPE doorbells allow the GICv5 hardware to notify KVM when an SPI or
LPI can be signalled to a non-resident VPE. This provides the
mechanism used to wake blocked vcpus once the hardware determines that
the interrupt is eligible to be delivered.
Add support for a per-VM VPE doorbell irq domain. The domain is
created under the GICv5 LPI domain, with one doorbell allocated per
VPE. Store the allocated doorbell base in the VM's GICv5 state so that
later patches can request per-vcpu doorbell IRQs and use them for IRS
commands and wakeups.
Add the per-VPE doorbell state to the GICv5 CPU interface state. The
doorbell IRQ number is populated when the IRQs are requested, and the
db_fired state is used by later patches once doorbell delivery is
wired up.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-9-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
KVM needs access to the host IRS configuration frame to configure
virtual-machine resources. The frame ID registers describe the
capabilities available to KVM.
Cache IRS_IDR2, IRS_IDR3, and IRS_IDR4 rather than extracting their
individual fields into kvm_vgic_global_state. Accessor helpers derive
the required capabilities from the cached registers.
Use the cached IRS_IDR4 VPE capacity to limit the maximum number of
vCPUs, while retaining VGIC_V5_MAX_CPUS as KVM's own upper bound.
Keeping the raw register values lets KVM adjust the exposed
capabilities when needed, as is done for ICH_VTR_EL2, and avoids tying
future EL2 code to the layout of kvm_vgic_global_state.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260904113404.4051341-8-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 changes for 7.3
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1
is present, guaranteeing that EL2-specific register bits are stateful
in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope
of those instructions remains 'in host' (i.e. L1 kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly in
the VGIC, our in-kernel generator of bugs (and sometimes interrupts)
|
|
* kvm-arm64/misc-7.3:
: Miscellaneous fixes for KVM/arm64, 7.3
:
: - Fixes for saving invalid table entries as part of saving the ITS
: tables (Fuad Tabba)
:
: - Don't reallocate the SPI array for re-attempted vgic_init(), avoiding
: a memory leak (Fuad Tabba)
:
: - Hold a reference on an LPI when saving the pending state (Qihang)
:
: - Don't WARN for out-of-range, guest-supplied INTID (Karl)
:
: - Avoid corrupting GPRs for 32-bit CP64 reads (Karl)
:
: - Reset 'in kernel' VGIC state when private IRQ allocation fails (Fuad)
:
: - Avoid kallsyms lookup in nVHE panic unless the host stage-2 is also
: disabled (Vincent)
:
: - Disregard Pending+Active state when computing maintenance IRQ for
: ICH_MISR_EL2.NP (Kajetan)
:
: - Various Sashiko-identified issues dealing with GICv5 (Sascha)
:
: - Fix CPU onlining in pKVM due to mismatched accesses when the MMU is
: disabled (Will)
KVM: arm64: Validate GICv5 timer PPIs before claiming ownership
KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs
KVM: arm64: vgic: Prevent speculative SPI array underflow
KVM: arm64: vgic: Free gic_kvm_info on initialization failure
KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'
KVM: arm64: vgic: Fix detection of MI on no pending LR
KVM: arm64: Drop %pB on nVHE panic when stage-2 is active
KVM: arm64: vgic: Reset in_kernel on private IRQ allocation failure
KVM: arm64: GICv2: Don't WARN on out-of-range GICV_DIR INTID
KVM: arm64: Preserve GPRs for AArch32 CP64 reads generating an UNDEF
KVM: arm64: vgic-v3: take an LPI reference in vgic_v3_save_pending_tables
KVM: arm64: vgic-its: Point saved ITEs at the next valid entry
KVM: arm64: vgic-its: Don't save collections the table cannot hold
KVM: arm64: vgic: Don't leak the SPI array when init is retried
KVM: arm64: vgic-its: Don't dereference a NULL collection on ITT save
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
* kvm-arm64/vtr-patch:
: Inline patching of ICH_VTR_EL2 constant, courtesy of Marc Zyngier
:
: Unify readers of ICH_VTR_EL2 on an instruction-patched constant value,
: avoiding system register accesses known to trap under nested
: virtualization and sharing the implementation between pKVM and 'regular'
: KVM.
KVM: arm64: vgic-v3: Kill kvm_vgic_global_state.ich_vtr_el2
KVM: arm64: vgic-v3: Simplify initial GICv3 configuration sampling
KVM: arm64: Convert most ICH_VTR_EL2 accesses to inlined literal value
KVM: arm64: Add a helper providing an inlined literal value for ICH_VTR_EL2
KVM: arm64: Move GICv3 broken SEIS implementation detection to a CPU errrata
KVM: arm64: vgic-v3: Make vtr_to_* helpers use architectural field symbols
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
* kvm-arm64/pkvm-7.3: (26 commits)
: pKVM updates for 7.3
:
: - Avoid name collision on trace_clock() when CONFIG_NVHE_EL2_TRACING is
: disabled (Mostafa Saleh)
:
: - Clean up state tracking for whether the EL2 shadow VM has been
: created (Fuad Tabba)
:
: - Synchronize SCTLR_EL1 when injecting an exception to use current
: PAN/SSBS state (Fuad Tabba)
:
: - Avoid unnecessary cache maintenance when I/D-cache are known to be
: coherent in pKVM (Mostafa Saleh)
:
: - Lazy vCPU context save/restore for pKVM (Fuad Tabba)
:
: - Various fixes to the stage-2 MMU for pKVM (Fuad Tabba)
:
: - Allow counter offsetting of non-protected guests in protected mode
: (Mostafa Saleh)
:
: - Condition the 'broken CNTVOFF' mitigation on a VM actually having a
: nonzero offset, fixing boot failures of pVMs on affected hardware
: (Mostafa Saleh)
KVM: arm64: Fix hvhe and broken CNTVOFF_EL2
KVM: arm64: Fix timer offsets for non-protected VMs
KVM: arm64: Make timer_get_offset() work in all contexts
KVM: arm64: selftests: Add stage-2 block transition test
KVM: arm64: Don't advertise eager page splitting under pKVM
KVM: arm64: Don't WARN on pKVM stage-2 map failures
KVM: arm64: Skip pKVM stage-2 flush when FWB is enabled
KVM: arm64: Top up stage-2 memcache for dirty logging faults
KVM: arm64: Top up the memcache for pKVM permission faults
KVM: arm64: Skip cache maintenance for non-cacheable pKVM mappings
KVM: arm64: Implement lazy vCPU state sync for non-protected guests
KVM: arm64: Add primitives to flush/sync the VGIC state at EL2
KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch
KVM: arm64: Add host and hypervisor vCPU lookup primitives
KVM: arm64: Move PSCI helper functions to a shared header
KVM: arm64: Factor out reusable vCPU reset helpers
KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code
KVM: arm64: Extract MPIDR computation into a shared header
KVM: arm64: selftests: Add a userspace watchpoint test
KVM: arm64: Flush external_mdscr_el1 to the pKVM hyp vCPU
...
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
GICv5 supports up to 128 PPIs, but KVM currently implements only the
first 64, which contain the architected PPIs it supports.
An encoded PPI with an ID outside that range passes irq_is_ppi(),
which only checks the encoded interrupt type. vgic_get_vcpu_irq()
therefore looks it up in private_irqs[], where array_index_nospec()
clamps the out-of-range index to zero and aliases PPI 0.
Include the supported PPI range in irq_is_ppi() so that KVM interfaces
reject unsupported PPIs. Also reject an out-of-range PPI in the lookup
as a safeguard against callers bypassing the predicate.
Fixes: 4d591252bacb ("KVM: arm64: gic-v5: Implement PPI interrupt injection")
Fixes: eb8bce08ecb1 ("KVM: arm64: gic: Introduce interrupt type helpers")
Link: https://sashiko.dev/#/patchset/20260724104819.1296803-1-sascha.bischoff@arm.com?part=27
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260811150941.941295-4-sascha.bischoff@arm.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
We currently have two implementations of get_timer offset(), one
in arm_arch_timer.h, and another one in switch.h.
These two only differ by a pair of kern_hyp_va(), which seems a
pretty weak reason to open-code it.
Turn this function into a macro to avoid the include dependency hell
on kern_hyp_va(), and make it work correctly in all contexts.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Link: https://patch.msgid.link/20260808085824.732659-2-smostafa@google.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
Move kvm_psci_valid_affinity() and kvm_psci_narrow_to_32bit() from
psci.c to include/kvm/arm_psci.h, and move psci_affinity_mask() there
too, renaming it kvm_psci_affinity_mask() now that it is no longer
file-local. A follow-up series handles some protected-guest PSCI calls
at EL2 using these helpers.
No functional change intended.
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260729131823.2021516-5-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
Introduce a new field pmmir_slots in struct kvm_arch to store
PMMIR_EL1.SLOTS. It only saves the actual hardware PMU value when
the VMM explicitly selects a PMU under KVM_ARM_VCPU_PMU_V3_STRICT.
Otherwise, it stays 0 after allocation.
Use this field to implement guest access, userspace get, and userspace
set for PMMIR_EL1:
- access_pmmir(): uses the value in kvm->arch.pmmir_slots directly. If
the VMM selected a PMU and KVM_ARM_VCPU_PMU_V3_STRICT is set, the guest
can correctly read the underlying core's SLOTS. Otherwise, it continues
to read 0 since the true SLOTS value can be nondeterministic.
- get_pmmir(): same as access_pmmir().
- set_pmmir(): only the SLOTS field is writable; a value setting any
other bit is rejected with -EINVAL, since get_pmmir() returns SLOTS
zero-extended. A value of 0 resets kvm->arch.pmmir_slots to 0 for
backward compatibility, as the register is RAZ in older KVM, a value
matching the current SLOTS is accepted as a no-op, and anything else is
rejected with -EINVAL. Once the VM has run PMMIR_EL1 is immutable, so a
mismatching write then returns -EBUSY.
The register is now exposed via KVM_GET_REG_LIST for PMUv3 vCPUs, so add
it to the get-reg-list selftest's PMU register list.
Signed-off-by: Congkai Tan <congkai@amazon.com>
Reviewed-by: Geoff Blake <blakgeof@amazon.com>
Reviewed-by: Haris Okanovic <harisokn@amazon.com>
Reviewed-by: Stanislav Spassov <stanspas@amazon.de>
Co-developed-by: Oliver Upton <oupton@kernel.org>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260722202702.4165917-2-congkai@amazon.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
Fix a potential race between decrementing an LPI's reference count and
evicting that structure from the LPI xarray.
LPI structures are maintained in the VGIC LPI xarray (dist->lpi_xa).
When the reference count of an LPI structure drops to zero,
vgic_release_lpi_locked() removes the structure from the xarray and
frees it under the xarray lock.
However, the release of an LPI can race with a concurrent LPI
re-registration with the same INTID via vgic_add_lpi() on another CPU,
since the reference count drop and the xarray eviction are not performed
in a single atomic step. This can happen e.g. if the guest issues a
DISCARD while the LPI is still referenced from a vCPU's active-pending
list (ap_list), and the same INTID is re-mapped via MAPTI.
Particularly, vgic_release_lpi_locked() is called from two distinct
paths: direct release via vgic_put_irq(), and deferred release via
vgic_release_deleted_lpis(). During direct release, the issue can result
in deleting a newly registered LPI from the xarray:
CPU0 (Releasing LPI) CPU1 (Adding new LPI)
==================== =====================
vgic_put_irq()
__vgic_put_irq()
refcount_dec_and_test()
vgic_add_lpi()
xa_lock_irqsave()
old_irq = xa_load(.., intid)
vgic_try_get_irq_ref(old_irq) == false
new IRQ inserted --> __xa_store(.., intid, ..)
xa_unlock_irqrestore()
xa_lock_irqsave();
vgic_release_lpi_locked()
__xa_erase(.., irq->intid) <-- BUG: new IRQ is erased
kfree_rcu(old_irq)
During the deferred release path, the old IRQ can be leaked:
CPU0 (Releasing LPI) CPU1 (Adding new LPI)
==================== =====================
vgic_put_irq_norelease()
__vgic_put_irq()
refcount_dec_and_test()
irq->pending_release = true
vgic_add_lpi()
xa_lock_irqsave()
old_irq = xa_load(.., intid)
vgic_try_get_irq_ref(oldirq) == false
BUG: old IRQ overwritten --> __xa_store(.., intid, ..)
xa_unlock_irqrestore()
vgic_release_deleted_lpis()
xa_lock_irqsave()
xa_for_each() { .. } <-- old IRQ with pending_release = true
is gone, so it cannot be released
To fix the direct release path, move the reference count drop inside
the xarray lock, making sure that vgic_add_lpi() never encounters the
to-be-released LPI.
In the deferred release path, the refcount drop must happen under a raw
spinlock, so the xarray lock cannot be grabbed, and the same solution
does not work. Instead, update vgic_add_lpi(), so that if it evicts
an LPI from the xarray, it takes on the responsibility of freeing it.
Consequently, an LPI may now be freed concurrently after a deferred
release drops the refcount, so accessing the pending_release field is no
longer safe from use-after-free. Delete all uses of the flag, and update
vgic_release_deleted_lpis() to identify orphaned LPIs purely based on
their refcount.
Reported-by: Claude:claude-opus-4-6
Fixes: 3a08a6ca7c37 ("KVM: arm64: vgic-v3: Use bare refcount for VGIC LPIs")
Fixes: d54594accf73 ("KVM: arm64: vgic-v3: Erase LPIs from xarray outside of raw spinlocks")
Signed-off-by: Carlos López <clopez@suse.de>
Link: https://patch.msgid.link/20260715105137.3973823-4-clopez@suse.de
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
kvm_vgic_global_state.ich_vtr_el2 is the last bit of caching that
we can get rid of. Not as bad as a sysreg access, but still worse
than a constant.
Move over to the inlined stuff and remove the cached value.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260721170754.3150521-7-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
* kvm-arm64/vgic-v5-PPI-fixes:
: .
: Substantial cleanup of the vgic-v5 PPI support. From the original
: cover letter:
:
: "With the GICv5 PPi support merged in, it has become obvious that a few
: things could be improved, both from the correctness and maintainability
: angles."
: .
KVM: arm64: Fix arch timer interrupts for GICv3-on-GICv5 guests
irqchip/gic-v5: Immediately exec priority drop following activate
Documentation: KVM: Clarify that PMU_V3_IRQ IntID requirements for GICv5
Documentation: KVM: Fix typos in VGICv5 documentation
KVM: arm64: selftests: Improve error handling for GICv5 PPI selftest
KVM: arm64: selftests: Cleanup unused vars in GICv5 PPI selftest
KVM: arm64: selftests: Add missing GIC CDEN to no-vgic-v5 selftest
KVM: arm64: vgic-v5: Atomically assign bits to PPI DVI bitmap
KVM: arm64: vgic-v5: Add missing trap handing for NV triage
KVM: arm64: vgic-v5: Limit support to 64 PPIs
KVM: arm64: vgic: Rationalise per-CPU irq accessor
KVM: arm64: vgic-v5: Drop defensive checks from vgic_v5_ppi_queue_irq_unlock()
KVM: arm64: vgic: Consolidate vgic_allocate_private_irqs_locked()
KVM: arm64: vgic: Constify struct irq_ops usage
KVM: arm64: vgic-v5: Drop pointless ARM64_HAS_GICV5_CPUIF check
KVM: arm64: vgic-v5: Remove use of __assign_bit() with a constant
KVM: arm64: vgic-v5: Move PPI caps into kvm_vgic_global_state
KVM: arm64: vgic-v5: Add for_each_visible_v5_ppi() iterator
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
vgic-v5 has introduced much more prevalent usage of the struct
irq_ops mechanism.
In the process, it becomes evident that suffers from two related
problems:
- it contains flags, rather than only callbacks
- it is mutable, because we need to update the above flags
Swap the flags for a helper retrieving the flags, and make all
irq_ops const, something that is slightly satisfying.
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Link: https://lore.kernel.org/r/20260520091949.542365-6-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Constant vgic properties are usually kept in kvm_vgic_global_state,
but the vgic-v5 code does its own thing.
Move the ppi_caps data into the global structure, which has the
modest additional advantage of making it ro_after_init.
Link: https://lore.kernel.org/r/20260520091949.542365-3-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Just like the timer, the PMU has an interrupt cache that serves little
purpose. Drop it.
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260520100200.543845-5-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
The timer code makes use of a per-timer irq level cache, which
looks like a very minor optimisation to avoid taking a lock upon
updating the GIC view of the interrupt when it is unchanged from
the previous state.
This is coming in the way of more important correctness issues,
so get rid of the cache, which simplifies a couple of minor things.
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260520100200.543845-4-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
The userspace notification of interrupts is has a few problems:
- it is utterly pointless
- it is annoyingly split between detecting the need for notification
and the population of the interrupts in the run structure
We can't do anything about the former (yet), but the latter can be
addressed. If we detect that we must notify userspace, we know that
we are going to exit, as we populate the exit status. Which means
we can also populate the interrupt state at this stage and be done
with it.
This simplifies the structure of the code.
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260520100200.543845-3-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
The newly introduced arch_timer_context::direct field is a bit pointless,
as it is always set on timers that are... err... direct, while
we already have a way to get to that by doing a get_map() operation.
Additionally, this field is:
- only set when get_map() is called
- never cleared
and the single point where it is actually checked doesn't call get_map()
at all.
At this stage, it is probably better to just kill it, and rely on
get_map() to give us the correct information.
Reviewed-by: Sascha Bischoff <sascha.bischoff@arm.com>
Fixes: 9491c63b6cd7b ("KVM: arm64: gic-v5: Enlighten arch timer for GICv5")
Link: https://sashiko.dev/#/patchset/20260319154937.3619520-1-sascha.bischoff%40arm.com
Link: https://patch.msgid.link/20260401103611.357092-12-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
GICv5 systems will likely not support the full set of PPIs. The
presence of any virtual PPI is tied to the presence of the physical
PPI. Therefore, the available PPIs will be limited by the physical
host. Userspace cannot drive any PPIs that are not implemented.
Moreover, it is not desirable to expose all PPIs to the guest in the
first place, even if they are supported in hardware. Some devices,
such as the arch timer, are implemented in KVM, and hence those PPIs
shouldn't be driven by userspace, either.
Provided a new UAPI:
KVM_DEV_ARM_VGIC_GRP_CTRL => KVM_DEV_ARM_VGIC_USERPSPACE_PPIs
This allows userspace to query which PPIs it is able to drive via
KVM_IRQ_LINE.
Additionally, introduce a check in kvm_vm_ioctl_irq_line() to reject
any PPIs not in the userspace mask.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260319154937.3619520-40-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Make it mandatory to use the architected PPI when running a GICv5
guest. Attempts to set anything other than the architected PPI (23)
are rejected.
Additionally, KVM_ARM_VCPU_PMU_V3_INIT is relaxed to no longer require
KVM_ARM_VCPU_PMU_V3_IRQ to be called for GICv5-based guests. In this
case, the architectued PPI is automatically used.
Documentation is bumped accordingly.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Link: https://patch.msgid.link/20260319154937.3619520-33-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Now that GICv5 has arrived, the arch timer requires some TLC to
address some of the key differences introduced with GICv5.
For PPIs on GICv5, the queue_irq_unlock irq_op is used as AP lists are
not required at all for GICv5. The arch timer also introduces an
irq_op - get_input_level. Extend the arch-timer-provided irq_ops to
include the PPI op for vgic_v5 guests.
When possible, DVI (Direct Virtual Interrupt) is set for PPIs when
using a vgic_v5, which directly inject the pending state into the
guest. This means that the host never sees the interrupt for the guest
for these interrupts. This has three impacts.
* First of all, the kvm_cpu_has_pending_timer check is updated to
explicitly check if the timers are expected to fire.
* Secondly, for mapped timers (which use DVI) they must be masked on
the host prior to entering a GICv5 guest, and unmasked on the return
path. This is handled in set_timer_irq_phys_masked.
* Thirdly, it makes zero sense to attempt to inject state for a DVI'd
interrupt. Track which timers are direct, and skip the call to
kvm_vgic_inject_irq() for these.
The final, but rather important, change is that the architected PPIs
for the timers are made mandatory for a GICv5 guest. Attempts to set
them to anything else are actively rejected. Once a vgic_v5 is
initialised, the arch timer PPIs are also explicitly reinitialised to
ensure the correct GICv5-compatible PPIs are used - this also adds in
the GICv5 PPI type to the intid.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260319154937.3619520-32-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Update kvm_vgic_create to create a vgic_v5 device. When creating a
vgic, FEAT_GCIE in the ID_AA64PFR2 is only exposed to vgic_v5-based
guests, and is hidden otherwise. GIC in ~ID_AA64PFR0_EL1 is never
exposed for a vgic_v5 guest.
When initialising a vgic_v5, skip kvm_vgic_dist_init as GICv5 doesn't
support one. The current vgic_v5 implementation only supports PPIs, so
no SPIs are initialised either.
The current vgic_v5 support doesn't extend to nested guests. Therefore,
the init of vgic_v5 for a nested guest is failed in vgic_v5_init.
As the current vgic_v5 doesn't require any resources to be mapped,
vgic_v5_map_resources is simply used to check that the vgic has indeed
been initialised. Again, this will change as more GICv5 support is
merged in.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260319154937.3619520-29-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
GICv5 adds support for directly injected PPIs. The mechanism for
setting this up is GICv5 specific, so rather than adding
GICv5-specific code to the common vgic code, we introduce a new
irq_op.
This new irq_op is intended to be used to enable or disable direct
injection for interrupts that support it. As it is an irq_op, it has
no effect unless explicitly populated in the irq_ops structure for a
particular interrupt. The usage is demonstracted in the subsequent
change.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260319154937.3619520-26-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
There are times when the default behaviour of vgic_queue_irq_unlock()
is undesirable. This is because some GICs, such a GICv5 which is the
main driver for this change, handle the majority of the interrupt
lifecycle in hardware. In this case, there is no need for a per-VCPU
AP list as the interrupt can be made pending directly. This is done
either via the ICH_PPI_x_EL2 registers for PPIs, or with the VDPEND
system instruction for SPIs and LPIs.
The vgic_queue_irq_unlock() function is made overridable using a new
function pointer in struct irq_ops. vgic_queue_irq_unlock() is
overridden if the function pointer is non-null.
This new irq_op is unused in this change - it is purely providing the
infrastructure itself. The subsequent PPI injection changes provide a
demonstration of the usage of the queue_irq_unlock irq_op.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260319154937.3619520-20-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
We only want to expose a subset of the PPIs to a guest. If a PPI does
not have an owner, it is not being actively driven by a device. The
SW_PPI is a special case, as it is likely for userspace to wish to
inject that.
Therefore, just prior to running the guest for the first time, we need
to finalize the PPIs. A mask is generated which, when combined with
trapping a guest's PPI accesses, allows for the guest's view of the
PPI to be filtered. This mask is global to the VM as all VCPUs PPI
configurations must match.
In addition, the PPI HMR is calculated.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260319154937.3619520-19-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
This change introduces GICv5 load/put. Additionally, it plumbs in
save/restore for:
* PPIs (ICH_PPI_x_EL2 regs)
* ICH_VMCR_EL2
* ICH_APR_EL2
* ICC_ICSR_EL1
A GICv5-specific enable bit is added to struct vgic_vmcr as this
differs from previous GICs. On GICv5-native systems, the VMCR only
contains the enable bit (driven by the guest via ICC_CR0_EL1.EN) and
the priority mask (PCR).
A struct gicv5_vpe is also introduced. This currently only contains a
single field - bool resident - which is used to track if a VPE is
currently running or not, and is used to avoid a case of double load
or double put on the WFI path for a vCPU. This struct will be extended
as additional GICv5 support is merged, specifically for VPE doorbells.
Co-authored-by: Timothy Hayes <timothy.hayes@arm.com>
Signed-off-by: Timothy Hayes <timothy.hayes@arm.com>
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260319154937.3619520-18-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Introduce the following hyp functions to save/restore GICv5 state:
* __vgic_v5_save_apr()
* __vgic_v5_restore_vmcr_apr()
* __vgic_v5_save_ppi_state() - no hypercall required
* __vgic_v5_restore_ppi_state() - no hypercall required
* __vgic_v5_save_state() - no hypercall required
* __vgic_v5_restore_state() - no hypercall required
Note that the functions tagged as not requiring hypercalls are always
called directly from the same context. They are either called via the
vgic_save_state()/vgic_restore_state() path when running with VHE, or
via __hyp_vgic_save_state()/__hyp_vgic_restore_state() otherwise. This
mimics how vgic_v3_save_state()/vgic_v3_restore_state() are
implemented.
Overall, the state of the following registers is saved/restored:
* ICC_ICSR_EL1
* ICH_APR_EL2
* ICH_PPI_ACTIVERx_EL2
* ICH_PPI_DVIRx_EL2
* ICH_PPI_ENABLERx_EL2
* ICH_PPI_PENDRx_EL2
* ICH_PPI_PRIORITYRx_EL2
* ICH_VMCR_EL2
All of these are saved/restored to/from the KVM vgic_v5 CPUIF shadow
state, with the exception of the PPI active, pending, and enable
state. The pending state is saved and restored from kvm_host_data as
any changes here need to be tracked and propagated back to the
vgic_irq shadow structures (coming in a future commit). Therefore, an
entry and an exit copy is required. The active and enable state is
restored from the vgic_v5 CPUIF, but is saved to kvm_host_data. Again,
this needs to by synced back into the shadow data structures.
The ICSR must be save/restored as this register is shared between host
and guest. Therefore, to avoid leaking host state to the guest, this
must be saved and restored. Moreover, as this can by used by the host
at any time, it must be save/restored eagerly. Note: the host state is
not preserved as the host should only use this register when
preemption is disabled.
As with GICv3, the VMCR is eagerly saved as this is required when
checking if interrupts can be injected or not, and therefore impacts
things such as WFI.
As part of restoring the ICH_VMCR_EL2 and ICH_APR_EL2, GICv3-compat
mode is also disabled by setting the ICH_VCTLR_EL2.V3 bit to 0. The
correspoinding GICv3-compat mode enable is part of the VMCR & APR
restore for a GICv3 guest as it only takes effect when actually
running a guest.
Co-authored-by: Timothy Hayes <timothy.hayes@arm.com>
Signed-off-by: Timothy Hayes <timothy.hayes@arm.com>
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260319154937.3619520-17-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Add in a sanitization function for ID_AA64PFR2_EL1, preserving the
already-present behaviour for the FPMR, MTEFAR, and MTESTOREONLY
fields. Add sanitisation for the GCIE field, which is set to IMP if
the host supports a GICv5 guest and NI, otherwise.
Extend the sanitisation that takes place in kvm_vgic_create() to zero
the ID_AA64PFR2.GCIE field when a non-GICv5 GIC is created. More
importantly, move this sanitisation to a separate function,
kvm_vgic_finalize_sysregs(), and call it from kvm_finalize_sys_regs().
We are required to finalize the GIC and GCIE fields a second time in
kvm_finalize_sys_regs() due to how QEMU blindly reads out then
verbatim restores the system register state. This avoids the issue
where both the GCIE and GIC features are marked as present (an
architecturally invalid combination), and hence guests fall over. See
the comment in kvm_finalize_sys_regs() for more details.
Overall, the following happens:
* Before an irqchip is created, FEAT_GCIE is presented if the host
supports GICv5-based guests.
* Once an irqchip is created, all other supported irqchips are hidden
from the guest; system register state reflects the guest's irqchip.
* Userspace is allowed to set invalid irqchip feature combinations in
the system registers, but...
* ...invalid combinations are removed a second time prior to the first
run of the guest, and things hopefully just work.
All of this extra work is required to make sure that "legacy" GICv3
guests based on QEMU transparently work on compatible GICv5 hosts
without modification.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260319154937.3619520-13-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
As part of booting the system and initialising KVM, create and
populate a mask of the implemented PPIs. This mask allows future PPI
operations (such as save/restore or state, or syncing back into the
shadow state) to only consider PPIs that are actually implemented on
the host.
The set of implemented virtual PPIs matches the set of implemented
physical PPIs for a GICv5 host. Therefore, this mask represents all
PPIs that could ever by used by a GICv5-based guest on a specific
host, albeit pre-filtered by what we support in KVM (see next
paragraph).
Only architected PPIs are currently supported in KVM with
GICv5. Moreover, as KVM only supports a subset of all possible PPIS
(Timers, PMU, GICv5 SW_PPI) the PPI mask only includes these PPIs, if
present. The timers are always assumed to be present; if we have KVM
we have EL2, which means that we have the EL1 & EL2 Timer PPIs. If we
have a PMU (v3), then the PMUIRQ is present. The GICv5 SW_PPI is
always assumed to be present.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260319154937.3619520-12-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
GICv5 has moved from using interrupt ranges for different interrupt
types to using some of the upper bits of the interrupt ID to denote
the interrupt type. This is not compatible with older GICs (which rely
on ranges of interrupts to determine the type), and hence a set of
helpers is introduced. These helpers take a struct kvm*, and use the
vgic model to determine how to interpret the interrupt ID.
Helpers are introduced for PPIs, SPIs, and LPIs. Additionally, a
helper is introduced to determine if an interrupt is private - SGIs
and PPIs for older GICs, and PPIs only for GICv5.
Additionally, vgic_is_v5() is introduced (which unsurpisingly returns
true when running a GICv5 guest), and the existing vgic_is_v3() check
is moved from vgic.h to arm_vgic.h (to live alongside the vgic_is_v5()
one), and has been converted into a macro.
The helpers are plumbed into the core vgic code, as well as the Arch
Timer and PMU code.
There should be no functional changes as part of this change.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Link: https://patch.msgid.link/20260319154937.3619520-10-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Prior to this change, the act of mapping a virtual IRQ to a physical
one also set the irq_ops. Unmapping then reset the irq_ops to NULL. So
far, this has been fine and hasn't caused any major issues.
Now, however, as GICv5 support is being added to KVM, it has become
apparent that conflating mapping/unmapping IRQs and setting/clearing
irq_ops can cause issues. The reason is that the upcoming GICv5
support introduces a set of default irq_ops for PPIs, and removing
this when unmapping will cause things to break rather horribly.
Split out the mapping/unmapping of IRQs from the setting/clearing of
irq_ops. The arch timer code is updated to set the irq_ops following a
successful map. The irq_ops are intentionally not removed again on an
unmap as the only irq_op introduced by the arch timer only takes
effect if the hw bit in struct vgic_irq is set. Therefore, it is safe
to leave this in place, and it avoids additional complexity when GICv5
support is introduced.
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Link: https://patch.msgid.link/20260319154937.3619520-6-sascha.bischoff@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
The current implementation uses `vgic_state_iter` in `struct
vgic_dist` to track the sequence position. This effectively makes the
iterator shared across all open file descriptors for the VM.
This approach has significant drawbacks:
- It enforces mutual exclusion, preventing concurrent reads of the
debugfs file (returning -EBUSY).
- It relies on storing transient iterator state in the long-lived
VM structure (`vgic_dist`).
Refactor the implementation to use the standard `seq_file` iterator.
Instead of storing state in `kvm_arch`, rely on the `pos` argument
passed to the `start` and `next` callbacks, which tracks the logical
index specific to the file descriptor.
This change enables concurrent access and eliminates the
`vgic_state_iter` field from `struct vgic_dist`.
Signed-off-by: Fuad Tabba <tabba@google.com>
Link: https://patch.msgid.link/20260202085721.3954942-4-tabba@google.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
The vgic-debug interface implementation uses XArray marks
(`LPI_XA_MARK_DEBUG_ITER`) to "snapshot" LPIs at the start of iteration.
This modifies global state for a read-only operation and complicates
reference counting, leading to leaks if iteration is aborted or fails.
Reimplement the iterator to use dynamic iteration logic:
- Remove `lpi_idx` from `struct vgic_state_iter`.
- Replace the XArray marking mechanism with dynamic iteration using
`xa_find_after(..., XA_PRESENT)`.
- Wrap XArray traversals in `rcu_read_lock()`/`rcu_read_unlock()` to
ensure safety against concurrent modifications (e.g., LPI unmapping).
- Handle potential races where an LPI is removed during iteration by
gracefully skipping it in `show()`, rather than warning.
- Remove the unused `LPI_XA_MARK_DEBUG_ITER` definition.
This simplifies the lifecycle management of the iterator and prevents
resource leaks associated with the marking mechanism, and paves the way
for using a standard seq_file iterator.
Signed-off-by: Fuad Tabba <tabba@google.com>
Link: https://patch.msgid.link/20260202085721.3954942-3-tabba@google.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Add the plumbing of GICv2 interrupt deactivation via GICV_DIR.
This requires adding a new device so that we can easily decode
the DIR address.
The deactivation itself is very similar to the GICv3 version.
Tested-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Tested-by: Mark Brown <broonie@kernel.org>
Link: https://msgid.link/20251120172540.2267180-39-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
