summaryrefslogtreecommitdiff
path: root/arch/arm64/kvm/nested.c
AgeCommit message (Collapse)Author
7 daysMerge branch kvm-arm64/cross-vcpu-sync into kvmarm-master/fixesMarc Zyngier
* kvm-arm64/cross-vcpu-sync: : \ : Fix a number of issues related to publication of state when a vcpu : is forced out of the guest, address other problems related to the : nesting and overhead of pausing the VM, and apply all of that to : some of the nasty corners of the VGIC handling. : : This eventually simplifies the handling of operations that are : inherently racy by pausing the involved vcpus. : / KVM: arm64: vgic-its: Remove the pending state of a discarded LPI KVM: arm64: vgic-its: Don't map an LPI from more than one event KVM: arm64: vgic-its: Stop the VM when handling MOVALL KVM: arm64: vgic: Stop the target vcpu when disabling LPIs KVM: arm64: Add helpers to halt a vCPU KVM: arm64: Turn vcpu->arch.pause into a counter KVM: arm64: Move IN_GUEST_MODE ahead of state being loaded KVM: arm64: Move OUTSIDE_GUEST_MODE publication past context being saved KVM: arm64: nv: Move L2->L1 switch on interrupt early Signed-off-by: Marc Zyngier <maz@kernel.org>
7 daysMerge branch kvm-arm64/hcr-cleanup into kvmarm-master/fixesMarc Zyngier
* kvm-arm64/hcr-cleanup: : \ : Cleanup the HCR_EL2 bit fields to exclusively use the generated : definitions. This also fixes a nasty regression introduced in : 6.16, where barriers were not upgraded from NSH to ISH anymore. : / KVM: arm64: Get rid of compatibility HCR_EL2 aliases KVM: arm64: Convert HCR_xyz to HCR_EL2_xyz KVM: arm64: Set HCR_EL2.BSU to Inner Shareable for guests Signed-off-by: Marc Zyngier <maz@kernel.org>
7 daysKVM: arm64: Convert HCR_xyz to HCR_EL2_xyzMarc Zyngier
Bulk convert all usages of HCR_xyz to HCR_EL2_xyz (the autogenerated version) using the following proof that I can't write shell scripts: awk 'match($3, /__HCR.*/) { print $2}' arch/arm64/include/asm/kvm_arm.h | while read i; do j=$(echo $i | sed -e "s/HCR_\(.*\)/\1/"); git grep -lw $i arch/arm64/ | xargs sed -i -e "s/\b$i\b/HCR_EL2_$j/g"; done ... and fixup the result. Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Tested-by: Fuad Tabba <fuad.tabba@linux.dev> Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com> Link: https://patch.msgid.link/20261015173605.1072963-2-maz@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-10-16Merge tag 'kvm-x86-coco-7.4' of https://github.com/kvm-x86/linux into HEADPaolo Bonzini
KVM guest_memfd and CoCo changes for 7.4 - Take the invalidate lock when unbinding a dying file to fix a race with kvm_gmem_invalidate_end() that results in KVM leaving the MMU invalidation count elevated. Not taking the lock in the first place was pure misguided paranoid. - When binding a memslot, purge the bindings if storing the new memslot fails, as Xarray's "store range" API can subtly fail partway through. - Establish memslot<=>guest_memfd bindings *after* the memslot is ready, as the memslot becomes reachable the instant the binding is installed. - Don't grab a page refcount when reloading an SNP VMSA and stop returning a "struct page" from guest_memfd's PFN lookup altogether, to ensure that KVM doesn't hold any unnecessary refcounts (the lifetime of the page is ensured by MMU notifiers and similar hooks). Preventing unnecessary refcounts greatly simplifies supporting in-place SHARED=>PRIVATE conversions. - Add support for in-place PRIVATE<=>SHARED conversion, i.e. allow a single guest_memfd instance to be used for both PRIVATE and SHARED variations of CoCo guests' memory (though not at the same time). This eliminates the need to double allocate guest memory for TDX and SNP, and is basically a hard requirement for pKVM, which relies on being able to preserve memory contents on conversions. - Hack-a-fix a deadlock bug in ->error_remove_folio() by returning -EBUSY if the filemap invalidate lock is contended. Punt on a proper fix for the moment, as a proper fix likely involves reworking truncate_error_folio(). - Add a pile of selftests for in-place conversions. - Misc guest_memfd cleanups Conflicts in arch/arm64/kvm/mmu.c: convert to use s2vi.
2026-10-15KVM: arm64: nv: Move L2->L1 switch on interrupt earlyMarc Zyngier
NV request generation to switch from L2 to L1 on interrupt is buried into the vgic flush code, while we should be able to deliver requests as early as possible. Similarly, MI generation happens *after* the need for a request has been evaluated, which means we potentially delay the maintenance interrupt by a full exit/entry cycle. Move these two things to happen much earlier in the entry code, and make sure the MI generation and PMU interrupt serve as input to the request switching back from L2 to L1. Cc: stable@vger.kernel.org Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Tested-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20261015100539.1011760-2-maz@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-10-11Merge branch kvm-arm64/pre-faulting into kvmarm-master/nextMarc Zyngier
* kvm-arm64/pre-faulting: : \ : Enable stage-2 pre-faulting in the canonical IPA space, implementing the : KVM_PRE_FAULT_MEMORY API. Patches courtesy of Lorenzo Stoakes, and based : on an initial work by Jack Thomson. : / KVM: selftests: Add nested pre-fault test for arm64 KVM: selftests: Add option for different backing in pre-fault tests KVM: selftests: Enable pre_fault_memory_test for arm64 Documentation: KVM: document arm64 KVM_PRE_FAULT_MEMORY KVM: arm64: Implement KVM_PRE_FAULT_MEMORY KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf() KVM: arm64: Propagate EHWPOISON in kvm_s2_fault_pin_pfn() KVM: arm64: Size the stage-2 memcache from the fault MMU KVM: arm64: Propagate and use kvm_s2_fault_result on S2 fault KVM: arm64: Propagate and use mmu in s2fd when handling guest aborts KVM: arm64: Propagate and use esr in s2fd when handling guest aborts KVM: arm64: Use ESR helpers in guest abort handling arm64: Add ESR fault helpers KVM: Allow architectures to disallow pre-fault Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-09-24KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf()Jack Thomson
Allow callers of kvm_pgtable_get_leaf() to specify the page-table walk flags, in preparation for performing walks under the MMU read lock. Reading a stage-2 leaf while only holding the read lock requires KVM_PGTABLE_WALK_SHARED: parallel faults (which also only hold the read lock) can unlink table pages and free them via RCU, so the walker must be inside an RCU read-side critical section, which the shared walk flag provides via kvm_pgtable_walk_begin(). All existing callers either hold the write lock, walk with interrupts disabled, or run at hyp where shared walks are rejected; they keep the current behaviour by passing no flags. No functional change intended. Signed-off-by: Jack Thomson <jackabt@amazon.com> [ljs: fixup new kvm_pgtable_get_leaf() invocation] Reviewed-by: Oliver Upton <oupton@kernel.org> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Tested-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260923-kvm-arm-prefault-v4-9-d4b0b4dfa8c3@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-09-24KVM: arm64: ptdump: Fix shadow ptdump sleep-in-atomic-context problemWei-Lin Chang
Previously we exposed shadow page tables by creating a debugfs ptdump file whenever a nested mmu instance gets bound to a new context, and deleting the debugfs file whose context was getting unbound. This turned out to be buggy, as the instance<->context binding process is done with the mmu_lock held, and debugfs creation/deletion can sleep. Change the approach and create a shadow ptdump file for each nested mmu instead during kvm_vcpu_init_nested(), in a sleepable context. The files will be named nested_mmu<index>, and reading it will return the nested context's VTTBR, VTCR, and s2 enabled or not before dumping the shadow page tables, given the nested mmu is valid. Fixes: 19e15dc73f0f ("KVM: arm64: nv: Expose shadow page tables in debugfs") Reported-by: Itaru Kitayama <itaru.kitayama@fujitsu.com> Closes: https://lore.kernel.org/kvmarm/aiuF0KSvvv-ZozI1@sm-arm-grace07/ Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com> Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260924094951.727352-3-weilin.chang@arm.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-09-22KVM: guest_memfd: Stop returning struct page from PFN lookupSean Christopherson
KVM currently expects guest_memfd PFN lookups to return a refcounted struct page, which callers hold across fault handling. Drop the page's refcount before returning from kvm_gmem_get_pfn() to prepare for the in-place conversions series. CoCo shared-to-private conversion handling must inspect folio refcounts to ensure exclusive ownership by guest_memfd. A concurrent guest page fault taking a temporary reference on the folio causes conversions to fail due to an elevated refcount. While this refcount is also taken on host userspace page faults, that refcount is taken on behalf of the host userspace page tables. This refcount will be dropped when conversions unmaps the page. Either way, once there's an mmap() or userspace mapping, the pages are open to way more refcounts, transient or not. This patch focuses on just dropping refcounts before handing KVM a page. guest_memfd already notifies KVM of page invalidations, so callers within KVM only need to respect the MMU invalidation protocol to safely rely on guest_memfd for page presence. Since the page refcounts are dropped, don't return the struct page pointer. Not returning the struct page from the guest_memfd PFN lookup moves KVM closer toward supporting memory backends that are not backed by struct page. Here are some notes on the cleanup in the callers of kvm_gmem_get_pfn(): kvm_release_faultin_page() in ARM's gmem_abort() originally also serves to set the page dirty and accessed under some conditions. The dirty and accessed flags don't matter for guest_memfd anyway, so it is safe to just drop the call to kvm_release_faultin_page(). For ARM's kvm_translate_vncr(), the local page pointer must be initialized to NULL so that the shared cleanup path that releases faulted-in pages safely no-ops for guest_memfd. For x86, no additional changes are required in the MMU fault path because the page fault tracking structure is zero-initialized at the start of page fault handling, ensuring the refcounted page pointer is already NULL. Reported-by: Yan Zhao <yan.y.zhao@intel.com> Closes: https://lore.kernel.org/all/anZ4W9o5pTWIEgMY@yzhao56-desk.sh.intel.com/ Co-developed-by: Yan Zhao <yan.y.zhao@intel.com> Signed-off-by: Yan Zhao <yan.y.zhao@intel.com> Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Tested-by: Michael Roth <michael.roth@amd.com> Tested-by: Yan Zhao <yan.y.zhao@intel.com> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Tested-by: Fuad Tabba <fuad.tabba@linux.dev> Co-developed-by: Ackerley Tng <ackerleytng@google.com> Signed-off-by: Ackerley Tng <ackerleytng@google.com> Reviewed-by: Gavin Shan <gshan@redhat.com> Link: https://patch.msgid.link/20260826-gmem-no-return-page-v4-5-3bb9c1ddb4e3@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-09-20KVM: arm64: nv: Drop kvm_s2_mmu pointer from kvm_guest_s2_mappingMarc Zyngier
As it appears that the kvm_guest_s2_mapping structure is quite large, and results in a 128 byte slab allocation, there is some incentive to shrink a bit. For this, replace the S2 MMU back-pointer with an index tucked into the low bits of the nested.start field. This allows us to shrink the structure by 8 bytes, and therefore to fit in a 96 byte slab. The index is also stored in the S2 MMU structure itself, which comes for free as it fits in an existing hole. The structure is also repacked to avoid other disgracious holes. Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Reviewed-by: Oliver Upton <oupton@kernel.org> Tested-by: Shuai Xue <xueshuai@linux.alibaba.com> Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com> Link: https://patch.msgid.link/20260915154305.3852871-7-weilin.chang@arm.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-09-20KVM: arm64: nv: Avoid full shadow stage-2 unmapWei-Lin Chang
With guest stage-2 tracking in place, we can improve MMU notifier unmaps from unmapping all existing shadow stage-2 mappings to only unmapping the ones affected by the given canonical IPA range. Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Reviewed-by: Oliver Upton <oupton@kernel.org> Tested-by: Shuai Xue <xueshuai@linux.alibaba.com> Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com> Link: https://patch.msgid.link/20260915154305.3852871-6-weilin.chang@arm.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-09-20KVM: arm64: nv: Track guest stage-2 mapping creationWei-Lin Chang
During shadow stage-2 faults, in addition to creating mappings in the shadow page tables, also allocate kvm_guest_s2_mapping objects, record the mapping ranges, and insert them into the canonical and nested mmu's guest_s2_mappings tree. Note that because we allow parallel faulting, the interval trees could store mappings that are not live in the shadow page tables. Storing a superset of the live mappings is fine because we will only over-unmap when we use this information later to do the targeted MMU notifier unmap. The mapping is also added to the interval trees if kvm_pgtable_stage2_map() returns -EAGAIN. This is required for example, when a 2M block map (A) races with a 4K page map (B): 1. (A) maps the 2M block in kvm_pgtable_visitor_cb() 2. (B) breaks that block into a table and maps 4K 3. (A) reloads and finds the table after kvm_pgtable_visitor_cb(), then descends into it. 4. (A) maps some 4K, but before it finishes it reads the entry mapped by (B). 5. (A) returns -EAGAIN although it had mapped a few pages. In this case, we don't know what subrange is mapped, just track the whole requested mapping range. Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Reviewed-by: Oliver Upton <oupton@kernel.org> Tested-by: Shuai Xue <xueshuai@linux.alibaba.com> Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com> Link: https://patch.msgid.link/20260915154305.3852871-4-weilin.chang@arm.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-09-20KVM: arm64: nv: Track guest stage-2 mapping removalWei-Lin Chang
kvm_stage2_unmap_range() is the helper to remove mappings from the stage-2 page tables. It is called during guest TLBI handling, memslot removal, nested mmu reuse, etc. Teach it about the guest stage-2 tracking trees and remove mappings from there when shadow mappings are removed. This keeps the tracking trees from having stale mappings pile up. Don't remove tracked mappings from the interval trees if they only partially overlap the removal range. For instance we can have a 4K shadow mapping tracked as a 2M range in the interval trees. It would be wrong to remove the tracked 2M range when a guest TLBI doesn't touch the 4K mapped. Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Reviewed-by: Oliver Upton <oupton@kernel.org> Tested-by: Shuai Xue <xueshuai@linux.alibaba.com> Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com> Link: https://patch.msgid.link/20260915154305.3852871-5-weilin.chang@arm.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-09-20KVM: arm64: nv: Introduce guest stage-2 tracking structuresWei-Lin Chang
In order to avoid unmapping all shadow stage-2 mappings when KVM receives a MMU notifier unmap call, we have to keep track of the canonical IPA -> nested IPA relationship of the shadow mappings created. This essentially means tracking the guest's stage-2. To do this, represent each mapping by struct kvm_guest_s2_mapping. It stores the mapping's canonical IPA range and the nested IPA range using two interval tree nodes. Both nodes will be inserted into their respective interval trees called guest_s2_mappings. The canonical IPA ranges will be stored in the tree within the canonical MMU, and the nested IPA ranges will be stored in the corresponding nested MMU's tree. For example: struct kvm_guest_s2_mapping mapping1, mapping2; ---------------------> mapping2.canonical | mapping1.canonical | ^ (both stored in canonical mmu's tree) | | --*****-----------------------*****----------- CIPA \\\\\ ||||| mapping1.nested_mmu \\\\\ \\\\\ | \\\\\ \\\\\ v ------\\\\\---------------------*****--------- NIPA #1 (nested mmu #1) \\\\\ | \\\\\ -> mapping1.nested \\\\\ (stored in nested mmu #1's tree) \\\\\ -----------*****------------------------------ NIPA #2 (nested mmu #2) | ^ -> mapping2.nested | (stored in nested mmu #2's tree) mapping2.nested_mmu Using the trees we can look up nodes in either of the IPA spaces, and for each node, find the corresponding range in the other IPA space from the other node in the enclosing kvm_guest_s2_mapping. Define kvm_guest_s2_mapping and the interval tree here. Guest stage-2 mapping tracking will come in subsequent patches. Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Reviewed-by: Oliver Upton <oupton@kernel.org> Tested-by: Shuai Xue <xueshuai@linux.alibaba.com> Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com> Link: https://patch.msgid.link/20260915154305.3852871-3-weilin.chang@arm.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-09-15KVM: arm64: nv: Delay freeing of shadow S2 structures until VM destructionMarc Zyngier
We free the shadow S2 structures from kvm_arch_flush_shadow_all(), which is a Bad Idea(tm). Freeing the page tables is fair game (this is what this callback is for), but freeing the container that could still be referenced by another part of the system is not great. Instead, grow separate destructors that gets called when we tear the VM down for good. From there, we can nuke both the individual MMUs as well as the global array that points to them, safe in the knowledge that the vcpus themselves have been destroyed already. Fixes: 4f128f8e1aaac ("KVM: arm64: nv: Support multiple nested Stage-2 mmu structures") Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com> Link: https://patch.msgid.link/20260911162203.1919330-3-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-09-15KVM: arm64: nv: Fix life cycle of the nested_mmus arrayMarc Zyngier
The nested_mmus array holds the shadow page tables that are used when a guest is running a nested context. These structures are allocated on VCPU_INIT for whole guest, which implies that they may have to be relocated as the array grows. Should a VCPU_INIT occur whilst a vcpu is actively running an L2 and that the allocation requires relocation, that vcpu will still be running with a pointer to the previous structure, which will have been freed. Fix this by turning the array of structures to an array of pointers, which is now allocated at VM creation, sized to the absolute maximum that KVM can handle. In turn, each VCPU_INIT contributes S2_MMU_PER_VCPU to the pool. No reallocation is ever performed, and the life cycle of each object is much clearer: - the nested_mmus array is allocated in kvm_init_nested(), and freed in kvm_arch_destroy_vm() - s2_mmu structures are allocated in kvm_vcpu_init_nested(), and freed on kvm_arch_flush_shadow_all() Finally, the freeing of vcpu->arch.vncr_array is made consistent rather than being done on some failure paths, but not others. Fixes: 4f128f8e1aaa ("KVM: arm64: nv: Support multiple nested Stage-2 mmu structures") Reported-by: Shen Yongchao <grayhat@foxmail.com> Reported-by: Karl Mehltretter <kmehltretter@gmail.com> Suggested-by: Karl Mehltretter <kmehltretter@gmail.com> Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Link: https://lore.kernel.org/r/20260803224405.41468-1-kmehltretter@gmail.com Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com> Link: https://patch.msgid.link/20260911162203.1919330-2-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-09-15KVM: arm64: nv: Fix null ptr deref on nested wp/unmap, teardown raceLorenzo Stoakes (ARM)
Commit 7270cc9157f4 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers") introduced VNCR_EL2 invalidation in both kvm_nested_s2_unmap() and kvm_nested_s2_wp(). However at the point of this being performed concurrent stage 2 teardown of a nested guest can cause kvm->arch.mmu.pgt to be set to NULL. This happens in kvm_flush_shadow_all() -> kvm_arch_flush_shadow_all() -> kvm_free_stage2_pgd() and is performed under the kvm->mmu_lock. Commit ec14c272408a ("KVM: arm64: nv: Unmap/flush shadow stage 2 page tables") introduced the teardown of the entire nested MMU range, which then invokes stage2_apply_range() with resched=true: mmu_notifier_invalidate_range_start() -> ... -> kvm_mmu_notifier_invalidate_range_start() -> kvm_mmu_unmap_gfn_range() -> kvm_unmap_gfn_range() -> kvm_nested_s2_unmap() -> kvm_stage2_unmap_range() -> __unmap_stage2_range() -> stage2_apply_range() This means that stage2_apply_range() can drop the kvm->mmu_lock and thus concurrent progress can be made in lockstep with kvm_arch_flush_shadow_all(). If kvm_arch_flush_shadow_all() advances ahead of stage2_apply_range() and completes its operation it guarantees a NULL pointer deref. Since kvm_free_stage2_pgd() is performed under the kvm->mmu_lock this will either be observed NULL or not and serialised against kvm_free_stage2_pgd(). Resolve the issue by abstracting the invalidation to a new function, kvm_invalidate_vncr_ipa_all(), and check that the pgt is non-NULL before dereferencing it. Fixes: 7270cc9157f4 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers") Cc: stable@vger.kernel.org Reviewed-by: Marc Zyngier <maz@kernel.org> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Tested-by: Jonathan Davies <jonathan.davies@nutanix.com> Link: https://patch.msgid.link/20260901-kvm-arm-nested-virt-fix-v3-2-b154676f7e4c@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-09-04treewide: refresh kmalloc_obj() conversionsKees Cook
This is another run of the Coccinelle script for converting kmalloc() family of allocations to kmalloc_obj() via the existing rules in scripts/coccinelle/api/kmalloc_objs.cocci This catches both the set of kmalloc() uses added since the first kmalloc_obj() conversions in v7.0 and adds a large group missed in the first pass due to Coccinelle not interacting well with the cleanup.h scoped_...() family of macros[1]. I worked around this with spatch's "--macro-file" argument to a file with all the scoped_...() macros mapped to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control flow indicator I could find. Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc, riscv, and s390 with no new warnings. Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1] Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2] Signed-off-by: Kees Cook <kees+treewide@kernel.org>
2026-08-19Merge branch 'kvm-arm64/vtr-patch' into nextOliver Upton
* kvm-arm64/vtr-patch: : Inline patching of ICH_VTR_EL2 constant, courtesy of Marc Zyngier : : Unify readers of ICH_VTR_EL2 on an instruction-patched constant value, : avoiding system register accesses known to trap under nested : virtualization and sharing the implementation between pKVM and 'regular' : KVM. KVM: arm64: vgic-v3: Kill kvm_vgic_global_state.ich_vtr_el2 KVM: arm64: vgic-v3: Simplify initial GICv3 configuration sampling KVM: arm64: Convert most ICH_VTR_EL2 accesses to inlined literal value KVM: arm64: Add a helper providing an inlined literal value for ICH_VTR_EL2 KVM: arm64: Move GICv3 broken SEIS implementation detection to a CPU errrata KVM: arm64: vgic-v3: Make vtr_to_* helpers use architectural field symbols Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/vncr-fixes' into nextOliver Upton
* kvm-arm64/vncr-fixes: : Fixes for handling L1 VNCR mappings, courtesy of Marc Zyngier : : - Recompute the VNCR software TLB when the MMU is toggled at stage-1 : : - Improve TLB invalidation intersection to handle TLBIs affecting the : end of the VA space : : - Fix race to invalidate the VNCR fixmap between TLBI emulation and : vcpu_put() : : - Add missing sign extension for computing TLBI ranges : : - Make VNCR invalidation participate in the MMU notifier seqcount, : preventing a concurrent VNCR TLB fill from consuming a stale : translation KVM: arm64: Correctly cap TLBI Range to the architural limit KVM: arm64: Add VNCR TLB tracking again KVM: arm64: Make VNCR invalidation participate in MMU invalidation retry KVM: arm64: Sign-extend VA for range-based TLBI invalidation KVM: arm64: Handle VNCR TLB invalidation race with vcpu_put() VNCR unmapping KVM: arm64: Correctly handle end of VA space TLBI invalidation KVM: arm64: Consider SCTLR_EL2.M when mapping the L1 VNCR page KVM: arm64: Handle negative S1 walk levels in VNCR TLB size evaluation KVM: arm64: Remove VM-wide VNCR mapping counter Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/feat-nv3' into nextOliver Upton
* kvm-arm64/feat-nv3: (29 commits) : Support for FEAT_NV2p1 and FEAT_NV3, courtesy of Marc Zyngier : : Couple of accelerations for nested virtualization: : : - Elide traps for EL1 alias registers when hardware guarantees that : EL2-only bits are stateful (NV2p1) : : - Use TGE-aware hardware to avoid unnecesary ERET/TLBI traps that : remain 'in-context' (i.e. L1 kernel/userspace) arm64: Add override for ID_AA64MMFR4_EL1.NV_frac KVM: arm64: selftest: Add NVHCR_EL2 to get-reg-list KVM: arm64: Expose FEAT_NV3 to guests KVM: arm64: Add FEAT_NV3 detection KVM: arm64: Engage NV3 TLBI trap elision KVM: arm64: Engage NV3 ERET trap elision KVM: arm64: Add NVHCR_EL2 context switching KVM: arm64: Add routing for NVHCR_EL2 trap KVM: arm64: Add NVHCR_EL2 handling to the sysreg array KVM: arm64: Add sanitisation for NVHCR_EL2 KVM: arm64: Make HCR_EL2 a non-VNCR register KVM: arm64: Add kvm_has_nv{2,3}() predicates KVM: arm64: Add NV3 control bits to HCRX_EL2 sanitisation KVM: arm64: Split NV-specific exit fixups from the non-NV handling arm64: Add ARM64_HAS_NV3 capability arm64: sysreg: Add HCRX_EL2 bits related to FEAT_NV3 arm64: sysreg: Add NVHCR_EL2 description as a mirror of HCR_EL2 arm64: Add FEAT_NV2p1 detection KVM: arm64: Expose FEAT_NV2p1 to NV guests KVM: arm64: Relax CNTHCTL_EL2 handling when FEAT_NV2p1 is present ... Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Add VNCR TLB tracking againMarc Zyngier
Having established that our VNCR TLB tracking was flawed and having dropped it from KVM, it is time to replace it with something that actually works in order to get some performance back, as the TLBI overhead is otherwise pretty high. The goal of such tracking is to hit the TLBI slow path if there are any VNCR TLBs in the guest, irrespective of their mapping state. For this purpose, we introduce an VM wide counter (vncr_tlb_count) that tracks how many valid VNCR TLB are present. This means that creating such TLB must increment the counter, and invalidation decrement it, and both these operations must be done with the MMU lock held for write. On TLBI handling affecting EL2 S1, a non-zero counter forces the handling to take the slow path to consider the VNCR TLBs. Note that the bumping up is done "speculatively" in order to make sure that a concurrent invalidation will be taken on the slow path. Not exactly rocket science. Hopefully I got it right this time. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260806091026.620700-9-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Make VNCR invalidation participate in MMU invalidation retryMarc Zyngier
A VNCR TLB invalidation can occur on one vcpu while another vcpu is faulting in this same page. Without correctly handling this, we can end up with the following scenario: - vcpu A walks the PTs to translate VNCR - before vcpu A is able to grab the MMU lock to insert the TLB, vcpu B updates the S1 PTs with an invalid entry, and issues a TLBI S1E2 for this VA - vcpu A inserts the TLB for something that is now invalid This isn't a new problem, and we manage S2 by having the MMU notifier to bump up mmu_invalidate_seq on invalidation so that the fault can be replayed. We can perform something similar here, and extend invalidate_vncr_va() to update the same counter, clearly indicating that the context has changed under our feet. This is safe as the invalidation always happen while holding the MMU lock for write, and that we sample the sequence number before walking S1. Fixes: 4ffa72ad8f37e ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2") Reported-by: sashiko-bot@kernel.org Link: https://lore.kernel.org/r/20260801130454.5D9F11F00AC4@smtp.kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260806091026.620700-8-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Handle VNCR TLB invalidation race with vcpu_put() VNCR unmappingMarc Zyngier
While VNCR TLB invalidation always occurs under the MMU lock, vcpu_put() doesn't, while it unmaps the VNCR page. The problem is that the invalidation evaluates vncr_tlb::cpu to decide whether an unmapping needs to take place (cpu != -1) before performing it. On the other hand, this_cpu_reset_vncr_fixmap() unconditionally unmaps if L1_VNCR_MAPPED is set. These two obviously can race, with a TOCTOU pattern on the TLBI path, and a BUG_ON() on the vcpu_put() path. And the two can end-up calling vncr_fixmap(-1), with extra lethal effects. Move the reset of vncr_tlb::cpu to -1 to a common function, and make this update atomic so that only a single thread can reset the field and perform the corresponding unmap. The vcpu_put() still need to unconditionally unmap the current VNCR to close another ugly race. Finally, the assignment of vncr_tlb::cpu is moved to be kept in sync with the actual mapping, similar to L1_VNCR_MAPPED being set. Fixes: 7270cc9157f47 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers") Reported-by: sashiko-bot@kernel.org Link: https://lore.kernel.org/r/20260801130237.0FD8F1F00ACA@smtp.kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Link: https://patch.msgid.link/20260806091026.620700-6-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Correctly handle end of VA space TLBI invalidationMarc Zyngier
Our TLB invalidation by VA code is based on comparing two ranges, one defined by the TLB, and one defined by the TLBI instruction. Each range is defined by a start and a size. However, the way the comparison is done doesn't account for address rollover, as it compares an address with (base + size). This works nicely until this expression represent the last page/block in the TTBR1 VA space, as the result is a big fat 0. And a failed TLB invalidation. Rewrite the comparison in a way that is immune to the address rollover (making the end address inclusive instead of exclusive), and move this into a common helper that is used by both VA and IPA invalidations, as suggested by Hyunwoo Kim (although the IPA version didn't suffer from this particular problem, obviously). Fixes: 4ffa72ad8f37e ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2") Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260806091026.620700-5-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Consider SCTLR_EL2.M when mapping the L1 VNCR pageMarc Zyngier
We record a VNCR TLB even when SCTLR_EL2.M is 0 in order to make our life easier. But this is not something that the architecture anticipate. As a consequence, a hypervisor is free to set VNCR_EL2 to some PA when SCTLR_EL2.M==0, use it to run a guest which indirectly accesses the VNCR page, then eventually set SCTLR_EL2.M==1 with the same VA. Yes, this is odd, but apparently legal. A common trick in HW is to invalidate the TLBs on SCTLR_ELx.M being flipped. But doing this is a not a good idea for us (we'd need to trap SCTLR accesses), and wouldn't scale as we nest deeper. Instead, use the fact that the S1 MMU being off at the point of translation is cached in our TLB, and if it doesn't match the current MMU state, leave the VNCR unmapped. Fixes: 2a359e072596f ("KVM: arm64: nv: Handle mapping of VNCR_EL2 at EL2") Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260806091026.620700-4-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Handle negative S1 walk levels in VNCR TLB size evaluationMarc Zyngier
Computing the effects of a TLB invalidation involves looking at the size of the mapping cached by the TLB. For S1 mappings such as VNCR, this is deducted from the combination of the base granule size and the mapping level. However, this implies that the S1 MMU is *on*. When the MMU is off, we indicate this with the level being set to a "creative" value of -127 (S1_MMU_DISABLED). This ends-up being misinterpreted by pgshift_level_to_ttl() as it doesn't handle negative levels at all (the level is immediately cast to a u8 and only the bottom two bits considered), leading to an invalidation size of 0. Not helpful. Tidy-up pgshift_level_to_ttl() to handle these negative levels, and ttl_to_size() to always return SZ_1G when no valid TTL is present. This allows the removal of open-coded checks for similar situations. Note that the check for a negative value not explicitely checking for S1_MMU_DISABLED is deliberate, so that actual negative levels introduced with LVA2 and D128 can take the same path if we ever support them. Fixes: 7270cc9157f47 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers") Reported-by: Hyunwoo Kim <imv4bel@gmail.com> Link: https://lore.kernel.org/r/ameGoxbn2wzBq2kL@v4bel Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260806091026.620700-3-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Remove VM-wide VNCR mapping counterMarc Zyngier
The global VNCR mapping counter is used to decide whether an L1 provided VNCR page is mapped in L0 on any CPU at the point of dealing with a TLB invalidation. It is incremented when a mapping is made in the fixmap, and decremented when unmapped. As it turns out, this tracking has several flaws: - we are trying to invalidate TLBs, and the mapping is only an opportunistic consequence of the TLB. Checking this counter to decide whether a TLB needs to be invalidated may result in missed invalidations. - an L1 vcpu invalidating its own TLB (a very likely case) will not succeed in invalidating the VNCR pseudo TLB because that page is not mapped in L0 at this stage. Given that this tracking fails at delivering the minimum guarantees that are required and is only a performance optimisation, remove it completely. Fixes: 4ffa72ad8f37e ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2") Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Link: https://patch.msgid.link/20260806091026.620700-2-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Expose FEAT_NV3 to guestsMarc Zyngier
Further enable FEAT_NV3 by making it visible to NV guests. Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-28-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Add sanitisation for NVHCR_EL2Marc Zyngier
Just like any other VNCR-based register, NVHCR_EL2 requires some level of sanitisation. Being specified as a live copy of HCR_EL2, it adopts the exact same format, but depends on FEAT_NV3 instead. A subtle aspect is that we only want to apply the sanitisation if FEAT_NV3 is actually present, as the VNCR location is otherwise used to back accesses to HCR_EL2. Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-21-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Expose FEAT_NV2p1 to NV guestsMarc Zyngier
Since NV2p1 is reducing the number of traps, it is valuable to expose it to NV guests. Do so. Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-12-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-21KVM: arm64: vgic-v3: Kill kvm_vgic_global_state.ich_vtr_el2Marc Zyngier
kvm_vgic_global_state.ich_vtr_el2 is the last bit of caching that we can get rid of. Not as bad as a sysreg access, but still worse than a constant. Move over to the inlined stuff and remove the cached value. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260721170754.3150521-7-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-15Merge tag 'kvmarm-fixes-7.2-1' of ↵Paolo Bonzini
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD KVM/arm64 fixes for 7.2, take #1 - Fix an accounting buglet when reclaiming pages from a protected guest - Fix a bunch of architectural compliance issues when injecting a synthesised exception, most of which were missing the PSTATE.IL bit indicating a 32bit-wide instruction - Another set of fixes addressing issues with translation of VNCR_EL2, including corner cases where the guest point that register at a RO page... - Don't warn when trapping accesses to ZCR_EL2 from an L2 guest, as that's not unexpected at all - Address a bunch of races with LPI migration vs LPIs being disabled - Fix a total howler of a bug combining FEAT_MOPS and NV, resulting in exception returning in the wrong place... - Coerce Fuad Tabba into a reviewer role, and may his Inbox catch fire!
2026-06-22KVM: arm64: nv: Mark VM as bugged for unexpected VNCR abortOliver Upton
KVM is unlikely to resolve an unexpected VNCR abort, meaning that returning to the guest will likely leave the vCPU stuck in an abort loop. Bug the VM and exit to userspace instead. Signed-off-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260618234207.1063941-6-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-22KVM: arm64: nv: Inject SEA if guest VNCR isn't normal memoryOliver Upton
When constructing an L1 VNCR mapping, KVM unconditionally uses cacheable memory attributes, even if the underlying PFN isn't memory. This gets particularly hairy if the endpoint doesn't support cacheable memory attributes, potentially throwing an SError on writeback... While KVM does permit cacheable memory attributes on certain PFNMAP VMAs, kvm_translate_vncr() isn't currently grabbing the VMA. So do the simpler thing for now and just reject everything that isn't memory. Cc: stable@vger.kernel.org Fixes: 2a359e072596 ("KVM: arm64: nv: Handle mapping of VNCR_EL2 at EL2") Signed-off-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260618234207.1063941-5-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-22KVM: arm64: nv: Re-translate VNCR before injecting abortOliver Upton
KVM faults in the VNCR page with FOLL_WRITE whenever the guest aborts for a write, similar to how a regular stage-2 mapping is handled. It is entirely possible that the guest reads from the VNCR before writing to it, in which case the PFN could only be read-only. Invalidate the VNCR TLB and re-fetch the translation upon taking a VNCR abort, allowing the host mapping to be faulted in for write the second time around. Interestingly enough, this also satisfies the ordering requirements of FEAT_ETS2/3 between descriptor updates and MMU faults. Cc: stable@vger.kernel.org Fixes: 2a359e072596 ("KVM: arm64: nv: Handle mapping of VNCR_EL2 at EL2") Reported-by: Sashiko <sashiko-bot@kernel.org> Signed-off-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260618234207.1063941-4-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-22KVM: arm64: nv: Inject SEA if kvm_translate_vncr() can't resolve PFNOliver Upton
kvm_handle_vncr_abort() assumes that s1_walk_result conveys an abort when kvm_translate_vncr() returns -EFAULT. This is not always the case as it's possible to encounter 'late' failures on the output of S1 translation, e.g. a GFN outside of the memslots. Fix it by preparing an external abort before returning from kvm_translate_vncr(). Get rid of the BUG_ON() in the fault injection path while at it. Cc: stable@vger.kernel.org Fixes: 2a359e072596 ("KVM: arm64: nv: Handle mapping of VNCR_EL2 at EL2") Signed-off-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260618234207.1063941-3-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-22KVM: arm64: nv: Respect read-only PFN when mapping L1 VNCROliver Upton
KVM currently maps the L1 VNCR into the host stage-1 by relying entirely on the permissions of the guest stage-1. At the same time, it is entirely possible that the backing PFN is read-only (e.g. RO memslot), meaning that the L1 VNCR should use at most a read-only mapping. Cache the writability of the PFN in the VNCR TLB and use it to constrain the resulting fixmap permissions. Promote VNCR permission faults to an SEA in the case where the guest attempts to write to a read-only endpoint. Conveniently, this also plugs a page leak found by Sashiko [*] resulting from the early return for a read-only PFN. Cc: stable@vger.kernel.org Fixes: 2a359e072596 ("KVM: arm64: nv: Handle mapping of VNCR_EL2 at EL2") Link: https://lore.kernel.org/kvm/20260608082603.16AEC1F00893@smtp.kernel.org/ Signed-off-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260618234207.1063941-2-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-12Merge tag 'kvmarm-7.2' of ↵Paolo Bonzini
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD KVM/arm64 updates for 7.2 * New features: - None. Zilch. Nada. Que dalle. * Fixes and other improvements: - Significant cleanup of the vgic-v5 PPI support which was merged in 7.1. This makes the code more maintainable, and squashes a couple of bugs in the meantime. - Set of fixes for the handling of the MMU in an NV context, particularly VNCR-triggered faults. S1POE support is fixed as well. - Large set of pKVM fixes, mostly addressing recurring issues around hypervisor tracking of donated pages in obscure cases where the donation could fail and leave things in a bizarre state. - Fixes for the so-called "lazy vgic init", which resulted in sleeping operations in non-preemptible sections. This turned out to be far more invasive than initially expected... - Reduce the overhead of L1/L2 context switch by not touching the FP registers. - Fix the way non-implemented page sizes are dealt with when a guest insist on using them for S2 translation. - The usual set of low-impact fixes and cleanups all over the map.
2026-06-12Merge branch kvm-arm64/nv-mmu-7.2 into kvmarm-master/nextMarc Zyngier
* kvm-arm64/nv-mmu-7.2: : . : Assorted collection of fixes for NV MMU bugs : : - Correctly plug AT S1E1A handling in the emulation backend : : - Make CPTR_EL2.E0POE depend on FEAT_S1POE : : - Drop the reference on the page if the VNCR translation : races with an MMU notifier : : - Correctly synthesise an SEA if a page table walk fails due : to a guest error : : - Fully invalidate the VNCR TLB and fixmap when translating : for a new VNCR : : - Restart S1 walk when the S2 walk fails due to a race condition : : - Correctly return -EAGAIN when a S1 walk fails : : - Fix block mapping validity check in stage-1 walker for 64kB pages : : - Fix potential NULL dereference when performing an EL2 TLBI targeting : the VNCR page : : - Hold kvm->mmu_lock while initialising the vncr_tlb pointer : . KVM: arm64: nv: Hold kvm->mmu_lock while initialising vcpu->arch.vncr_tlb KVM: arm64: nv: Avoid dereferencing NULL VNCR pseudo-TLB KVM: arm64: Fix block mapping validity check in stage-1 walker KVM: arm64: nv: Restart stage-1 walk if stage-2 desc update fails KVM: arm64: Restart instruction upon race in __kvm_at_s12() KVM: arm64: nv: Inject SEA TTW when desc update can't write to GPA KVM: arm64: nv: Fully update VNCR fixmap state in kvm_translate_vncr() KVM: arm64: Don't leak PFN when kvm_translate_vncr() races MMU notifier arm64: cpufeature: Expose ID_AA64ISAR2_EL1.ATS1A to KVM KVM: arm64: Wire AT S1E1A in the system instruction handling table KVM: arm64: Key CPTR_EL2.E0POE propagation on FEAT_S1POE Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-10KVM: arm64: nv: Hold kvm->mmu_lock while initialising vcpu->arch.vncr_tlbMarc Zyngier
Sashiko reports that there is a race between initialising vncr_tlb and making use of it, as we don't hold the mmu_lock at this point. Additionally, it identifies a memory leak, should userspace repeatedly invokes the KVM_RUN ioctl after a failure of kvm_arch_vcpu_run_pid_change(), as we assign vncr_tlb blindly on first run, irrespective of prior allocations. Slap the two bugs in one go by taking the kvm->mmu_lock on assigning vncr_tlb, preventing the race for good, and by checking that vncr_tlb is indeed NULL prior to allocation. Reported-by: Sashiko <sashiko-bot@kernel.org> Link: https://lore.kernel.org/r/20260607180815.85FBC1F00893@smtp.kernel.org Reviewed-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260608081108.2244133-1-maz@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-10KVM: arm64: nv: Avoid dereferencing NULL VNCR pseudo-TLBMarc Zyngier
VNCR TLB invalidation occurs from MMU notifiers or TLBI instructions, and either can race against a vcpu not being onlined yet (no pseudo-TLB allocated). Similarly, the TLB might be invalid, and the invalidation should be skipped in this case. Both kvm_invalidate_vncr_ipa() and kvm_invalidate_vncr_va() are expected to perform the same checks, except that the latter doesn't check for the allocation and blindly dereferences the pointer. Solve this by introducing a new iterator built on top of the usual kvm_for_each_vcpu() that checks for both of the above conditions, and convert the two users to it. Reported-by: Hyunwoo Kim <imv4bel@gmail.com> Link: https://lore.kernel.org/r/aiUvSbrWndQeUPc8@v4bel Fixes: 4ffa72ad8f37 ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2") Cc: stable@vger.kernel.org Reviewed-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260607175745.297793-1-maz@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-05KVM: arm64: Reassign nested_mmus array behind mmu_lockHyunwoo Kim
kvm->arch.nested_mmus[] is walked under kvm->mmu_lock, including from the MMU notifier path (kvm_unmap_gfn_range() -> kvm_nested_s2_unmap()), which can run at any time. kvm_vcpu_init_nested() reallocates the array and frees the old buffer while holding only kvm->arch.config_lock, so such a walker can reference the freed array. Allocate the new array outside of mmu_lock, as the allocation can sleep. Under the lock, copy the existing entries, fix up the back pointers and reassign the array. Free the old buffer after dropping the lock, as kvfree() can sleep as well. Fixes: 4f128f8e1aaac ("KVM: arm64: nv: Support multiple nested Stage-2 mmu structures") Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com> Reviewed-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/aiKIVVeIr1aAB1yp@v4bel Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger,kernel.org
2026-06-03KVM: arm64: nv: Inject SEA TTW when desc update can't write to GPAOliver Upton
Similar to the handling of descriptor reads, inject an SEA during TTW when the descriptor access fails for reasons other than a race, such as a read-only memslot or a bad HVA. Fixes: bff8aa213dee ("KVM: arm64: Implement HW access flag management in stage-1 SW PTW") Fixes: e4c7dfac2f1a ("KVM: arm64: nv: Implement HW access flag management in stage-2 SW PTW") Signed-off-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260602235450.103057-4-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-03KVM: arm64: nv: Fully update VNCR fixmap state in kvm_translate_vncr()Oliver Upton
kvm_translate_vncr() first invalidates the pseudo-TLB entry and corresponding fixmap in anticipation of installing a new translation. While the fixmap invalidation does clear the mapping from host stage-1, it does not clear the L1_VNCR_MAPPED flag. Depending on the state of the VNCR TLB at vcpu_put(), this could potentially precipitate a BUG_ON() if vt->cpu is reset. Share a helper with kvm_vcpu_put_hw_mmu(), ensuring that KVM's view of the VNCR fixmap is in sync with the state of the VNCR TLB. Give it a slightly verbose name to make it obvious that it is meant to be used local to a CPU, unlike other VNCR TLB maintenance. Fixes: 069a05e535496 ("KVM: arm64: nv: Handle VNCR_EL2-triggered faults") Signed-off-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260602235450.103057-3-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-03KVM: arm64: Don't leak PFN when kvm_translate_vncr() races MMU notifierOliver Upton
In the case that kvm_translate_vncr() races with an MMU notifier the early return does not release a reference on the faulted in PFN. Add the necessary call to kvm_release_faultin_page() for the unused PFN. Cc: stable@vger.kernel.org Fixes: 069a05e535496 ("KVM: arm64: nv: Handle VNCR_EL2-triggered faults") Reported-by: Sashiko (local):gemini-3.1-pro Signed-off-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260602235450.103057-2-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-05-29KVM: arm64: Correctly cap ZCR_EL2 provided by a guest hypervisorMark Brown
ZCR_EL2 can be updated by a VHE guest hypervisor either using ZCR_EL2 (which traps) or ZCR_EL1 (which does not trap). KVM handles both in different way: - on ZCR_EL2 trap, ZCR_EL2.LEN is immediately capped at the VM's own VL limit. This has the potential to break existing SW that relies on the full LEN field to be stateful. - on ZCR_EL1 access, we do absolutely nothing. On restoring the SVE context for an L2 guest, we directly restore the guest hypervisor's view of ZCR_EL2 into the physical ZCR_EL2. If the guest's view of the register was updated using the ZCR_EL2 accessor, the value has already been sanitised (with the caveat mentioned above). But if the guest used ZCR_EL1, the raw value is written into the HW, and the L2 guest can now access VLs that it shouldn't. Fix all the above by moving the VL capping to the restore points, ensuring that: - the HW is always programmed with a capped value, irrespective of the accessor being used, - the ZCR_EL2.LEN field is always completely stateful, irrespective of the accessor being used. Additionally, move ZCR_EL2 to be a sanitised register, ensuring that only the LEN field is actually stateful. This requires some creative construction of the RES0 mask, as the sysreg generation script does not yet generate RAZ/WI fields. Fixes: b3d29a823099 ("KVM: arm64: nv: Handle ZCR_EL2 traps") Signed-off-by: Mark Brown <broonie@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260529-kvm-arm64-fix-zcr-len-nv-v2-1-86cad51992bd@kernel.org [maz: rewrote commit message, tidy up access_zcr_el2()] Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-05-28KVM: arm64: Fallback to a supported value for unsupported guest TGxWei-Lin Chang
When KVM derives the translation granule for emulated stage-1 and stage-2 walks, it decodes TCR/VTCR.TGx and treats the granule as-is. This is wrong when the guest programs a granule size that is not advertised in the guest's ID_AA64MMFR0_EL1.TGRAN* fields. Architecturally, such a value must be treated as an implemented granule size. Choose an available one while prioritizing PAGE_SIZE. Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Link: https://patch.msgid.link/20260414000334.3947257-5-weilin.chang@arm.com [maz: minor tidying up] Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-05-28KVM: arm64: nv: Use literal granule size in TLBI range calculationWei-Lin Chang
TLBI handling derives the invalidation range from guest VTCR_EL2.TG0 in get_guest_mapping_ttl() and compute_tlb_inval_range(). Switch these to use a helper that returns the decoded VTCR_EL2.TG0 granule size instead of decoding it inline. This keeps the granule size derivation in one place and prepares for following changes that adjust the effective granule size. Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Link: https://patch.msgid.link/20260414000334.3947257-4-weilin.chang@arm.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-05-28KVM: arm64: Factor out TG0/1 decoding of VTCR and TCRWei-Lin Chang
The current code decodes TCR.TG0/TG1 and VTCR.TG0 inline at several places. Extract this logic into helpers so the granule size can be derived in one place. This enables us to alter the effective granule size in the same place, which we will do in a later patch. Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Link: https://patch.msgid.link/20260414000334.3947257-3-weilin.chang@arm.com Signed-off-by: Marc Zyngier <maz@kernel.org>