| Age | Commit message (Collapse) | Author |
|---|
|
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git
# Conflicts:
# arch/arm64/include/asm/ptrace.h
# arch/arm64/include/asm/sysreg.h
# arch/arm64/kvm/hyp/nvhe/hyp-main.c
|
|
|
|
* kvm-arm64/pre-faulting:
: \
: Enable stage-2 pre-faulting in the canonical IPA space, implementing the
: KVM_PRE_FAULT_MEMORY API. Patches courtesy of Lorenzo Stoakes, and based
: on an initial work by Jack Thomson.
: /
KVM: selftests: Add nested pre-fault test for arm64
KVM: selftests: Add option for different backing in pre-fault tests
KVM: selftests: Enable pre_fault_memory_test for arm64
Documentation: KVM: document arm64 KVM_PRE_FAULT_MEMORY
KVM: arm64: Implement KVM_PRE_FAULT_MEMORY
KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf()
KVM: arm64: Propagate EHWPOISON in kvm_s2_fault_pin_pfn()
KVM: arm64: Size the stage-2 memcache from the fault MMU
KVM: arm64: Propagate and use kvm_s2_fault_result on S2 fault
KVM: arm64: Propagate and use mmu in s2fd when handling guest aborts
KVM: arm64: Propagate and use esr in s2fd when handling guest aborts
KVM: arm64: Use ESR helpers in guest abort handling
arm64: Add ESR fault helpers
KVM: Allow architectures to disallow pre-fault
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Neither the VCPU_RUN in kvm_main.c nor the VCPU RUN in kvm_host.h names
an ioctl that exists. The one that enters the guest is KVM_RUN, so refer
to it by that name.
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260915084441.3846947-2-fuad.tabba@linux.dev
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Signed-off-by: Christian Brauner <brauner@kernel.org>
|
|
* misc: (45 commits)
KVM: x86: Request event processing for exception VM-Exits
KVM: selftests: Add coverage for the EFER_LMSLE_MBZ defeature
KVM: selftests: Add module param API to check if nested virtualization is enabled
KVM: selftests: Rename svm_nested_clear_efer_svme to svm_nested_efer_test
KVM: x86: Honor the guest's EFER_LMSLE_MBZ
KVM: x86: Advertise EFER_LMSLE_MBZ when KVM disallows EFER.LMSLE
KVM: SVM: Add support for virtualizating Bus Lock Detect
KVM: SVM: Add a vCPU-aware helper to get supported DEBUGCTL bits
KVM: nSVM: Don't assume all active-low bits DR6 are fixed-1
KVM: nSVM: Open code check on LBR virtualization being enabled in vmcb12
KVM: nSVM: Disable LBRV in nested control cache when unsupported
KVM: SVM: Add helper to query if LBR virtualization needs to be enabled
KVM: x86: Kill off DR6_FIXED_1 to prevent future misuse
KVM: x86: Set guest's fixed-1 DR6 bits when delivering #DB payload
KVM: x86: Force fixed-1 bits in DR6 after synchronizing with hardware
KVM: Return an "unsigned long", not "u64" for the fixed-1 DR6 bits
KVM: x86: Rename kvm_dr6_fixed() => kvm_get_dr6_fixed_1()
KVM: x86: Preserve DR6.BLD (Bus Lock Detect) when delivering #DB payload
KVM: x86: Restrict saved GPA writes to hardware write faults
KVM: x86/pmu: Don't retry a counter whose config was rejected
...
|
|
* fixes: (69 commits)
KVM: x86: Use active memslots for the per-vCPU MMIO cache
KVM: SEV: Fix page dirtying in sev_gmem_post_populate()
KVM: SVM: Clear AVIC Physical ID table entry if vCPU creation fails
KVM: SVM: Use "is AVIC-addressable" helper to sanity check load()/put()
KVM: SVM: Add paranoid helper for checking if vCPU is AVIC-addressable
KVM: SEV: Do cache maintenance on the source VM *before* clearing SEV state
KVM: SEV: Nullify "have run CPUs" mask pointer when freeing it
KVM: selftests: Extend nested x2APIC test to validate using eVMCS for vmcs12
KVM: selftests: Extend nested x2APIC test to validate disabling x2APIC virt
KVM: selftests: Verify that L0's TPR doesn't get clobbered
KVM: selftests: Run the nested x2APIC with and without APICv being inhibited in L2
KVM: selftests: Add x2APIC MSR test for inhibiting APICv while nested
KVM: nVMX: Force MSR bitmap refresh if runtime eVMCS controls are modified
KVM: SVM: Use the active VMCB's MSR bitmap when checking if MSR is intercepted
KVM: SVM: Sync guest's PERF_CNTR_GLOBAL_CTL from h/w only on successful VMRUN
KVM: SVM: Don't mark ASID fields as dirty when setting control.tlb_ctl
KVM: SVM: Update control fields on #VMEXIT if and only if VMRUN succeeded
KVM: SVM: Preserve TLB control (i.e. pending TLB flush) on failed VMRUN
KVM: x86/mmu: Bail from shadow walks if the root is invalid or a dummy
KVM: WARN if vCPU creation is in-progress when locking all vCPUs
...
|
|
* coco: (59 commits)
KVM: guest_memfd: Avoid deadlock in error_remove_folio()
KVM: guest_memfd: Fix the in-place conversion documentation
KVM: selftests: Update private memory exits test to work with per-gmem attributes
KVM: selftests: Update private_mem_conversions_test for in-place conversions
KVM: selftests: Set up page size and alignment independently for guest_memfd
KVM: selftests: Make TEST_EXPECT_SIGBUS thread-safe
KVM: selftests: Provide common function to set memory attributes
KVM: selftests: Provide function to look up guest_memfd details from gpa
KVM: selftests: Reset shared memory after hole-punching
KVM: selftests: Test conversion with elevated page refcount
KVM: selftests: Add helpers to pin pages with CONFIG_GUP_TEST
KVM: selftests: Test that shared/private status is consistent across processes
KVM: selftests: Test that truncation does not change shared/private status
KVM: selftests: Convert with allocated folios in different layouts
KVM: selftests: Test conversion before allocation
KVM: selftests: Test conversion precision in guest_memfd
KVM: selftests: Test conversion flow when INIT_SHARED
KVM: selftests: Test basic single-page conversion flow
KVM: selftests: Add helpers for calling ioctls on guest_memfd
KVM: selftests: Add selftests global for guest memory attributes capability
...
|
|
Remove the now unused file_is_kvm function.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
Switch kvm_get_kvm, kvm_get_kvm_safe, and kvm_put_kvm from
EXPORT_SYMBOL_GPL to EXPORT_SYMBOL_FOR_KVM_INTERNAL. These symbols are
now only used within KVM itself. No external module needs them now that
VFIO uses file-based reference counting. Remove the corresponding
exemptions from the x86 KVM Makefile exports check.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
Replace manual module reference counting with file-based reference
counting for KVM integration. Previously, VFIO used symbol_get() to
obtain function pointers for kvm_get_kvm_safe() and kvm_put_kvm(),
then manually tracked module references through these symbols. This
approach required storing the put_kvm function pointer in each device
and carefully managing symbol references.
Pass struct file pointers instead of struct kvm pointers throughout the
VFIO-KVM interface, leveraging the kernel's existing file reference
counting via get_file()/get_file_active() and fput(). Convert the x86
page-track API and update s390 vfio to use file_to_kvm_<arch>(). This
simplifies the code.
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Acked-by: Alex Williamson <alex@shazbot.org>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
[borntraeger@linux.ibm.com remove _safe from vfio_device_get_kvm_safe]
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
Add kvm->file as a no-reference back-pointer to the VM file in struct
kvm. The pointer is set at VM creation time and cleared under
WRITE_ONCE() in kvm_vm_release() before the file is freed.
Callers storing the file must take their own reference.
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
Add a macro mechanism that generates an opt-in and arch-namespaced
file_to_kvm_<arch>() to convert a file reference to a kvm object if the
file handle represents a kvm handle.
Activate for x86 and for s390 (native KVM only).
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
memory_failure() invokes ->error_remove_folio() while holding the
poisoned folio lock. KVM's guest_memfd callback then takes
mapping->invalidate_lock for read before invalidating guest mappings.
This can deadlock with paths that hold mapping->invalidate_lock for write
and then try to lock the same folio, e.g. truncation from MADV_REMOVE or
FALLOC_FL_PUNCH_HOLE. Because rwsems are fair, the problem is not limited
to guest_memfd punch-hole: a pending writer can also block future readers,
so ->error_remove_folio() must not block on the invalidate lock while
holding the poisoned folio lock.
Use filemap_invalidate_trylock_shared() in kvm_gmem_error_folio(). If the
invalidate lock is contended, fail the recovery instead of potentially
deadlocking the memory-failure path.
Fixes: a7800aa80ea4 ("KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory")
Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
Link: https://lore.kernel.org/all/tencent_7DA064CAC9B463EB9B651801011E2CE4230A@qq.com/
Link: https://patch.msgid.link/ambEdSPjerZIVN0b@192.168.1.215
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Now that KVM holds kvm->lock for the entirety of vCPU creation, from when
created_vcpus is incremented until the new vCPU is fully onlined, WARN if
the impossible happens and KVM somehow sees a discrepancy between the
number of vCPUs "created" and "onlined".
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-8-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Now that KVM uses kvm_get_vcpu_by_id() to check for an existing vCPU ID
before doing any meaningful work, which was made possible by holding
kvm->lock for the entirety of vCPU creation, revert the now-redundant
"early" vCPU ID tracking. The claims about the impact of kvm->vcpu_ids on
the memory footprint were a wee bit wrong: the worst case scenario isn't
256 bytes per VM, it's 256 "unsigned longs" per VM, i.e. 2048 bytes per VM.
Increasing the size of "struct kvm" by 2048 nearly doubled the total size
on many architectures, and tripped x86's KVM_SANITY_CHECK_VM_STRUCT_SIZE,
which was added to detect this *exact* scenario, where a single change
significantly increased the size of "struct kvm". I.e. attempting to build
KVM with CONFIG_DEBUG_KERNEL=n fails on x86 (the build failures got missed
because all build bots apparently test only CONFIG_DEBUG_KERNEL=y kernels,
and maintainers' test flows were similarly lacking).
This reverts commit 97d65b544f48b2ee49f6aea32145e3e7969955dc.
Fixes: 97d65b544f48 ("KVM: Check for duplicate vcpu_id as early as possible")
Reported-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Closes: https://lore.kernel.org/all/56a4bc35ee605588b7cc36c8e45c12b5f3b506cb.camel@guillain.net
Reported-by: Paweł S <spawel523@gmail.com>
Closes: https://lore.kernel.org/all/CABD%3DWFOS4j4hDv%2BpW-eEM9HAM2q2GY_iYdAG%2BqvYcUEinUrcQQ@mail.gmail.com
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-7-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Now that kvm->lock is held for the entirety of vCPU creation, check for a
conflicting vCPU ID at the begnning of vCPU creation, before the arch
precreate() hook is invoked. This will allow reverting commit 97d65b544f48
("KVM: Check for duplicate vcpu_id as early as possible"). For now, keep
the redundant vcpu_ids tracking as a sanity check.
No functional change intended (absent KVM bugs, checking vcpu_ids and
walking kvm_get_vcpu_by_id() should yield the same result).
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-6-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
When creating a vCPU, don't drop kvm->lock to when doing the bulk of actual
vCPU creation, as allowing multiple vCPUs to be created in parallel adds
significant complexity in KVM (as evidenced by the many related bugs), and
all known VMMs fully serialize vCPU creation. Remove all manually locking
of kvm->lock from kvm_arch_vcpu_{post,}create() for obvious reasons.
For many years, "everyone" has assumed that dropping kvm->lock was done for
performance reasons optimization, e.g. to allow userspace to create all
vCPUs concurrently for latency purposes. But as above, no known VMM does
that. Looking at the history of this code, before commit 11ec28047118
("KVM: Convert vm lock to a mutex"), kvm->lock was a spinlock. I.e. KVM
*had* to drop kvm->lock when doing the bulk of vCPU creation, otherwise KVM
couldn't do normal memory allocations. When kvm->lock got turned into a
mutex for unrelated reasons, no one took advantage updated of the change to
simplify vCPU creation. And 19 years later, everyone just assumed that KVM
continued to deal with the complexity for performance reasons.
Furthermore, naively parallelizing vCPU creation in userspace is likely a
net negative due to the overheads of task creation. Unless a VMM carefully
avoids the extra overhead related to parallelization, e.g. spawns each
vCPU's thread before creating the vCPU, creating vCPUs concurrently is a
net *negative* up until about ~64 vCPUs, after which the times are a wash.
The absolute speed of light _is_ faster if KVM doesn't hold kvm-lock, but
at vCPU counts of ~16 or less, it's probably in the noise when considering
total VM creation time, as the added latency is less than 1ms up until 16
or so vCPUs.
On top of all that, KVM has had a *lot* of fatal bugs (most often found by
syzkaller) related to vCPUs being created while trying to do per-VM
operations (basically, see every flow that locks all vCPUs). I.e. the
parallel vCPU creation "support" is actively harmful as the only "use case"
is for misbehaving userspace to exploit KVM bugs.
Serializing vCPU creation will allow reverting commit 97d65b544f48 ("KVM:
Check for duplicate vcpu_id as early as possible"), which had "minor" math
error: the worst case scenario isn't "256 bytes per VM", it's "256 unsigned
longs per VM", i.e. 2048 bytes per VM, which doubles the size of each VM
and pushes several architectures into order-1 allocations.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Reject locking of all vCPUs if vCPU creation is in-progress, i.e. if the
number of "created" vCPUs doesn't match the number of "onlined" vCPUs.
It's simply not possible to guarantee that KVM has truly locked all vCPUs
if one or more vCPUs are actively being created. Holding kvm->lock does
prevent in-flight vCPUs from being fully onlined, but it's infeasible for
common KVM to know whether or not that provides sufficient protection.
In practice, this is likely a minor bug fix for the ARM and RISC-V usage of
kvm_trylock_all_vcpus(), and a glorified nop for everything else. E.g.
ARM's kvm_timer_vcpu_init() can race kvm_vm_ioctl_set_counter_offset() with
respect to observing KVM_ARCH_FLAG_VM_COUNTER_OFFSET.
Opportunistically drop x86's existing manual checks on vCPU creation being
in-progress as all of x86's checks immediately precede or follow locking of
all vCPUs. Leave arm64 and RISC-V alone for the moment, as their checks
aren't as obviously redundant/equivalent.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-2-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
All existing architectures which implement KVM pre-fault (x86, s390) have
mechanisms for disallowing pre-faulting.
Currently these are open coded as part of kvm_arch_vcpu_pre_fault_memory().
Formalise this by moving them into a new kvm_arch_pre_fault_allowed() hook,
which every architecture selecting CONFIG_KVM_GENERIC_PRE_FAULT_MEMORY must
implement, returning an error code if the operation is disallowed or 0
otherwise.
The hook is called early in the generic code, allowing architectures to
disallow the operation prior to vCPU load.
This is important, as kvm_vcpu_pre_fault_memory() is the only place where
generic code can call vcpu_load() on a vCPU that has not yet been
initialised.
This lays the foundation for a future change which implements pre-faulting
for arm64 which needs to disallow the mechanism for uninitialised vCPUs.
Suggested-by: Oliver Upton <oupton@kernel.org>
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260923-kvm-arm-prefault-v4-1-d4b0b4dfa8c3@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-24-54ccd48e100b@kernel.org
Acked-by: Paul Moore <paul@paul-moore.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Skip the xarray reservation loop when clearing all memory attributes, as
storing NULL only erases the entry and never needs to allocate, so no
reservation (and no cleanup of a failed one) is required in that case.
Suggested-by: Sean Christopherson <seanjc@google.com>
Cc: David Ballesteros <davimaba.v@proton.me>
Signed-off-by: Zeng Chi <zengchi@kylinos.cn>
Link: https://patch.msgid.link/20260921102442.1232375-1-zeng_chi911@163.com
[sean: split to separate patch]
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Explicitly instantiate the memory attributes xarray with XA_FLAGS_ACCOUNT
to ensure that all allocations are accounted to the memcg. Frustratingly,
memory allocations done in the "fastpath" do not honor the passed in gfp,
even for an explicit xa_reserve(). Only the rare, slow path __xas_nomem()
honors the original gfp. E.g.
xa_reserve(..., GFP_KERNEL_ACCOUNT)
|
-> ...
|
-> __xa_cmpxchg_raw()
|
-> xas_store() <== does not take @gfp
|
-> xas_create()
|
-> xas_alloc()
The bug was confirmed by observing that a process in a cgroup limited to
256 MiB grew radix_tree_node slab by ~512 MiB while its memory.current
stayed near 0.
Fixes: 5a475554db1e ("KVM: Introduce per-page memory attributes")
Cc: stable@vger.kernel.org
Assisted-by: Claude-Code:claude-opus-5
Signed-off-by: David Ballesteros <davimaba.v@proton.me>
Link: https://patch.msgid.link/20260915175335.138547-4-davimaba.v@proton.me
[sean: rewrite changelog, tag for stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
kvm_vm_set_mem_attributes() reserves an xarray entry for every gfn in
the range before storing the new attributes, so that the store loop
can't fail partway through. If one of the reservations fails, e.g. with
-ENOMEM, the entries that were already reserved are left in the array.
That is harmless as far as xa_reserve() is concerned, as the reserved
entries read back as NULL via xa_load(), but it confuses the "does this
range have no attributes at all" check:
if (!attrs)
return !xas_find(&xas, end - 1);
A reserved entry is XA_ZERO_ENTRY, not NULL, and xas_find() returns it
as present. So a leftover reservation makes KVM report that a fully
shared range has attributes even though kvm_get_memory_attributes()
returns none for every gfn in the range. On x86, the next time
mixed-attribute tracking is recomputed for the range (memslot creation,
or a later attribute change that straddles the 2MiB page),
hugepage_has_attrs() treats a fully shared 2MiB range as mixed and
refuses to map it with a hugepage, until userspace happens to set
attributes on the range again.
Drop the shortcut and handle the !attrs case in the per-index loop,
using xas_next_entry() to find the next non-NULL entry. xas_next_entry()
is essentially an optimized xas_find(), so the effective change is that
the !attrs lookup now goes through xas_retry() like the attrs != 0 case,
i.e. reserved entries are skipped and retry entries restart the walk.
Don't check the index when no entry is found, as the xarray leaves the
xas index in a bogus state in that case; no entry simply means the rest
of the range has no attributes.
KVM never stores a non-NULL entry with a value of zero (clearing stores
NULL), but such an entry would be returned by xas_next_entry() and trip
the index check, so WARN if one is ever seen.
Fixes: 5a475554db1e ("KVM: Introduce per-page memory attributes")
Cc: stable@vger.kernel.org
Suggested-by: Sean Christopherson <seanjc@google.com>
Cc: David Ballesteros <davimaba.v@proton.me>
Signed-off-by: Zeng Chi <zengchi@kylinos.cn>
Link: https://patch.msgid.link/20260921102442.1232375-1-zeng_chi911@163.com
[sean: expand comment to elaborate on xarray APIs, split optimization out]
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Allow the user to disable KVM_VM_MEMORY_ATTRIBUTES even when KVM supports
PRIVATE and SHARED attributes, and expose gmem_in_place_conversion as a
module parameter when per-VM attributes are supported. I.e. let userspace
enable in-place PRIVATE<=>SHARED conversion of guest_memfd pages.
Provide both a Kconfig option and a (conditional) module param so that
deployments that use a custom kernel can fully disable per-VM tracking,
while not forcing distros to ship two separate kernels in order to provide
backwards compatibility for downstream users.
Don't allow running VMs with mixed tracking for a given instance of KVM,
i.e. disallow toggling the module param after KVM is loaded, as the extra
complexity needed to handle per-VM behavior far outweighs any potential
benefit. E.g. neither TDX nor SNP supports live migration, so in effect
the requirement is that existing deployments that want to support both the
old and the new models would need to tell their VMM which flavor of
tracking to use.
[ Xiaoyao: Define module_param only if CONFIG_KVM_VM_MEMORY_ATTRIBUTES is enabled ]
Suggested-by: Xiaoyao Li <xiaoyao.li@intel.com>
Co-developed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-22-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Bury KVM_VM_MEMORY_ATTRIBUTES in x86 to discourage other architectures
from adding support for per-VM memory attributes, because tracking private
vs. shared memory on a per-VM basis is now deprecated in favor of tracking
on a per-guest_memfd basis, and while RWX memory attributes are on the
horizon, they too are expected to be x86-only.
This will also allow modifying KVM_VM_MEMORY_ATTRIBUTES to be
user-selectable (in x86) without creating weirdness in KVM's Kconfigs.
Now that guest_memfd supports in-place conversions, it's entirely possible
to run x86 CoCo VMs without support for KVM_VM_MEMORY_ATTRIBUTES.
Leave the code itself in common KVM so that it's trivial to undo this
change if new per-VM attributes do come along.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-21-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Make the source page for populating an SNP guest_memfd instance optional
if in-place conversion/population is enabled. If KVM can convert the page
in-place, then it's possible for guest memory to be initialized directly
from userspace by mmap()'ing the guest_memfd and writing to it while the
corresponding GPA ranges are in a 'shared' state, before converting them
to the 'private' state expected by KVM_SEV_SNP_LAUNCH_UPDATE.
Update the handling/documentation for KVM_SEV_SNP_LAUNCH_UPDATE to allow
for 'uaddr' to be set to NULL when in-place conversion is enabled, which
SNP_LAUNCH_UPDATE will then use to determine when it should/shouldn't
copy in data from a separate memory location. Continue to enforce
non-NULL when PRIVATE is tracked per-VM, not per-guest_memfd.
[ Sean: Moved condition to snp_launch_update ]
Signed-off-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-19-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Move the folio initialization logic from kvm_gmem_get_pfn() into
__kvm_gmem_get_pfn() to also zero pages if the page is to be used in
kvm_gmem_populate().
With in-place conversion, the existing data in a guest_memfd page can be
populated into guest memory through platform-specific ioctls.
Without first zeroing the page obtained using __kvm_gmem_get_pfn(), it
might contain uninitialized host memory, which would leak to the guest if
the populate completes.
guest_memfd pages are zeroed at most once in the page's entire lifetime
with guest_memfd, and that is tracked using the uptodate flag.
Zeroing the page in __kvm_gmem_get_pfn() is chosen over zeroing in
kvm_gmem_get_folio() since other flows, such as a future write() syscall,
can get a page, write to the page and then set page uptodate without
zeroing.
There may be some performance penalty due to redundant zeroing, but this
would pale in comparison to the cost of actually assigning the page to the
VM.
This aligns with the concept of zeroing before first use - the other place
where zeroing happens is in kvm_gmem_fault_user_mapping().
On populate failure, the page is not re-zeroed, since on SNP, if firmware
rejects a CPUID page, the expected CPUID values provided by firmware are
returned to userspace via page contents. More generally, page contents may
be modified on populate failure.
Don't mark the page uptodate again after populating, since the page would
already be marked uptodate before the post_populate() call.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-18-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Provide a function to check that a range has given attributes.
Optimize setting memory attributes by returning early if all pages in the
requested range already have the requested attributes.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-17-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
When doing in-place conversion from PRIVATE to SHARED, immediately inform
arch code of the conversion for all allocated pages/folios, e.g. so that
arch code can put hardware metadata tables in the correct state. Eagerly
updating the table for to SHARED conversions avoids having to implement
on-demand updates, e.g. when faulting in host userspace mappings. Skip the
entire flow if the arch doesn't implement conversion callbacks, as getting
folios from the filemap is noticeably expensive, especially when converting
large chunks of memory.
Deliberately don't eagerly update the metadata table on conversions from
SHARED to PRIVATE, because assigning a page to a VM (versus "returning" it
to the host) requires the exact GFN associated with the page, i.e would
require walking the memslot bindings. And because KVM *must* do on-demand
metadata updates when getting a PFN for KVM-internal usage, as that's the
only time a relevant memslot binding is guaranteed to exist.
Note! Inform arch code of the conversion within the protection of the
invalidation sequence, to ensure that any existing mappings are dropped
before hardware is updated, and to ensure that new mappings can't be
established until after the conversion is complete.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-16-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
When converting memory to private in guest_memfd, it is necessary to ensure
that the pages are not currently being accessed by any other part of the
kernel or userspace to avoid any current user writing to guest private
memory.
guest_memfd checks for any outstanding references to determine whether a
page is still in use. The only expected references after unmapping the
range requested for conversion are those that are held by guest_memfd
itself.
A folio will have outstanding references if it is present in a per-CPU
lru_add fbatch. guest_memfd does not actually participate in LRU, but
freshly-allocated folios are still added to the lru_add fbatch for batch
LRU statistics processing.
A folio may also have extra refcounts if it is on the mlock fbatch.
These two known usages of the folio are handled by draining both the
lru_add and mlock fbatches for the folio. After draining, if the refcount
is still elevated, then there are truly outstanding references.
If the page may be DMA-pinned, DMA is using it and hence there are
outstanding references. Checking if the page may be DMA-pinned can have
false positives, but that is only with a significant number of refcounts,
at which point draining LRU is not going to move the needle - it can still
be concluded that the folio has outstanding references.
If the page is still mapped after guest_memfd tried to unmap it earlier in
the conversion process, it also has outstanding references.
Exit early to avoid unnecessary draining in these two cases. If an
outstanding reference is detected, stop scanning, record the failing
offset, and return immediately.
Track the drain status to avoid repeatedly draining LRU caches across
multiple folios while scanning the requested range.
Update the kvm_memory_attributes2 structure to include an error_offset
field. This allows KVM to report the exact offset where a conversion
failed. If the safety check fails, return -EAGAIN and copy the error_offset
back to userspace so that it can potentially retry the operation or handle
the failure gracefully.
Report error_offset if preallocating memory to track attributes fails with
-ENOMEM as well, in which case the start offset of the range is returned.
Update documentation to document the error_offset field and the possible
-EAGAIN error.
Suggested-by: David Hildenbrand <david@kernel.org>
Co-developed-by: Vishal Annapurve <vannapurve@google.com>
Signed-off-by: Vishal Annapurve <vannapurve@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-15-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Add a new ioctl (and matching struct), KVM_SET_MEMORY_ATTRIBUTES2, using
the same base ioctl number (0xd2), but with R/W semantics for the kernel
instead of just read semantics. "Officially" documenting that KVM writes
to the payload will allow KVM to support partial/incremental conversions,
instead of all-or-nothing updates (which requires complex unwinding), by
recording the failing offset if an error occurs.
Opportunistically add a new struct as well, even though KVM could squeeze
the error offset into "struct kvm_memory_attributes", as there's no cost to
doing so in practice. Pad the struct with a pile of extra space to try and
avoid ending up with "struct kvm_memory_attributes3" in the future. Use
the same layout for the fields common to version 1 of the struct, e.g. to
ease upgrading userspace, and to provide flexibility if KVM ever adds
support for KVM_SET_MEMORY_ATTRIBUTES2 at VM scope.
Introduce KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES to advertise the
availability of the KVM_SET_MEMORY_ATTRIBUTES2 ioctl.
Update the KVM API documentation to define the new ioctl and its behavior,
and add the necessary UAPI definitions and capability checks.
The process of setting memory attributes has a clear point of no return
because, for CoCo VMs, zapping stage 2 page tables is a destructive
operation. Unlike regular VMs, where re-faulting pages into the stage 2
page tables merely incurs a performance penalty, CoCo guests must
(re-):accept pages after every fault. To preserve CoCo security guarantees,
guests will not accept pages they did not explicitly request faults
for. Consequently, during memory conversions, any operation that could
cause the process to abort must be completed before the stage 2 page tables
are zapped.
Zap only the ranges that are not already in the requested state to avoid
inadvertently destroying (CoCo) data. ARM CCA guests will try to mark the
entire DRAM as private at boot. If there are no shared pages at all, the
to-private conversion can be skipped, but the existence of a single shared
page would require the conversion process to proceed, and if it proceeds,
zapping both shared and private pages would destroy data and break the
guest.
Suggested-by: Michael Roth <michael.roth@amd.com>
Suggested-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Co-developed-by: Vishal Annapurve <vannapurve@google.com>
Signed-off-by: Vishal Annapurve <vannapurve@google.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-14-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Accept the mapping type filter as a parameter in the invalidation start
helper instead of querying it internally. This allows callers to specify
which mappings (shared, private, or both) should be invalidated.
In the next patch, the conversion process will use this new parameter to
invalidate mappings only when they're different from the target state of
the conversion, i.e. invalidate only shared mappings on a shared to private
conversion and not both.
No functional change intended.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-13-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
enabled
If a guest_memfd memslot is created but the guest_memfd does not have the
GUEST_MEMFD_FLAG_MMAP set, KVM still fulfils guest faults by looking up the
memslot's userspace_addr.
Set KVM_MEMSLOT_GMEM_ONLY if in-place conversion is enabled so that the
guest_memfd's memory will be used for both shared and private memory. With
in-place conversion, guest_memfd will be the only backing memory for the
memslot.
No validation is performed to require userspace_addr to be a mapping from
the associated guest_memfd because even after validation, userspace is free
to remap something else at the provided userspace_addr.
userspace_addr will still be used by functions like kvm_read_guest(), and
if userspace_addr does not match up with the corresponding memory in the
memslot's guest_memfd (whether userspace_addr points to the wrong offset or
some non-guest_memfd memory, etc), that is a user error.
Requiring both shared and private memory to come from the only associated
guest_memfd simplifies invalidation in stage 2 page tables. On a PUNCH_HOLE
operation on a guest_memfd, the invalidation is now guaranteed to be
invalidating only memory mapped from the given guest_memfd.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-12-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
conversions
When removing one or more folios from a guest_memfd instance, invalidate
both SHARED and PRIVATE mappings if in-place conversion is enabled, because
stating the obvious, KVM needs to ensure that all mappings to the folio(s)
are dropped.
Opportunistically rename the helper to capture that it returns a filter for
all gfns in anticipation of zapping only the previous mapping types on
conversion. I.e. when doing in-place conversion to PRIVATE, only SHARED
mappings need to be zapped (ignoring that KVM would ideally not invalidate
ranges whose attributes aren't changing in the first place).
Note, precisely zapping only the possible mapping types when in-place
conversion is disabled is important for functional correctness, not just
for performance. Specifically, if KVM zaps both when SHARED vs. PRIVATE is
tracked per-VM, then a PUNCH_HOLE operation on a PRIVATE guest_memfd will
incorrectly zap SHARED mappings that have nothing to do with that gmem
instance (because they're mapped via a VMA, not a gmem fd).
The incorrect over-zapping of SHARED memory that doesn't belong to the gmem
fd requesting the zapping will be resolved in a later patch, where, if
in-place conversion is enabled, KVM will use both shared and private memory
from the guest_memfd. If both shared and private memory are from the
guest_memfd, invalidation will only zap memory belonging to the given gmem
instance.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-11-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Stub in global variable to enable in-place guest_memfd private<=>shared
memory conversion, which will eventually be exposed to userspace via a
module param, and wire up the __kvm_is_private_gfn() static call to the
guest_memfd version when in-place conversion is enabled, i.e. when gmem is
the sole authority on private vs. shared memory.
Co-developed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Fuad Tabba <tabba@google.com>
Cc: Xiaoyao Li <xiaoyao.li@intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-9-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Introduce a generic kvm_is_private_gfn() interface using a static call to
determine if a GFN is private. This allows the implementation for checking
a GFN's private/shared status to be set at runtime.
In preparation for choosing implementations between a guest_memfd lookup
and the existing VM attribute lookup, rename the existing
VM-attribute-based check to kvm_vm_is_private_gfn() to emphasize that it
looks up VM attributes.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-8-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Rename kvm_mem_is_private() to kvm_is_private_gfn() to prepare for in-place
conversion, where there will be two lookup functions,
kvm_vm_is_private_gfn() and kvm_gmem_is_private_gfn().
This renaming allows consistent prefixing of "vm" vs "gmem" for
kvm_*_is_private_gfn(), as opposed to kvm_gmem_mem_is_private(), which
looks like a typo.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-7-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Rename memory attribute APIs to add a "vm_" in the name in anticipation of
moving PRIVATE tracking into guest_memfd, to allow in-place conversion
between SHARED and PRIVATE. At that point, there will effectively be two
(potential) sources of memory attributes: the VM and guest_memfd.
kvm_vm_set_mem_attributes() already has "vm" in the name to indicate that
it is a VM ioctl. Rename it to kvm_set_vm_mem_attributes() to show that it
is setting the VM's memory attributes. (Drop the VM-ioctl scoping since it
is a helper local to the file.) Update the accompanying trace function to
match.
No functional change intended.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-6-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
defined
Explicitly guard reporting support for KVM_MEMORY_ATTRIBUTE_PRIVATE based
on kvm_arch_has_private_mem being #defined in anticipation of tracking
PRIVATE vs. SHARED state per-guest_memfd, not per-VM (to allow in-place
conversion).
guest_memfd support for memory attributes is expected to be unconditional
to avoid yet more macros (all architectures that support guest_memfd are
expected to use per-gmem attributes at some point), at which point
enumerating support KVM_MEMORY_ATTRIBUTE_PRIVATE based solely on memory
attributes being supported by KVM at-large would result in a system-scope
check (NULL @kvm) over-reporting support on arm64.
Give architectures full control over overriding the default definition of
kvm_arch_has_private_mem() by removing the coupling with
CONFIG_KVM_VM_MEMORY_ATTRIBUTES.
In a later patch, kvm_arch_has_private_mem() will be defined based on
whether architectural features are compiled in, and made orthogonal to
CONFIG_KVM_VM_MEMORY_ATTRIBUTES.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-5-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Rename the per-VM memory attributes Kconfig to make it explicitly about
per-VM attributes in anticipation of adding memory attributes support to
guest_memfd, at which point it will be possible (and desirable) to have
memory attributes without the per-VM support, even in x86.
No functional change intended.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-4-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Start plumbing in guest_memfd support for in-place private<=>shared
conversions by tracking attributes via a maple tree. KVM currently tracks
private vs. shared attributes on a per-VM basis, which made sense when a
guest_memfd _only_ supported private memory, but tracking per-VM simply
can't work for in-place conversions as the shared/private status of a given
page needs to be per-gmem_inode, not per-VM.
Use the filemap invalidation lock to protect the maple tree, as taking the
lock for read when faulting in memory (for userspace or the guest) isn't
expected to result in meaningful contention, and using a separate lock
would add significant complexity (avoiding deadlock is quite difficult).
In kvm_gmem_get_pfn(), drop the folio refcount before releasing
filemap_invalidate_lock(). This ensures that a competing conversion request
from userspace (to be added in a later patch), which also takes the
filemap_invalidate_lock(), will never see an elevated refcount due to
kvm_gmem_get_pfn().
Co-developed-by: Vishal Annapurve <vannapurve@google.com>
Signed-off-by: Vishal Annapurve <vannapurve@google.com>
Co-developed-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Co-developed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Tested-by: Shivank Garg <shivankg@amd.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-3-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Use kvm_mem_is_private() when populating guest_memfd instead of using an
open coded equivalent. In addition to simplifying the populate code *now*,
this avoids the need to provide a range-based gmem lookup API in the future
as well.
No functional change intended.
Suggested-by: Xiaoyao Li <xiaoyao.li@intel.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-2-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Add and use kvm_arch_has_gmem_convert() to guard guest_memfd's invocation
of arch hooks related to converting memory between private and shared, as
only one half of the x86 CoCo duo needs the runtime hooks (any pre-work is
pure overhead for TDX). At this exact moment, the overhead is negligible,
but that will change when in-place conversion comes along, at which point
to-shared conversions will "need" to find all affected folios prior to
calling into arch code. In quotes because very technically that work could
be pushed to arch code, but that would bleed guest_memfd details into arch
code and would be far worse than adding yet another kvm_arch_has... hook.
Opportunistically provide the kvm_arch_gmem_make_private() declaration, and
rely on dead-code elimination to eliminate the call to non-existent code
when CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT=n.
Reported-by: Binbin Wu <binbin.wu@linux.intel.com>
Closes: https://lore.kernel.org/all/1ec08cd8-3072-4753-ad5e-cd34956647f8@linux.intel.com
Suggested-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260910-gmem-inplace-conversion-v13-1-dd6fbf94f4e1@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
KVM currently expects guest_memfd PFN lookups to return a refcounted struct
page, which callers hold across fault handling.
Drop the page's refcount before returning from kvm_gmem_get_pfn() to
prepare for the in-place conversions series.
CoCo shared-to-private conversion handling must inspect folio refcounts to
ensure exclusive ownership by guest_memfd. A concurrent guest page fault
taking a temporary reference on the folio causes conversions to fail due to
an elevated refcount.
While this refcount is also taken on host userspace page faults, that
refcount is taken on behalf of the host userspace page tables. This
refcount will be dropped when conversions unmaps the page. Either way, once
there's an mmap() or userspace mapping, the pages are open to way more
refcounts, transient or not. This patch focuses on just dropping refcounts
before handing KVM a page.
guest_memfd already notifies KVM of page invalidations, so callers within
KVM only need to respect the MMU invalidation protocol to safely rely on
guest_memfd for page presence.
Since the page refcounts are dropped, don't return the struct page pointer.
Not returning the struct page from the guest_memfd PFN lookup moves KVM
closer toward supporting memory backends that are not backed by struct
page.
Here are some notes on the cleanup in the callers of kvm_gmem_get_pfn():
kvm_release_faultin_page() in ARM's gmem_abort() originally also serves to
set the page dirty and accessed under some conditions. The dirty and
accessed flags don't matter for guest_memfd anyway, so it is safe to just
drop the call to kvm_release_faultin_page().
For ARM's kvm_translate_vncr(), the local page pointer must be initialized
to NULL so that the shared cleanup path that releases faulted-in pages
safely no-ops for guest_memfd.
For x86, no additional changes are required in the MMU fault path because
the page fault tracking structure is zero-initialized at the start of page
fault handling, ensuring the refcounted page pointer is already NULL.
Reported-by: Yan Zhao <yan.y.zhao@intel.com>
Closes: https://lore.kernel.org/all/anZ4W9o5pTWIEgMY@yzhao56-desk.sh.intel.com/
Co-developed-by: Yan Zhao <yan.y.zhao@intel.com>
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Tested-by: Michael Roth <michael.roth@amd.com>
Tested-by: Yan Zhao <yan.y.zhao@intel.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Co-developed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Link: https://patch.msgid.link/20260826-gmem-no-return-page-v4-5-3bb9c1ddb4e3@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Drop the superfluous WRITE_ONCE() when setting a memslot's guest_memfd file
during initial binding, as the memslot *must* be inactive and unreachable.
The superfluous WRITE_ONCE() was added by commit 67b43038ce14 ("KVM:
guest_memfd: Remove RCU-protected attribute from slot->gmem.file") to
maintain rough "parity" with the existing rcu_assign_pointer(), not
realizing that the only reason rcu_assign_pointer() was used was to make
sparse and other checkers happy.
Cc: Yan Zhao <yan.y.zhao@intel.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Yan Zhao <yan.y.zhao@intel.com>
Link: https://patch.msgid.link/20260922001332.1121266-7-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
is ready
Wait to bind a memslot to a guest_memfd instance until *after* the memslot
is fully prepared, as creating the binding in guest_memfd will effectively
expose the memslot to readers. As pointed out by Sashiko, binding the
memslot before it's ready to be exposed to the rest of the world can break
various memslot assumption and rules. E.g. x86 could observe a NULL rmap
pointer if a PUNCH_HOLE hit the guest_memfd after the binding was created,
but before KVM made it through kvm_prepare_memory_region().
Fixes: a7800aa80ea4 ("KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory")
Cc: stable@vger.kernel.org
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260826170551.BEF801F000E9@smtp.kernel.org
Link: https://patch.msgid.link/20260922001332.1121266-6-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Split binding a memslot to a guest_memfd instance into prepare() and
commit() phases so that KVM can separate preparing the memslot from binding
the memslot to the gmem instance, i.e. from committing the memslot. This
will allow waiting to commit the memslot+gmem binding until the memslot is
fully prepared, which is necessary as the memslot becomes reachable when
the binding is created.
As a bonus, drop the unwind-on-failure from the commit phase (other than
nullifying the bindings), as the only reason bind() did the full unwind is
because it technically didn't own the memslot, i.e. "needed" to leave
memslot in the same state it started in.
No functional change intended (the unwinding down on bind() failure was
effectively dead code since KVM simply deletes the memslot on failure,
i.e. there was nothing that could actually observe the unwind).
Cc: stable@vger.kernel.org
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Link: https://patch.msgid.link/20260922001332.1121266-5-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
