path: root/Documentation/virt/kvm/arm/pkvm.rst

.. SPDX-License-Identifier: GPL-2.0

====================
Protected KVM (pKVM)
====================

**NOTE**: pKVM is currently an experimental, development feature and
subject to breaking changes as new isolation features are implemented.
Please reach out to the developers at kvmarm@lists.linux.dev if you have
any questions.

Overview
========

Booting a host kernel with '``kvm-arm.mode=protected``' enables
"Protected KVM" (pKVM). During boot, pKVM installs a stage-2 identity
map page-table for the host and uses it to isolate the hypervisor
running at EL2 from the rest of the host running at EL1/0. pKVM supports
GICv3 only: GICv2 is not supported in protected mode, and neither are
GICv5-based guests.

pKVM permits creation of protected virtual machines (pVMs) by passing
the ``KVM_VM_TYPE_ARM_PROTECTED`` machine type identifier to the
``KVM_CREATE_VM`` ioctl(). The hypervisor isolates pVMs from the host by
unmapping pages from the stage-2 identity map as they are accessed by a
pVM. Hypercalls are provided for a pVM to share specific regions of its
IPA space back with the host, allowing for communication with the VMM.
A Linux guest must be configured with ``CONFIG_ARM_PKVM_GUEST=y`` in
order to issue these hypercalls.

See hypercalls.rst for more details.

Isolation mechanisms
====================

pKVM relies on a number of mechanisms to isolate pVMs from the host:

CPU memory isolation
--------------------

Status: Isolation of anonymous memory and metadata pages.

Metadata pages (e.g. page-table pages and '``struct kvm_vcpu``' pages)
are donated from the host to the hypervisor during pVM creation and
are consequently unmapped from the stage-2 identity map until the pVM is
destroyed.

Similarly to regular KVM, pages are lazily mapped into the guest in
response to stage-2 page faults handled by the host. However, when
running a pVM, these pages are first pinned and then unmapped from the
stage-2 identity map as part of the donation procedure. This gives rise
to some user-visible differences when compared to non-protected VMs,
largely due to the lack of MMU notifiers:

* Memslots cannot be moved or deleted once the pVM has started running
  (``-EPERM``).
* Read-only memslots and dirty logging are not supported (``-EPERM``).
* With the exception of swap, file-backed pages cannot be mapped into a
  pVM.
* Donated pages are accounted against ``RLIMIT_MLOCK`` and so the VMM
  must have a sufficient resource limit or be granted ``CAP_IPC_LOCK``.
  The lack of a runtime reclaim mechanism means that memory locked for
  a pVM will remain locked until the pVM is destroyed.
* Changes to the VMM address space (e.g. a ``MAP_FIXED`` mmap() over a
  mapping associated with a memslot) are not reflected in the guest and
  may lead to loss of coherency.
* Accessing pVM memory that has not been shared back will result in the
  delivery of a SIGSEGV.
* If a system call accesses pVM memory that has not been shared back
  then it will either return ``-EFAULT`` or forcefully reclaim the
  memory pages. Reclaimed memory is zeroed by the hypervisor and a
  subsequent attempt to access it in the pVM will return ``-EFAULT``
  from the ``KVM_RUN`` ioctl().

CPU state isolation
-------------------

Status: CPU register state of protected vCPUs is managed entirely at EL2.

pKVM performs the complete context switch for protected vCPUs within the
hypervisor. The hypervisor sets a protected vCPU's initial state, not the
host, and only what each exit needs is synchronised back.

The user-visible consequences are described under `API behaviour for
protected VMs`_.

DMA isolation using an IOMMU
----------------------------

Status: **Unimplemented.**

Proxying of Trustzone services
------------------------------

Status: FF-A and PSCI calls from the host are proxied by the pKVM
hypervisor.

The FF-A proxy ensures that the host cannot share pVM or hypervisor
memory with Trustzone as part of a "confused deputy" attack.

The PSCI proxy ensures that CPUs always have the stage-2 identity map
installed when they are executing in the host. This proxy is distinct
from the PSCI handling provided to protected guests, which is described
under `API behaviour for protected VMs`_.

Protected VM firmware (pvmfw)
-----------------------------

Status: **Unimplemented.**

API behaviour for protected VMs
===============================

Protected vCPU state is owned by EL2 (see `CPU state isolation`_). The VMM
configures a vCPU before its first ``KVM_RUN``; afterwards the state is
private to the guest and the ioctls that access it return ``-EPERM``. The
vCPU ioctls' errors follow one rule: ``-EPERM`` means the host asked for
state that the guest owns, ``-EINVAL`` means the request is not valid for a
protected VM, and a feature pKVM doesn't support for protected VMs is
rejected the way KVM rejects one that isn't implemented (``-ENXIO`` from
``KVM_ARM_VCPU_PVTIME_CTRL``, for example). The ioctls themselves are
described in Documentation/virt/kvm/api.rst.

Boot
----

A protected VM boots from a single primary vCPU. Before the first
``KVM_RUN``, the VMM prepares the boot state:

* Set ``KVM_MP_STATE_RUNNABLE`` on the primary vCPU and
  ``KVM_MP_STATE_STOPPED`` on every other vCPU. EL2 allows only one
  RUNNABLE primary per protected VM. A second RUNNABLE vCPU fails at its
  first ``KVM_RUN``.
* Set the primary vCPU's boot state with ``KVM_SET_ONE_REG``: the kernel
  entry address in ``PC`` and the DTB pointer in ``x0``.

``PC`` and ``x0`` are the only registers EL2 takes from the host. Other
pre-run writes are accepted, but the guest starts from the state the
hypervisor resets it to.

Secondary vCPUs are started by the guest itself through PSCI ``CPU_ON``
(see `Power state`_), which supplies their entry point and context ID.
The VMM cannot choose where they boot.

vCPU state
----------

* ``KVM_GET_ONE_REG`` and ``KVM_SET_ONE_REG`` return ``-EPERM`` once the
  vCPU has run. Before that, they access the host-side copy from which
  EL2 builds the guest's boot state (see `Boot`_).
* ``KVM_ARM_VCPU_INIT`` accepts only the vCPU features that a protected
  guest supports and returns ``-EINVAL`` otherwise.
  ``KVM_ARM_VCPU_PSCI_0_2`` is required, as EL2 implements PSCI 1.1 for
  the guest (see `Power state`_). ``KVM_ARM_VCPU_EL1_32BIT`` is not
  supported: protected guests run in AArch64 only and see no AArch32
  support in ``ID_AA64PFR0_EL1``. Once the vCPU has run,
  ``KVM_ARM_VCPU_INIT`` returns ``-EPERM``, as re-initialising it would
  reset the host-side copy alone.
* ``KVM_SET_VCPU_EVENTS`` returns ``-EPERM`` for external-abort injection
  (``ext_dabt_pending``), and ``KVM_CAP_ARM_INJECT_EXT_DABT`` is not
  advertised to a protected VM. SError injection is unaffected.
* ``KVM_SET_GUEST_DEBUG`` returns ``-EPERM``, and ``KVM_CAP_SET_GUEST_DEBUG``
  is not advertised to a protected VM (see `Debug`_).

Power state
-----------

EL2 implements PSCI 1.1 for a protected guest. The calls that move a
vCPU's power state, ``CPU_ON`` and ``CPU_OFF``, are handled at EL2, and
the host cannot change the outcome: for ``CPU_ON`` the host only
schedules the target, which EL2 has already reset to the entry point
the guest chose, and for ``CPU_OFF`` it only stops scheduling it.
``AFFINITY_INFO`` is handled by the host, which returns ``OFF`` once
it has stopped scheduling every vCPU the query covers, as for a
non-protected VM. A vCPU becomes a valid ``CPU_ON`` target at its first
``KVM_RUN``, and before that EL2 returns ``INVALID_PARAMETERS``. The
platform calls, ``CPU_SUSPEND``, ``SYSTEM_OFF``, ``SYSTEM_RESET`` and
``SYSTEM_RESET2``, are forwarded to the host and behave as for a
non-protected VM, with the ``SYSTEM_*`` calls exiting to the VMM as
``KVM_EXIT_SYSTEM_EVENT``; the ``SYSTEM_RESET2`` reset type and cookie
are in the guest's registers, which ``KVM_GET_ONE_REG`` rejects once
the vCPU has run. Any other function returns ``NOT_SUPPORTED``, and
``PSCI_FEATURES`` reports the same set. Because the host handles those
forwarded calls, ``KVM_SET_ONE_REG`` on ``KVM_REG_ARM_PSCI_VERSION``
returns ``-EINVAL`` for a version below 1.1.

Once a vCPU has run, its power state follows the guest's PSCI calls, not
the VMM's. A vCPU that the guest has powered off with ``CPU_OFF``, or has
not yet brought online with ``CPU_ON``, reads as
``KVM_MP_STATE_UNINITIALIZED``, and only an in-guest ``CPU_ON`` can power
it on: ``KVM_SET_MP_STATE`` returns ``-EPERM`` for ``KVM_MP_STATE_RUNNABLE``
and ``KVM_MP_STATE_SUSPENDED`` on it, accepts ``KVM_MP_STATE_STOPPED`` and
leaves it as it is, and never takes ``KVM_MP_STATE_UNINITIALIZED`` from
the VMM. ``KVM_MP_STATE_STOPPED`` still stops a vCPU EL2 has powered on,
and ``KVM_MP_STATE_RUNNABLE`` resumes it, so the VMM can pause it.

Other interface differences
---------------------------

* ``KVM_CHECK_EXTENSION`` reports only the arm64 capabilities that pKVM
  supports for protected guests, and ``KVM_ENABLE_CAP`` accepts only
  those. Query them on the VM file descriptor: the system file descriptor
  has no VM to filter against. The capabilities the generic KVM code
  answers itself (``KVM_CAP_SYNC_MMU`` or ``KVM_CAP_DEVICE_CTRL``, for
  example) are reported as for any VM, and don't mean the feature works
  for a protected one. The filter does not cover every interface either:
  device-fd configuration (for example the VGIC after
  ``KVM_CREATE_DEVICE``) and vCPU attributes are unfiltered, so a feature
  the filter excludes is rejected by the attribute itself.
  ``KVM_ARM_VCPU_PVTIME_CTRL`` returns ``-ENXIO``, for example, as it does
  where steal time isn't implemented.
* The vGIC of a protected VM remains host-managed: device creation,
  configuration and interrupt injection all work as they do for a
  non-protected VM.
* ``KVM_ARM_SET_COUNTER_OFFSET`` and ``KVM_ARM_GET_REG_WRITABLE_MASKS``
  return ``-EINVAL``: their capabilities are not advertised to a protected
  VM, whose counter offset and ID registers are set by EL2. A protected
  guest sees the physical timebase.
* A protected guest's first access to each page of memory exits to the
  host, since the hypervisor cannot tell memory from a device before the
  page is mapped. For a store, the host sees the value of the register
  the syndrome names, clamped to the access width, and nothing else from
  the register file. For a load, the register the syndrome names takes
  the value the host returns, as for any emulated access.
* A protected guest that uses a feature it was not given, or executes an
  ``SMC``, takes an undefined instruction exception from the hypervisor;
  the host is not involved.
* Apart from PSCI, the hypervisor handles a protected guest's SMCCC
  calls itself and does not involve the host. A function it does not
  implement returns ``NOT_SUPPORTED``, and the host's SMCCC filter does
  not apply: the VM device attribute ioctls (``KVM_ARM_VM_SMCCC_CTRL``)
  return ``-EINVAL`` for a protected VM.
* The hypervisor can decode a trapped guest access only from the CPU's
  instruction syndrome, which is provided only for a load or store of a
  single general-purpose register. An access without one (for example a
  load/store pair or a SIMD/FP access) cannot be decoded. For a
  non-protected VM it can exit to the VMM as ``KVM_EXIT_ARM_NISV``. For a
  protected VM it cannot be emulated by the VMM, so the guest takes a
  synchronous external abort instead.

Debug
-----

Hardware-assisted debugging is not available to protected guests.
``MDSCR_EL1``, ``OSLAR_EL1``, ``OSLSR_EL1``, ``OSDLR_EL1``, and breakpoint
and watchpoint 0 are RAZ/WI; any other debug register access takes an
undefined instruction exception.

Resources
=========

Quentin Perret's KVM Forum 2022 talk entitled "Protected KVM on arm64: A
technical deep dive" remains a good resource for learning more about
pKVM, despite some of the details having changed in the meantime:

https://www.youtube.com/watch?v=9npebeVFbFw