summaryrefslogtreecommitdiff
path: root/arch/arm64/kernel/cpu_errata.c
AgeCommit message (Collapse)Author
12 daysMerge tag 'arm64-upstream' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux Pull arm64 updates from Catalin Marinas: "Here are the arm64 updates for 7.4. The more interesting bits are the removal of big-endian kernel support (already marked BROKEN for some time), preemptible this_cpu_*() operations and MPAM support for the firmware-backed (Fb) interface. CCA (Arm's confidential compute) is slowly trickling in with host RMI (Realm Management Interface) support and some refactoring, including a dedicated SMCCC bus to advertise firmware services. There are also the usual PMU updates for new hardware and a few fixes/cleanups. Core features: - Remove big-endian kernel support: CONFIG_CPU_BIG_ENDIAN has depended on BROKEN for a while, so drop the remaining code (BE hypervisor object, BPF JIT, assembly/crypto/library routines, endianness helper macros, selftests) and force a little-endian build - Add preemptible this_cpu_*() operations (read/write, RMW, xchg, cmpxchg and cmpxchg128) that no longer need to disable preemption around the per-CPU address generation and access - Preparation patches for implementing CONFIG_HOTPLUG_PARALLEL on arm64 Memory management: - Treat faults taken by EFI runtime firmware as kernel faults - Clear PTE_WRITE for EFI_MEMORY_RO mappings (with hardware dirty state management enabled, this is not truly read-only) - Raise the default ARCH_MMAP_RND_BITS for 48-bit and 52-bit VA configurations (28/26/24 bits for 4K/16K/64K pages) - Implement clear_page[s]() using memset() - Enable BBML2 no-abort (a.k.a. BBML3) for the NVIDIA Rigel CPU Perf and PMU: - Support for HiSilicon's ITS PMU - Add support for some new events: INST_SIMD_ALU_VEC for Apple PMUs and a cycle counter for the latest version of HiSilicon's MN PMU hardware - A bunch of fixes to the CXL PMU driver to address pre-existing issues from Sashiko - Fix generation of PERF_RECORD_ITRACE_START records for SPE and ETM drivers - Reduce excessive stack consumption in the Marvell DDR PMU driver - Minor driver updates and cleanups all over Arm Confidential Compute Architecture (CCA): - Add an Arm SMCCC bus for firmware services discovered through SMCCC and move the TRNG and CCA guest (TSM provider) drivers onto it, removing the dummy CCA platform device and adding a sysfs ABI for Realm guests - Move the Realm Service Interface (RSI) code out of arch/arm64/ into drivers/firmware/arm_rmm/ - Add the host side of the Realm Management Interface (RMI): detect, configure and activate the RMM, add SRO support and wrappers for the Realm RMI commands - Block hibernation (new arch hook in the PM core) and kexec while the RMM is active. To be revisited in the future with proper support for deactivation - Handle Granule Protection Faults (GPFs) taken by the host, caused by accesses to memory delegated to the RMM. In a future patch, the exception table lookup will be limited to is_kdump_kernel() cases MPAM (Memory Partitioning And Monitoring): - Add support for the MPAM firmware-backed (Fb) interface, allowing MSC accesses to be mediated by firmware - Fix a list of theoretical issues with the driver (MSC access error propagation, MBWU monitor save/restore and locking, resctrl class handling, error interrupt restore, maximum number of PARTIDs) - Add the mailing list and a missing file to the MAINTAINERS entry - Document when to set arm64.nompam Errata and CPU features: - Add the NXP i.MX8QM/8QP workaround for the Cortex-A53 cache coherency issue (ERR050104), using a new ARM64_CPUCAP_SOC_ERRATUM capability type for SoC-level errata Miscellaneous: - Mask debug exceptions (DAIF) around the hardware breakpoint register manipulations when called from generic kernel code - Don't zero the SVE state buffer when the SVE state is live - Return -EPERM instead of -EBUSY from the GCS prctl() locking failures (matching x86), with the selftest updated. Slight ABI change but no software deployment yet - kselftest: fix unsound MTE fault skipping by introducing MTE-safe memory accessors, avoid copying each signal test binary more than once - Update sysreg definitions (fix three enum values and two field names, mark architecturally dropped fields as RES0) and remove the TME transaction state check from the BRBE driver - Fix the description of ARM64_HAS_LPA2 and remove a redundant ID_AA64MMFR0 define" * tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (144 commits) arm64: cpufeature: Enable BBML2 for NVIDIA Rigel arm64: cputype: Add NVIDIA Rigel definitions arm64: raise default ARCH_MMAP_RND_BITS for 48-bit and 52-bit VA arm64/debug: mask debug exceptions when called from common code kselftest/arm64: signal: copy each test binary only once kselftest/arm64: Check for -EPERM not -EBUSY in the locking test arm64: gcs: Return -EPERM not -EBUSY for prctl() locking failures perf: arm_brbe: Remove the check for TME transaction states arm64: sysreg: Mark fields the architecture dropped as RES0 arm64: sysreg: Fix the names of two misspelt fields arm64: sysreg: Fix three incorrect enum values arm64: smp: Defer RCU registration during secondary CPU bringup arm64: smp: Don't bother printing the I-cache policy for each CPU arm64: smp: Defer update of secondary CPU capabilities arm64: smp: Tidy up cpuinfo init and cpufeature updates arm64: smp: Tidy up smp_prepare_cpus() arm64: cpufeature: Store every ID register with its overrides applied arm64: cpufeature: Read MPAMIDR_EL1 in __cpuinfo_store_cpu() arm64: cpufeature: Add read_cpuid_with_overrides() arm64: cpufeature: Check arm64_ftr_regs[] before the first store ...
2026-10-12arm64: errata: Add NXP i.MX8QM/8QP workaround for A53 cache coherency issuePeng Fan
According to NXP errata document IMX8_1N94W[1], the i.MX8QuadMax and i.MX8QuadPlus SoCs suffer from a cache coherency issue (ERR050104). The upper bits, above bit 35, of the ARADDR and ACADDR buses within the Arm A53 subsystem have been incorrectly connected. This causes some TLBI and IC maintenance operations exchanged between the A53 and A72 core clusters to be corrupted. The workaround requires: - Downgrading targeted TLBI operations to broadcast-all variants. Instead of patching the low-level __TLBI_1 macro (which causes excessive over-invalidation), redirect high-level TLB flush functions (flush_tlb_mm, __do_flush_tlb_range, flush_tlb_kernel_range, __flush_tlb_kernel_pgtable) to use VMALLE1IS via alternatives. - Upgrading IC IVAU to IC IALLUIS for both kernel (via ALTERNATIVE in invalidate_icache_by_line) and EL0 userspace (via trap-and-upgrade in user_cache_maint_handler with SCTLR_EL1.UCI=0). - Disabling KVM since correct TLB maintenance cannot be guaranteed for guests. - No need to touch SMMU Broadcast TLB Maintenance (BTM) since the arm-smmu driver does not use it. SoC detection uses devicetree compatible string "fsl,imx8qm" or "fsl,imx8qp" since the boot CPU MIDR_EL1 (0x410fd034) and AIDR_EL1 (0) are not unique to these SoCs. [1] https://www.nxp.com/docs/en/errata/IMX8_1N94W.pdf [ Reworked per review feedback from Will Deacon and Mark Rutland: - Move TLBI workaround from __TLBI_1 macro to high-level flush functions to avoid excessive over-invalidation - Add kernel IC IVAU upgrade via ALTERNATIVE in assembler.h - Add cpucap_is_possible() entry for compile-time elimination] Tested-by: Franz Schnyder <franz.schnyder@toradex.com> # Toradex Apalis iMX8QM Co-developed-by: Ivan T. Ivanov <iivanov@suse.de> Signed-off-by: Ivan T. Ivanov <iivanov@suse.de> Signed-off-by: Peng Fan <peng.fan@nxp.com> [catalin.marinas@arm.com: Kconfig and errata text, sysreg_clear_set(), of_machine_compatible_match()] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2026-10-12arm64: cpufeature: Add ARM64_CPUCAP_SOC_ERRATUM for SoC-level errataCatalin Marinas
Add a new capability type for SoC-level (not CPU-level) errata. Unlike CPU errata which need to be checked on every CPU, SoC errata are probed once on the boot CPU and the result applies to all CPUs. Secondary and late CPUs skip the matches() check but still get cpu_enable() called. This avoids redundant probing on secondary and late CPUs for errata that are SoC-wide properties, not per-CPU. Convert ARM64_ERRATUM_4311569 from ARM64_CPUCAP_SYSTEM_FEATURE to ARM64_CPUCAP_SOC_ERRATUM since it is a SoC-level property already determined at boot. This changes its scope from SCOPE_SYSTEM to SCOPE_BOOT_CPU, so the workaround is now enabled before the secondary CPUs are brought up. This is safe as the erratum is enabled via an early_param, parsed before the boot CPU capabilities are detected. Co-developed-by: Peng Fan <peng.fan@nxp.com> Signed-off-by: Peng Fan <peng.fan@nxp.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2026-10-01arm64: errata: Add Cortex-A725 erratum 3821522 workaroundBeata Michalska
Cortex-A725 erratum 3821522 affects the CNT_CYCLES event, which can incur a significant increment error when a CPU enters and subsequently exits WFE or WFI, and may no longer track the system counter frequency. The AMEVCNTR01_EL0 counter is used as the AMU constant counter for frequency invariance and CPPC FFH feedback counters. Wire the affected Cortex-A725 range into the shared broken AMU constant-counter capability so the affected counter is treated as unavailable by returning zero in the AMU counter paths. This prevents the broken counter from being used as a reference source. The erratum can also affect PMUv3 users of the CNT_CYCLES event, but this workaround intentionally does not change PMU event handling. Hiding or rejecting the PMU event from the erratum code would change the perf-visible PMU event interface, including raw event selection, and would need a separate PMU-specific approach rather than being folded into the AMU reference-counter workaround. Cc: stable@vger.kernel.org Signed-off-by: Beata Michalska <beata.michalska@arm.com> Reviewed-by: Vladimir Murzin <vladimir.murzin@arm.com> Signed-off-by: Will Deacon <will@kernel.org>
2026-10-01arm64: errata: Factor out broken AMU const counter capBeata Michalska
Move the workaround from the erratum 2457168-specific cpucap to a generic broken AMU constant-counter one. This keeps the existing Cortex-A510 handling unchanged while allowing other errata with similar AMU constant counter issue to share the capability bit and call sites. Signed-off-by: Beata Michalska <beata.michalska@arm.com> Reviewed-by: Vladimir Murzin <vladimir.murzin@arm.com> Signed-off-by: Will Deacon <will@kernel.org>
2026-09-18arm64: errata: match the target implementation CPU's own MIDRDavid Carlier
__is_affected_midr_range() is handed the MIDR and REVIDR of one target implementation CPU, but tests the erratum's range with is_midr_in_range(), which re-scans all of target_impl_cpus[] and ignores the @midr argument. The range test is thus constant across the per-CPU loop in is_affected_midr_range() and only answers "is any target CPU in range". Since just the fixed_revs REVIDR check uses the iteration's own registers, an out-of-range target CPU can decide whether a MIDR_FIXED() exemption applies. A VM then enables a workaround whose only in-range CPU is fixed silicon, e.g. erratum 2658417 on a Cortex-A510 r1p1 with REVIDR_EL1[25] set. Factor the range test into __is_midr_in_range(), which takes an explicit MIDR, and use it in __is_affected_midr_range(). Fixes: 86edf6bdcf05 ("smccc/kvm_guest: Enable errata based on implementation CPUs") Cc: stable@vger.kernel.org Assisted-by: Claude:claude-opus-5 Signed-off-by: David Carlier <devnexen@gmail.com> Signed-off-by: Will Deacon <will@kernel.org>
2026-09-03arm64: errata: pass REVIDR when matching target implementation CPUsKhushit Shah
When target implementation CPUs are provided, is_affected_midr_range() accidentally passed the MIDR as both arguments to __is_affected_midr_range(), so the REVIDR mask check operated on the wrong register. Pass REVIDR as intended. Fixes: 86edf6bdcf05 ("smccc/kvm_guest: Enable errata based on implementation CPUs") Cc: stable@vger.kernel.org Signed-off-by: Khushit Shah <khushit.shah@nutanix.com> Reviewed-by: Zenghui Yu (Huawei) <zenghui.yu@linux.dev> Acked-by: Marc Zyngier <maz@kernel.org> Reviewed-by: Shameer Kolothum <skolothumtho@nvidia.com> Signed-off-by: Will Deacon <will@kernel.org>
2026-08-25Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvmLinus Torvalds
Pull kvm updates from Paolo Bonzini: "ARM64: - Add support for 'slot' based PMU events, paired with new UAPI that compels the user to select a specific PMU implementation - Lazy save/restore of vCPU state for pKVM, along with various fixes and cleanups to the management of vCPU state between the untrusted host and pKVM hypervisor - Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1 is present, guaranteeing that EL2-specific register bits are stateful in the EL1 counterpart - Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope of those instructions remains 'in host' (i.e. L1 kernel/userspace) - Pile of fixes for the management of the VNCR pseudo-TLB, such as under-invalidations and races with concurrent TLBIs on other vCPUs - Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a runtime-patched constant, allowing the same data to be shared with pKVM prior to dropping host privileges - Considerable pile of LLM-assisted fixes around the shop but mostly in the VGIC, our in-kernel generator of bugs (and sometimes interrupts) LoongArch: - Advertise already-supported capabilities - Some bug fixes about timer and MMIO - Some hardening about interrupt injection - Replace kvm_err() with kvm_pr_unimpl() - Add FPU/LSX/LASX test cases for selftests RISC-V: - Svadu/Zicfiss/Zicfilp FWFT support for Guest - Use try_cmpxchg for IMSIC MRIF RMW - More arch-specific tracepoints in KVM RISC-V - Eager page splitting when enabling dirty logging - Optimize hfence request handling for SMP Guests - Improve dirty log clearing by skipping zero bits in mask - Guard HFENCE range loops against overflow - CPU PM notifiers in KVM RISC-V for non-retentive idle states - Fix kernel-mode vector context save/restore for Guest s390: - Fixes for vfio-ap - Fixes for the gmap rework - Fixes for vsie - AI triggered fixes all over - diag9c tracing - code move preparation for the additional arm64 support - enable CONTEXT_ANALYSIS x86: - Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding regs.c (the kvm_cache_regs.h => regs.h is already applied) and msrs.{c,h}, and moving relevant code out of x86.c - Split kvm_mmu in three parts, respectively to describe the format of page tables, walking the guest page tables and building the page tables. Always use the same page table walker kvm->arch.gva_walk as the entry point to convert a guest's virtual address, where the previous code used two different kvm_mmu structs depending on whether the walk included nested EPT/NPT or not. Make page fault vmexits reuse the permission checking machinery that is used for guest page faults. This is both a cleanup and a baby step towards supporting XS/XU memory permissions - Document some of the "fun" gotchas with the APIC base when creating IRQCHIPs on x86 - Remove a defunct masterclock update from kvm_xen_shared_info_init(). It could result in incorrect kvmclock due to triggering an unnecessary switch to/from masterclock mode - Skip Xen runstate time updates if time has effectively gone backwards, so that the guest doesn't report 100% steal time for a very, very long time - Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM was updating the wrong sub-leaf, and upstream KVM will soon provide all the information needed by userspace to populate the CPUID field itself - Fix a bug where KVM would walk a newly created rmap without holding the rmap lock (or mmu_lock) during aging - Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs - Fix a variety of #DB priority bugs - Fix a class of races related to enabling Hyper-V emulation on a vCPU after the vCPU is visible to the rest of KVM - Use static calls for nested virtualization ops - Move more KVM-internal code out of x86's kvm_host.h - Enumerate support for a variety of Zhaoxin instructions that don't require explicit virtualization - Fix missing EFER validation bugs, including in the KVM_SET_SREGS* path - Harden kvm_vcpu_map() against double-mapping and thus leaking references - Misc fixes and cleanups, e.g. for largely benign syzkaller splats x86 (Intel): - Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used for IPI virtualization when the vCPU is freed, to fix a use-after-free where hardware will write to a freed vCPU's PID - Service local TLB flushes on a failed nested VM-Enter to fix a bug where KVM could miss a TLB on a future, successful VM-Enter with the same L2 VPID - Cap the maximum value shoved into the VMX Preemption Timer to workaround an erratum that affects all existing Intel CPUs that support CPUID 0x15 - Fix VPID virtualization bugs where KVM would fail to flush hardware TLBs - Harden the TDX "populate" ioctls against bad input, and to prepare for supporting in-place private<=>shared conversion x86 (AMD): - Forcefully invalidate SNP VMSA pages if their backing guest_memfd page is zapped/invalidated, e.g. due to a PUNCH_HOLE in response to a Page-State Change request - Remove a dying VM from the GA Log notifier list before the VM is actually destroyed, to fix a potential use-after-free - While FOLL_WRITE was needed in the past to trigger CoW unsharing, nowadays FOLL_LONGTERM does that already even without FOLL_WRITE, and in fact, get_user_pages() actually disallows FOLL_WRITE together with FOLL_LONGTERM. So don't pass FOLL_WRITE when registering encrypted memory regions, i.e. when pinning SEV/SEV-ES guest memory, to fix a regression with file-backed memory introduced by KVM's (correct) usage of long-term pins (This was reviewed by mm maintainers; for more information, see commit ee1a586dd1fa "KVM: SEV: Drop FOLL_WRITE for encrypted region registration") - Allocate full pages for SEV/SEV-ES {DE,EN}CRYPT ops on SNP-enabled hosts to fix a data corruption issue due to the PSP driver assigning to-be-written pages to firmware (as required by the SNP specs) - Unconditionally intercept ICBEP so that KVM generates the correct guest RIP when handling an ICEBP-induced TASK_SWITCH #VMEXIT - Harden the SNP "populate" ioctls against bad input, and to prepare for supporting in-place private<=>shared conversion Generic: - Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's debugfs - Add a per-VM bitmap to track which vCPU IDs have been "claimed" but for which the vCPU isn't yet online, and use the bitmap to reject duplicate IDs before calling into arch code. This allows arch code to consume vcpu_id without having to worry about cross-vCPU clobbering (at least s390 and x86 have had related bugs) - Rework the so called "prepare" and "invalidate" guest_memfd hooks to prepare for in-place private<=>shared conversion, and clean up a few warts along the way Selftests: - Automatically allocate a full page for L2 guest stacks on x86 instead of requiring test-specific L1 guest code to carve out a portion of the L1 stack for L2 usage, and to ensure the L2 stack also adheres to the x86-64 calling convention ABI - Add a selftest to verify {Guest,Host}-Only behavior in x86's mediated PMU - Clean up nested SVM's handling of GPRs on L2<=>L1 transitions, reuse the functionality for nested VMX, and drop the ucall hack that was fudging around the lack of GPR switching on nVMX - Add a stress test to verify KVM doesn't clobber/drop #PF state, e.g. CR2, across save/restore, including when L2 is active - Add a test to verify KVM_CREATE_VM accepts exactly what is reported by KVM_CAP_VM_TYPES - Misc selftests fixes and cleanups - Fix several issues with seeding the pRNG, and rework the pRNG APIs to that the pRNG can be sanely used in host code, not just guest code - Add an IRQ test to validate virtual IRQ deliverty for IRQs wired up via KVM_IRQFD + KVM_SET_GSI_ROUTING, with optional support for triggering IRQs via writes to an assigned VFIO device - Add syscall wrappers to assert success on a variety of pthreads and CPU affinity APIs - Set vCPU pthread affinity as early as possible to reduce contention issues that were surfaced by PREEMPT_LAZY, which result in runtimes of over a minute on large hosts, versus the expected ~5 seconds - Rework the PMU counters test to run each testcase using a single VM with many vCPUs for each sub-testcase, instead of using a unique VM for each sub-testcase. This cuts the runtime by ~20x Miscellaneous: - MAINTAINERS updates for vfio-ap, guest_memfd, kvm-x86. Mostly representing the status quo more accurately, but also... welcome David Hildenbrand as guest_memfd reviewer!" * tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (413 commits) KVM: arm64: Validate GICv5 timer PPIs before claiming ownership KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs KVM: arm64: vgic: Prevent speculative SPI array underflow KVM: arm64: vgic: Free gic_kvm_info on initialization failure KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params' s390/vfio-ap: Fix NULL deref in status_show() during queue probe s390/vfio-ap: Fix hot-unplug skipped when last AP adapter or domain removed s390/vfio-ap: fix potential use of uninitialized apm_filtered bitmap s390/vfio-ap: Fix control domain removal in vfio_ap_mdev_cfg_remove s390/vfio-ap: Fix required lock not held during update of ap_matrix_mdev object s390/vfio-ap: Fix missing lock required to access list of ap_matrix_mdev objects s390/vfio-ap: Fix dereference matrix_mdev->kvm without checking for NULL s390/vfio-ap: Fix stale do_remove flag across iterations in vfio_ap_mdev_cfg_remove RISC-V: KVM: fix vcpu vector context handling for kernel-mode vector riscv: vector: allow non-preemptible kernel-mode vector with IRQs off riscv: vector: refactor riscv_v_start_kernel_context KVM: s390: gmap: Make prefix handling optional KVM: s390: gmap: Make CMMA optional KVM: s390: gmap: Make storage keys optional KVM: s390: Prepare gmap for a second KVM implementation ...
2026-07-22arm64: errata: work around NVIDIA Olympus device store/load orderingShanker Donthineni
On systems with NVIDIA Olympus cores, a Device-nGnR* load can be observed by a peripheral before an older, non-overlapping Device-nGnR* store to the same peripheral. This breaks the program-order guarantee that software expects for Device-nGnR* accesses and can leave a peripheral in an incorrect state. The erratum can occur only when all of the following apply: - A PE executes a Device-nGnR* store followed by a younger Device-nGnR* load. - The store is not a store-release. - The accesses target the same peripheral and do not overlap in bytes. - There is at most one intervening Device-nGnR* store in program order, and there are no intervening Device-nGnR* loads. - There is no DSB or full DMB between the store and the load. - Specific microarchitectural and timing conditions occur. Insert a DMB OSH immediately before each raw MMIO load on affected CPUs. As a full barrier, DMB OSH orders the older Device store before the younger Device load and prevents the erroneous observation. Add the barrier directly to the __raw_read*() helpers, independently of the existing device-load-acquire alternative. On affected CPUs this adds one DMB OSH per raw MMIO load, including each load used by memcpy_fromio(). On unaffected CPUs the alternative remains a NOP. Co-developed-by: Vikram Sethi <vsethi@nvidia.com> Signed-off-by: Vikram Sethi <vsethi@nvidia.com> Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com> Link: https://lore.kernel.org/all/akPQ8F3OgER621UP@willie-the-truck/ Reviewed-by: Vladimir Murzin <vladimir.murzin@arm.com> Signed-off-by: Will Deacon <will@kernel.org>
2026-07-21arm64: Clarify ARM64_WORKAROUND_REPEAT_TLBI semanticsMark Rutland
Will notes that the ARM64_WORKAROUND_REPEAT_TLBI name is potentially misleading, and that it would be nice to rename that and add some documentation. See: https://lore.kernel.org/linux-arm-kernel/ajKn_Pt50CmOUrsP@willie-the-truck/ To that end, I've renamed the Kconfig symbol and hwcap from: [CONFIG_]ARM64_WORKAROUND_REPEAT_TLBI ... to: [CONFIG_]ARM64_WORKAROUND_REPEAT_TLBI_SYNC ... and I've added some rationale alongside the Kconfig. As the Kconfig symbol isn't user selectable, the usual 'help' section won't appear in menuconfig, so I've added this as a comment. The rename was scripted with: git grep -l REPEAT_TLBI | while read F; do sed -i '{ s/WORKAROUND_REPEAT_TLBI\>/WORKAROUND_REPEAT_TLBI_SYNC/g }' $F; done Bikeshedding-wise, I considered a few names, including: * ARM64_WORKAROUND_REPEAT_TLBI_SYNC * ARM64_WORKAROUND_TLBI_REPEAT_SYNC * ARM64_WORKAROUND_BROADCAST_TLBI_REPEAT_SYNC ... and I settled on ARM64_WORKAROUND_REPEAT_TLBI_SYNC to try keep things simple, and to avoid unnecessary churn caused by moving definitions to retain alphabetical order. I'm happy to defer to Will and Catalin's preference. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2026-07-21KVM: arm64: Move GICv3 broken SEIS implementation detection to a CPU errrataMarc Zyngier
Using is_midr_in_range_list() in a noinstr function isn't a good idea. And yet kvm_compute_ich_hcr_trap_bits() is doing that by calling vgic_v3_broken_seis(). Move all the broken SEIS detection logic to the errata detection framework, and use a new ARM64_WORKAROUND_GICv3_BROKEN_SEIS cap to indicate that we're running on broken CPUs. This reuses the MIDR list used for IMPDEF PMU detection, which has a 100% overlap with the SEIS stuff... Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260721170754.3150521-3-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-06-10arm64: errata: Mitigate TLBI errata on Microsoft Azure Cobalt 100 CPUWill Deacon
Commit fb091ff39479 ("arm64: Subscribe Microsoft Azure Cobalt 100 to ARM Neoverse N2 errata") states that Microsoft Azure Cobalt 100 CPU "is a Microsoft implemented CPU based on r0p0 of the ARM Neoverse N2 CPU, and therefore suffers from all the same errata.". So enable the workaround for the latest broadcast TLB invalidation bug on these parts. Signed-off-by: Will Deacon <will@kernel.org>
2026-06-10arm64: errata: Mitigate TLBI errata on NVIDIA Olympus CPUShanker Donthineni
NVIDIA Olympus cores are affected by the TLBI completion issue tracked as CVE-2025-10263. The existing ARM64_ERRATUM_4118414 handling already uses ARM64_WORKAROUND_REPEAT_TLBI to issue an additional broadcast TLBI;DSB sequence and ensure affected memory write effects are globally observed. Add MIDR_NVIDIA_OLYMPUS to the repeat-TLBI match list so the same mitigation is enabled on affected Olympus systems. Also document the NVIDIA Olympus erratum in the arm64 silicon errata table and list it in the Kconfig help text. Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Mark Rutland <mark.rutland@arm.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Will Deacon <will@kernel.org>
2026-06-10arm64: errata: Mitigate TLBI errata on various Arm CPUsMark Rutland
A number of CPUs developed by Arm suffer from errata whereby a broadcast TLBI;DSB sequence may complete before the global observation of writes which are translated by an affected TLB entry. These errata ONLY affect the completion of memory accesses which have been translated by an invalidated TLB entry, and these errata DO NOT affect the actual invalidation of TLB entries. TLB entries are removed correctly. This issue has been assigned CVE ID CVE-2025-10263. To mitigate this issue, Arm recommends that software follows any affected TLBI;DSB sequence with an additional TLBI;DSB, which will ensure that all memory write effects affected by the first TLBI have been globally observed. The additional TLBI can use any operation that is broadcast to affected CPUs, and the additional DSB can use any option that is sufficient to complete the additional TLBI. The ARM64_WORKAROUND_REPEAT_TLBI workaround is sufficient to mitigate the issue. Enable this workaround for affected CPUs, and update the silicon errata documentation accordingly. Note that due to the manner in which Arm develops IP and tracks errata, some CPUs share a common erratum number. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2026-06-03arm64: kernel: Disable CNP on HiSilicon HIP09Zeng Heng
HiSilicon HIP09 implements TLB entry matching behavior that deviates from the ARM architecture specification when the CNP (Common not Private) bit is set in TTBRx_ELx. When TTBRx.CNP=1, TLB entries may be incorrectly shared between CPU cores, leading to TLB conflicts and stale mappings. This affects coherency and can result in incorrect translations. Add the hardware erratum workaround (Hisilicon erratum 162100125) to disable CNP on affected HIP09 cores. Co-developed-by: Tong Tiangen <tongtiangen@huawei.com> Signed-off-by: Tong Tiangen <tongtiangen@huawei.com> Signed-off-by: Zeng Heng <zengheng4@huawei.com> Reviewed-by: Vladimir Murzin <vladimir.murzin@arm.com> Acked-by: Wei Xu <xuwei5@hisilicon.com> Signed-off-by: Will Deacon <will@kernel.org>
2026-06-03arm64: cpufeature: Add WORKAROUND_DISABLE_CNP capabilityZeng Heng
The NVIDIA Carmel CNP erratum is not the only case requiring CNP to be disabled. Abstract this into a common WORKAROUND_DISABLE_CNP capability to facilitate adding errata for future chips and reduce duplicate checks in has_useable_cnp(). This serves as a prerequisite for the subsequent Hisilicon erratum 162100125. Suggested-by: Vladimir Murzin <vladimir.murzin@arm.com> Signed-off-by: Zeng Heng <zengheng4@huawei.com> Reviewed-by: Vladimir Murzin <vladimir.murzin@arm.com> Acked-by: Wei Xu <xuwei5@hisilicon.com> Signed-off-by: Will Deacon <will@kernel.org>
2026-04-10arm64: errata: Work around early CME DVMSync acknowledgementCatalin Marinas
C1-Pro acknowledges DVMSync messages before completing the SME/CME memory accesses. Work around this by issuing an IPI to the affected CPUs if they are running in EL0 with SME enabled. Note that we avoid the local DSB in the IPI handler as the kernel runs with SCTLR_EL1.IESB=1. This is sufficient to complete SME memory accesses at EL0 on taking an exception to EL1. On the return to user path, no barrier is necessary either. See the comment in sme_set_active() and the more detailed explanation in the link below. To avoid a potential IPI flood from malicious applications (e.g. madvise(MADV_PAGEOUT) in a tight loop), track where a process is active via mm_cpumask() and only interrupt those CPUs. Link: https://lore.kernel.org/r/ablEXwhfKyJW1i7l@J2N7QTR9R3 Cc: Will Deacon <will@kernel.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Mark Brown <broonie@kernel.org> Reviewed-by: Will Deacon <will@kernel.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2026-01-23arm64: errata: Workaround for SI L1 downstream coherency issueLucas Wei
When software issues a Cache Maintenance Operation (CMO) targeting a dirty cache line, the CPU and DSU cluster may optimize the operation by combining the CopyBack Write and CMO into a single combined CopyBack Write plus CMO transaction presented to the interconnect (MCN). For these combined transactions, the MCN splits the operation into two separate transactions, one Write and one CMO, and then propagates the write and optionally the CMO to the downstream memory system or external Point of Serialization (PoS). However, the MCN may return an early CompCMO response to the DSU cluster before the corresponding Write and CMO transactions have completed at the external PoS or downstream memory. As a result, stale data may be observed by external observers that are directly connected to the external PoS or downstream memory. This erratum affects any system topology in which the following conditions apply: - The Point of Serialization (PoS) is located downstream of the interconnect. - A downstream observer accesses memory directly, bypassing the interconnect. Conditions: This erratum occurs only when all of the following conditions are met: 1. Software executes a data cache maintenance operation, specifically, a clean or clean&invalidate by virtual address (DC CVAC or DC CIVAC), that hits on unique dirty data in the CPU or DSU cache. This results in a combined CopyBack and CMO being issued to the interconnect. 2. The interconnect splits the combined transaction into separate Write and CMO transactions and returns an early completion response to the CPU or DSU before the write has completed at the downstream memory or PoS. 3. A downstream observer accesses the affected memory address after the early completion response is issued but before the actual memory write has completed. This allows the observer to read stale data that has not yet been updated at the PoS or downstream memory. The implementation of workaround put a second loop of CMOs at the same virtual address whose operation meet erratum conditions to wait until cache data be cleaned to PoC. This way of implementation mitigates performance penalty compared to purely duplicate original CMO. Signed-off-by: Lucas Wei <lucaswei@google.com> Signed-off-by: Will Deacon <will@kernel.org>
2025-09-22arm64: errata: Apply workarounds for Neoverse-V3AEMark Rutland
Neoverse-V3AE is also affected by erratum #3312417, as described in its Software Developer Errata Notice (SDEN) document: Neoverse V3AE (MP172) SDEN v9.0, erratum 3312417 https://developer.arm.com/documentation/SDEN-2615521/9-0/ Enable the workaround for Neoverse-V3AE, and document this. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com> Signed-off-by: Will Deacon <will@kernel.org>
2025-09-18arm64: errata: Expand speculative SSBS workaround for Cortex-A720AEKuninori Morimoto
It is same as Cortex-A720. Link: https://lore.kernel.org/all/aMlFwbDjJ6yKuxTv@J2N7QTR9R3.cambridge.arm.com/ Signed-off-by: Kuninori Morimoto <kuninori.morimoto.gx@renesas.com> Signed-off-by: Will Deacon <will@kernel.org>
2025-05-23Merge branch kvm-arm64/misc-6.16 into kvmarm-master/nextMarc Zyngier
* kvm-arm64/misc-6.16: : . : Misc changes and improvements for 6.16: : : - Add a new selftest for the SVE host state being corrupted by a guest : : - Keep HCR_EL2.xMO set at all times for systems running with the kernel at EL2, : ensuring that the window for interrupts is slightly bigger, and avoiding : a pretty bad erratum on the AmpereOne HW : : - Replace a couple of open-coded on/off strings with str_on_off() : : - Get rid of the pKVM memblock sorting, which now appears to be superflous : : - Drop superflous clearing of ICH_LR_EOI in the LR when nesting : : - Add workaround for AmpereOne's erratum AC04_CPU_23, which suffers from : a pretty bad case of TLB corruption unless accesses to HCR_EL2 are : heavily synchronised : : - Add a per-VM, per-ITS debugfs entry to dump the state of the ITS tables : in a human-friendly fashion : . KVM: arm64: Fix documentation for vgic_its_iter_next() KVM: arm64: vgic-its: Add debugfs interface to expose ITS tables arm64: errata: Work around AmpereOne's erratum AC04_CPU_23 KVM: arm64: nv: Remove clearing of ICH_LR<n>.EOI if ICH_LR<n>.HW == 1 KVM: arm64: Drop sort_memblock_regions() KVM: arm64: selftests: Add test for SVE host corruption KVM: arm64: Force HCR_EL2.xMO to 1 at all times in VHE mode KVM: arm64: Replace ternary flags with str_on_off() helper Signed-off-by: Marc Zyngier <maz@kernel.org>
2025-05-19arm64: errata: Work around AmpereOne's erratum AC04_CPU_23D Scott Phillips
On AmpereOne AC04, updates to HCR_EL2 can rarely corrupt simultaneous translations for data addresses initiated by load/store instructions. Only instruction initiated translations are vulnerable, not translations from prefetches for example. A DSB before the store to HCR_EL2 is sufficient to prevent older instructions from hitting the window for corruption, and an ISB after is sufficient to prevent younger instructions from hitting the window for corruption. Signed-off-by: D Scott Phillips <scott@os.amperecomputing.com> Reviewed-by: Oliver Upton <oliver.upton@linux.dev> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20250513184514.2678288-1-scott@os.amperecomputing.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2025-04-18arm64: Rework checks for broken Cavium HW in the PI codeMarc Zyngier
Calling into the MIDR checking framework from the PI code has recently become much harder, due to the new fancy "multi-MIDR" support that relies on tables being populated at boot time, but not that early that they are available to the PI code. There are additional issues with this framework, as the code really isn't position independend *at all*. This leads to some ugly breakages, as reported by Ada. It so appears that the only reason for the PI code to call into the MIDR checking code is to cope with The Most Broken ARM64 System Ever, aka Cavium ThunderX, which cannot deal with nG attributes that result of the combination of KASLR and KPTI as a consequence of Erratum 27456. Duplicate the check for the erratum in the PI code, removing the dependency on the bulk of the MIDR checking framework. This allows dropping that same check from kaslr_requires_kpti(), as the KPTI code already relies on the ARM64_WORKAROUND_CAVIUM_27456 cap. Fixes: c8c2647e69bed ("arm64: Make  _midr_in_range_list() an exported function") Reported-by: Ada Couprie Diaz <ada.coupriediaz@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/3d97e45a-23cf-419b-9b6f-140b4d88de7b@arm.com Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> Cc: Oliver Upton <oliver.upton@linux.dev> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20250418093129.1755739-1-maz@kernel.org Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-03-19Merge branch 'kvm-arm64/pmuv3-asahi' into kvmarm/nextOliver Upton
* kvm-arm64/pmuv3-asahi: : Support PMUv3 for KVM guests on Apple silicon : : Take advantage of some IMPLEMENTATION DEFINED traps available on Apple : parts to trap-and-emulate the PMUv3 registers on behalf of a KVM guest. : Constrain the vPMU to a cycle counter and single event counter, as the : Apple PMU has events that cannot be counted on every counter. : : There is a small new interface between the ARM PMU driver and KVM, where : the PMU driver owns the PMUv3 -> hardware event mappings. arm64: Enable IMP DEF PMUv3 traps on Apple M* KVM: arm64: Provide 1 event counter on IMPDEF hardware drivers/perf: apple_m1: Provide helper for mapping PMUv3 events KVM: arm64: Remap PMUv3 events onto hardware KVM: arm64: Advertise PMUv3 if IMPDEF traps are present KVM: arm64: Compute synthetic sysreg ESR for Apple PMUv3 traps KVM: arm64: Move PMUVer filtering into KVM code KVM: arm64: Use guard() to cleanup usage of arm_pmus_lock KVM: arm64: Drop kvm_arm_pmu_available static key KVM: arm64: Use a cpucap to determine if system supports FEAT_PMUv3 KVM: arm64: Always support SW_INCR PMU event KVM: arm64: Compute PMCEID from arm_pmu's event bitmaps drivers/perf: apple_m1: Support host/guest event filtering drivers/perf: apple_m1: Refactor event select/filter configuration Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-03-11arm64: Enable IMP DEF PMUv3 traps on Apple M*Oliver Upton
Apple M1 and M2 CPUs support IMPDEF traps of the PMUv3 sysregs, allowing a hypervisor to virtualize an architectural PMU for a VM. Flip the appropriate bit in HACR_EL2 on supporting hardware. Tested-by: Janne Grunau <j@jannau.net> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20250305203040.428448-1-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-02-26smccc/kvm_guest: Enable errata based on implementation CPUsShameer Kolothum
Retrieve any migration target implementation CPUs using the hypercall and enable associated errata. Reviewed-by: Cornelia Huck <cohuck@redhat.com> Reviewed-by: Sebastian Ott <sebott@redhat.com> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20250221140229.12588-6-shameerali.kolothum.thodi@huawei.com Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-02-26arm64: Make  _midr_in_range_list() an exported functionShameer Kolothum
Subsequent patch will add target implementation CPU support and that will require _midr_in_range_list() to access new data. To avoid exporting the data make _midr_in_range_list() a normal function and export it. No functional changes intended. Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20250221140229.12588-5-shameerali.kolothum.thodi@huawei.com Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-02-26arm64: Modify _midr_range() functions to read MIDR/REVIDR internallyShameer Kolothum
These changes lay the groundwork for adding support for guest kernels, allowing them to leverage target CPU implementations provided by the VMM. No functional changes intended. Suggested-by: Oliver Upton <oliver.upton@linux.dev> Reviewed-by: Sebastian Ott <sebott@redhat.com> Reviewed-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20250221140229.12588-2-shameerali.kolothum.thodi@huawei.com Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2025-01-02KVM: arm64: Work around x1e's CNTVOFF_EL2 bogosityMarc Zyngier
It appears that on Qualcomm's x1e CPU, CNTVOFF_EL2 doesn't really work, specially with HCR_EL2.E2H=1. A non-zero offset results in a screaming virtual timer interrupt, to the tune of a few 100k interrupts per second on a 4 vcpu VM. This is also evidenced by this CPU's inability to correctly run any of the timer selftests. The only case this doesn't break is when this register is set to 0, which breaks VM migration. When HCR_EL2.E2H=0, the timer seems to behave normally, and does not result in an interrupt storm. As a workaround, use the fact that this CPU implements FEAT_ECV, and trap all accesses to the virtual timer and counter, keeping CNTVOFF_EL2 set to zero, and emulate accesses to CVAL/TVAL/CTL and the counter itself, fixing up the timer to account for the missing offset. And if you think this is disgusting, you'd probably be right. Acked-by: Oliver Upton <oliver.upton@linux.dev> Link: https://lore.kernel.org/r/20241217142321.763801-12-maz@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2024-10-04arm64: Subscribe Microsoft Azure Cobalt 100 to erratum 3194386Easwar Hariharan
Add the Microsoft Azure Cobalt 100 CPU to the list of CPUs suffering from erratum 3194386 added in commit 75b3c43eab59 ("arm64: errata: Expand speculative SSBS workaround") CC: Mark Rutland <mark.rutland@arm.com> CC: James More <james.morse@arm.com> CC: Will Deacon <will@kernel.org> CC: stable@vger.kernel.org # 6.6+ Signed-off-by: Easwar Hariharan <eahariha@linux.microsoft.com> Link: https://lore.kernel.org/r/20241003225239.321774-1-eahariha@linux.microsoft.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2024-10-01arm64: errata: Expand speculative SSBS workaround once moreMark Rutland
A number of Arm Ltd CPUs suffer from errata whereby an MSR to the SSBS special-purpose register does not affect subsequent speculative instructions, permitting speculative store bypassing for a window of time. We worked around this for a number of CPUs in commits: * 7187bb7d0b5c7dfa ("arm64: errata: Add workaround for Arm errata 3194386 and 3312417") * 75b3c43eab594bfb ("arm64: errata: Expand speculative SSBS workaround") * 145502cac7ea70b5 ("arm64: errata: Expand speculative SSBS workaround (again)") Since then, a (hopefully final) batch of updates have been published, with two more affected CPUs. For the affected CPUs the existing mitigation is sufficient, as described in their respective Software Developer Errata Notice (SDEN) documents: * Cortex-A715 (MP148) SDEN v15.0, erratum 3456084 https://developer.arm.com/documentation/SDEN-2148827/1500/ * Neoverse-N3 (MP195) SDEN v5.0, erratum 3456111 https://developer.arm.com/documentation/SDEN-3050973/0500/ Enable the existing mitigation by adding the relevant MIDRs to erratum_spec_ssbs_list, and update silicon-errata.rst and the Kconfig text accordingly. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/20240930111705.3352047-3-mark.rutland@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2024-08-30arm64: errata: Enable the AC03_CPU_38 workaround for ampere1aD Scott Phillips
The ampere1a cpu is affected by erratum AC04_CPU_10 which is the same bug as AC03_CPU_38. Add ampere1a to the AC03_CPU_38 workaround midr list. Cc: <stable@vger.kernel.org> Signed-off-by: D Scott Phillips <scott@os.amperecomputing.com> Acked-by: Oliver Upton <oliver.upton@linux.dev> Link: https://lore.kernel.org/r/20240827211701.2216719-1-scott@os.amperecomputing.com Signed-off-by: Will Deacon <will@kernel.org>
2024-08-01arm64: errata: Expand speculative SSBS workaround (again)Mark Rutland
A number of Arm Ltd CPUs suffer from errata whereby an MSR to the SSBS special-purpose register does not affect subsequent speculative instructions, permitting speculative store bypassing for a window of time. We worked around this for a number of CPUs in commits: * 7187bb7d0b5c7dfa ("arm64: errata: Add workaround for Arm errata 3194386 and 3312417") * 75b3c43eab594bfb ("arm64: errata: Expand speculative SSBS workaround") Since then, similar errata have been published for a number of other Arm Ltd CPUs, for which the same mitigation is sufficient. This is described in their respective Software Developer Errata Notice (SDEN) documents: * Cortex-A76 (MP052) SDEN v31.0, erratum 3324349 https://developer.arm.com/documentation/SDEN-885749/3100/ * Cortex-A77 (MP074) SDEN v19.0, erratum 3324348 https://developer.arm.com/documentation/SDEN-1152370/1900/ * Cortex-A78 (MP102) SDEN v21.0, erratum 3324344 https://developer.arm.com/documentation/SDEN-1401784/2100/ * Cortex-A78C (MP138) SDEN v16.0, erratum 3324346 https://developer.arm.com/documentation/SDEN-1707916/1600/ * Cortex-A78C (MP154) SDEN v10.0, erratum 3324347 https://developer.arm.com/documentation/SDEN-2004089/1000/ * Cortex-A725 (MP190) SDEN v5.0, erratum 3456106 https://developer.arm.com/documentation/SDEN-2832921/0500/ * Cortex-X1 (MP077) SDEN v21.0, erratum 3324344 https://developer.arm.com/documentation/SDEN-1401782/2100/ * Cortex-X1C (MP136) SDEN v16.0, erratum 3324346 https://developer.arm.com/documentation/SDEN-1707914/1600/ * Neoverse-N1 (MP050) SDEN v32.0, erratum 3324349 https://developer.arm.com/documentation/SDEN-885747/3200/ * Neoverse-V1 (MP076) SDEN v19.0, erratum 3324341 https://developer.arm.com/documentation/SDEN-1401781/1900/ Note that due to the manner in which Arm develops IP and tracks errata, some CPUs share a common erratum number and some CPUs have multiple erratum numbers for the same HW issue. On parts without SB, it is necessary to use ISB for the workaround. The spec_bar() macro used in the mitigation will expand to a "DSB SY; ISB" sequence in this case, which is sufficient on all affected parts. Enable the existing mitigation by adding the relevant MIDRs to erratum_spec_ssbs_list. The list is sorted alphanumerically (involving moving Neoverse-V3 after Neoverse-V2) so that this is easy to audit and potentially extend again in future. The Kconfig text is also updated to clarify the set of affected parts and the mitigation. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Will Deacon <will@kernel.org> Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/20240801101803.1982459-4-mark.rutland@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2024-06-12arm64: errata: Expand speculative SSBS workaroundMark Rutland
A number of Arm Ltd CPUs suffer from errata whereby an MSR to the SSBS special-purpose register does not affect subsequent speculative instructions, permitting speculative store bypassing for a window of time. We worked around this for Cortex-X4 and Neoverse-V3, in commit: 7187bb7d0b5c7dfa ("arm64: errata: Add workaround for Arm errata 3194386 and 3312417") ... as per their Software Developer Errata Notice (SDEN) documents: * Cortex-X4 SDEN v8.0, erratum 3194386: https://developer.arm.com/documentation/SDEN-2432808/0800/ * Neoverse-V3 SDEN v6.0, erratum 3312417: https://developer.arm.com/documentation/SDEN-2891958/0600/ Since then, similar errata have been published for a number of other Arm Ltd CPUs, for which the mitigation is the same. This is described in their respective SDEN documents: * Cortex-A710 SDEN v19.0, errataum 3324338 https://developer.arm.com/documentation/SDEN-1775101/1900/?lang=en * Cortex-A720 SDEN v11.0, erratum 3456091 https://developer.arm.com/documentation/SDEN-2439421/1100/?lang=en * Cortex-X2 SDEN v19.0, erratum 3324338 https://developer.arm.com/documentation/SDEN-1775100/1900/?lang=en * Cortex-X3 SDEN v14.0, erratum 3324335 https://developer.arm.com/documentation/SDEN-2055130/1400/?lang=en * Cortex-X925 SDEN v8.0, erratum 3324334 https://developer.arm.com/documentation/109108/800/?lang=en * Neoverse-N2 SDEN v17.0, erratum 3324339 https://developer.arm.com/documentation/SDEN-1982442/1700/?lang=en * Neoverse-V2 SDEN v9.0, erratum 3324336 https://developer.arm.com/documentation/SDEN-2332927/900/?lang=en Note that due to shared design lineage, some CPUs share the same erratum number. Add these to the existing mitigation under CONFIG_ARM64_ERRATUM_3194386. As listing all of the erratum IDs in the runtime description would be unwieldy, this is reduced to: "SSBS not fully self-synchronizing" ... matching the description of the errata in all of the SDENs. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/20240603111812.1514101-6-mark.rutland@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2024-06-12arm64: errata: Unify speculative SSBS errata logicMark Rutland
Cortex-X4 erratum 3194386 and Neoverse-V3 erratum 3312417 are identical, with duplicate Kconfig text and some unsightly ifdeffery. While we try to share code behind CONFIG_ARM64_WORKAROUND_SPECULATIVE_SSBS, having separate options results in a fair amount of boilerplate code, and this will only get worse as we expand the set of affected CPUs. To reduce this boilerplate, unify the two behind a common Kconfig option. This removes the duplicate text and Kconfig logic, and removes the need for the intermediate ARM64_WORKAROUND_SPECULATIVE_SSBS option. The set of affected CPUs is described as a list so that this can easily be extended. I've used ARM64_ERRATUM_3194386 (matching the Neoverse-V3 erratum ID) as the common option, matching the way we use ARM64_ERRATUM_1319367 to cover Cortex-A57 erratum 1319537 and Cortex-A72 erratum 1319367. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Will Deacon <wilL@kernel.org> Link: https://lore.kernel.org/r/20240603111812.1514101-5-mark.rutland@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2024-05-10arm64: errata: Add workaround for Arm errata 3194386 and 3312417Mark Rutland
Cortex-X4 and Neoverse-V3 suffer from errata whereby an MSR to the SSBS special-purpose register does not affect subsequent speculative instructions, permitting speculative store bypassing for a window of time. This is described in their Software Developer Errata Notice (SDEN) documents: * Cortex-X4 SDEN v8.0, erratum 3194386: https://developer.arm.com/documentation/SDEN-2432808/0800/ * Neoverse-V3 SDEN v6.0, erratum 3312417: https://developer.arm.com/documentation/SDEN-2891958/0600/ To workaround these errata, it is necessary to place a speculation barrier (SB) after MSR to the SSBS special-purpose register. This patch adds the requisite SB after writes to SSBS within the kernel, and hides the presence of SSBS from EL0 such that userspace software which cares about SSBS will manipulate this via prctl(PR_GET_SPECULATION_CTRL, ...). Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/20240508081400.235362-5-mark.rutland@arm.com Signed-off-by: Will Deacon <will@kernel.org>
2024-02-15arm64: Subscribe Microsoft Azure Cobalt 100 to ARM Neoverse N2 errataEaswar Hariharan
Add the MIDR value of Microsoft Azure Cobalt 100, which is a Microsoft implemented CPU based on r0p0 of the ARM Neoverse N2 CPU, and therefore suffers from all the same errata. CC: stable@vger.kernel.org # 5.15+ Signed-off-by: Easwar Hariharan <eahariha@linux.microsoft.com> Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Acked-by: Marc Zyngier <maz@kernel.org> Reviewed-by: Oliver Upton <oliver.upton@linux.dev> Link: https://lore.kernel.org/r/20240214175522.2457857-1-eahariha@linux.microsoft.com Signed-off-by: Will Deacon <will@kernel.org>
2024-01-12arm64: errata: Add Cortex-A510 speculative unprivileged load workaroundRob Herring
Implement the workaround for ARM Cortex-A510 erratum 3117295. On an affected Cortex-A510 core, a speculatively executed unprivileged load might leak data from a privileged load via a cache side channel. The issue only exists for loads within a translation regime with the same translation (e.g. same ASID and VMID). Therefore, the issue only affects the return to EL0. The erratum and workaround are the same as ARM Cortex-A520 erratum 2966298, so reuse the existing workaround. Cc: stable@vger.kernel.org Signed-off-by: Rob Herring <robh@kernel.org> Reviewed-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20240110-arm-errata-a510-v1-2-d02bc51aeeee@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2024-01-12arm64: Rename ARM64_WORKAROUND_2966298Rob Herring
In preparation to apply ARM64_WORKAROUND_2966298 for multiple errata, rename the kconfig and capability. No functional change. Cc: stable@vger.kernel.org Signed-off-by: Rob Herring <robh@kernel.org> Reviewed-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20240110-arm-errata-a510-v1-1-d02bc51aeeee@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2023-11-01Merge tag 'arm64-upstream' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux Pull arm64 updates from Catalin Marinas: "No major architecture features this time around, just some new HWCAP definitions, support for the Ampere SoC PMUs and a few fixes/cleanups. The bulk of the changes is reworking of the CPU capability checking code (cpus_have_cap() etc). - Major refactoring of the CPU capability detection logic resulting in the removal of the cpus_have_const_cap() function and migrating the code to "alternative" branches where possible - Backtrace/kgdb: use IPIs and pseudo-NMI - Perf and PMU: - Add support for Ampere SoC PMUs - Multi-DTC improvements for larger CMN configurations with multiple Debug & Trace Controllers - Rework the Arm CoreSight PMU driver to allow separate registration of vendor backend modules - Fixes: add missing MODULE_DEVICE_TABLE to the amlogic perf driver; use device_get_match_data() in the xgene driver; fix NULL pointer dereference in the hisi driver caused by calling cpuhp_state_remove_instance(); use-after-free in the hisi driver - HWCAP updates: - FEAT_SVE_B16B16 (BFloat16) - FEAT_LRCPC3 (release consistency model) - FEAT_LSE128 (128-bit atomic instructions) - SVE: remove a couple of pseudo registers from the cpufeature code. There is logic in place already to detect mismatched SVE features - Miscellaneous: - Reduce the default swiotlb size (currently 64MB) if no ZONE_DMA bouncing is needed. The buffer is still required for small kmalloc() buffers - Fix module PLT counting with !RANDOMIZE_BASE - Restrict CPU_BIG_ENDIAN to LLVM IAS 15.x or newer move synchronisation code out of the set_ptes() loop - More compact cpufeature displaying enabled cores - Kselftest updates for the new CPU features" * tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (83 commits) arm64: Restrict CPU_BIG_ENDIAN to GNU as or LLVM IAS 15.x or newer arm64: module: Fix PLT counting when CONFIG_RANDOMIZE_BASE=n arm64, irqchip/gic-v3, ACPI: Move MADT GICC enabled check into a helper perf: hisi: Fix use-after-free when register pmu fails drivers/perf: hisi_pcie: Initialize event->cpu only on success drivers/perf: hisi_pcie: Check the type first in pmu::event_init() arm64: cpufeature: Change DBM to display enabled cores arm64: cpufeature: Display the set of cores with a feature perf/arm-cmn: Enable per-DTC counter allocation perf/arm-cmn: Rework DTC counters (again) perf/arm-cmn: Fix DTC domain detection drivers: perf: arm_pmuv3: Drop some unused arguments from armv8_pmu_init() drivers: perf: arm_pmuv3: Read PMMIR_EL1 unconditionally drivers/perf: hisi: use cpuhp_state_remove_instance_nocalls() for hisi_hns3_pmu uninit process clocksource/drivers/arm_arch_timer: limit XGene-1 workaround arm64: Remove system_uses_lse_atomics() arm64: Mark the 'addr' argument to set_ptes() and __set_pte_at() as unused drivers/perf: xgene: Use device_get_match_data() perf/amlogic: add missing MODULE_DEVICE_TABLE arm64/mm: Hoist synchronization out of set_ptes() loop ...
2023-10-16arm64: Fixup user features at boot timeMark Rutland
For ARM64_WORKAROUND_2658417, we use a cpu_enable() callback to hide the ID_AA64ISAR1_EL1.BF16 ID register field. This is a little awkward as CPUs may attempt to apply the workaround concurrently, requiring that we protect the bulk of the callback with a raw_spinlock, and requiring some pointless work every time a CPU is subsequently hotplugged in. This patch makes this a little simpler by handling the masking once at boot time. A new user_feature_fixup() function is called at the start of setup_user_features() to mask the feature, matching the style of elf_hwcap_fixup(). The ARM64_WORKAROUND_2658417 cpucap is added to cpucap_is_possible() so that code can be elided entirely when this is not possible. Note that the ARM64_WORKAROUND_2658417 capability is matched with ERRATA_MIDR_RANGE(), which implicitly gives the capability a ARM64_CPUCAP_LOCAL_CPU_ERRATUM type, which forbids the late onlining of a CPU with the erratum if the erratum was not present at boot time. Therefore this patch doesn't change the behaviour for late onlining. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Mark Brown <broonie@kernel.org> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> Cc: Will Deacon <will@kernel.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2023-09-29arm64: errata: Add Cortex-A520 speculative unprivileged load workaroundRob Herring
Implement the workaround for ARM Cortex-A520 erratum 2966298. On an affected Cortex-A520 core, a speculatively executed unprivileged load might leak data from a privileged load via a cache side channel. The issue only exists for loads within a translation regime with the same translation (e.g. same ASID and VMID). Therefore, the issue only affects the return to EL0. The workaround is to execute a TLBI before returning to EL0 after all loads of privileged data. A non-shareable TLBI to any address is sufficient. The workaround isn't necessary if page table isolation (KPTI) is enabled, but for simplicity it will be. Page table isolation should normally be disabled for Cortex-A520 as it supports the CSV3 feature and the E0PD feature (used when KASLR is enabled). Cc: stable@vger.kernel.org Signed-off-by: Rob Herring <robh@kernel.org> Link: https://lore.kernel.org/r/20230921194156.1050055-2-robh@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2023-06-16arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2Oliver Upton
AmpereOne has an erratum in its implementation of FEAT_HAFDBS that required disabling the feature on the design. This was done by reporting the feature as not implemented in the ID register, although the corresponding control bits were not actually RES0. This does not align well with the requirements of the architecture, which mandates these bits be RES0 if HAFDBS isn't implemented. The kernel's use of stage-1 is unaffected, as the HA and HD bits are only set if HAFDBS is detected in the ID register. KVM, on the other hand, relies on the RES0 behavior at stage-2 to use the same value for VTCR_EL2 on any cpu in the system. Mitigate the non-RES0 behavior by leaving VTCR_EL2.HA clear on affected systems. Cc: stable@vger.kernel.org Cc: D Scott Phillips <scott@os.amperecomputing.com> Cc: Darren Hart <darren@os.amperecomputing.com> Acked-by: D Scott Phillips <scott@os.amperecomputing.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20230609220104.1836988-2-oliver.upton@linux.dev Signed-off-by: Oliver Upton <oliver.upton@linux.dev>
2023-01-06arm64: errata: Workaround possible Cortex-A715 [ESR|FAR]_ELx corruptionAnshuman Khandual
If a Cortex-A715 cpu sees a page mapping permissions change from executable to non-executable, it may corrupt the ESR_ELx and FAR_ELx registers, on the next instruction abort caused by permission fault. Only user-space does executable to non-executable permission transition via mprotect() system call which calls ptep_modify_prot_start() and ptep_modify _prot_commit() helpers, while changing the page mapping. The platform code can override these helpers via __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION. Work around the problem via doing a break-before-make TLB invalidation, for all executable user space mappings, that go through mprotect() system call. This overrides ptep_modify_prot_start() and ptep_modify_prot_commit(), via defining HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION on the platform thus giving an opportunity to intercept user space exec mappings, and do the necessary TLB invalidation. Similar interceptions are also implemented for HugeTLB. Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Mark Rutland <mark.rutland@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: linux-doc@vger.kernel.org Cc: linux-kernel@vger.kernel.org Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> Link: https://lore.kernel.org/r/20230102061651.34745-1-anshuman.khandual@arm.com Signed-off-by: Will Deacon <will@kernel.org>
2022-12-15Revert "arm64: errata: Workaround possible Cortex-A715 [ESR|FAR]_ELx corruption"Will Deacon
This reverts commit 44ecda71fd8a70185c270f5914ac563827fe1d4c. All versions of this patch on the mailing list, including the version that ended up getting merged, have portions of code guarded by the non-existent CONFIG_ARM64_WORKAROUND_2645198 option. Although Anshuman says he tested the code with some additional debug changes [1], I'm hesitant to fix the CONFIG option and light up a bunch of code right before I (and others) disappear for the end of year holidays, during which time we won't be around to deal with any fallout. So revert the change for now. We can bring back a fixed, tested version for a later -rc when folks are thinking about things other than trees and turkeys. [1] https://lore.kernel.org/r/b6f61241-e436-5db1-1053-3b441080b8d6@arm.com Reported-by: Lukas Bulwahn <lukas.bulwahn@gmail.com> Link: https://lore.kernel.org/r/20221215094811.23188-1-lukas.bulwahn@gmail.com Signed-off-by: Will Deacon <will@kernel.org>
2022-11-18arm64: errata: Workaround possible Cortex-A715 [ESR|FAR]_ELx corruptionAnshuman Khandual
If a Cortex-A715 cpu sees a page mapping permissions change from executable to non-executable, it may corrupt the ESR_ELx and FAR_ELx registers, on the next instruction abort caused by permission fault. Only user-space does executable to non-executable permission transition via mprotect() system call which calls ptep_modify_prot_start() and ptep_modify _prot_commit() helpers, while changing the page mapping. The platform code can override these helpers via __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION. Work around the problem via doing a break-before-make TLB invalidation, for all executable user space mappings, that go through mprotect() system call. This overrides ptep_modify_prot_start() and ptep_modify_prot_commit(), via defining HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION on the platform thus giving an opportunity to intercept user space exec mappings, and do the necessary TLB invalidation. Similar interceptions are also implemented for HugeTLB. Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Mark Rutland <mark.rutland@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: linux-doc@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20221116140915.356601-3-anshuman.khandual@arm.com Signed-off-by: Will Deacon <will@kernel.org>
2022-10-07arm64: errata: Add Cortex-A55 to the repeat tlbi listJames Morse
Cortex-A55 is affected by an erratum where in rare circumstances the CPUs may not handle a race between a break-before-make sequence on one CPU, and another CPU accessing the same page. This could allow a store to a page that has been unmapped. Work around this by adding the affected CPUs to the list that needs TLB sequences to be done twice. Signed-off-by: James Morse <james.morse@arm.com> Cc: <stable@vger.kernel.org> Link: https://lore.kernel.org/r/20220930131959.3082594-1-james.morse@arm.com Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2022-09-16arm64: errata: remove BF16 HWCAP due to incorrect result on Cortex-A510James Morse
Cortex-A510's erratum #2658417 causes two BF16 instructions to return the wrong result in rare circumstances when a pair of A510 CPUs are using shared neon hardware. The two instructions affected are BFMMLA and VMMLA, support for these is indicated by the BF16 HWCAP. Remove it on affected platforms. Signed-off-by: James Morse <james.morse@arm.com> Link: https://lore.kernel.org/r/20220909165938.3931307-4-james.morse@arm.com [catalin.marinas@arm.com: add revision to the Kconfig help; remove .type] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2022-08-23arm64: errata: add detection for AMEVCNTR01 incrementing incorrectlyIonela Voinescu
The AMU counter AMEVCNTR01 (constant counter) should increment at the same rate as the system counter. On affected Cortex-A510 cores, AMEVCNTR01 increments incorrectly giving a significantly higher output value. This results in inaccurate task scheduler utilization tracking and incorrect feedback on CPU frequency. Work around this problem by returning 0 when reading the affected counter in key locations that results in disabling all users of this counter from using it either for frequency invariance or as FFH reference counter. This effect is the same to firmware disabling affected counters. Details on how the two features are affected by this erratum: - AMU counters will not be used for frequency invariance for affected CPUs and CPUs in the same cpufreq policy. AMUs can still be used for frequency invariance for unaffected CPUs in the system. Although unlikely, if no alternative method can be found to support frequency invariance for affected CPUs (cpufreq based or solution based on platform counters) frequency invariance will be disabled. Please check the chapter on frequency invariance at Documentation/scheduler/sched-capacity.rst for details of its effect. - Given that FFH can be used to fetch either the core or constant counter values, restrictions are lifted regarding any of these counters returning a valid (!0) value. Therefore FFH is considered supported if there is a least one CPU that support AMUs, independent of any counters being disabled or affected by this erratum. Clarifying comments are now added to the cpc_ffh_supported(), cpu_read_constcnt() and cpu_read_corecnt() functions. The above is achieved through adding a new erratum: ARM64_ERRATUM_2457168. Signed-off-by: Ionela Voinescu <ionela.voinescu@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: James Morse <james.morse@arm.com> Link: https://lore.kernel.org/r/20220819103050.24211-1-ionela.voinescu@arm.com Signed-off-by: Will Deacon <will@kernel.org>
2022-08-17arm64: Fix match_list for erratum 1286807 on Arm Cortex-A76Zenghui Yu
Since commit 51f559d66527 ("arm64: Enable repeat tlbi workaround on KRYO4XX gold CPUs"), we failed to detect erratum 1286807 on Cortex-A76 because its entry in arm64_repeat_tlbi_list[] was accidently corrupted by this commit. Fix this issue by creating a separate entry for Kryo4xx Gold. Fixes: 51f559d66527 ("arm64: Enable repeat tlbi workaround on KRYO4XX gold CPUs") Cc: Shreyas K K <quic_shrekk@quicinc.com> Signed-off-by: Zenghui Yu <yuzenghui@huawei.com> Acked-by: Marc Zyngier <maz@kernel.org> Link: https://lore.kernel.org/r/20220809043848.969-1-yuzenghui@huawei.com Signed-off-by: Will Deacon <will@kernel.org>