| Age | Commit message (Collapse) | Author |
|---|
|
git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux
Pull crypto library updates from Eric Biggers:
- Migrate the x86 and riscv optimized implementations of various AES
modes (ECB, CBC, CBC-CTS, CTR, XCTR, XTS) into lib/crypto/.
This makes the library APIs be properly accelerated on these
architectures, while still accelerating crypto_skcipher as well.
It removes a lot of redundant glue code, since the crypto API
boilerplate no longer needs to be duplicated for these architectures.
It also fixes the longstanding issue where optimizations for these
AES modes were disabled by default on these architectures.
- Optimize AES-XTS on SPARC64 using the SPARC crypto opcodes
- Move the SipHash code into lib/crypto/
- Update the ChaCha and Poly1305 functions to take size_t lengths,
resolving issues where lengths over INT_MAX were mishandled
- Make frame pointer unwinding work properly in some of the x86
assembly functions, by following the standard %rbp convention
- Add helper functions for zeroizing structs
- Make a few documentation improvements
* tag 'libcrypto-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux: (49 commits)
lib/crypto: md5: remove unused header
lib/crypto: mips/chacha: Handle lengths of 2 GiB or more
lib/crypto: arm/chacha: Handle lengths of 2 GiB or more in scalar code
lib/crypto: chacha: Use size_t length
lib/crypto: poly1305: Use size_t length
lib/crypto: x86/sha512: Follow the normal %rbp convention
lib/crypto: x86/sha256: Follow the normal %rbp convention
lib/crypto: x86/sha1: Follow the normal %rbp convention
lib/crypto: chacha20poly1305: zeroize state in __chacha20poly1305_decrypt
lib/crypto: sparc/aes-xts: Add optimization using the AES opcodes
lib: Move SipHash into lib/crypto/
lib/crypto: x86/aes-ctr: Remove some unreachable code
crypto: x86/aesni-intel - Add transitional selections
crypto: aes - Boost priority of encryption modes further
lib/crypto: x86/aes: Set RNDKEY in aes_cbc_encrypt_aesni()
lib/crypto: riscv/aes-xts: Migrate optimized code into library
lib/crypto: riscv/aes-ctr: Migrate optimized code into library
lib/crypto: riscv/aes-cbc: Migrate optimized code into library
lib/crypto: riscv/aes-ecb: Migrate optimized code into library
lib/crypto: riscv/aes: Pass key struct to assembly code
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 FPU state updates from Borislav Petkov:
- Document and cleanup the signal frame handling with the goal to allow
process migration between CPUs with different FPU capabilities but
which use only a common set of FPU features, i.e., think
checkpoint/restore (Andrei Vagin)
- Consolidate how XSTATE features are checked by doing them once during
boot with the intent of removing cpu_has_xfeatures(). Latter will
happen in the next cycle (Eric Biggers)
- Other cleanups
* tag 'x86_fpu_for_v7.4_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/fpu: Remove additional remnants of math emulation
x86/fpu: Remove unnecessary checks for X86_FEATURE_FPU
selftests/x86: Add tests for signal frame FPU portability
x86/fpu: Pre-fault only required size of xstate buffer
x86/fpu: Fix potential underflow in xstate_calculate_size()
x86/fpu: Document reasoning of FX-only fallback
x86/fpu: Extract restore_from_ia32_fxstate() and clean up fpu__restore_sig()
x86/fpu: Clean up and rename variables in signal frame handling
x86/fpu: Document signal frame layout and portability
xor: Add AVX-512 optimized xor_gen()
xor: Remove redundant X86_FEATURE_OSXSAVE check
lib/crc: x86: Stop using cpu_has_xfeatures()
lib/crypto: x86: Stop using cpu_has_xfeatures()
crypto: x86 - Stop using cpu_has_xfeatures()
um: Check for missing AVX and AVX-512 xstate bits
x86/fpu: Check for missing AVX and AVX-512 xstate bits
|
|
The `<linux/unaligned.h>` header is included but never actually used.
The code handles memory accesses using memcpy() instead of the unaligned
helpers.
Remove this unused header to avoid false dependencies.
Signed-off-by: Mohamad Raizudeen <raizudeen.kerneldev@gmail.com>
Link: https://patch.msgid.link/20261016023558.8797-1-raizudeen.kerneldev@gmail.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
The MIPS ChaCha assembly, which is chacha_crypt_arch() itself, tests
the remaining length with signed branches, so for 2 GiB + 64 bytes or
more it takes the final partial block path too early and processes less
than requested. It is only used on 32-bit kernels, where size_t is 32
bits, so declaring the length as size_t does not avoid this. No
in-tree caller passes such lengths.
The partial block code also keeps the remaining length as a negative
value, so instead of changing the assembly, rename it to
chacha_crypt_mips() and add a C chacha_crypt_arch() that passes it at
most 1 GiB per call.
Tested in QEMU user mode: with a length of 0xfffff045 the assembly
processed about 4 KiB and advanced the block counter by 64; with the
same 1 GiB loop in C, a 1.25 GiB + 4101 byte buffer matches a C
reference. An LLM review of this code found the bug.
Assisted-by: LLM
Signed-off-by: Vladimir Kukharenko <vladimir@genproof.com>
Link: https://patch.msgid.link/20261014-libcrypto-chacha-32bit-2gib-v1-2-e0719ca0be89@genproof.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
chacha_doarm() compares the remaining length with signed conditions
(blt, movle/movgt, bgt), and DIV_ROUND_UP(bytes, CHACHA_BLOCK_SIZE) in
chacha_crypt_arch() wraps above 0xffffffc0, so lengths of 2 GiB or
more are mishandled. size_t is 32 bits on arm, so declaring the length
as size_t does not avoid this. On a Cortex-A7, an in-place
chacha20_crypt() of 2 GiB + 4101 bytes changed only one byte.
Use unsigned conditions (blo, movls/movhi, bhi) and
DIV_ROUND_UP_POW2().
This affects the scalar code, which is used when NEON isn't usable and
always on Cortex-A5 and Cortex-A7. No in-tree caller passes such
lengths. Tested in QEMU on a Cortex-A7 with CONFIG_VMSPLIT_1G. An LLM
review of this code found the bug.
Assisted-by: LLM
Signed-off-by: Vladimir Kukharenko <vladimir@genproof.com>
Link: https://patch.msgid.link/20261014-libcrypto-chacha-32bit-2gib-v1-1-e0719ca0be89@genproof.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Upgrade the ChaCha functions from 32-bit lengths to size_t.
'size_t' is the best practice for buffer lengths, and it's what the rest
of the crypto library uses.
Moreover, all the 64-bit ChaCha assembly functions actually already
treated the length as 64-bit, despite it being declared as 32-bit in C
on arm64, ppc64, and x86_64. For arm64 and x86_64, this caused
unspecified behavior due to the high 32 bits of the corresponding
register having an unspecified value in those architectures' ABIs.
However, in practice these bits are always zero and no issue occurred.
(ppc64 was fine too, since there the ABI zero-extends 32-bit arguments.)
These issues date to the original introduction of the ChaCha code.
However, as noted above, they don't have any real-world impact
currently; so this is an API robustness improvement only.
Reported-by: Vladimir Kukharenko <vladimir@genproof.com>
Closes: https://lore.kernel.org/r/20261012-libcrypto-4gib-lengths-v1-0-ea566c129a2e@genproof.com
Tested-by: Vladimir Kukharenko <vladimir@genproof.com>
Link: https://patch.msgid.link/20261014152604.323310-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Upgrade the Poly1305 functions from 32-bit lengths to size_t.
'size_t' is the best practice for buffer lengths, and it's what the rest
of the crypto library uses.
Moreover, all the 64-bit Poly1305 assembly functions actually already
treated the length as 64-bit, despite it being declared as 32-bit in C
on arm64, mips64, ppc64, and riscv64:
- On arm64, this caused unspecified behavior due to the high 32 bits
having an unspecified value in that architecture's ABI. However, in
practice these bits are always zero and no issue occurred. (ppc64 was
fine too, since there the ABI zero-extends 32-bit arguments.)
- On mips64 and riscv64, this caused incorrect behavior on lengths above
S32_MAX, due to the sign extension done by those architectures' ABIs.
Note that no current callers pass lengths over S32_MAX.
Also, the expression 'desc->buflen + nbytes >= POLY1305_BLOCK_SIZE' in
poly1305_update() could overflow if nbytes was in [UINT_MAX - 14,
UINT_MAX]. Note that again, no current callers pass lengths this large.
These issues date to the original introduction of the Poly1305 code.
However, as noted above, they don't have any real-world impact
currently; so this is an API robustness improvement only.
Reported-by: Vladimir Kukharenko <vladimir@genproof.com>
Closes: https://lore.kernel.org/r/20261012-libcrypto-4gib-lengths-v1-0-ea566c129a2e@genproof.com
Tested-by: Vladimir Kukharenko <vladimir@genproof.com>
Link: https://patch.msgid.link/20261014152604.323310-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
The SHA-512 assembly functions push callee-save GPRs to the stack
*before* they set up the frame pointer (%rbp). (They are leaf
functions, but they need to set up a frame pointer anyway because they
allocate a 32-byte-aligned buffer on the stack.)
As a result, during these functions, 8(%rbp) doesn't contain a return
address like it is supposed to.
This differs from the convention used by C code, and it interferes with
the frame pointer unwinder (CONFIG_UNWINDER_FRAME_POINTER=y).
Specifically, if a call trace is dumped in these functions, then all
entries are populated purely from the fallback scan of the entire stack:
RIP: 0010:sha512_transform_rorx+0x673/0xdf0
[...]
Call Trace:
<TASK>
? sha512_blocks_avx2+0x37/0x60
? __sha512_update+0x80/0x230
? sha512+0x31/0x190
? sha512+0x114/0x190
[...]
? do_syscall_64+0x6f/0x3e0
With this fix to follow the normal %rbp convention, the call trace is
generated reliably:
RIP: 0010:sha512_transform_rorx+0x90c/0xdf0
[...]
Call Trace:
<TASK>
sha512_blocks_avx2+0x37/0x60
__sha512_update+0x80/0x230
sha512+0x114/0x190
[...]
do_syscall_64+0x6f/0x3e0
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20261011134843.166585-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Some of the SHA-256 assembly functions push callee-save GPRs to the
stack *before* they set up the frame pointer (%rbp). (They are leaf
functions, but they need to set up a frame pointer anyway because they
allocate a 16- or 32-byte-aligned buffer on the stack.)
As a result, during these functions, 8(%rbp) doesn't contain a return
address like it is supposed to.
This differs from the convention used by C code, and it interferes with
the frame pointer unwinder (CONFIG_UNWINDER_FRAME_POINTER=y).
Specifically, if a call trace is dumped in these functions, then all
entries are populated purely from the fallback scan of the entire stack:
RIP: 0010:sha256_transform_rorx+0xd78/0x1120
[...]
Call Trace:
<TASK>
? sha256_blocks_avx2+0x37/0x70
? __sha256_update+0x77/0x220
? sha256+0x9d/0x110
[...]
? do_syscall_64+0x6f/0x3e0
With this fix to follow the normal %rbp convention, the call trace is
generated reliably:
RIP: 0010:sha256_transform_rorx+0xdae/0x1120
[...]
Call Trace:
<TASK>
sha256_blocks_avx2+0x37/0x70
__sha256_update+0x77/0x220
sha256+0x9d/0x110
[...]
do_syscall_64+0x6f/0x3e0
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20261011134843.166585-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Some of the SHA-1 assembly functions push callee-save GPRs to the stack
*before* they set up the frame pointer (%rbp). (They are leaf
functions, but they need to set up a frame pointer anyway because they
allocate a 16- or 32-byte-aligned buffer on the stack.)
As a result, during these functions, 8(%rbp) doesn't contain a return
address like it is supposed to.
This differs from the convention used by C code, and it interferes with
the frame pointer unwinder (CONFIG_UNWINDER_FRAME_POINTER=y).
Specifically, if a call trace is dumped in these functions, then all
entries are populated purely from the fallback scan of the entire stack:
RIP: 0010:sha1_transform_avx2+0x1921/0x1ea0
[...]
Call Trace:
<TASK>
? sha1_transform_avx+0x2a9/0xc90
? sha1_blocks_avx2+0x78/0x140
? kernel_fpu_end+0x2d/0x40
? sha1_blocks_avx2+0x52/0x140
? sha1_update+0x77/0x220
[...]
? do_syscall_64+0x6f/0x3e0
With this fix to follow the normal %rbp convention, the call trace is
generated reliably:
RIP: 0010:sha1_transform_avx2+0x66e/0x1ea0
[...]
Call Trace:
<TASK>
sha1_blocks_avx2+0x78/0x140
sha1_update+0x77/0x220
[...]
do_syscall_64+0x6f/0x3e0
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20261011134843.166585-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a test case which reproduces the bug fixed by commit 24f07e8c07a0
("lib/crypto: arm64: Fix lost Poly1305 carry when resuming NEON state").
It is written in a fairly general way, such that it should also be able
to reproduce similar bugs in other implementations.
Update the comment on test_poly1305_allones_keys_and_message() to more
accurately describe what it does and differentiate it from the new test.
Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
Link: https://patch.msgid.link/20261009125354.315476-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
When poly1305_blocks_neon() resumes from a base 2^26 accumulator and the
next update contains an odd number of full 16-byte blocks, it processes
one leading block through poly1305_mult() so that the remaining NEON
work has an even block count. That requires converting the accumulator
to the base 2^64 form used by poly1305_mult(). The final ADC of that
conversion stores the carry into d2, but the following accumulation and
poly1305_mult() use h2. If the conversion carries across bit 128, the
carry is dropped and the emitted tag is wrong.
This was introduced by Cryptogams commit 03dc4adf91c8
("arm/poly1305-armv*.pl: optimize branches."), which removed the
reduction that consumed d2 shortly before Linux imported the code.
OpenSSL's copy did not take that change.
The carry is possible because the NEON code stores the accumulator [h0,
h1, h2, h3, h4] in a lazily reduced, redundant base 2^26 form: its final
carry chain leaves h0, h2, h3 < 2^26, but h1 and h4 may slightly exceed
2^26 - 1. For example, the reachable state
h0 = 4
h1 = 2^26
h2 = 2^26 - 1
h3 = 2^26 - 1
h4 = 2^24 - 1
represents 4 + 2^26*2^26 + (2^26-1)*2^52 + (2^26-1)*2^78 +
(2^24-1)*2^104 = 2^128 + 4, despite the bit representing the 2^128 term
(bit 24 of h4) not being set; the extra comes from the h1 limb being
above the normal base 2^26 range. When this value is converted into the
non-redundant base 2^64 form, a carry into the new 2^128 bit (in bit 0
of h2) is necessary to preserve the value.
Given the above bounds, a carry into bit 128 happens exactly when
h1 >= 2^26, h2 = h3 = 2^26 - 1, and h4 mod 2^24 = 2^24 - 1.
Store the carry into h2, matching the other base 2^26 to base 2^64
conversions in poly1305_blocks() and poly1305_emit().
This bug causes an incorrect tag to be emitted. However, this is
unlikely to happen in practice where Poly1305 is used with a random key,
since the random key tends to distribute the accumulator values
throughout their valid range. There isn't an obvious way for an
attacker without knowledge of the r_key to cause [h0, h1, h2, h3, h4] to
land on the precise values where this bug occurs.
[EB: Rewrote the commit message to more clearly describe the bug and
its impact, and simplified explanation about why there's a carry bit.]
Fixes: f569ca164751 ("crypto: arm64/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON implementation")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://patch.msgid.link/20261009125354.315476-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Use memzero_explicit() instead of a plain struct assignment to guarantee
zeroization of the 'struct poly1305_desc_ctx'.
Note that dead store elimination was only theoretically possible with
link-time optimization here. But it's still worth fixing.
Fixes: a1d93064094c ("crypto: poly1305 - expose init/update/final library interface")
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-c3a6bd76-f6ab-4203-892a-cd06f7aa0c34
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20261006083432.16328-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
The `__chacha20poly1305_decrypt` function does not zeroize the chacha
state, unlike its encrypt counterpart. The regular
`chacha20poly1305_decrypt` function handles this by manually calling
chacha_zeroize_state(). However, `xchacha20poly1305_decrypt` returns
the result directly without clearing the state, leaving the derived
chacha20 subkey on the stack.
Because this function is EXPORT_SYMBOL()ed, this is an ABI robustness
improvement. To ensure consistent and safe cleanup for any future
callers, move the chacha_zeroize_state() call into
`__chacha20poly1305_decrypt()` itself, so the helper cleans up after its
own state just like `__chacha20poly1305_encrypt()` does.
Additionally, moved the src_len check to the callers so the state is not
initialized on invalid inputs, matching the logic in
chacha20poly1305_crypt_sg_inplace().
Suggested-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Mohamad Raizudeen <raizudeen.kerneldev@gmail.com>
Link: https://patch.msgid.link/20261005150324.5621-1-raizudeen.kerneldev@gmail.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Implement aes_xts_encrypt_arch() and aes_xts_decrypt_arch() with the AES
opcodes, for AES-128 and AES-256, two blocks per iteration. AES-192,
which IEEE 1619 does not specify, and data that is not 8-byte aligned
are left to the generic code.
The tweak is kept as little-endian words, so that multiplying it by x is
a 128-bit shift done with addcc and the VIS3 addxc. A little-endian
store and an ordinary load through a stack slot give it in the byte
order of the data, and the slot is cleared on return. The routines open
a register window, as %g4-%g6 belong to the kernel.
With that, xts-aes-lib no longer has to be kept off SPARC. Register it
there again, with the priority x86 and RISC-V use, so that it also
outranks an xts(ecb-aes-sparc64) instance (priority 300).
On a T7-1 (M7), MiB/s unless noted:
xts(ecb-aes-sparc64) xts-aes-lib
AF_ALG, 64 KiB, AES-128 415 1092
AF_ALG, 64 KiB, AES-256 390 936
dm-crypt on brd, AES-256:
sequential read 1399 2559
sequential write 1472 3480
4 KiB write latency, QD1 (us) 40.6 32.7
dm-crypt now matches aes-cbc-essiv on the same ramdisk.
On a T4-1, AF_ALG AES-256 goes from 238 to 544 MiB/s, and AES-128
from 252 to 616.
Tested with CONFIG_CRYPTO_SELFTESTS_FULL, and against OpenSSL over
random keys, tweaks and lengths, on both machines.
Link: https://github.com/sparclinux/issues/issues/106
Signed-off-by: Stian Halseth <stian@itx.no>
Link: https://patch.msgid.link/20261004192812.4145406-1-stian@itx.no
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
The addition of SipHash to lib/ predated the existence of the
lib/crypto/ directory. Since SipHash is a cryptographic algorithm, move
it from the top-level lib/ directory to lib/crypto/.
(This does mean HalfSipHash comes with it too, as it's in the same
files. But lib/crypto/ still seems like the proper place for both.)
Specifically:
- Move both siphash.c and siphash_kunit.c. Keep the header
<linux/siphash.h> as-is for now.
- Drop the dedicated MAINTAINERS entry for SipHash, as per the
suggestion from Jason
(https://lore.kernel.org/r/asB7QVFMlgKgbje2@zx2c4.com/). The C files
now just fold up to the "CRYPTO LIBRARY" entry. Add the header
explicitly, as it otherwise would be left unmaintained.
- Move the kconfig option that controls the KUnit test. Put
"CRYPTO_LIB" in its name and update the prompt and help text to make
it consistent with the other crypto library test options.
- Enable the KUnit test in lib/crypto/.kunitconfig.
- Link to siphash.rst from the appropriate place in libcrypto-hash.rst.
- Use obj-y instead of lib-y, for consistency with the fact that
currently lib/crypto/ doesn't use kbuild's support for library file
goals. We could keep it as lib-y, but virtually every kernel needs
SipHash support anyway (with vsprintf, printk, IPv4, IPv6, etc.
depending on it) so there is no practical difference in this case.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Jason A. Donenfeld <Jason@zx2c4.com>
Link: https://patch.msgid.link/20261003162827.155527-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In the AES-CTR and AES-XCTR code that handles a remainder of '4*VL < LEN
< 8*VL', first 4 full vectors are processed, leaving 0 < LEN < 4*VL.
After that, five different cases are handled depending on whether there
are 0, 1, 2, 3, or 4 full vectors remaining. But the case of 4 vectors
is never reached there, making the code handling it unused. Remove it.
With that done, the '.Lxor_tail_partial_vec_3\@' block would be reached
only via an unconditional jump. Therefore, relocate it to the only
place that jumps to it. This allows removing the jump at the end of the
'.Lxor_tail_partial_vec_2\@' block and making it fall through.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20261002034405.49886-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Make aes_cbc_encrypt_aesni() set the RNDKEY register alias explicitly,
rather than reusing the alias left by the previous function (which
worked by chance).
Also consistently comment the register aliases needed by _do_aes.
Link: https://patch.msgid.link/20260930170542.140403-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the riscv-optimized AES-XTS code via a
riscv-specific crypto_skcipher algorithm, just implement the AES-XTS
library functions. This is simpler, it makes the AES-XTS library
functions be riscv-optimized, and it also fixes the longstanding issue
where the riscv-optimized AES-XTS code was disabled by default. AES-XTS
support still remains available through crypto_skcipher via
crypto/aes.c, but individual architectures no longer need to handle it.
To match what the library expects, update the assembly functions to
operate on struct aes_key rather than struct crypto_aes_ctx, adjust the
argument order, and remove the redundant ciphertext stealing support
which is already implemented in a generic way in the library.
Bump up the priority of the corresponding library-based algorithm on
riscv now that it no longer has to be lower than arch/riscv/crypto/.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-21-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the riscv-optimized AES-CTR code via a
riscv-specific crypto_skcipher algorithm, just implement the AES-CTR
library functions. This is simpler, it makes the AES-CTR library
functions be riscv-optimized, and it also fixes the longstanding issue
where the riscv-optimized AES-CTR code was disabled by default. AES-CTR
support still remains available through crypto_skcipher via
crypto/aes.c, but individual architectures no longer need to handle it.
To match what the library expects, update the assembly functions to
operate on struct aes_enckey rather than struct crypto_aes_ctx, and
adjust the argument order.
Bump up the priority of the corresponding library-based algorithm on
riscv now that it no longer has to be lower than arch/riscv/crypto/.
Also re-enable the library-based "ccm(aes)" and "gcm(aes)".
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-20-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the riscv-optimized AES-CBC and AES-CBC-CTS code via
a riscv-specific crypto_skcipher algorithm, just implement the AES-CBC
and AES-CBC-CTS library functions. This is simpler, it makes the
AES-CBC and AES-CBC-CTS library functions be riscv-optimized, and it
also fixes the longstanding issue where the riscv-optimized AES-CBC and
AES-CBC-CTS code was disabled by default. AES-CBC and AES-CBC-CTS
support still remains available through crypto_skcipher via
crypto/aes.c, but individual architectures no longer need to handle it.
To match what the library expects, update the assembly functions to
operate on struct aes_enckey or struct aes_key rather than struct
crypto_aes_ctx, and adjust the argument order.
Remove the single-block special case from the assembly code, since it's
already handled by lib/crypto/aes.c before calling aes_cbc_cts_*_arch().
Bump up the priority of the corresponding library-based algorithms on
riscv now that they no longer have to be lower than arch/riscv/crypto/.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-19-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the riscv-optimized AES-ECB code via a
riscv-specific crypto_skcipher algorithm, just implement the AES-ECB
library functions. This is simpler, it makes the AES-ECB library
functions be riscv-optimized, and it also fixes the longstanding issue
where the riscv-optimized AES-ECB code was disabled by default. AES-ECB
support still remains available through crypto_skcipher via
crypto/aes.c, but individual architectures no longer need to handle it.
To match what the library expects, update the assembly functions to
operate on struct aes_enckey or struct aes_key rather than struct
crypto_aes_ctx, and adjust the argument order.
Bump up the priority of the corresponding library-based algorithm on
riscv now that it no longer has to be lower than arch/riscv/crypto/.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-18-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Make the assembly code take the AES key struct directly, rather than the
round keys pointer and key length separately. Make the aes_begin macro
assume this convention, and remove support for the legacy
'struct crypto_aes_ctx' from it since that isn't used here.
This aligns with the convention that is being used (and will continue to
be used) for the AES modes, it makes the C glue code slightly simpler,
and it avoids the unnecessary shuffling around of arguments.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-17-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Copy arch/riscv/crypto/aes-macros.S to lib/crypto/riscv/aes-macros.S and
make lib/crypto/riscv/aes-riscv64-zvkned.S include the latter copy.
This makes it possible to change these macros without interfering with
the remaining code in arch/riscv/crypto/.
Of course, the copy in arch/riscv/crypto/ will be removed once the rest
of the AES code there is migrated to the library.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-16-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Migrate aes-xts-avx-x86_64.S into lib/crypto/, wiring it up to the XTS
library functions instead of the crypto_skcipher API. It still remains
available through crypto_skcipher via crypto/aes.c.
Some adjustments to the assembly code were needed:
- Take 'struct aes_key' instead of 'struct crypto_aes_ctx'.
- Remove the ciphertext stealing support from the assembly code, as the
library implements it in a generic way instead. (This does slightly
reduce performance when the length isn't a multiple of 16 bytes;
however, that case seems to never be reached in practice in the
kernel. So it makes sense to not extensively optimize for it yet.)
- Change 'int len' to 'long nblocks' for compatibility with the
library's use of size_t lengths.
- Remove the CFI stubs, as the functions are now called directly.
- Adjust the argument order to match the caller. Not strictly required,
but it's easiest to handle this now when changing the function
prototypes anyway and adding the new glue code.
This makes the remaining code in aesni-intel_asm.S (which just handled
key expansion) unused, so remove that too.
Don't remove the selection of CRYPTO_SKCIPHER from CRYPTO_AES_NI_INTEL
yet, since it's still needed for the "skcipher_walk" functions.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-15-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Migrate aes-ctr-avx-x86_64.S into lib/crypto/, wiring it up to the CTR
and XCTR library functions instead of the crypto_skcipher API. It still
remains available through crypto_skcipher via crypto/aes.c.
Some slight adjustments to the assembly code were needed:
- Take 'struct aes_enckey' instead of 'struct crypto_aes_ctx'.
- Upgrade the length argument from 32-bit to 64-bit so that it's
compatible with the library's use of size_t (at least assuming no
lengths over S64_MAX, which seems quite safe to assume...)
- Remove the CFI stubs, as the functions are now called directly.
- Adjust the argument order to match the caller. Not strictly required,
but it's easiest to handle this now when changing the function
prototypes anyway and adding the new glue code.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-14-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Optimize the crypto library's AES-XTS support with AES-NI, bringing its
performance on par with the "xts-aes-aesni" skcipher algorithm it will
supersede.
The new assembly code is written from scratch to fit well into the
crypto library and to be more consistent with aes-xts-avx-x86_64.S than
the code in arch/x86/crypto/aesni-intel_asm.S that it will supersede.
At a high level it is quite similar though, including doing 4 blocks per
iteration and supporting 32-bit mode for parity with the old code.
The new assembly code also fixes the flaw the old code had where the
encrypted tweaks were spilled to the destination buffer, rather than
kept entirely in registers (64-bit mode) or spilled to the stack (32-bit
mode). The encrypted tweaks are secret values that should not be
exposed to any code that doesn't have access to the key itself.
Note: the priority of xts-aes-lib is left unchanged at 110 temporarily.
It will be increased when the AVX-optimized AES-XTS code is migrated
too. Most systems use the AVX-optimized code; this commit just deals
with support for older CPUs that have AES-NI but not AVX (and 32-bit).
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-8-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Optimize the crypto library's AES-CTR support with AES-NI, bringing its
performance on par with the "ctr-aes-aesni" skcipher algorithm it will
supersede.
The new assembly function is written from scratch to fit well into the
crypto library and to be more consistent with aes-ctr-avx-x86_64.S than
the code in arch/x86/crypto/aesni-intel_asm.S that it will supersede.
That includes using the "ctr64" convention, where the assembly code is
simplified by making the C code handle incrementing the high 64 bits of
the counter. Unlike the ECB, CBC, and XTS code, 32-bit support is *not*
included for this one, as the existing CTR code didn't have it.
Note: the priority of ctr-aes-lib is left unchanged at 110 temporarily.
It will be increased when the AVX-optimized code is migrated too.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-7-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Optimize the crypto library's AES-CBC and AES-CBC-CTS support with
AES-NI, bringing their performance on par with the "cbc-aes-aesni" and
"cts-cbc-aes-aesni" skcipher algorithms they will supersede.
The new assembly functions are written from scratch to fit well into the
crypto library. However, they are functionally very similar to the
functions in arch/x86/crypto/aesni-intel_asm.S that they will supersede
and are intended to provide parity with those -- including supporting
32-bit mode, having the inner loops do 4 AES blocks per iteration, etc.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Optimize the crypto library's AES-ECB support with AES-NI, bringing its
performance on par with the "ecb-aes-aesni" skcipher algorithm it will
supersede.
The new assembly functions are written from scratch to fit well into the
crypto library. However, they are functionally very similar to the
functions in arch/x86/crypto/aesni-intel_asm.S that they will supersede
and are intended to provide parity with those -- including supporting
32-bit mode, having the inner loops do 4 AES blocks per iteration, etc.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Various miscellaneous updates to the assembly code in preparation for
adding implementations of AES modes to the same file:
- Define macros for the function argument registers, function prologues,
and function epilogues to centralize some of the handling of 32-bit vs
64-bit.
- Refactor the actual AES encryption and AES decryption logic into
macros _do_aes and _do_aes_ecb so that some of the modes can reuse it.
- Rename mask to expandkey_mask to differentiate it from the bswap_mask
that will be added.
- Update the prototypes of aes_encrypt_aesni() and aes_decrypt_aesni()
to be dst, src, key so that they will match the mode functions.
Note that this means passing a pointer to the key struct instead of a
(nrounds, rndkeys) pair, similar to what arch/x86/crypto/aes*.S do.
Although this makes the assembly code depend on the format of the key
struct, having one fewer argument makes it easier to accommodate
32-bit mode, and the C glue code becomes slightly simpler.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Update the calling convention for aes_xctr_arch() to pass the counter by
value, then do the increment in generic code. This aligns better with
the x86_64 and arm64 assembly code for XCTR, which takes the counter by
value and thus has to be paired with an increment in C code anyway.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code needs to zeroize their local hmac_sha1_key
or hmac_sha1_ctx structures after use to avoid leaking sensitive material.
Provide hmac_sha1_zeroize_key() and hmac_sha1_zeroize_ctx() helper
functions that e.g. can be used with __cleanup() to automatically zeroize
the structures when they go out of scope.
While we're at it, replace the related memzero_explicit() call in
lib/crypto/sha1.c with a call to the new helper function.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-9-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code needs to zeroize their local blake2b_ctx
or blake2s_ctx structures after use to avoid leaking sensitive material.
Provide blake2b_zeroize_ctx() and blake2s_zeroize_ctx() helper functions
that e.g. can be used with __cleanup() to automatically zeroize the
structures when they go out of scope.
While we're at it, replace the related memzero_explicit() calls in
lib/crypto/blake2*.c with calls to the new helper functions.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-8-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code functions need to zeroize their local
sm3_ctx structure after use to avoid leaking sensitive material. Provide
a sm3_zeroize_ctx() helper function that e.g. can be used with __cleanup()
to automatically zeroize the structure when it goes out of scope.
While we're at it, replace the related memzero_explicit() call in
lib/crypto/sm3.c with a call to the new helper function.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-7-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code functions need to zeroize their local
hmac_md5_key or hmac_md5_ctx structures after use to avoid leaking
sensitive material on the stack.
Provide hmac_md5_zeroize_key() and hmac_md5_zeroize_ctx() helper
functions that e.g. can be used with __cleanup() to automatically
zeroize the structure when it goes out of scope.
While we're at it, replace the related memzero_explicit() call in
lib/crypto/md5.c with a call to the new helper function.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-6-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code needs to zeroize their local aes_ccm_key
or aes_ccm_ctx structures after use to avoid leaking sensitive material.
Provide aes_ccm_zeroize_key() and aes_ccm_zeroize_ctx() helper functions
that e.g. can be used with __cleanup() to automatically zeroize the
structures when they go out of scope.
While we're at it, replace the related memzero_explicit() calls in
lib/crypto/aes.c with calls to the new helper functions.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-5-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code needs to zeroize their local aes_gcm_key
or aes_gcm_ctx structures after use to avoid leaking sensitive material.
Provide aes_gcm_zeroize_key() and aes_gcm_zeroize_ctx() helper functions
that e.g. can be used with __cleanup() to automatically zeroize the
structures when they go out of scope.
While we're at it, replace the related memzero_explicit() calls in
lib/crypto/aes.c with calls to the new helper functions.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-4-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code functions need to zeroize their local
aes_xts_key structures after use to avoid leaking sensitive material.
Provide an aes_xts_zeroize_key() helper function that e.g. can be
used with __cleanup() to automatically zeroize the struct when it
goes out of scope.
While we're at it, replace the related memzero_explicit() call in
lib/crypto/aes.c with a call to the new helper function.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-3-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Some crypto functions need to zeroize their local aes_key or aes_enckey
structures after use to avoid leaking sensitive material on the stack.
Provide aes_zeroize_key() and aes_zeroize_enckey() helper functions that
can be used with __cleanup() to automatically zeroize the structs when
they go out of scope.
While we're at it, replace the memzero_explicit() calls in lib/crypto/aes.c
with the new helper functions.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-2-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Checking both boot_cpu_has() and cpu_has_xfeatures() has never really been
needed in practice, and it's never been universally done (e.g., lib/raid/
omits cpu_has_xfeatures()). Nevertheless, both x86 and UML now explicitly
clear the AVX and AVX-512 flags if their xfeatures are missing, which should
remove any remaining doubts.
Thus, remove all the calls to cpu_has_xfeatures().
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260901145740.42337-5-ebiggers@kernel.org
|
|
By using __cleanup(aes_cmac_zeroize_key) for clearing the key data, we
can save one line of code here.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260807125845.1477067-6-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a KUnit test suite for the AES-GCM library API.
It consists of:
- All the shared test cases from aead-test-template.h. These include
extensive consistency tests, a "Monte-Carlo test", and a benchmark.
- Tests against hardcoded AES-GCM test vectors from external sources.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a KUnit test suite for the AES-CCM library API.
It consists of:
- All the shared test cases from aead-test-template.h. These include
extensive consistency tests, a "Monte-Carlo test", and a benchmark.
- Tests against hardcoded AES-CCM test vectors from external sources.
- Tests for CCM-specific message length validation.
To generate the expected aes_ccm_monte_carlo_checksum[] value, add a
script gen-aead-testvecs.py which computes it using python-cryptography.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add aead-test-template.h which contains shared test logic for AEAD
algorithms. It's similar to the existing hash-test-template.h, which
contains shared test logic for hash algorithms.
The initial use cases for this will be AES-CCM and AES-GCM.
It may be used by the existing ChaCha20Poly1305 test suite in the future
as well, though the ChaCha20Poly1305 API currently lacks some of the
functionality that the "template" expects to be present.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Make the test cases more self-contained by replacing the test suite
scoped guarded buffer ('test_buf') with buffers allocated by each test
case using the helper functions in test-utils.h.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Move hash-test-template's test RNG code and the buffer allocation code
from various test suites into a header test-utils.h. This allows them
to be shared by other test suites.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Upcoming changes will wire up architecture-optimized implementations of
GCM and CCM. FIPS labs can consider such designs to meet the threshold
for separate self-tests to be needed.
Therefore, add FIPS self-tests for encryption and decryption in these
modes.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802222408.91757-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Upcoming changes will wire up architecture-optimized implementations of
ECB, CBC, CBC-CTS, CTR, and XTS. FIPS labs can consider such designs to
meet the threshold for separate self-tests to be needed.
The inverse direction of the block cipher also needs to be exercised,
which the existing CMAC self-test doesn't do.
Therefore, add FIPS self-tests for encryption and decryption in these
modes as well as the "bare" AES.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802222408.91757-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In preparation for adding FIPS self-tests for AES encryption modes,
split fips.h into separate files for the AES and SHA test vectors.
They are still generated by the same script, but this keeps things a bit
more organized.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802222408.91757-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
