| Age | Commit message (Collapse) | Author |
|---|
|
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
# Conflicts:
# Documentation/scheduler/index.rst
# arch/arm64/configs/defconfig
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
|
|
Add a test case which reproduces the bug fixed by commit 24f07e8c07a0
("lib/crypto: arm64: Fix lost Poly1305 carry when resuming NEON state").
It is written in a fairly general way, such that it should also be able
to reproduce similar bugs in other implementations.
Update the comment on test_poly1305_allones_keys_and_message() to more
accurately describe what it does and differentiate it from the new test.
Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
Link: https://patch.msgid.link/20261009125354.315476-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
When poly1305_blocks_neon() resumes from a base 2^26 accumulator and the
next update contains an odd number of full 16-byte blocks, it processes
one leading block through poly1305_mult() so that the remaining NEON
work has an even block count. That requires converting the accumulator
to the base 2^64 form used by poly1305_mult(). The final ADC of that
conversion stores the carry into d2, but the following accumulation and
poly1305_mult() use h2. If the conversion carries across bit 128, the
carry is dropped and the emitted tag is wrong.
This was introduced by Cryptogams commit 03dc4adf91c8
("arm/poly1305-armv*.pl: optimize branches."), which removed the
reduction that consumed d2 shortly before Linux imported the code.
OpenSSL's copy did not take that change.
The carry is possible because the NEON code stores the accumulator [h0,
h1, h2, h3, h4] in a lazily reduced, redundant base 2^26 form: its final
carry chain leaves h0, h2, h3 < 2^26, but h1 and h4 may slightly exceed
2^26 - 1. For example, the reachable state
h0 = 4
h1 = 2^26
h2 = 2^26 - 1
h3 = 2^26 - 1
h4 = 2^24 - 1
represents 4 + 2^26*2^26 + (2^26-1)*2^52 + (2^26-1)*2^78 +
(2^24-1)*2^104 = 2^128 + 4, despite the bit representing the 2^128 term
(bit 24 of h4) not being set; the extra comes from the h1 limb being
above the normal base 2^26 range. When this value is converted into the
non-redundant base 2^64 form, a carry into the new 2^128 bit (in bit 0
of h2) is necessary to preserve the value.
Given the above bounds, a carry into bit 128 happens exactly when
h1 >= 2^26, h2 = h3 = 2^26 - 1, and h4 mod 2^24 = 2^24 - 1.
Store the carry into h2, matching the other base 2^26 to base 2^64
conversions in poly1305_blocks() and poly1305_emit().
This bug causes an incorrect tag to be emitted. However, this is
unlikely to happen in practice where Poly1305 is used with a random key,
since the random key tends to distribute the accumulator values
throughout their valid range. There isn't an obvious way for an
attacker without knowledge of the r_key to cause [h0, h1, h2, h3, h4] to
land on the precise values where this bug occurs.
[EB: Rewrote the commit message to more clearly describe the bug and
its impact, and simplified explanation about why there's a carry bit.]
Fixes: f569ca164751 ("crypto: arm64/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON implementation")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://patch.msgid.link/20261009125354.315476-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Use memzero_explicit() instead of a plain struct assignment to guarantee
zeroization of the 'struct poly1305_desc_ctx'.
Note that dead store elimination was only theoretically possible with
link-time optimization here. But it's still worth fixing.
Fixes: a1d93064094c ("crypto: poly1305 - expose init/update/final library interface")
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-c3a6bd76-f6ab-4203-892a-cd06f7aa0c34
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20261006083432.16328-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
The `__chacha20poly1305_decrypt` function does not zeroize the chacha
state, unlike its encrypt counterpart. The regular
`chacha20poly1305_decrypt` function handles this by manually calling
chacha_zeroize_state(). However, `xchacha20poly1305_decrypt` returns
the result directly without clearing the state, leaving the derived
chacha20 subkey on the stack.
Because this function is EXPORT_SYMBOL()ed, this is an ABI robustness
improvement. To ensure consistent and safe cleanup for any future
callers, move the chacha_zeroize_state() call into
`__chacha20poly1305_decrypt()` itself, so the helper cleans up after its
own state just like `__chacha20poly1305_encrypt()` does.
Additionally, moved the src_len check to the callers so the state is not
initialized on invalid inputs, matching the logic in
chacha20poly1305_crypt_sg_inplace().
Suggested-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Mohamad Raizudeen <raizudeen.kerneldev@gmail.com>
Link: https://patch.msgid.link/20261005150324.5621-1-raizudeen.kerneldev@gmail.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Implement aes_xts_encrypt_arch() and aes_xts_decrypt_arch() with the AES
opcodes, for AES-128 and AES-256, two blocks per iteration. AES-192,
which IEEE 1619 does not specify, and data that is not 8-byte aligned
are left to the generic code.
The tweak is kept as little-endian words, so that multiplying it by x is
a 128-bit shift done with addcc and the VIS3 addxc. A little-endian
store and an ordinary load through a stack slot give it in the byte
order of the data, and the slot is cleared on return. The routines open
a register window, as %g4-%g6 belong to the kernel.
With that, xts-aes-lib no longer has to be kept off SPARC. Register it
there again, with the priority x86 and RISC-V use, so that it also
outranks an xts(ecb-aes-sparc64) instance (priority 300).
On a T7-1 (M7), MiB/s unless noted:
xts(ecb-aes-sparc64) xts-aes-lib
AF_ALG, 64 KiB, AES-128 415 1092
AF_ALG, 64 KiB, AES-256 390 936
dm-crypt on brd, AES-256:
sequential read 1399 2559
sequential write 1472 3480
4 KiB write latency, QD1 (us) 40.6 32.7
dm-crypt now matches aes-cbc-essiv on the same ramdisk.
On a T4-1, AF_ALG AES-256 goes from 238 to 544 MiB/s, and AES-128
from 252 to 616.
Tested with CONFIG_CRYPTO_SELFTESTS_FULL, and against OpenSSL over
random keys, tweaks and lengths, on both machines.
Link: https://github.com/sparclinux/issues/issues/106
Signed-off-by: Stian Halseth <stian@itx.no>
Link: https://patch.msgid.link/20261004192812.4145406-1-stian@itx.no
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
The addition of SipHash to lib/ predated the existence of the
lib/crypto/ directory. Since SipHash is a cryptographic algorithm, move
it from the top-level lib/ directory to lib/crypto/.
(This does mean HalfSipHash comes with it too, as it's in the same
files. But lib/crypto/ still seems like the proper place for both.)
Specifically:
- Move both siphash.c and siphash_kunit.c. Keep the header
<linux/siphash.h> as-is for now.
- Drop the dedicated MAINTAINERS entry for SipHash, as per the
suggestion from Jason
(https://lore.kernel.org/r/asB7QVFMlgKgbje2@zx2c4.com/). The C files
now just fold up to the "CRYPTO LIBRARY" entry. Add the header
explicitly, as it otherwise would be left unmaintained.
- Move the kconfig option that controls the KUnit test. Put
"CRYPTO_LIB" in its name and update the prompt and help text to make
it consistent with the other crypto library test options.
- Enable the KUnit test in lib/crypto/.kunitconfig.
- Link to siphash.rst from the appropriate place in libcrypto-hash.rst.
- Use obj-y instead of lib-y, for consistency with the fact that
currently lib/crypto/ doesn't use kbuild's support for library file
goals. We could keep it as lib-y, but virtually every kernel needs
SipHash support anyway (with vsprintf, printk, IPv4, IPv6, etc.
depending on it) so there is no practical difference in this case.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Jason A. Donenfeld <Jason@zx2c4.com>
Link: https://patch.msgid.link/20261003162827.155527-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In the AES-CTR and AES-XCTR code that handles a remainder of '4*VL < LEN
< 8*VL', first 4 full vectors are processed, leaving 0 < LEN < 4*VL.
After that, five different cases are handled depending on whether there
are 0, 1, 2, 3, or 4 full vectors remaining. But the case of 4 vectors
is never reached there, making the code handling it unused. Remove it.
With that done, the '.Lxor_tail_partial_vec_3\@' block would be reached
only via an unconditional jump. Therefore, relocate it to the only
place that jumps to it. This allows removing the jump at the end of the
'.Lxor_tail_partial_vec_2\@' block and making it fall through.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20261002034405.49886-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Make aes_cbc_encrypt_aesni() set the RNDKEY register alias explicitly,
rather than reusing the alias left by the previous function (which
worked by chance).
Also consistently comment the register aliases needed by _do_aes.
Link: https://patch.msgid.link/20260930170542.140403-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the riscv-optimized AES-XTS code via a
riscv-specific crypto_skcipher algorithm, just implement the AES-XTS
library functions. This is simpler, it makes the AES-XTS library
functions be riscv-optimized, and it also fixes the longstanding issue
where the riscv-optimized AES-XTS code was disabled by default. AES-XTS
support still remains available through crypto_skcipher via
crypto/aes.c, but individual architectures no longer need to handle it.
To match what the library expects, update the assembly functions to
operate on struct aes_key rather than struct crypto_aes_ctx, adjust the
argument order, and remove the redundant ciphertext stealing support
which is already implemented in a generic way in the library.
Bump up the priority of the corresponding library-based algorithm on
riscv now that it no longer has to be lower than arch/riscv/crypto/.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-21-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the riscv-optimized AES-CTR code via a
riscv-specific crypto_skcipher algorithm, just implement the AES-CTR
library functions. This is simpler, it makes the AES-CTR library
functions be riscv-optimized, and it also fixes the longstanding issue
where the riscv-optimized AES-CTR code was disabled by default. AES-CTR
support still remains available through crypto_skcipher via
crypto/aes.c, but individual architectures no longer need to handle it.
To match what the library expects, update the assembly functions to
operate on struct aes_enckey rather than struct crypto_aes_ctx, and
adjust the argument order.
Bump up the priority of the corresponding library-based algorithm on
riscv now that it no longer has to be lower than arch/riscv/crypto/.
Also re-enable the library-based "ccm(aes)" and "gcm(aes)".
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-20-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the riscv-optimized AES-CBC and AES-CBC-CTS code via
a riscv-specific crypto_skcipher algorithm, just implement the AES-CBC
and AES-CBC-CTS library functions. This is simpler, it makes the
AES-CBC and AES-CBC-CTS library functions be riscv-optimized, and it
also fixes the longstanding issue where the riscv-optimized AES-CBC and
AES-CBC-CTS code was disabled by default. AES-CBC and AES-CBC-CTS
support still remains available through crypto_skcipher via
crypto/aes.c, but individual architectures no longer need to handle it.
To match what the library expects, update the assembly functions to
operate on struct aes_enckey or struct aes_key rather than struct
crypto_aes_ctx, and adjust the argument order.
Remove the single-block special case from the assembly code, since it's
already handled by lib/crypto/aes.c before calling aes_cbc_cts_*_arch().
Bump up the priority of the corresponding library-based algorithms on
riscv now that they no longer have to be lower than arch/riscv/crypto/.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-19-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the riscv-optimized AES-ECB code via a
riscv-specific crypto_skcipher algorithm, just implement the AES-ECB
library functions. This is simpler, it makes the AES-ECB library
functions be riscv-optimized, and it also fixes the longstanding issue
where the riscv-optimized AES-ECB code was disabled by default. AES-ECB
support still remains available through crypto_skcipher via
crypto/aes.c, but individual architectures no longer need to handle it.
To match what the library expects, update the assembly functions to
operate on struct aes_enckey or struct aes_key rather than struct
crypto_aes_ctx, and adjust the argument order.
Bump up the priority of the corresponding library-based algorithm on
riscv now that it no longer has to be lower than arch/riscv/crypto/.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-18-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Make the assembly code take the AES key struct directly, rather than the
round keys pointer and key length separately. Make the aes_begin macro
assume this convention, and remove support for the legacy
'struct crypto_aes_ctx' from it since that isn't used here.
This aligns with the convention that is being used (and will continue to
be used) for the AES modes, it makes the C glue code slightly simpler,
and it avoids the unnecessary shuffling around of arguments.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-17-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Copy arch/riscv/crypto/aes-macros.S to lib/crypto/riscv/aes-macros.S and
make lib/crypto/riscv/aes-riscv64-zvkned.S include the latter copy.
This makes it possible to change these macros without interfering with
the remaining code in arch/riscv/crypto/.
Of course, the copy in arch/riscv/crypto/ will be removed once the rest
of the AES code there is migrated to the library.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-16-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Migrate aes-xts-avx-x86_64.S into lib/crypto/, wiring it up to the XTS
library functions instead of the crypto_skcipher API. It still remains
available through crypto_skcipher via crypto/aes.c.
Some adjustments to the assembly code were needed:
- Take 'struct aes_key' instead of 'struct crypto_aes_ctx'.
- Remove the ciphertext stealing support from the assembly code, as the
library implements it in a generic way instead. (This does slightly
reduce performance when the length isn't a multiple of 16 bytes;
however, that case seems to never be reached in practice in the
kernel. So it makes sense to not extensively optimize for it yet.)
- Change 'int len' to 'long nblocks' for compatibility with the
library's use of size_t lengths.
- Remove the CFI stubs, as the functions are now called directly.
- Adjust the argument order to match the caller. Not strictly required,
but it's easiest to handle this now when changing the function
prototypes anyway and adding the new glue code.
This makes the remaining code in aesni-intel_asm.S (which just handled
key expansion) unused, so remove that too.
Don't remove the selection of CRYPTO_SKCIPHER from CRYPTO_AES_NI_INTEL
yet, since it's still needed for the "skcipher_walk" functions.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-15-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Migrate aes-ctr-avx-x86_64.S into lib/crypto/, wiring it up to the CTR
and XCTR library functions instead of the crypto_skcipher API. It still
remains available through crypto_skcipher via crypto/aes.c.
Some slight adjustments to the assembly code were needed:
- Take 'struct aes_enckey' instead of 'struct crypto_aes_ctx'.
- Upgrade the length argument from 32-bit to 64-bit so that it's
compatible with the library's use of size_t (at least assuming no
lengths over S64_MAX, which seems quite safe to assume...)
- Remove the CFI stubs, as the functions are now called directly.
- Adjust the argument order to match the caller. Not strictly required,
but it's easiest to handle this now when changing the function
prototypes anyway and adding the new glue code.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-14-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Optimize the crypto library's AES-XTS support with AES-NI, bringing its
performance on par with the "xts-aes-aesni" skcipher algorithm it will
supersede.
The new assembly code is written from scratch to fit well into the
crypto library and to be more consistent with aes-xts-avx-x86_64.S than
the code in arch/x86/crypto/aesni-intel_asm.S that it will supersede.
At a high level it is quite similar though, including doing 4 blocks per
iteration and supporting 32-bit mode for parity with the old code.
The new assembly code also fixes the flaw the old code had where the
encrypted tweaks were spilled to the destination buffer, rather than
kept entirely in registers (64-bit mode) or spilled to the stack (32-bit
mode). The encrypted tweaks are secret values that should not be
exposed to any code that doesn't have access to the key itself.
Note: the priority of xts-aes-lib is left unchanged at 110 temporarily.
It will be increased when the AVX-optimized AES-XTS code is migrated
too. Most systems use the AVX-optimized code; this commit just deals
with support for older CPUs that have AES-NI but not AVX (and 32-bit).
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-8-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Optimize the crypto library's AES-CTR support with AES-NI, bringing its
performance on par with the "ctr-aes-aesni" skcipher algorithm it will
supersede.
The new assembly function is written from scratch to fit well into the
crypto library and to be more consistent with aes-ctr-avx-x86_64.S than
the code in arch/x86/crypto/aesni-intel_asm.S that it will supersede.
That includes using the "ctr64" convention, where the assembly code is
simplified by making the C code handle incrementing the high 64 bits of
the counter. Unlike the ECB, CBC, and XTS code, 32-bit support is *not*
included for this one, as the existing CTR code didn't have it.
Note: the priority of ctr-aes-lib is left unchanged at 110 temporarily.
It will be increased when the AVX-optimized code is migrated too.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-7-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Optimize the crypto library's AES-CBC and AES-CBC-CTS support with
AES-NI, bringing their performance on par with the "cbc-aes-aesni" and
"cts-cbc-aes-aesni" skcipher algorithms they will supersede.
The new assembly functions are written from scratch to fit well into the
crypto library. However, they are functionally very similar to the
functions in arch/x86/crypto/aesni-intel_asm.S that they will supersede
and are intended to provide parity with those -- including supporting
32-bit mode, having the inner loops do 4 AES blocks per iteration, etc.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Optimize the crypto library's AES-ECB support with AES-NI, bringing its
performance on par with the "ecb-aes-aesni" skcipher algorithm it will
supersede.
The new assembly functions are written from scratch to fit well into the
crypto library. However, they are functionally very similar to the
functions in arch/x86/crypto/aesni-intel_asm.S that they will supersede
and are intended to provide parity with those -- including supporting
32-bit mode, having the inner loops do 4 AES blocks per iteration, etc.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Various miscellaneous updates to the assembly code in preparation for
adding implementations of AES modes to the same file:
- Define macros for the function argument registers, function prologues,
and function epilogues to centralize some of the handling of 32-bit vs
64-bit.
- Refactor the actual AES encryption and AES decryption logic into
macros _do_aes and _do_aes_ecb so that some of the modes can reuse it.
- Rename mask to expandkey_mask to differentiate it from the bswap_mask
that will be added.
- Update the prototypes of aes_encrypt_aesni() and aes_decrypt_aesni()
to be dst, src, key so that they will match the mode functions.
Note that this means passing a pointer to the key struct instead of a
(nrounds, rndkeys) pair, similar to what arch/x86/crypto/aes*.S do.
Although this makes the assembly code depend on the format of the key
struct, having one fewer argument makes it easier to accommodate
32-bit mode, and the C glue code becomes slightly simpler.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Update the calling convention for aes_xctr_arch() to pass the counter by
value, then do the increment in generic code. This aligns better with
the x86_64 and arm64 assembly code for XCTR, which takes the counter by
value and thus has to be paired with an increment in C code anyway.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260927224418.109759-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code needs to zeroize their local hmac_sha1_key
or hmac_sha1_ctx structures after use to avoid leaking sensitive material.
Provide hmac_sha1_zeroize_key() and hmac_sha1_zeroize_ctx() helper
functions that e.g. can be used with __cleanup() to automatically zeroize
the structures when they go out of scope.
While we're at it, replace the related memzero_explicit() call in
lib/crypto/sha1.c with a call to the new helper function.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-9-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code needs to zeroize their local blake2b_ctx
or blake2s_ctx structures after use to avoid leaking sensitive material.
Provide blake2b_zeroize_ctx() and blake2s_zeroize_ctx() helper functions
that e.g. can be used with __cleanup() to automatically zeroize the
structures when they go out of scope.
While we're at it, replace the related memzero_explicit() calls in
lib/crypto/blake2*.c with calls to the new helper functions.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-8-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code functions need to zeroize their local
sm3_ctx structure after use to avoid leaking sensitive material. Provide
a sm3_zeroize_ctx() helper function that e.g. can be used with __cleanup()
to automatically zeroize the structure when it goes out of scope.
While we're at it, replace the related memzero_explicit() call in
lib/crypto/sm3.c with a call to the new helper function.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-7-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code functions need to zeroize their local
hmac_md5_key or hmac_md5_ctx structures after use to avoid leaking
sensitive material on the stack.
Provide hmac_md5_zeroize_key() and hmac_md5_zeroize_ctx() helper
functions that e.g. can be used with __cleanup() to automatically
zeroize the structure when it goes out of scope.
While we're at it, replace the related memzero_explicit() call in
lib/crypto/md5.c with a call to the new helper function.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-6-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code needs to zeroize their local aes_ccm_key
or aes_ccm_ctx structures after use to avoid leaking sensitive material.
Provide aes_ccm_zeroize_key() and aes_ccm_zeroize_ctx() helper functions
that e.g. can be used with __cleanup() to automatically zeroize the
structures when they go out of scope.
While we're at it, replace the related memzero_explicit() calls in
lib/crypto/aes.c with calls to the new helper functions.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-5-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code needs to zeroize their local aes_gcm_key
or aes_gcm_ctx structures after use to avoid leaking sensitive material.
Provide aes_gcm_zeroize_key() and aes_gcm_zeroize_ctx() helper functions
that e.g. can be used with __cleanup() to automatically zeroize the
structures when they go out of scope.
While we're at it, replace the related memzero_explicit() calls in
lib/crypto/aes.c with calls to the new helper functions.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-4-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In certain cases crypto code functions need to zeroize their local
aes_xts_key structures after use to avoid leaking sensitive material.
Provide an aes_xts_zeroize_key() helper function that e.g. can be
used with __cleanup() to automatically zeroize the struct when it
goes out of scope.
While we're at it, replace the related memzero_explicit() call in
lib/crypto/aes.c with a call to the new helper function.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-3-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Some crypto functions need to zeroize their local aes_key or aes_enckey
structures after use to avoid leaking sensitive material on the stack.
Provide aes_zeroize_key() and aes_zeroize_enckey() helper functions that
can be used with __cleanup() to automatically zeroize the structs when
they go out of scope.
While we're at it, replace the memzero_explicit() calls in lib/crypto/aes.c
with the new helper functions.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260916095022.604354-2-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Checking both boot_cpu_has() and cpu_has_xfeatures() has never really been
needed in practice, and it's never been universally done (e.g., lib/raid/
omits cpu_has_xfeatures()). Nevertheless, both x86 and UML now explicitly
clear the AVX and AVX-512 flags if their xfeatures are missing, which should
remove any remaining doubts.
Thus, remove all the calls to cpu_has_xfeatures().
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Acked-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260901145740.42337-5-ebiggers@kernel.org
|
|
By using __cleanup(aes_cmac_zeroize_key) for clearing the key data, we
can save one line of code here.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260807125845.1477067-6-thuth@redhat.com
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a KUnit test suite for the AES-GCM library API.
It consists of:
- All the shared test cases from aead-test-template.h. These include
extensive consistency tests, a "Monte-Carlo test", and a benchmark.
- Tests against hardcoded AES-GCM test vectors from external sources.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a KUnit test suite for the AES-CCM library API.
It consists of:
- All the shared test cases from aead-test-template.h. These include
extensive consistency tests, a "Monte-Carlo test", and a benchmark.
- Tests against hardcoded AES-CCM test vectors from external sources.
- Tests for CCM-specific message length validation.
To generate the expected aes_ccm_monte_carlo_checksum[] value, add a
script gen-aead-testvecs.py which computes it using python-cryptography.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add aead-test-template.h which contains shared test logic for AEAD
algorithms. It's similar to the existing hash-test-template.h, which
contains shared test logic for hash algorithms.
The initial use cases for this will be AES-CCM and AES-GCM.
It may be used by the existing ChaCha20Poly1305 test suite in the future
as well, though the ChaCha20Poly1305 API currently lacks some of the
functionality that the "template" expects to be present.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Make the test cases more self-contained by replacing the test suite
scoped guarded buffer ('test_buf') with buffers allocated by each test
case using the helper functions in test-utils.h.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Move hash-test-template's test RNG code and the buffer allocation code
from various test suites into a header test-utils.h. This allows them
to be shared by other test suites.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802233005.161467-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Upcoming changes will wire up architecture-optimized implementations of
GCM and CCM. FIPS labs can consider such designs to meet the threshold
for separate self-tests to be needed.
Therefore, add FIPS self-tests for encryption and decryption in these
modes.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802222408.91757-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Upcoming changes will wire up architecture-optimized implementations of
ECB, CBC, CBC-CTS, CTR, and XTS. FIPS labs can consider such designs to
meet the threshold for separate self-tests to be needed.
The inverse direction of the block cipher also needs to be exercised,
which the existing CMAC self-test doesn't do.
Therefore, add FIPS self-tests for encryption and decryption in these
modes as well as the "bare" AES.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802222408.91757-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
In preparation for adding FIPS self-tests for AES encryption modes,
split fips.h into separate files for the AES and SHA test vectors.
They are still generated by the same script, but this keeps things a bit
more organized.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260802222408.91757-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
The old AES-GCM library code is no longer used, so remove it.
Tested-by: Nikunj A Dadhania <nikunj@amd.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://patch.msgid.link/20260722025338.33354-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-CCM to the crypto library.
This will be used to provide a streamlined implementation of the
"ccm(aes)" crypto_aead algorithm. Most users of "ccm(aes)" will also be
able to switch to the library, which as usual will be faster and
simpler, e.g.:
- fs/smb/client/
- fs/smb/server/
- net/mac80211/
- net/mac802154/
(I've already written proof-of-concept patches for all the above, and
they helped inform the API design.)
As in the AES-GCM API, incremental operation is supported. It has to be
used carefully, especially when decrypting, but it makes the API general
enough to work well for all users.
The AES-CCM library code calls aes_cbcmac_blocks() directly, bypassing
the higher-level aes_cbcmac_init(), aes_cbcmac_update(), and
aes_cbcmac_final(). The latter set of functions is useful only for
AES-CCM, so they don't make sense to keep around and will be removed
once the "ccm(aes)" crypto_aead starts using the AES-CCM library.
Initial test coverage is provided by the crypto_aead support added in a
later commit. I'm planning a KUnit test suite as well.
Link: https://patch.msgid.link/20260715221153.246410-8-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-GCM to the crypto library.
This will be used to provide streamlined implementations of the
"gcm(aes)" and "rfc4106(gcm(aes))" crypto_aead algorithms. Most users
of these will also be able to switch to the library, which as usual will
be faster and simpler, e.g.:
- drivers/net/macsec.c
- fs/smb/client/
- fs/smb/server/
- net/ceph/messenger_v2.c
- net/mac80211/ (for both GMAC and GCMP)
- net/tipc/crypto.c
- security/keys/trusted-keys/trusted_dcp.c
(I've already written proof-of-concept patches for all the above, and
they helped inform the API design.)
As usual, the architecture-optimized AES-GCM code will be migrated into
the library as well (using the hooks provided in this commit as well as
the GHASH ones), eliminating lots of repetitive boilerplate code.
Incremental en/decryption is supported. Incremental operation is a bit
controversial in AEAD APIs because users have to be careful not to
consume any decrypted data that hasn't been authenticated yet. But I do
think it's the right choice here. It's not fundamentally different from
the existing incremental MAC APIs, and it's the only approach that's
general enough to work well for all users in the kernel:
- An array of virtually-addressed buffers (like that used by
BoringSSL's EVP_AEAD_CTX_sealv() and EVP_AEAD_CTX_openv()) doesn't
work in the kernel in general, since in some cases the data for a
single AES-GCM message is contained in a large number of highmem
pages that each need to be mapped into memory individually. That
can be done efficiently only by using CPU-local mappings, but there
is a limited number of those.
Ceph messenger v2 is a great example, as it can send or receive up
to 32 MiB in a single AES-GCM message. And it needs the
en/decrypted data to go into a (potentially large) number of bvecs
provided by a custom iterator, as well as into four
virtually-addressed buffers, two of which can be large buffers in
the vmalloc region.
Even just allocating an array big enough to store all the pointers
can be problematic in the kernel. There are cases in which
decryption runs in GFP_NOIO context or even in softirq context,
where memory allocations are not as reliable as they normally are.
- Meanwhile, 'struct scatterlist' (the choice of crypto_aead) has
turned out to be really inconvenient for anyone who *does* just have
virtually-addressed buffers. This is especially true if they can be
in the vmalloc region, including the stack, as in that case the
conversion to a scatterlist has to be done page-by-page.
And even for users who have all of their data in bare 'struct page',
none of them actually use 'struct scatterlist' as their native data
structure anyway. They actually use skbs, bvecs, or other formats.
- iov_iter is attractive, but ultimately not general enough either
(considering the Ceph case for example), but also too general in
some ways (like having support for userspace addresses). Additional
iter types like ITER_SKB would help a bit, but bloating iov_iter
with more types would reduce performance elsewhere in the kernel.
Initial test coverage is provided by the crypto_aead support added in a
later commit. I'm planning a KUnit test suite as well.
Link: https://patch.msgid.link/20260715221153.246410-7-ebiggers@kernel.org
Link: https://patch.msgid.link/20260722021730.16897-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-XTS to the crypto library.
This will be used to provide a streamlined implementation of the
"xts(aes)" crypto_skcipher algorithm. I'm also planning to use this
directly in fscrypt and blk-crypto-fallback.
As usual, the architecture-optimized AES-XTS code will be migrated into
the library as well (using the hooks provided in this commit),
eliminating lots of repetitive boilerplate code. Compared to direct
implementation of "xts(aes)", I've also eliminated the requirement for
architectures to implement ciphertext stealing, as the library just
handles it portably instead. That will simplify things considerably.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Link: https://patch.msgid.link/20260715221153.246410-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-CTR and AES-XCTR to the crypto library.
These will be used to provide streamlined implementations of the
"ctr(aes)" and "xctr(aes)" crypto_skcipher algorithms. Most users of
"ctr(aes)" will also be able to switch to the library, which as usual
will be simpler and faster, e.g.:
- net/mac80211/fils_aead.c
- net/mac802154/llsec.c
As usual, the architecture-optimized AES-CTR and AES-XCTR code will be
migrated into the library as well (using the hooks provided in this
commit), eliminating lots of repetitive boilerplate code.
This is also a prerequisite for supporting AES-GCM, AES-CCM, and
AES-HCTR2 in the crypto library.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260715221153.246410-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-CBC and AES-CBC-CTS to the crypto library.
These will be used to provide streamlined implementations of the
"cbc(aes)" and "cts(cbc(aes))" crypto_skcipher algorithms. Most users
of these crypto_skcipher algorithms will also be able to switch to the
library, which as usual will be simpler and faster, e.g.:
- block/blk-crypto-fallback.c (for AES-128-CBC-ESSIV)
- fs/crypto/crypto.c (for AES-128-CBC-ESSIV)
- fs/crypto/fname.c (for AES-256-CTS and AES-128-CBC)
- kernel/bpf/crypto.c
- net/ceph/crypto.c
- security/keys/encrypted-keys/encrypted.c
As usual, the architecture-optimized AES-CBC and AES-CBC-CTS code will
be migrated into the library as well (using the hooks provided in this
commit), eliminating lots of repetitive boilerplate code.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260715221153.246410-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add support for AES-ECB to the crypto library.
This will be used to provide a streamlined implementation of the
"ecb(aes)" crypto_skcipher algorithm. fs/crypto/keysetup_v1.c will also
use aes_ecb_encrypt() directly.
As usual, the architecture-optimized AES-ECB code will be migrated into
the library as well (using the hooks provided in this commit),
eliminating lots of repetitive boilerplate code.
ECB is obsolete of course, but we need this for parity with the
traditional API and to support some odd users of ECB in the kernel.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Create a documentation file libcrypto-unauth-encryption.rst to hold the
documentation for this and other unauthenticated encryption modes.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://patch.msgid.link/20260715221153.246410-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Starting in commit 7137cbf2b5c9 ("crypto: aes - Add cmac, xcbc, and
cbcmac algorithms using library"), the aes module (CRYPTO_AES) supports
CBC based MACs using the corresponding library functions.
To avoid including unneeded functionality, that support honors the
existing CRYPTO_CMAC, CRYPTO_XCBC, and CRYPTO_CCM kconfig options. The
dependencies are selected if at least one of those is enabled.
However, the select statements don't correctly handle the case where
CRYPTO_AES=y and (for example) CRYPTO_CMAC=m. In that case the
dependencies get selected at level 'm', due to how the kconfig language
works. That causes a linker error.
Fix this by changing the selection conditions to use '!= n'.
A similar issue also exists for CRYPTO_LIB_AES's conditional selection
of CRYPTO_LIB_UTILS. The same '!= n' would work, but instead just make
CRYPTO_LIB_AES always select CRYPTO_LIB_UTILS. CRYPTO_LIB_UTILS is
lightweight, and it's needed by most AES modes and many other things.
Fixes: 7137cbf2b5c9 ("crypto: aes - Add cmac, xcbc, and cbcmac algorithms using library")
Fixes: 309a7e514da7 ("lib/crypto: aes: Add support for CBC-based MACs")
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260709022954.45113-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
