| Age | Commit message (Collapse) | Author |
|---|
|
Fix undefined behavior caused by shifting a 32-bit integer by 32 bits
during decompression. This prevents potential kernel decompression
failures or corruption when parsing malicious or malformed bzip2 archives.
Link: https://lkml.kernel.org/r/20260308165012.2872633-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Remove the unnecessary initialization of 'rcu' to false in
report_bug_entry() and report_bug(), as it is assigned by warn_rcu_enter()
before its first use.
Link: https://lkml.kernel.org/r/20260306162418.2815979-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Use lowercase "kernel BUG" consistently in pr_crit() messages. The
verbose path already uses "kernel BUG at %s:%u!" but the non-verbose
fallback uses "Kernel BUG" with an uppercase 'K'.
Link: https://lkml.kernel.org/r/20260306162327.2815553-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Fix "This results of this trade" to "The results of this trade" in the
comment describing the lbits and dbits tuning parameters.
Link: https://lkml.kernel.org/r/20260306161732.2812132-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Fix "all variable" to "all variables" in the file header comment.
Link: https://lkml.kernel.org/r/20260306161707.2812005-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
When inflate_codes() fails in inflate_dynamic(), the code jumps to the
'out' label which only frees 'll', leaking the Huffman tables 'tl' and
'td'. Restructure the code so that the decoding tables are always freed
before reaching the 'out' label.
Link: https://lkml.kernel.org/r/20260306161647.2811874-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
When inflate_codes() fails in inflate_fixed(), only the length list 'l' is
freed, but the Huffman tables 'tl' and 'td' are leaked. Add the missing
huft_free() calls on the error path.
Link: https://lkml.kernel.org/r/20260306161612.2811703-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Fix a typo in __uuid_gen_common() where "reversion" (meaning to revert)
was used instead of "revision" when describing the UUID variant field.
Link: https://lkml.kernel.org/r/20260306161250.2811500-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The Baikal SoC and platform support was dropped from the kernel, remove
the reference to non-exist file. While at it, fix spelling.
Link: https://lkml.kernel.org/r/20260302092831.2267785-4-andriy.shevchenko@linux.intel.com
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
Reviewed-by: Kuan-Wei Chiu <visitorckw@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Update header inclusions to follow IWYU (Include What You Use) principle.
Link: https://lkml.kernel.org/r/20260302092831.2267785-3-andriy.shevchenko@linux.intel.com
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Kuan-Wei Chiu <visitorckw@gmail.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "lib: polynomial: Move to math/ and clean up", v2.
While removing Baikal SoC and platform code pieces I found that this code
belongs to lib/math/ rather than generic lib/. Hence the move and
followed up cleanups.
This patch (of 3):
The algorithm behind polynomial belongs to our collection of math
equations and expressions handling. Move it to math/ subfolder where
others of the kind are located.
Link: https://lkml.kernel.org/r/20260302092831.2267785-2-andriy.shevchenko@linux.intel.com
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Kuan-Wei Chiu <visitorckw@gmail.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Replace the implicit 'bool' to 'int' conversion with an explicit ternary
operator. This makes the pointer arithmetic clearer and avoids relying on
boolean memory representation for logic flow.
Link: https://lkml.kernel.org/r/20260301203845.2617217-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Using bitwise OR (|=) on a boolean variable is valid C, but replacing it
with a direct logical assignment makes the intent clearer and appeases
strict static analysis tools.
Link: https://lkml.kernel.org/r/20260301152143.2572137-2-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Include <linux/export.h> explicitly instead of relying on it being
implicitly included by <linux/module.h> for the EXPORT_SYMBOL macro.
Link: https://lkml.kernel.org/r/20260301152143.2572137-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Fix a missing article ('a') in the comment describing the glob
implementation, and replace 'blacklists' with 'denylists' to align with
the kernel's inclusive terminology guidelines.
Link: https://lkml.kernel.org/r/20260301154553.2592681-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Add the missing dual MIT/GPL license identifier to glob.c.
Link: https://lkml.kernel.org/r/20260228195300.2468310-1-objecting@objecting.org
Signed-off-by: Josh Law <objecting@objecting.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Like gcc, clang-22 now also warns about a function that it incorrectly
identifies as a printf-style format:
lib/bug.c:190:22: error: diagnostic behavior may be improved by adding the 'format(printf, 1, 0)' attribute to the declaration of '__warn_printf' [-Werror,-Wmissing-format-attribute]
179 | static void __warn_printf(const char *fmt, struct pt_regs *regs)
| __attribute__((format(printf, 1, 0)))
180 | {
181 | if (!fmt)
182 | return;
183 |
184 | #ifdef HAVE_ARCH_BUG_FORMAT_ARGS
185 | if (regs) {
186 | struct arch_va_list _args;
187 | va_list *args = __warn_args(&_args, regs);
188 |
189 | if (args) {
190 | vprintk(fmt, *args);
| ^
Revert the change that added a gcc-specific workaround, and instead add
the generic annotation that avoid the warning.
Link: https://lkml.kernel.org/r/20260323205534.1284284-1-arnd@kernel.org
Fixes: d36067d6ea00 ("bug: Hush suggest-attribute=format for __warn_printf()")
Suggested-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Suggested-by: Brendan Jackman <jackmanb@google.com>
Link: https://lore.kernel.org/all/20251208141618.2805983-1-andriy.shevchenko@linux.intel.com/T/#u
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Cc: Bill Wendling <morbo@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Justin Stitt <justinstitt@google.com>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Since the ChaCha permutation is invertible, the local variable
'permuted_state' is sufficient to compute the original 'state', and thus
the key, even after the permutation has been done.
While the kernel is quite inconsistent about zeroizing secrets on the
stack (and some prominent userspace crypto libraries don't bother at all
since it's not guaranteed to work anyway), the kernel does try to do it
as a best practice, especially in cases involving the RNG.
Thus, explicitly zeroize 'permuted_state' before it goes out of scope.
Fixes: c08d0e647305 ("crypto: chacha20 - Add a generic ChaCha20 stream cipher implementation")
Cc: stable@vger.kernel.org
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260326032920.39408-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
As a preparation of the untangling of time namespaces and the vDSO, move
the glue functions between those subsystems into a new file.
While at it, switch the mutex lock and mmap_read_lock() in the vDSO
namespace code to guard().
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260326-vdso-timens-decoupling-v2-1-c82693a7775f@linutronix.de
|
|
IS_ERR_OR_NULL() already uses likely(!ptr) internally. checkpatch points
out the nesting: Remove the explicit use of likely().
Change generated with coccinelle.
Signed-off-by: Philipp Hahn <phahn-oss@avm.de>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260310-b4-is_err_or_null-v1-59-bd63b656022d@avm.de
|
|
In most real-life cases, 0-length bitmap provided by user is a sign of
an error. The API doesn't provide any guarantees on returned value, and
the bitmap pointers are not dereferenced.
Signed-off-by: Yury Norov <ynorov@nvidia.com>
|
|
When enabling Context Analysis (CONTEXT_ANALYSIS := y) in arch/x86/kvm
code, Clang's Thread Safety Analysis failed to recognize that identical
per_cpu() accesses refer to the same lock:
| CC [M] arch/x86/kvm/vmx/posted_intr.o
| arch/x86/kvm/vmx/posted_intr.c:186:2: error: releasing raw_spinlock '__ptr + __per_cpu_offset[vcpu->cpu]' that was not held [-Werror,-Wthread-safety-analysis]
| 186 | raw_spin_unlock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu));
| | ^
| ./include/linux/spinlock.h:276:32: note: expanded from macro 'raw_spin_unlock'
| 276 | #define raw_spin_unlock(lock) _raw_spin_unlock(lock)
| | ^
| arch/x86/kvm/vmx/posted_intr.c:207:1: error: raw_spinlock '__ptr + __per_cpu_offset[vcpu->cpu]' is still held at the end of function [-Werror,-Wthread-safety-analysis]
| 207 | }
| | ^
| arch/x86/kvm/vmx/posted_intr.c:182:2: note: raw_spinlock acquired here
| 182 | raw_spin_lock_nested(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu),
| | ^
| ./include/linux/spinlock.h:235:2: note: expanded from macro 'raw_spin_lock_nested'
| 235 | _raw_spin_lock(((void)(subclass), (lock)))
| | ^
| 2 errors generated.
This occurred because the default RELOC_HIDE() implementation (used by
the per-CPU macros) is a statement expression containing an intermediate
'unsigned long' variable (this version appears to predate Git history).
While the analysis strips away inner casts when resolving pointer
aliases, it stops when encountering intermediate non-pointer variables
(this is Thread Safety Analysis specific and irrelevant for codegen).
This prevents the analysis from concluding that the pointers passed to
e.g. raw_spin_lock() and raw_spin_unlock() were identical when per-CPU
accessors are used.
Simplify RELOC_HIDE() to a single expression. This preserves the intent
of obfuscating UB-introducing out-of-bounds pointer calculations from
the compiler via the 'unsigned long' cast, but allows the alias analysis
to successfully resolve the pointers.
Using a recent Clang version, I observe that generated code remains the
same for vmlinux; the intermediate variable was already being optimized
away (for any respectable modern compiler, not doing so would be an
optimizer bug). Note that GCC provides its own version of RELOC_HIDE(),
so this change only affects Clang builds.
Add a test case to lib/test_context-analysis.c to catch any regressions.
Reported-by: Bart Van Assche <bvanassche@acm.org>
Reported-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Marco Elver <elver@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
Link: https://lore.kernel.org/all/e3946223-4543-4a76-a328-9c6865e95192@acm.org/
Link: https://patch.msgid.link/20260319135245.1420780-1-elver@google.com
|
|
printf_kunit already covers IPv4 address formatting, but the ip6()
test case is empty even though printk-formats.rst documents %pI6,
%pi6, %pI6c, and generic %pIS variants.
Add focused IPv6 checks for raw and generic formatting, compressed
output, the single-zero %pI6c corner case, and bracketed port
formatting for sockaddr_in6.
Signed-off-by: Shuvam Pandey <shuvampandey1@gmail.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Tested-by: Petr Mladek <pmladek@suse.com>
Link: https://patch.msgid.link/20260317114548.98919-1-shuvampandey1@gmail.com
[pmladek@suse.com: Removed non-necessary details from the commit message.]
Signed-off-by: Petr Mladek <pmladek@suse.com>
|
|
Currently the kconfig options for the crypto library KUnit tests appear
in the menu:
-> Library routines
-> Crypto library routines
However, this is the only content of "Crypto library routines". I.e.,
it is empty when CONFIG_KUNIT=n. This is because the crypto library
routines themselves don't have (or need to have) prompts.
Since this usually ends up as an unnecessary empty menu, let's remove
this menu and instead source the lib/crypto/tests/Kconfig file from
lib/Kconfig.debug inside the "Runtime Testing" menu:
-> Kernel hacking
-> Kernel Testing and Coverage
-> Runtime Testing
This puts the prompts alongside the ones for most of the other lib/
KUnit tests. This seems to be a much better match to how the kconfig
menus are organized.
Acked-by: Randy Dunlap <rdunlap@infradead.org>
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Link: https://lore.kernel.org/r/20260322032438.286296-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Since the architecture-optimized SM3 code was migrated into lib/crypto/,
sm3_block_generic() is no longer called. Remove it. Then, since this
frees up the name, rename sm3_transform() to sm3_block_generic()
(matching the naming convention used in other hash algorithms).
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260321040935.410034-12-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the x86-optimized SM3 code via an x86-specific
crypto_shash algorithm, instead just implement the sm3_blocks() library
function. This is much simpler, it makes the SM3 library functions be
x86-optimized, and it fixes the longstanding issue where the
x86-optimized SM3 code was disabled by default. SM3 still remains
available through crypto_shash, but individual architectures no longer
need to handle it.
Tweak the prototype of sm3_transform_avx() to match what the library
expects, including changing the block count to size_t. Note that the
assembly code actually already treated this argument as size_t.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260321040935.410034-10-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the riscv-optimized SM3 code via a riscv-specific
crypto_shash algorithm, instead just implement the sm3_blocks() library
function. This is much simpler, it makes the SM3 library functions be
riscv-optimized, and it fixes the longstanding issue where the
riscv-optimized SM3 code was disabled by default. SM3 still remains
available through crypto_shash, but individual architectures no longer
need to handle it.
Tweak the prototype of sm3_transform_zvksh_zvkb() to match what the
library expects, including changing the block count to size_t.
Note that the assembly code already treated it as size_t.
Note: to see the diff from arch/riscv/crypto/sm3-riscv64-glue.c to
lib/crypto/riscv/sm3.h, view this commit with 'git show -M10'.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260321040935.410034-9-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Instead of exposing the arm64-optimized SM3 code via arm64-specific
crypto_shash algorithms, instead just implement the sm3_blocks() library
function. This is much simpler, it makes the SM3 library functions be
arm64-optimized, and it fixes the longstanding issue where the
arm64-optimized SM3 code was disabled by default. SM3 still remains
available through crypto_shash, but individual architectures no longer
need to handle it.
Tweak the SM3 assembly function prototypes to match what the library
expects, including changing the block count from 'int' to 'size_t'.
sm3_ce_transform() had to be updated to access 'x2' instead of 'w2',
while sm3_neon_transform() already used 'x2'.
Remove the CFI stubs which are no longer needed because the SM3 assembly
functions are no longer ever indirectly called.
Remove the dependency on KERNEL_MODE_NEON. It was unnecessary, because
KERNEL_MODE_NEON is always enabled on arm64.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260321040935.410034-8-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a KUnit test suite for the SM3 library. It closely mirrors the test
suites for the other cryptographic hash functions. The actual test and
benchmark logic is already in hash-test-template.h; this just wires it
up for SM3 in the usual way.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260321040935.410034-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a straightforward library API for SM3, mirroring the ones for the
other hash algorithms. It uses the existing generic implementation of
SM3's compression function in lib/crypto/sm3.c. Hooks are added for
architecture-optimized implementations, which later commits will wire up
to the existing optimized SM3 code for arm64, riscv, and x86.
Note that the rationale for this is *not* that SM3 should be used, or
that any kernel subsystem currently seems like a candidate for switching
from the sm3 crypto_shash to SM3 library. (SM3, in fact, shouldn't be
used. Likewise you shouldn't use MD5, SHA-1, RC4, etc...)
Rather, it's just that this will simplify how the kernel's existing SM3
code is integrated and make it much easier to maintain and test. SM3 is
one of the only hash algorithms with arch-optimized code that is still
integrated in the old way. By converting it to the new lib/crypto/ code
organization, we'll only have to keep track of one way of doing things.
The library will also get a KUnit test suite (as usual for lib/crypto/),
so it will become more easily and comprehensively tested as well.
Skip adding functions for HMAC-SM3 for now, though. There's not as much
point in adding those right now.
Note: similar to the other hash algorithms, the library API uses
'struct sm3_ctx', not 'struct sm3_state'. The existing 'struct
sm3_state' and the sm3_block_generic() function which uses it are
temporarily kept around until their users are updated by later commits.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260321040935.410034-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Make the AES-GCM library use the GHASH library instead of directly
calling gf128mul_lle(). This allows the architecture-optimized GHASH
implementations to be used, or the improved generic implementation if no
architecture-optimized implementation is usable.
Note: this means that <crypto/gcm.h> no longer needs to include
<crypto/gf128mul.h>. Remove that inclusion, and include
<crypto/gf128mul.h> explicitly from arch/x86/crypto/aesni-intel_glue.c
which previously was relying on the transitive inclusion.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-20-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Remove the 4k_lle multiplication functions and the associated
gf128mul_table_le data table. Their only user was the generic
implementation of GHASH, which has now been changed to use a different
implementation based on standard integer multiplication.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-18-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Remove the "ghash-pclmulqdqni" crypto_shash algorithm. Move the
corresponding assembly code into lib/crypto/, and wire it up to the
GHASH library.
This makes the GHASH library be optimized with x86's carryless
multiplication instructions. It also greatly reduces the amount of
x86-specific glue code that is needed, and it fixes the issue where this
GHASH optimization was disabled by default.
Rename and adjust the prototypes of the assembly functions to make them
fit better with the library. Remove the byte-swaps (pshufb
instructions) that are no longer necessary because the library keeps the
accumulator in POLYVAL format rather than GHASH format.
Rename clmul_ghash_mul() to polyval_mul_pclmul() to reflect that it
really does a POLYVAL style multiplication. Wire it up to both
ghash_mul_arch() and polyval_mul_arch().
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-15-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Remove the "ghash-s390" crypto_shash algorithm, and replace it with an
implementation of ghash_blocks_arch() for the GHASH library.
This makes the GHASH library be optimized with CPACF. It also greatly
reduces the amount of s390-specific glue code that is needed, and it
fixes the issue where this GHASH optimization was disabled by default.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-14-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Remove the "ghash-riscv64-zvkg" crypto_shash algorithm. Move the
corresponding assembly code into lib/crypto/, modify it to take the
length in blocks instead of bytes, and wire it up to the GHASH library.
This makes the GHASH library be optimized with the RISC-V Vector
Cryptography Extension. It also greatly reduces the amount of
riscv-specific glue code that is needed, and it fixes the issue where
this optimized GHASH code was disabled by default.
Note that this RISC-V code has multiple opportunities for improvement,
such as adding more parallelism, providing an optimized multiplication
function, and directly supporting POLYVAL. But for now, this commit
simply tweaks ghash_zvkg() slightly to make it compatible with the
library, then wires it up to ghash_blocks_arch().
ghash_preparekey_arch() is also implemented to store the copy of the raw
key needed by the vghsh.vv instruction.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-13-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Remove the "p8_ghash" crypto_shash algorithm. Move the corresponding
assembly code into lib/crypto/, and wire it up to the GHASH library.
This makes the GHASH library be optimized for POWER8. It also greatly
reduces the amount of powerpc-specific glue code that is needed, and it
fixes the issue where this optimized GHASH code was disabled by default.
Note that previously the C code defined the POWER8 GHASH key format as
"u128 htable[16]", despite the assembly code only using four entries.
Fix the C code to use the correct key format. To fulfill the library
API contract, also make the key preparation work in all contexts.
Note that the POWER8 assembly code takes the accumulator in GHASH
format, but it actually byte-reflects it to get it into POLYVAL format.
The library already works with POLYVAL natively. For now, just wire up
this existing code by converting it to/from GHASH format in C code.
This should be cleaned up to eliminate the unnecessary conversion later.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-12-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Remove the "ghash-neon" crypto_shash algorithm. Move the corresponding
assembly code into lib/crypto/, and wire it up to the GHASH library.
This makes the GHASH library be optimized on arm64 (though only with
NEON, not PMULL; for now the goal is just parity with crypto_shash). It
greatly reduces the amount of arm64-specific glue code that is needed,
and it fixes the issue where this optimization was disabled by default.
To integrate the assembly code correctly with the library, make the
following tweaks:
- Change the type of 'blocks' from int to size_t
- Change the types of 'dg' and 'h' to polyval_elem. Note that this
simply reflects the format that the code was already using.
- Remove the 'head' argument, which is no longer needed.
- Remove the CFI stubs, as indirect calls are no longer used.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-10-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Remove the "ghash-neon" crypto_shash algorithm. Move the corresponding
assembly code into lib/crypto/, and wire it up to the GHASH library.
This makes the GHASH library be optimized on arm (though only with NEON,
not PMULL; for now the goal is just parity with crypto_shash). It
greatly reduces the amount of arm-specific glue code that is needed, and
it fixes the issue where this optimization was disabled by default.
To integrate the assembly code correctly with the library, make the
following tweaks:
- Change the type of 'blocks' from int to size_t.
- Change the types of 'dg' and 'h' to polyval_elem. Note that this
simply reflects the format that the code was already using, at least
on little endian CPUs. For big endian CPUs, add byte-swaps.
- Remove the 'head' argument, which is no longer needed.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-8-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add a KUnit test suite for the GHASH library functions.
It closely mirrors the POLYVAL test suite.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Add GHASH support to the gf128hash module.
This will replace the GHASH support in the crypto_shash API. It will be
used by the "gcm" template and by the AES-GCM library (when an
arch-optimized implementation of the full AES-GCM is unavailable).
This consists of a simple API that mirrors the existing POLYVAL API, a
generic implementation of that API based on the existing efficient and
side-channel-resistant polyval_mul_generic(), and the framework for
architecture-optimized implementations of the GHASH functions.
The GHASH accumulator is stored in POLYVAL format rather than GHASH
format, since this is what most modern GHASH implementations actually
need. The few implementations that expect the accumulator in GHASH
format will just convert the accumulator to/from GHASH format
temporarily. (Supporting architecture-specific accumulator formats
would be possible, but doesn't seem worth the complexity.)
However, architecture-specific formats of struct ghash_key will be
supported, since a variety of formats will be needed there anyway. The
default format is just the key in POLYVAL format.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Currently, some architectures (arm64 and x86) have optimized code for
both GHASH and POLYVAL. Others (arm, powerpc, riscv, and s390) have
optimized code only for GHASH. While POLYVAL support could be
implemented on these other architectures, until then we need to support
the case where arch-optimized functions are present only for GHASH.
Therefore, update the support for arch-optimized POLYVAL functions to
allow architectures to opt into supporting these functions individually.
The new meaning of CONFIG_CRYPTO_LIB_GF128HASH_ARCH is that some level
of GHASH and/or POLYVAL acceleration is provided.
Also provide an implementation of polyval_mul() based on
polyval_blocks_arch(), for when polyval_mul_arch() isn't implemented.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-3-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Currently, the standalone GHASH code is coupled with crypto_shash. This
has resulted in unnecessary complexity and overhead, as well as the code
being unavailable to library code such as the AES-GCM library. Like was
done with POLYVAL, it needs to find a new home in lib/crypto/.
GHASH and POLYVAL are closely related and can each be implemented in
terms of each other. Optimized code for one can be reused with the
other. But also since GHASH tends to be difficult to implement directly
due to its unnatural bit order, most modern GHASH implementations
(including the existing arm, arm64, powerpc, and x86 optimized GHASH
code, and the new generic GHASH code I'll be adding) actually
reinterpret the GHASH computation as an equivalent POLYVAL computation,
pre and post-processing the inputs and outputs to map to/from POLYVAL.
Given this close relationship, it makes sense to group the GHASH and
POLYVAL code together in the same module. This gives us a wide range of
options for implementing them, reusing code between the two and properly
utilizing whatever instructions each architecture provides.
Thus, GHASH support will be added to the library module that is
currently called "polyval". Rename it to an appropriate name:
"gf128hash". Rename files, options, functions, etc. where appropriate
to reflect the upcoming sharing with GHASH. (Note: polyval_kunit is not
renamed, as ghash_kunit will be added alongside it instead.)
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20260319061723.1140720-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
|
|
Bitmap API handles nbits == 0 in most cases correctly, i.e. it doesn't
dereferene underlying bitmap and returns a sane value where convenient,
or implementation defined, or undef.
Implicitly testing nbits == 0 case, however, may make an impression that
this is a regular case. This is wrong. In most cases nbits == 0 is a
sign of an error on a client side. The tests should not make such an
implression.
This patch reworks the existing tests to not test nbits == 0. The
following patch adds an explicit test for it with an appropriate
precaution.
Signed-off-by: Yury Norov <ynorov@nvidia.com>
|
|
Test the function for correctness when some bits are set in the last word
of bitmap beyond nbits. This is motivated by commit a9dadc1c512807f9
("powerpc/xive: Fix the size of the cpumask used in
xive_find_target_in_mask()").
Signed-off-by: Yury Norov <ynorov@nvidia.com>
|
|
count_leading_zeros() is based on fls(), which is defined for x == 0,
contrary to __ffs() family. The comment in crypto/mpi erroneously
states that the function may return undef in such case.
Fix the comment together with the outdated function signature, and now
that COUNT_LEADING_ZEROS_0 is not referenced in the codebase, get rid of
it too.
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Acked-by: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
|
|
The function calculates a Hamming weight of a bitmap starting from an
arbitrary bit.
Signed-off-by: Yury Norov <ynorov@nvidia.com>
|
|
test_find_first_bit()
test_find_first_bit() searches for a set bit from the beginning of the
test bitmap and clears it repeatedly, eventually clearing the entire
bitmap.
After test_find_first_bit() is executed, test_find_first_and_bit() and
test_find_next_and_bit() are executed without randomly reinitializing the
cleared bitmap.
In the first phase (testing find_bit() with a random-filled bitmap),
test_find_first_bit() only operates on 1/10 of the entire size of the
testing bitmap, so this isn't a big problem.
However, in the second phase (testing find_bit() with a sparse bitmap),
test_find_first_bit() clears the entire test bitmap, so the subsequent
test_find_first_and_bit() and test_find_next_and_bit() will not find any
set bits. This is probably not the intended benchmark.
To fix this issue, test_find_first_bit() operates on a duplicated bitmap
and does not clear the original test bitmap.
The same is already done in test_find_first_and_bit().
While we're at it, add const qualifiers to the bitmap pointer arguments
in the test functions.
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk>
Signed-off-by: Yury Norov <ynorov@nvidia.com>
|
|
Remove find_nth_andnot_bit() leftovers.
CC: Rasmus Villemoes <linux@rasmusvillemoes.dk>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Fixes: b0c85e99458a ("cpumask: Remove unnecessary cpumask_nth_andnot()")
Signed-off-by: Yury Norov <ynorov@nvidia.com>
|
|
Different subtests print output in slightly different formats. Unify the
format for better visual representation.
The test output before:
[ 0.553474] test_bitmap: parselist: 14: input is '0-2047:128/256' OK, Time: 202
[ 0.555121] test_bitmap: bitmap_print_to_pagebuf: input is '0-32767
[ 0.555121] ', Time: 1278
[ 0.578392] test_bitmap: Time spent in test_bitmap_read_perf: 427864
[ 0.580137] test_bitmap: Time spent in test_bitmap_write_perf: 793554
[ 0.581957] test_bitmap: all 390447 tests passed
And after:
[ 0.314982] test_bitmap: parselist('0-2047:128/256'): 135
[ 0.315517] test_bitmap: scnprintf("%*pbl", '0-32767'): 342
[ 0.330045] test_bitmap: test_bitmap_read_perf: 252294
[ 0.331132] test_bitmap: test_bitmap_write_perf: 539001
[ 0.332163] test_bitmap: all 390447 tests passed
Signed-off-by: Yury Norov <ynorov@nvidia.com>
|
|
scnprintf("%*pbl") is more verbose than bitmap_print_to_pagebuf().
Switch the test to using it. This also improves the test output
because bitmap_print_to_pagebuf() adds \n at the end of the printed
bitmap, which breaks the test format.
Signed-off-by: Yury Norov <ynorov@nvidia.com>
|
