hfsplus: canonicalize decomposed catalog names like fsck_hfs expects

hfsplus_decompose_table (Apple Technote #1150) decomposes each source
character entirely on its own. It never reorders the result, and it
predates a handful of later corrections to the decomposition standard.
Two concrete consequences, both of which macOS's own fsck_hfs treats as
"Illegal name" (CatalogCheck.c: CheckCatalogName(), FixDecomps()):

 - When combining marks contributed by two different source characters
   end up adjacent, this driver stores them in input order rather than
   ascending Unicode combining-class order.
 - A small, fixed set of characters either aren't decomposed at all,
   or are decomposed into a sequence macOS corrected back in
   Mac OS X 10.2 ("Jaguar").

This is reliably reproducible: xfstests generic/339 exercises dirhash
collisions by creating many files with randomized Unicode names, and
the resulting HFS+ volume fails _check_generic_filesystem afterward
with a long run of "Illegal name" reports from fsck.hfsplus.

Fix hfsplus_asc2uni() to store names the way current macOS does, using
the Unicode tables:

 - hfsplus_canonical_reorder() applies the Unicode Canonical Ordering
   Algorithm: each maximal run of nonzero-combining-class code units is
   stable-sorted into ascending class order.
 - decompose_unichar() falls back to hfsplus_legacy_decompose() for the
   characters missing from Apple Technote #1150's table.
 - hfsplus_fixup_legacy_sequences() substitutes any of fsck_hfs's known
   bad sequences with their corrected form.

hfsplus_decompose_str() ties these together and is now shared by
hfsplus_asc2uni(), hfsplus_hash_dentry() and hfsplus_compare_dentry(),
so hashing and comparison always agree with what actually gets stored -
otherwise two byte-for-byte different but canonically-equivalent
spellings of a name could hash differently or fail to compare equal
against the catalog entry a create() of either would produce.

cc: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
cc: Yangtao Li <frank.li@vivo.com>
cc: linux-fsdevel@vger.kernel.org
Signed-off-by: Viacheslav Dubeyko <slava@dubeyko.com>
Link: https://lore.kernel.org/r/20260919001607.2777138-3-slava@dubeyko.com
Signed-off-by: Viacheslav Dubeyko <slava@dubeyko.com>

0 files changed, 0 insertions, 0 deletions