Folding/Normalizing Ligatures (e.g. Æ to ae) Using (Core)Foundation

Question

I am writing a helper that performs a number of transformations on an input string, in order to create a search-friendly representation of that string.

Think of the

Answer 1

Congratulations, you've found one of the more painful bits of text processing!

First off, NamesList.txt and CaseFolding.txt are indispensable resources for things like this, if you haven't already seen them.

Part of the problem is you're trying to do something almost correct that works in all the languages/locales you care about, whereas Unicode is more concerned about doing the correct thing when displaying strings in a single language-locale.

For (2), ß has canonically case-folded to ss since the earliest CaseFolding.txt I can find (3.0-Update1/CaseFolding-2.txt). CFStringFold() and -[NSString stringByFoldingWithOptions:] ought to do the right thing, but if not, a "locale-independent" s.upper().lower() appears to give a sensible answer for all inputs (and also handles the infamous "Turkish I").

For (5), you're a little out of luck: Unicode 6.2 doesn't appear to contain a normative mapping from Æ to AE and has changed from "letter" to "ligature" and back again (U+00C6 is LATIN CAPITAL LETTER A E in 1.0, LATIN CAPITAL LIGATURE AE in 1.1, and LATIN CAPITAL LETTER AE in 2.0). You could search NamesList.txt for "ligature" and add a bunch of special cases.

Notes:

• CFStringNormalize() doesn't do what you want. You do want to normalize strings before adding them to the index; I suggest NFKC at the start and end of other processing.
• CFStringTransform() doesn't quite do what you want either; all the scripts are "latin"
• CFStringFold() is order-dependent: The combining ypogegrammeni and prosgegrammeni are stripped by kCFCompareDiacriticInsensitive but converted to a lowercase iota by kCFCompareCaseInsensitive. The "correct" thing appears to be to do the case-fold first followed by the others, although stripping it may make more sense linguistically.
• You almost certainly do not want to use kCFCompareLocalized unless you want to rebuild the search index every time the locale changes.

Readers from other languages note: Check that the function you use is not dependent on the user's current locale! Java users should use something like s.toUpperCase(Locale.ENGLISH), .NET users should use s.ToUpperInvariant(). If you actually want the user's current locale, specify it explicitly.