UTF8 Filenames in PHP and Different Unicode Encodings

Question

I have a file containing Unicode characters on a server running linux. If I SSH into the server and use tab-completion to navigate to the file/folder containing unicode char

Answer 1

Firstly: You should try to avoid imposing semantics on the names of files. I can't really tell why PHP is generating filenames in your scenario, so I can't suggest how you should apply this rule.

The different (two byte and three byte) representations of é are UTF-8 encodings of the composed and decomposed variations of this character in Unicode. In Unicode these are distinct ways to represent the same visual character. Unicode has the concept of "canonicalisation" in which all representations of the same character are converted to a single representation, sort of like squashing two strings to lowercase to perform a caseless comparison.

Linux does not perform canonicalisation or any other processing automatically for file names, so a file may be named with precomposed (like the two byte sequence) or decomposed (like the three byte sequence) characters or any mix of the two, it's up to whoever named the file. If you are creating the files, you could set a policy (e.g. always use precomposed characters) and write some code to enforce it. Otherwise, you can't rely on any particular rule here.