Check if byte sequence contains utf-16

Question

I am reading a byte sequence from a stream. Assume for the sake of argument, that the sequence is of a fixed length and I read the whole thing into a byte array (in my case

Answer 1

If the contents are expected to be written in a language using the Latin script, simply counting nulls will detect UTF-16. In UTF-8, null bytes will decode to NUL control character, and they don't appear in text normally.

Languages written in other scripts cannot be fully valid in both UTF-16 and UTF-8 unless it's artificially constructed to be so.

So, first detect if it's fully valid UTF-8 sequence on its own:

• If yes, check for null bytes, and if there are some, it's UTF-16. Otherwise it's UTF-8.
• If not, it's UTF-16.

If the above resulted in UTF-16, that's not enough as you have to know the endianess as well. With languages written in Latin script, the amount of odd or even null bytes will tell this.