C# big-endian UCS-2

Question

The project I\'m currently working on needs to interface with a client system that we don\'t make, so we have no control over how data is sent either way. The problem is tha

Answer 1

EDIT: Now we know that the problem isn't in the encoding of the text data but in the encoding of the length. There are a few options:

• Reverse the bytes and then use the built-in BitConverter code (which I assume is what you're using now; that or BinaryReader)
• Perform the conversion yourself using repeated "add and shift" operations
• Use my EndianBitConverter or EndianBinaryReader classes from MiscUtil, which are like BitConverter and BinaryReader, but let you specify the endianness.

---

You may be looking for Encoding.BigEndianUnicode. That's the big-endian UTF-16 encoding, which isn't strictly speaking the same as UCS-2 (as pointed out by Marc) but should be fine unless you give it strings including characters outside the BMP (i.e. above U+FFFF), which can't be represented in UCS-2 but are represented in UTF-16.

From the Wikipedia page:

The older UCS-2 (2-byte Universal Character Set) is a similar character encoding that was superseded by UTF-16 in version 2.0 of the Unicode standard in July 1996.2 It produces a fixed-length format by simply using the code point as the 16-bit code unit and produces exactly the same result as UTF-16 for 96.9% of all the code points in the range 0-0xFFFF, including all characters that had been assigned a value at that time.

I find it highly unlikely that the client system is sending you characters where there's a difference (which is basically the surrogate pairs, which are permanently reserved for that use anyway).

Answer 2

UCS-2 is so close to UTF-16 that Encoding.BigEndianUnicode will almost always suffice.

The issue (comments) around reading the length prefix (as big-endian) is more correctly resolved via shift operations, which will do the right thing on all systems. For example:

Read4BytesIntoBuffer(buffer);
int len =(buffer[0] << 24) | (buffer[1] << 16) | (buffer[2] << 8) | (buffer[3]); 

This will then work the same (at parsing a big-endian 4 byte int) on any system, regardless of local endianness.

Answer 3

string x = "abc";
byte[] data = Encoding.BigEndianUnicode.GetBytes(x);

In other direction:

string decodedX = Encoding.BigEndianUnicode.GetString(data);

It is not exactly UCS-2 but it is enough for most cases.

UPD: Unicode FAQ

Q: What is the difference between UCS-2 and UTF-16?

A: UCS-2 is obsolete terminology which refers to a Unicode implementation up to Unicode 1.1, before surrogate code points and UTF-16 were added to Version 2.0 of the standard. This term should now be avoided.

UCS-2 does not define a distinct data format, because UTF-16 and UCS-2 are identical for purposes of data exchange. Both are 16-bit, and have exactly the same code unit representation.

Sometimes in the past an implementation has been labeled "UCS-2" to indicate that it does not support supplementary characters and doesn't interpret pairs of surrogate code points as characters. Such an implementation would not handle processing of character properties, code point boundaries, collation, etc. for supplementary characters.