std::string and UTF-8 encoded unicode

后端 未结 3 1294
耶瑟儿~
耶瑟儿~ 2021-01-05 11:27

If I understand well, it is possible to use both string and wstring to store UTF-8 text.

  • With char, ASCII characters take a single byte, some chinese charac

3条回答
  •  甜味超标
    2021-01-05 11:55

    You are talking about Unicode. Unicode uses 32 bits to represent a character. However since that is wasting memory there are more compact encodings. UTF-8 is one such encoding. It assumes that you are using byte units and it maps Unicode characters to 1, 2, 3 or 4 bytes. UTF-16 is another that is using words as units and maps Unicode characters to 1 or 2 words (2 or 4 bytes). You can use both encoding with both string and wchar_t. UTF-8 tends to be more compact for english text/numbers.

    Some things will work regardless of encoding and type used (compare). However all functions that need to understand one character will be broken. I.e the 5th character is not always the 5th entry in the underlying array. It might look like it's working with certain examples but It will eventually break. string::compare will work but do not expect to get alphabetical ordering. That is language dependent. string::find_first_of will work for some but not all. Long string will likely work just because they are long while shorter ones might get confused by character alignment and generate very hard to find bugs.

    Best thing is to find a library that handles it for you and ignore the type underneath (unless you have strong reasons to pick one or the other).

提交回复
热议问题