utf-16

Convert UTF-8 with BOM to UTF-8 with no BOM in Python

由 懵懂的女人 提交于 2019-11-26 11:44:00
Two questions here. I have a set of files which are usually UTF-8 with BOM. I'd like to convert them (ideally in place) to UTF-8 with no BOM. It seems like codecs.StreamRecoder(stream, encode, decode, Reader, Writer, errors) would handle this. But I don't really see any good examples on usage. Would this be the best way to handle this? source files: Tue Jan 17$ file brh-m-157.json brh-m-157.json: UTF-8 Unicode (with BOM) text Also, it would be ideal if we could handle different input encoding wihtout explicitly knowing (seen ASCII and UTF-16). It seems like this should all be feasible. Is

Convert UTF-16 to UTF-8 and remove BOM?

由 让人想犯罪 __ 提交于 2019-11-26 11:09:49
问题 We have a data entry person who encoded in UTF-16 on Windows and would like to have utf-8 and remove the BOM. The utf-8 conversion works but BOM is still there. How would I remove this? This is what I currently have: batch_3={\'src\':\'/Users/jt/src\',\'dest\':\'/Users/jt/dest/\'} batches=[batch_3] for b in batches: s_files=os.listdir(b[\'src\']) for file_name in s_files: ff_name = os.path.join(b[\'src\'], file_name) if (os.path.isfile(ff_name) and ff_name.endswith(\'.json\')): print ff_name

UTF-8, UTF-16, and UTF-32

由 狂风中的少年 提交于 2019-11-26 10:58:16
What are the differences between UTF-8, UTF-16, and UTF-32? I understand that they will all store Unicode, and that each uses a different number of bytes to represent a character. Is there an advantage to choosing one over the other? AnthonyWJones UTF-8 has an advantage in the case where ASCII characters represent the majority of characters in a block of text, because UTF-8 encodes all characters into 8 bits (like ASCII). It is also advantageous in that a UTF-8 file containing only ASCII characters has the same encoding as an ASCII file. UTF-16 is better where ASCII is not predominant, since

Why does the Java char primitive take up 2 bytes of memory?

由 白昼怎懂夜的黑 提交于 2019-11-26 10:57:54
问题 Is there any reason why Java char primitive data type is 2 bytes unlike C which is 1 byte? Thanks 回答1: When Java was originally designed, it was anticipated that any Unicode character would fit in 2 bytes (16 bits), so char and Character were designed accordingly. In fact, a Unicode character can now require up to 4 bytes. Thus, UTF-16, the internal Java encoding, requires supplementary characters use 2 code units. Characters in the Basic Multilingual Plane (the most common ones) still use 1.

Convert UTF-16 to UTF-8 under Windows and Linux, in C

由 江枫思渺然 提交于 2019-11-26 10:36:26
问题 I was wondering if there is a recommended \'cross\' Windows and Linux method for the purpose of converting strings from UTF-16LE to UTF-8? or one should use different methods for each environment? I\'ve managed to google few references to \'iconv\' , but for somreason I can\'t find samples of basic conversions, such as - converting a wchar_t UTF-16 to UTF-8. Anybody can recommend a method that would be \'cross\', and if you know of references or a guide with samples, would very appreciate it.

Correctly reading a utf-16 text file into a string without external libraries?

由 强颜欢笑 提交于 2019-11-26 09:38:09
问题 I\'ve been using StackOverflow since the beginning, and have on occasion been tempted to post questions, but I\'ve always either figured them out myself or found answers posted eventually... until now. This feels like it should be fairly simple, but I\'ve been wandering around the internet for hours with no success, so I turn here: I have a pretty standard utf-16 text file, with a mixture of English and Chinese characters. I would like those characters to end up in a string (technically, a

Convert UTF-8 with BOM to UTF-8 with no BOM in Python

由 拥有回忆 提交于 2019-11-26 08:50:55
问题 Two questions here. I have a set of files which are usually UTF-8 with BOM. I\'d like to convert them (ideally in place) to UTF-8 with no BOM. It seems like codecs.StreamRecoder(stream, encode, decode, Reader, Writer, errors) would handle this. But I don\'t really see any good examples on usage. Would this be the best way to handle this? source files: Tue Jan 17$ file brh-m-157.json brh-m-157.json: UTF-8 Unicode (with BOM) text Also, it would be ideal if we could handle different input

Manually converting unicode codepoints into UTF-8 and UTF-16

由 浪尽此生 提交于 2019-11-26 08:48:51
问题 I have a university programming exam coming up, and one section is on unicode. I have checked all over for answers to this, and my lecturer is useless so that’s no help, so this is a last resort for you guys to possibly help. The question will be something like: The string \'mЖ丽\' has these unicode codepoints U+006D , U+0416 and U+4E3D , with answers written in hexadecimal, manually encode the string into UTF-8 and UTF-16. Any help at all will be greatly appreciated as I am trying to get my

Difference between UTF-8 and UTF-16?

由 谁说我不能喝 提交于 2019-11-26 08:42:58
问题 Difference between UTF-8 and UTF-16? Why do we need these? MessageDigest md = MessageDigest.getInstance(\"SHA-256\"); String text = \"This is some text\"; md.update(text.getBytes(\"UTF-8\")); // Change this to \"UTF-16\" if needed byte[] digest = md.digest(); 回答1: I believe there are a lot of good articles about this around the Web, but here is a short summary. Both UTF-8 and UTF-16 are variable length encodings. However, in UTF-8 a character may occupy a minimum of 8 bits, while in UTF-16

Difference between Big Endian and little Endian Byte order

由 淺唱寂寞╮ 提交于 2019-11-26 06:15:50
问题 What is the difference between Big Endian and Little Endian Byte order ? Both of these seem to be related to Unicode and UTF16. Where exactly do we use this? 回答1: Big-Endian (BE) / Little-Endian (LE) are two ways to organize multi-byte words. For example, when using two bytes to represent a character in UTF-16, there are two ways to represent the character 0x1234 as a string of bytes (0x00-0xFF): Byte Index: 0 1 --------------------- Big-Endian: 12 34 Little-Endian: 34 12 In order to decide
订阅 utf-16