cjk

I'm a beginner having trouble decoding several dozen CSV file with numbers + (Simplified) Chinese characters to UTF-8 in Python 2.7. I do not know the encoding of the input files so I have tried all the possible encodings I am aware of -- GB18030, UTF-7, UTF-8, UTF-16 & UTF-32 (LE & BE). Also, for good measure, GBK and GB3212, though these should be a subset of GB18030. The UTF ones all stop when they get to the first Chinese characters. The other encodings stop somewhere in the first line except GB18030. I thought this would be the solution because it read through the first few files and

Where can I find a Unicode table showing only the simplified Chinese characters? I have searched everywhere but cannot find anything. UPDATE : I have found that there is another encoding called GB 2312 - http://en.wikipedia.org/wiki/GB_2312 - which contains only simplified characters. Surely I can use this to get what I need? I have also found this file which maps GB2312 to Unicode - http://cpansearch.perl.org/src/GUS/Unicode-UTF8simple-1.06/gb2312.txt - but I'm not sure if it's accurate or not. If that table isn't correct maybe someone could point me to one that is, or maybe just a table of

I want to split a sentence into a list of words. For English and European languages this is easy, just use split() >>> "This is a sentence.".split() ['This', 'is', 'a', 'sentence.'] But I also need to deal with sentences in languages such as Chinese that don't use whitespace as word separator. >>> u"这是一个句子".split() [u'\u8fd9\u662f\u4e00\u4e2a\u53e5\u5b50'] Obviously that doesn't work. How do I split such a sentence into a list of words? UPDATE: So far the answers seem to suggest that this requires natural language processing techniques and that the word boundaries in Chinese are ambiguous. I'm

DOMDocument seems to convert Chinese characters into codes, for instance, 你的乱发 will become ä½ çš„ä¹±å‘ How can I keep the Chinese or other foreign language as they are instead of converting them into codes? Below is my simple test, $dom = new DOMDocument(); $dom->loadHTML($html); If I add this below before loadHTML(), $html = mb_convert_encoding($html, "HTML-ENTITIES", "UTF-8"); I get, 你的乱发 Even though the coverted codes will be displayed as Chinese characters, 你的乱发 still are not 你的乱发 what I am after.... DOMDocument seems to convert Chinese characters into codes [...]. How can I keep the

How to determine if a character is a Chinese character using ruby？ An interesting article on encodings in Ruby: http://blog.grayproductions.net/articles/bytes_and_characters_in_ruby_18 (it's part of a series - check the table of contents at the start of the article also) I haven't used chinese characters before but this seems to be the list supported by unicode: http://en.wikipedia.org/wiki/List_of_CJK_Unified_Ideographs . Also take note that it's a unified system including Japanese and Korean characters (some characters are shared between them) - not sure if you can distinguish which are

I created an application that works perfectly until the user selects 125% or 150%. It would break my application. I later found a way to find the font size by detecting the DPI. This was working great until people with Chinese versions of Windows 7 started using my application. The entire application breaks on Chinese Windows 7. From what I can tell (I can't really test it for I only have the English version and installation the language packs does not cause the problem) Chinese characters are causing a weird DPI that breaks my application. My current code works like this: if (dpi.DpiX == 120)