cjk

I needed to strip the Chinese out of a bunch of strings today and was looking for a simple Python regex. Any suggestions? cryo The short, but relatively comprehensive answer for narrow Unicode builds of python (excluding ordinals > 65535 which can only be represented in narrow Unicode builds via surrogate pairs): RE = re.compile(u'[⺀-⺙⺛-⻳⼀-⿕々〇〡-〩〸-〺〻㐀-䶵一-鿃豈-鶴侮-頻並-龎]', re.UNICODE) nochinese = RE.sub('', mystring) The code for building the RE, and if you need to detect Chinese characters in the supplementary plane for wide builds: # -*- coding: utf-8 -*- import re LHan = [[0x2E80, 0x2E99], # Han

In your experience which Unicode characters, codepoints, ranges outside the BMP (Basic Multilingual Plane) are the most common so far? These are the ones which require 4 bytes in UTF-8 or surrogates in UTF-16. I would've expected the answer to be Chinese and Japanese characters used in names but not included in the most widespread CJK multibyte character sets, but on the project I do most work on, the English Wiktionary, we have found that the Gothic alphabet is far more common so far. UPDATE I've written a couple of software tools to scan entire Wikipedias for non-BMP characters and found to

I'm using R 2.15.0 on Windows 7 64-bit. I would like to output unicode (CJK) text to a file. The following code shows how a Unicode character sent to write on a UTF-8 file connection does not work as (I) expected: rty <- file("test.txt",encoding="UTF-8") write("在", file=rty) close(rty) rty <- file("test.txt",encoding="UTF-8") scan(rty,what=character()) close(rty) As shown by the output of scan: Read 1 item [1] "<U+5728>" The file was not written with the UTF character itself, but some kind of ANSI-compliant fallback. Can I make it work right the first time (i.e. with a text file that has "在"

To match A to Z, we will use regex: [A-Za-z] How to allow regex to match utf8 characters entered by user? For example Chinese words like 环保部 stema What you are looking for are Unicode properties. e.g. \p{L} is any kind of letter from any language So a regex to match such a Chinese word could be something like \p{L}+ There are many such properties, for more details see regular-expressions.info Another option is to use the modifier Pattern.UNICODE_CHARACTER_CLASS In Java 7 there is a new property Pattern.UNICODE_CHARACTER_CLASS that enables the Unicode version of the predefined character classes