Easiest way to remove unicode representations from a string in python 3?

Question

I have a string in python 3 that has several unicode representations in it, for example:

t = 'R\\u00f3is\\u00edn'

and I want to convert t so that it has the proper representation when I print it, ie:

>>> print(t)
Róisín

However I just get the original string back. I've tried re.sub and some others, but I can't seem to find a way that will change these characters without having to iterate over each one. What would be the easiest way to do so?

Answer 1

You want to use the built-in codec unicode_escape.

If t is already a bytes (an 8-bit string), it's as simple as this:

>>> print(t.decode('unicode_escape'))
Róisín

If t has already been decoded to Unicode, you can to encode it back to a bytes and then decode it this way. If you're sure that all of your Unicode characters have been escaped, it actually doesn't matter what codec you use to do the encode. Otherwise, you could try to get your original byte string back, but it's simpler, and probably safer, to just force any non-encoded characters to get encoded, and then they'll get decoded along with the already-encoded ones:

>>> print(t.encode('unicode_escape').decode('unicode_escape')
Róisín

In case you want to know how to do this kind of thing with regular expressions in the future, note that sub lets you pass a function instead of a pattern for the repl. And you can convert any hex string into an integer by calling int(hexstring, 16), and any integer into the corresponding Unicode character with chr (note that this is the one bit that's different in Python 2—you need unichr instead). So:

>>> re.sub(r'(\\u[0-9A-Fa-f]+)', lambda matchobj: chr(int(matchobj.group(0)[2:], 16)), t)
Róisín

Or, making it a bit more clear:

>>> def unescapematch(matchobj):
...     escapesequence = matchobj.group(0)
...     digits = escapesequence[2:]
...     ordinal = int(digits, 16)
...     char = chr(ordinal)
...     return char
>>> re.sub(r'(\\u[0-9A-Fa-f]+)', unescapematch, t)
Róisín

The unicode_escape codec actually handles \U, \x, \X, octal (\066), and special-character (\n) sequences as well as just \u, and it implements the proper rules for reading only the appropriate max number of digits (4 for \u, 8 for \U, etc., so r'\\u22222' decodes to '∢2' rather than '𢈢'), and probably more things I haven't thought of. But this should give you the idea.

Answer 2

First of all, it is rather confused what you what to convert to.

Just imagine that you may want to convert to 'o' and 'i'. In this case you can just make a map:

mp = {u'\u00f3':'o', u'\u00ed':'i'}

Than you may apply the replacement like:

t = u'R\u00f3is\u00edn'
for i in range(len(t)):
    if t[i] in mp: t[i]=mp[t[i]]
print t

Answer 3

I apologize for posting as a second answer, I don't have the reputation to comment on abarnert's solution.

After using his function to process approximately 50K android strings I noticed that there is yet another small improvement possible for certain use-cases.

I changed the + to {1,4} to deal with the case where valid hex characters follow a 4-digit escape.

I also changed int(escapesequence) to read int(digits)

>>> def unescapematch(matchobj):
...     escapesequence = matchobj.group(0)
...     digits = escapesequence[2:]
...     ordinal = int(digits, 16)
...     char = unichr(ordinal)
...     return char

>>> print re.sub(r'(\\u[0-9A-Fa-f]{1,4})', unescapematch, "Wi\u2011Fi")
Wi‑Fi

>>> print re.sub(r'(\\u[0-9A-Fa-f]+)', unescapematch, "Wi\u2011Fi")
Traceback (most recent call last):
  File "<pyshell#102>", line 1, in <module>
    print re.sub(r'(\\u[0-9A-Fa-f]+)', unescapematch, "Wi\u2011Fi")
  File "C:\Python27\lib\re.py", line 151, in sub
     return _compile(pattern, flags).sub(repl, string, count)
  File "<pyshell#99>", line 5, in unescapematch
    char = unichr(ordinal)
ValueError: unichr() arg not in range(0x10000) (narrow Python build)