removing accent and special characters [duplicate]

Question

Possible Duplicate:
What is the best way to remove accents in a python unicode string?
Python and character normalization

I would like to remove accents, turn all characters to lowercase, and delete any numbers and special characters.

Example :

Frédér8ic@ --> frederic

Proposal:

def remove_accents(data):
    return ''.join(x for x in unicodedata.normalize('NFKD', data) if \
    unicodedata.category(x)[0] == 'L').lower()

Is there any better way to do this?

Answer 1

A possible solution would be

def remove_accents(data):
    return ''.join(x for x in unicodedata.normalize('NFKD', data) if x in string.printable).lower()

Using NFKD AFAIK is the standard way to normalize unicode to convert it to compatible characters. The rest as to remove the special characters numbers and unicode characters that originated from normalization, you can simply compare with string.ascii_letters and remove any character's not in that set.

Answer 2

Can you convert the string into HTML entities? If so, you can then use a simple regular expression.

The following replacement would work in PHP/PCRE (see my other answer for an example):

'~&([a-z]{1,2})(?:acute|cedil|circ|grave|lig|orn|ring|slash|th|tilde|uml);~i' => '$1'

Then simply convert back from HTML entities and remove any non a-Z char (demo @ CodePad).

Sorry I don't know Python enough to provide a Pythonic answer.