pysqlite2: ProgrammingError - You must not use 8-bit bytestrings

Question

I'm currently persisting filenames in a sqlite database for my own purposes. Whenever I try to insert a file that has a special character (like é etc.), it throws the following error:

pysqlite2.dbapi2.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

When I do "switch my application over to Unicode strings" by wrapping the value sent to pysqlite with the unicode method like: unicode(filename), it throws this error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 66: ordinal not in range(128)

Is there something I can do to get rid of this? Modifying all of my files to conform isn't an option.

UPDATE If I decode the text via filename.decode("utf-8"), I'm still getting the ProgrammingError above.

My actual code looks like this:

cursor.execute("select * from musiclibrary where absolutepath = ?;",
    [filename.decode("utf-8")])

What should my code here look like?

Answer 1

You need to specify the encoding of filename for conversion to Unicode, for example: filename.decode('utf-8'). Just using unicode(...) picks the console encoding, which is often unreliable (and often ascii).

Answer 2

You should pass as Unicode the arguments of your SQL statement.

Now, it all depends on how you obtain the filename list. Perhaps you're reading the filesystem using os.listdir or os.walk? If that is the case, there is a way to have directly the filenames as Unicode just by passing a Unicode argument to either of these functions:
Examples:

• os.listdir(u'.')
• os.walk(u'.')

Of course, you can substitute the u'.' directory with the actual directory whose contents you are reading. Just make sure it's a Unicode string.

Answer 3

Have you tried to pass the unicode string directly:

cursor.execute("select * from musiclibrary where absolutepath = ?;",(u'namé',))

You will need to add the file encoding at the beginning of the script:

# coding: utf-8

Answer 4

You figured this out already, but:

I don't think you could actually get that ProgrammingError exception from cursor.execute("select * from musiclibrary where absolutepath = ?;", [filename.decode("utf-8")]), as the question currently states.

Either the utf-8 decode would explode, or the cursor.execute call would be happy with the result.

Answer 5

Try to change to this:

cursor.execute("select * from musiclibrary where absolutepath = ?;",
    [unicode(filename,'utf8')])

In your filename origin not encode with utf8, change utf8 to your encoding.