Difference between open and codecs.open in Python

Question

There are two ways to open a text file in Python:

f = open(filename)

And

import codecs
f = codecs.open(filename, encoding=\         


        

Answer 1

In Python 2 there are unicode strings and bytestrings. If you just use bytestrings, you can read/write to a file opened with open() just fine. After all, the strings are just bytes.

The problem comes when, say, you have a unicode string and you do the following:

>>> example = u'Μου αρέσει Ελληνικά'
>>> open('sample.txt', 'w').write(example)
Traceback (most recent call last):
File "", line 1, in 
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

So here obviously you either explicitly encode your unicode string in utf-8 or you use codecs.open to do it for you transparently.

If you're only ever using bytestrings then no problems:

>>> example = 'Μου αρέσει Ελληνικά'
>>> open('sample.txt', 'w').write(example)
>>>

It gets more involved than this because when you concatenate a unicode and bytestring string with the + operator you get a unicode string. Easy to get bitten by that one.

Also codecs.open doesn't like bytestrings with non-ASCII chars being passed in:

codecs.open('test', 'w', encoding='utf-8').write('Μου αρέσει')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.7/codecs.py", line 691, in write
    return self.writer.write(data)
  File "/usr/lib/python2.7/codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 0: ordinal not in range(128)

The advice about strings for input/ouput is normally "convert to unicode as early as possible and back to bytestrings as late as possible". Using codecs.open allows you to do the latter very easily.

Just be careful that you are giving it unicode strings and not bytestrings that may have non-ASCII characters.