Why does locale.getpreferredencoding() return 'ANSI_X3.4-1968' instead of 'UTF-8'?

Question

Whenever I try to read UTF-8 encoded text files, using open(file_name, encoding=\'utf-8\'), I always get an error saying ASCII codec can\'t decode some characte

Answer 1

I solved it by running the following:

apt install locales-all

Answer 2

I think you are misreading the error message. Be careful at distinguishing UnicodeDecodeError and UnicodeEncodeError.

You say that Python complains that “ascii codec can't decode some characters”. However, there is no such error message, as far as I know. Compare the following two cases:

>>> b = 'é'.encode('utf8')
>>> b.decode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can’t decode byte 0xc3 in position 0: ordinal not in range(128)
>>> 'é'.encode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can’t encode character '\xe9' in position 0: ordinal not in range(128)

It's either “can't decode byte” or “can't encode character”, but it's never “can't decode character”.

This might seem pedantic, but in this line,

for line in f: print(line)

you have both decoding (before the colon) and encoding (the print expression). So you need to be sure which process is causing trouble. One possibility would be to write this in two lines.

However, if f is opened with encoding='utf-8', as you write, then I'm pretty sure the problem is caused by the print expression. print() writes to sys.stdout by default. Since this stream is already open when Python is started, its encoding is already set as well – depending on your environment. Since in your locale LC_ALL is not set, the ASCII default (“ANSI X3.4-1968”) is used (this might answer your question in the title).

If you can't or don't want to change the locale, here's what you can do to send UTF-8 text to STDOUT from within Python:

• use the underlying binary stream:

  for line in f:
      sys.stdout.buffer.write(line.encode('utf-8')
  

• re-encode sys.stdout (actually: replace sys.stdout with a re-encoded version):

  import codecs
  sys.stdout = codecs.getwriter('utf-8')(sys.stdout.buffer)
  

In any case, it's still possible that your terminal is unable to properly display UTF-8 text, either because it's uncapable of that or because it's not configured to do so. In that case, you'll probably see question marks or mojibake. But that's a different story, outside of Python's control...

Answer 3

I had a similar problem. For me, initially the environtment variable LANG was not set (you can check this by running env)

$ python3 -c 'import locale; print(locale.getdefaultlocale())'
(None, None)
$ python3 -c 'import locale; print(locale.getpreferredencoding())'
ANSI_X3.4-1968

The available locales for me was (on a fresh Ubuntu 18.04 Docker image):

$ locale -a
C
C.UTF-8
POSIX

So i picked the utf-8 one:

$ export LANG="C.UTF-8"

And then things work

$ python3 -c 'import locale; print(locale.getdefaultlocale())'
('en_US', 'UTF-8')
$ python3 -c 'import locale; print(locale.getpreferredencoding())'
UTF-8

---

If you pick a locale that is not avaiable, such as

export LANG="en_US.UTF-8"

it will not work:

$ python3 -c 'import locale; print(locale.getdefaultlocale())'
('en_US', 'UTF-8')
$ python3 -c 'import locale; print(locale.getpreferredencoding())'
ANSI_X3.4-1968

and this is why locale is giving the error messages:

locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory