UnicodeEncodeError [Python3/Gunicorn/Nginx/Django]

Question

Solved

This error happened when I follow the instruction in here. I set Unit file. I understood that LAN env valuable doens't transfer properly and it used default ascii. You can fix this problem with adding one line in unit file. It was a really long journey to find out....

[service]
Environment="LANG=en_US.UTF-8"

---

I didn't have any problem when I set my environment with mod_wsgi and Apache2. I just had to add "export LANG='en_US.UTF-8' export LC_ALL='en_US.UTF-8'" in /etc/apache2/envars path to upload file with non-ASCII filename.

This time, I set my env with Nginx and Gunicorn. But It occurs UnicodeEncoderError in every part where shows non-ASCII characters.

'ascii' codec can't encode characters in position 57-59: ordinal not in range(128)

I don't understand why 'ascii' codec is used to encode in Python3 and Django environment.

I searched and searched and searched. I checked and tried like below.

1. PostgreSQL check : Encoding UTF8
2. Django default encoding : utf-8
3. Ubuntu locale check : en_US.UTF-8 (I also tried 'ko_KR.UTF-8', because it has an error with Korean)
4. checked python 3 sys.getdefaultencoding, sys.stdout.encoding, sys.stdin.encoding : utf-8
5. adding charset utf-8; in /etc/nginx/sites-available/myproject
6. checked if Gunicorn call python2 instead of python3 : installing gunicorn with pip3 virtualenv and checked '#!/home/username/venv/bin/python3' in gunicorn file.
7. write import sys reload(sys) sys.setdefaultencoding('utf-8') in views.py : I understood it can apply to python2 and it is not recommended way.

I was suspicous that Gunicorn call python2 instead of python3 because python2 is also installed in my Ubuntu. If I check python -c 'import sys; print(sys.getdefaultencoding());', I can see 'ascii'. Python 2 has str and unicode. I concluded that Gunicorn doesn't have problem with it at the end.

I am still suspicious of python 2.7 which was preinstalled in Ubuntu16.04. Can you explain that What i am missing or misunderstanding?? I understood that ascii codec encode error could happen with Python2, not Python3.

Env Python 3.5 Django 1.11.1 Gunicorn 19.7.1 Nginx 1.10.3 Ubuntu 16.04 LTS

Answer 1

For people who are having the encoding problem with gunicorn but not using it as a daemon (for example if you run gunicorn in docker you just launch the command), and therefore, not using this configuration files:

I'm using the openpyxl library to generate a spreadsheet, and I got this error when testing it in the server in which I have the service dockerized with gunicorn. After confirming that every other encoding requirement was fine (python, system locales, etc.) it was clear that the problem was gunicorn. Everything else in the app works fine, it's only the spreadsheet exportation that throws this.

I was about to try replacing gunicorn with something else but then I realized that the problem is with file names and not with the process of generating the contents of the file.

I was naming the generated file "exportació", and just replacing it with "exportacio" made it work like a charm.

Given that I really don't need non-ascii characters in the file name, that works enough for me.