Python Requests encoding POST data

Question

Version: Python 2.7.3

Other libraries: Python-Requests 1.2.3, jinja2 (2.6)

I have a script that submits data to a forum and the problem is that non-ascii cha

Answer 1

Your client behaves as it should e.g. running nc -l 8888 as a server and making a request:

import requests

requests.post('http://localhost:8888', data={u'post': u'Andr\xe9 T\xe9chin\xe9'})

shows:

POST / HTTP/1.1
Host: localhost:8888
Content-Length: 33
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.2.3 CPython/2.7.3

post=Andr%C3%A9+T%C3%A9chin%C3%A9

You can check that it is correct:

>>> import urllib
>>> urllib.unquote_plus(b"Andr%C3%A9+T%C3%A9chin%C3%A9").decode('utf-8')
u'Andr\xe9 T\xe9chin\xe9'

• check the server decodes the request correctly. You could try to specify the charset:

  headers = {"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8"}
  

  the body contains only ascii characters so it shouldn't hurt and the correct server would ignore any parameters for x-www-form-urlencoded type anyway. Look for gory details in URL-encoded form data

• check the issue is not a display artefact i.e., the value is correct but it displays incorrectly

Answer 2

Try to decode into utf8:

unicode(my_string_variable, "utf8")

or decode and encode:

sometext = gettextfromsomewhere().decode('utf-8')
env = jinja2.Environment(loader=jinja2.PackageLoader('jinjaapplication', 'templates'))
template = env.get_template('mypage.html')
print template.render( sometext = sometext ).encode('utf-8')