parsing XML file gets UnicodeEncodeError (ElementTree) / ValueError (lxml)

Question

I send a GET request to the CareerBuilder API :

import requests

url = \"http://api.careerbuilder.com/v1/jobsearch\"
payload = {\'DeveloperKey\': \'MY_DEVLOP         


        

Answer 1

Understand the question has already got its answer, I faced this similar issue on Python3 and it worked fine on Python2. My resolution was: str_xml.encode() and then xml = etree.fromstring(str_xml) and then the parsing and extractions of tags and attributes.

Answer 2

You are using the decoded unicode value. Use r.raw raw response data instead:

r = requests.get(url, params=payload, stream=True)
r.raw.decode_content = True
etree.parse(r.raw)

which will read the data from the response directly; do note the stream=True option to .get().

Setting the r.raw.decode_content = True flag ensures that the raw socket will give you the decompressed content even if the response is gzip or deflate compressed.

You don't have to stream the response; for smaller XML documents it is fine to use the response.content attribute, which is the un-decoded response body:

r = requests.get(url, params=payload)
xml = etree.fromstring(r.content)

XML parsers always expect bytes as input as the XML format itself dictates how the parser is to decode those bytes to Unicode text.

Answer 3

Correction!

See below how I got it all wrong. Basically, when we use the method .text then the result is a unicode encoded string. Using it raises the following exception in lxml

ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

Which basically means that @martijn-pieters was right, we must use the raw response as returned by .content

Incorrect answer (but might be interesting to someone)

For whoever is interested. I believe the reason this error occurs is probably an invalid guess taken by requests as explained in Response.text documentation:

Content of the response, in unicode.

If Response.encoding is None, encoding will be guessed using chardet.

The encoding of the response content is determined based solely on HTTP headers, following RFC 2616 to the letter. If you can take advantage of non-HTTP knowledge to make a better guess at the encoding, you should set r.encoding appropriately before accessing this property.

So, following this, one could also make sure requests' r.text encodes the response content correctly by explicitly setting the encoding with r.encoding = 'UTF-8'

This approach adds another validation that the received response is indeed in the correct encoding prior to parsing it with lxml.