Python 3 chokes on CP-1252/ANSI reading

Question

I\'m working on a series of parsers where I get a bunch of tracebacks from my unit tests like:

  File \"c:\\Python31\\lib\\encodings\\cp1252.py\", line 23, i         


        

Answer 1

All files are "not Unicode". Unicode is an internal representation which must be encoded. You need to determine for each file what encoding has been used, and specify that where necessary when the file is opened.

As the traceback and error message indicate, the file in question is NOT encoded in cp1252.

If it is encoded in latin1, the "\x81" that it is complaining about is a C1 control character that doesn't even have a name (in Unicode). Consider latin1 extremely unlikely to be valid.

You say "some of the files are parsed with xml.dom.minidom" -- parsed successfully or unsuccessfully?

A valid XML file should declare its encoding (default is UTF-8) in the first line, and you should not need to specify an encoding in your code. Show us the code that you are using to do the xml.dom.minidom parsing.

"others read directly as iterables" -- sample code please.

Suggestion: try opening some of each type of file in your browser. Then click View and click Character Encoding (Firefox) or Encoding (Internet Explorer). What encoding has the browser guessed [usually reliably]?

Other possible encoding clues: What languages are used in the text in the files? Where did you get the files from?

Note: please edit your question with clarifying information; don't answer in the comments.