Pandas df.to_csv(“file.csv” encode=“utf-8”) still gives trash characters for minus sign

Question

I\'ve read something about a Python 2 limitation with respect to Pandas\' to_csv( ... etc ...). Have I hit it? I\'m on Python 2.7.3

This turns out trash characters

Answer 1

Your "bad" output is UTF-8 displayed as CP1252.

On Windows, many editors assume the default ANSI encoding (CP1252 on US Windows) instead of UTF-8 if there is no byte order mark (BOM) character at the start of the file. While a BOM is meaningless to the UTF-8 encoding, its UTF-8-encoded presence serves as a signature for some programs. For example, Microsoft Office's Excel requires it even on non-Windows OSes. Try:

df.to_csv('file.csv',encoding='utf-8-sig')

That encoder will add the BOM.