Python ASCII codec can't encode character error during write to CSV

Question

I\'m not entirely sure what I need to do about this error. I assumed that it had to do with needing to add .encode(\'utf-8\'). But I\'m not entirely sure if that\'s what I n

Answer 1

I found the easiest option, in addition to Alastair's excellent suggestions, to be using python3 instead of python 2. all it required in my script was to change wb in the open statement to simply w in accordance with Python3's syntax.

Answer 2

Python 2.x CSV library is broken. You have three options. In order of complexity:

1. Edit: See below Use the fixed library https://github.com/jdunck/python-unicodecsv (pip install unicodecsv). Use as a drop-in replacement - Example:

   with open("myfile.csv", 'rb') as my_file:    
       r = unicodecsv.DictReader(my_file, encoding='utf-8')
   

2. Read the CSV manual regarding Unicode: https://docs.python.org/2/library/csv.html (See examples at the bottom)

3. Manually encode each item as UTF-8:

   for cell in row.findAll('td'):
       text = cell.text.replace('[','').replace(']','')
       list_of_cells.append(text.encode("utf-8"))
   

Edit, I found python-unicodecsv is also broken when reading UTF-16. It complains about any 0x00 bytes.

Instead, use https://github.com/ryanhiebert/backports.csv, which more closely resembles Python 3 implementation and uses io module..

Install:

pip install backports.csv

Usage:

from backports import csv
import io

with io.open(filename, encoding='utf-8') as f:
    r = csv.reader(f):