UnicodeDecodeError Sentiment140 Kaggle

问题

I am trying to read the Sentiment140.csv available on Kaggle: https://www.kaggle.com/kazanova/sentiment140

My code is this one:

import pandas as pd
import os

cols = ['sentiment','id','date','query_string','user','text']
BASE_DIR = ''
df = pd.read_csv(os.path.join(BASE_DIR, 'Sentiment140.csv'),header=None, names=cols)

And it gives me this error:

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 80-81: invalid continuation byte

The things I would like to understand are:

1) How do I solve this issue?

2) Where can I see which type of encoding should I use instead of "utf-8", based on the error?

3) Using other encoding methods will cause me other issues later on?

Thanks in advance

P.s. I am using python3 on a mac

回答1:

This works: https://investigate.ai/investigating-sentiment-analysis/cleaning-the-sentiment140-data/

Turns out encoding="latin-1" and you have to specify column names, otherwise it will use the first row as column names. This is how lousy real-world dataset can be haha

来源：https://stackoverflow.com/questions/52016160/unicodedecodeerror-sentiment140-kaggle