pandas to_sql gives unicode decode error

Question

I have a pandas dataframe I loaded via read_csv that I am trying to push to a database via to_sql when I attempt

df.to_sql(\"assessmentinfo_pivot\", util.ENGINE)         


        

Answer 1

I experienced a similar problem on python 3.7.: UnicodeEncodeError: 'charmap' codec can't encode character '\ufffd' in position 0: character maps to

It was the way I defined my engine. I had charset defined to utf-8 in my engine, yet it did not pick it up:

# Connecting to the database(reference for checkout_listener not added)
def MysqlConnection(DbName):
    DB_TYPE = 'mysql'
    DB_DRIVER = 'mysqldb'
    DB_NAME = DbName
    POOL_SIZE = 100
    CHARSET = 'utf-8'

    SQLALCHEMY_DATABASE_URI = '%s+%s://%s:%s@%s:%s/%s?%s' % (DB_TYPE, DB_DRIVER, DB_USER,
                                                             DB_PASS, DB_HOST, DB_PORT, DB_NAME, CHARSET)
    ENGINE1 = create_engine(
        SQLALCHEMY_DATABASE_URI, pool_size=POOL_SIZE, pool_recycle=3600, echo=False)
    event.listen(ENGINE1, 'checkout', checkout_listener)
    return (ENGINE1);

This worked fine on python 2 but on python 3, the charmap error would occur. The only solution I found was to write engine in a different manner, and add charset to the definition string:

connection_string = f"{mysql_user}:{mysql_password}@localhost:3306/{db_name}?charset=utf8"
engine = create_engine(f'mysql://{connection_string}')