how to transform pandas dataframe for insertion via executemany() statement?

Question

I have a fairly big pandas dataframe - 50 or so headers and a few hundred thousand rows of data - and I\'m looking to transfer this data to a database using the

Answer 1

Might be a little late to answer this question, but maybe it can still help someone. executemany() is not implemented by many ODBC. One of the ones that does have it is MySQL. When they refer to sequence of parameters they mean:

parameters=[{'name':'Jorge', 'age':22, 'sex':'M'}, 
            {'name':'Karen', 'age':25, 'sex':'F'}, 
            {'name':'James', 'age':29, 'sex':'M'}]

and for a query statement it would look something like:

SQL = INSERT IGNORE INTO WORKERS (NAME, AGE, SEX) VALUES (%(name)s, %(age)s, %(sex)s)

Which looks like you got there. A couple things though I want to point out in case it helps: pandas has a to_sql function that inserts into a db if you provide it the connector object, and chunks the data as well.

To rapidly create a sequence of parameters from a pandas dataframe I found the following two methods helpful:

# creates list of dict, list of parameters
# REF: https://groups.google.com/forum/#!topic/pydata/qna3Z3WmVpM
parameters = [df.iloc[line, :].to_dict() for line in range(len(df))]

# Cleaner Way
parameters = df.to_dict(orient='records')