Pandas update sql

Question

Is there any way to do an SQL update-where from a dataframe without iterating through each line? I have a postgresql database and to update a table in the db from a datafram

Answer 1

I have so far not seen a case where the pandas sql connector can be used in any scalable way to update database data. It may have seemed like a good idea to build one, but really, for operational work it just does not scale.

What I would recommend is to dump your entire dataframe as CSV using

df.to_csv('filename.csv', encoding='utf-8')

Then loading the CSV into the database using COPY for PostgreSQL or LOAD DATA INFILE for MySQL.

If you do not make other changes to the table in question while the data is being manipulated by pandas, you can just load into the table.

If there are concurrency issues, you will have to load the data into a staging table that you then use to update your primary table from.

In the later case, your primary table needs to have a datetime which tells you when the latest modification to it was so you can determine if your pandas changes are the latest or if the database changes should remain.