Speed up to_sql() when writing Pandas DataFrame to Oracle database using SqlAlchemy and cx_Oracle

Question

Using pandas dataframe\'s to_sql method, I can write a small number of rows to a table in oracle database pretty easily:

from sqlalchemy import create_engine         


        

Answer 1

Just commenting here for posterity. I'm on python 3.6.8, pandas 1.1.3, sqlalchemy 1.3.20. When I tried implementing the solution from MaxU, I was initially encountering an error:

raise ValueError(f"{col} ({my_type}) not a string")

I honestly don't know why. After spending a couple hours debugging, this is what finally worked for me. In my case, I was trying to read from a CSV and insert to Oracle:

import cx_Oracle
import numpy as np
import pandas as pd
import sqlalchemy as sa
from sqlalchemy import create_engine

conn = create_engine('oracle://{}:{}@{}'.format(USERNAME, PASSWORD, DATABASE))

df = pd.read_csv(...)
object_columns = [c for c in df.columns[df.dtypes == 'object'].tolist()]
dtyp = {c:sa.types.VARCHAR(df[c].str.len().max()) for c in object_columns}

df.to_sql(..., dtype=dtyp)

To be honest, I didn't really change much so not 100% sure why I was getting the original error, but just posting here in case it's helpful.