SQLAlchemy INSERT IGNORE

Question

How can I insert multiple data records into table ignoring duplicates. I am using SQLAlchemy.

Thank you!

Answer 1

To always replace INSERT by INSERT OR IGNORE, you can use a compiler extension:

from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql.expression import Insert

@compiles(Insert)
def _prefix_insert_with_ignore(insert, compiler, **kw):
    return compiler.visit_insert(insert.prefix_with('OR IGNORE'), **kw)

Or to do this only temporarily, call the compiles decorator manually and use deregister once you're done:

from sqlalchemy.ext.compiler import compiles, deregister
from sqlalchemy.sql.expression import Insert

def _prefix_insert_with_ignore(insert, compiler, **kw):
    return compiler.visit_insert(insert.prefix_with('OR IGNORE'), **kw)

compiles(Insert)(_prefix_insert_with_replace)
try:
    # do some inserts...
finally:
    deregister(Insert)

This does feel hacky because it's still a global change, but as long as you don't use threads and make sure everything is properly committed before the deregister call, it's probably okay.

Answer 2

prefix_with("TEXT") adds arbitrary text between INSERT and the rest of the SQL. execute() accepts a list of dictionaries with the records you would like to insert or a single dictionary if you only want to insert a single record.

The SQLite syntax for the behavior you're looking for:

inserter = table_object.insert().prefix_with("OR REPLACE")
inserter.execute([{'column1':'value1'}, {'column1':'value2'}])