Why is executemany slow in Python MySQLdb?

Question

I am developing a program in Python that accesses a MySQL database using MySQLdb. In certain situations, I have to run an INSERT or REPLACE command on many rows. I am curren

Answer 1

Try lowercasing the word 'values' in your query - this appears to be a bug/regression in MySQL-python 1.2.3.

MySQL-python's implementation of executemany() matches the VALUES clause with a regular expression and then just clones the list of values for each row of data, so you end up executing exactly the same query as with your first approach.

Unfortunately the regular expression lost its case-insensitive flag in that release (subsequently fixed in trunk r622 but never backported to the 1.2 branch) so it degrades to iterating over the data and firing off a query per row.

Answer 2

Strongly do not recommend to use executeMany in pyodbc as well as ceodbc both slow and contains a lot of bugs.

Instead consider use execute and manually construct SQL query using simple string format.

transaction = "TRANSACTION BEGIN {0} COMMIT TRANSACTION

bulkRequest = ""
for i in range(0, 100)
    bulkRequest = bulkRequest + "INSERT INTO ...... {0} {1} {2}"

ceodbc.execute(transaction.format(bulkRequest))

Current implementation is very simple fast and reliable.

Answer 3

Your first example is a single (large) statement that is generated and then sent to the database.

The second example is a much simpler statement that inserts/replaces a single row but is executed multiple times. Each command is sent to the database separately so you have to pay the turnaround time from client to server and back for every row inserted. I would think that this extra latency introduced between the commands is the main reason for the decreased performance of the second example.

Answer 4

In case you're using mysqlclient-python (fork of MySQLdb1), also the recommended driver for Django (by Django), there's the following usecase you need to know of:

cursor.executemany falls back to using cursor.execute (silently) in case your query is of the form:

INSERT INTO testdb.test (type, some_field, status, some_char_field) VALUES (%s, hex(%s), %s, md5(%s));

The driver employs a python regex that doesn't seem to support the use of mysql functions in the VALUES clause.

    RE_INSERT_VALUES = re.compile(
    r"\s*((?:INSERT|REPLACE)\b.+\bVALUES?\s*)" +
    r"(\(\s*(?:%s|%\(.+\)s)\s*(?:,\s*(?:%s|%\(.+\)s)\s*)*\))" +
    r"(\s*(?:ON DUPLICATE.*)?);?\s*\Z",
    re.IGNORECASE | re.DOTALL)

Link to the relevant github issue https://github.com/PyMySQL/mysqlclient-python/issues/334