Getting ExecuteBatch to execute faster

Question

I\'m trying to read a table from a sybase server, process the rows, and output the results to another table. (Below is my code)

The code retrieves the table pretty

Answer 1

To work with batches effectively you should turn AutoCommit option off and turn it on after executing the batch (or alternatively use connection.commit() method)

connection.setAutoCommit(false);
while(rs.next())
    {
     .....
     ps.addBatch();     
    }
int[] results = ps.executeBatch();
connection.setAutoCommit(true);

Answer 2

Add ?rewriteBatchedStatements=true to the end of your JDBC url. It'll give you a serious performance improvement. Note that this is specific to MySql, won't have any effect with any other JDBC drivers.

Eg : jdbc:mysql://server:3306/db_name?rewriteBatchedStatements=true

It improved my performance by more than 15 times

Answer 3

I had this same problem, finally figured it out though I also was not able to find the right explanation anywhere.

The answer is that for simple un-conditioned inserts .executeBatch() should not be used. What batch mode is doing is making lots of individual "insert into table x ..." statements and that is why it is running slow. However if the insert statements were more complex, possibly with conditions that affect each row differently, then it might require individual insert statements and a batch execution would actually be useful.

An example of what works, try the following which creates a single insert statement as a PreparedStatement (but same concept as a Statement object would require), and solves the problem of running slow:

public boolean addSetOfRecords(String tableName, Set<MyObject> objects) {
    StringBuffer sql = new StringBuffer("INSERT INTO " + tableName + " VALUES (?,?,?,?)");
    for(int i=1;i<objects.size();i++) {
        sql.append(",(?,?,?,?)");
    }
    try {
        PreparedStatement p = db.getConnection().prepareStatement(sql.toString());
        int i = 1;
        for(MyObject obj : objects) {
            p.setString(i++, obj.getValue());
            p.setString(i++, obj.getType());
            p.setString(i++, obj.getId());
            p.setDate(i++, new Date(obj.getRecordDate().getTime()));
        }
        p.execute();
        p.close();
        return true;
    } catch (SQLException e) {
        e.printStackTrace();
        return false;
    }
}

Answer 4

There is a commercial solution from Progress DataDirect to translate JDBC batches into the database's native bulk load protocol to significantly improve performance. It's very popular with SQL Server since it does not require BCP. I am employed by that vendor and wrote a blog on how to bulk insert JDBC batches.