How do I batch sql statements with Golang's database.sql package?
In Java I would do it like this :
// Create a prepared statement String sql = "INSERT INTO my_table VALUES(?)"; PreparedStatement pstmt = connection.prepareStatement(sql); // Insert 10 rows of data for (int i=0; iHow would I achieve the same in Golang?
Notabene: Although this answer is marked as accepted, it is wrong (see discussion in comments below). The only reason this answer still exists is because I'm unable to delete it since it was marked as accepted.
I don't know exactly what batching in Java does on an SQL level, but you can use transactions to achieve a batch execution of several statements at once. Just make sure your DB engine supports it.
func (db *DB) Begin() (*Tx, error)
Begin starts a transaction. The isolation level is dependent on the driver.
func (tx *Tx) Prepare(query string) (*Stmt, error)
Prepare creates a prepared statement for use within a transaction.
Commit commits the transaction.
Since the db.Exec function is variadic, one option (that actually does only make a single network roundtrip) is to construct the statement yourself and explode the arguments and pass them in.
Sample code:
func BulkInsert(unsavedRows []*ExampleRowStruct) error { valueStrings := make([]string, 0, len(unsavedRows)) valueArgs := make([]interface{}, 0, len(unsavedRows) * 3) for _, post := range unsavedRows { valueStrings = append(valueStrings, "(?, ?, ?)") valueArgs = append(valueArgs, post.Column1) valueArgs = append(valueArgs, post.Column2) valueArgs = append(valueArgs, post.Column3) } stmt := fmt.Sprintf("INSERT INTO my_sample_table (column1, column2, column3) VALUES %s", strings.Join(valueStrings, ",")) _, err := db.Exec(stmt, valueArgs...) return err } In a simple test I ran, this solution is about 4 times faster at inserting 10,000 rows than the Begin, Prepare, Commit presented in the other answer - though the actual improvement will depend a lot on your individual setup, network latencies, etc.
If you’re using PostgreSQL then pq supports bulk imports.
Batching is not possible via the interfaces available in database/sql. A particular database driver may support it separately, however. For instance https://github.com/ziutek/mymysql appears to support batching with MySQL.
Expanding on Avi Flax's answer, I needed an ON CONFLICT DO UPDATE clause in my INSERT.
The solution to this is to COPY to a temporary table (set to delete at the end of the transaction) then INSERT from the temporary table to the permanent table.
Here's the code I settled on:
func (fdata *FDataStore) saveToDBBulk(items map[fdataKey][]byte) (err error) { tx, err := fdata.db.Begin() if err != nil { return errors.Wrap(err, "begin transaction") } txOK := false defer func() { if !txOK { tx.Rollback() } }() // The ON COMMIT DROP clause at the end makes sure that the table // is cleaned up at the end of the transaction. // While the "for{..} state machine" goroutine in charge of delayed // saving ensures this function is not running twice at any given time. _, err = tx.Exec(sqlFDataMakeTempTable) // CREATE TEMPORARY TABLE fstore_data_load // (map text NOT NULL, key text NOT NULL, data json) // ON COMMIT DROP if err != nil { return errors.Wrap(err, "create temporary table") } stmt, err := tx.Prepare(pq.CopyIn(_sqlFDataTempTableName, "map", "key", "data")) for key, val := range items { _, err = stmt.Exec(string(key.Map), string(key.Key), string(val)) if err != nil { return errors.Wrap(err, "loading COPY data") } } _, err = stmt.Exec() if err != nil { return errors.Wrap(err, "flush COPY data") } err = stmt.Close() if err != nil { return errors.Wrap(err, "close COPY stmt") } _, err = tx.Exec(sqlFDataSetFromTemp) // INSERT INTO fstore_data (map, key, data) // SELECT map, key, data FROM fstore_data_load // ON CONFLICT DO UPDATE SET data = EXCLUDED.data if err != nil { return errors.Wrap(err, "move from temporary to real table") } err = tx.Commit() if err != nil { return errors.Wrap(err, "commit transaction") } txOK = true return nil } Adapting Andrew's solution for PostgreSQL, which doesn't support the ? placeholder, the following works:
func BulkInsert(unsavedRows []*ExampleRowStruct) error { valueStrings := make([]string, 0, len(unsavedRows)) valueArgs := make([]interface{}, 0, len(unsavedRows) * 3) i := 0 for _, post := range unsavedRows { valueStrings = append(valueStrings, fmt.Sprintf("($%d, $%d, $%d)", i*3+1, i*3+2, i*3+3)) valueArgs = append(valueArgs, post.Column1) valueArgs = append(valueArgs, post.Column2) valueArgs = append(valueArgs, post.Column3) i++ } stmt := fmt.Sprintf("INSERT INTO my_sample_table (column1, column2, column3) VALUES %s", strings.Join(valueStrings, ",")) _, err := db.Exec(stmt, valueArgs...) return err }