What is the fastest way to insert 100 000 records into an MDB file in C#

Question

I know this questions is rather general, but I have searched the whole day and I haven't been able to find the proper way to do this.

Here is my code to insert some 100 000 dummy records into an MDB file using C#.

OleDbConnection con = new OleDbConnection();
string dbProvider = "PROVIDER=Microsoft.Jet.OLEDB.4.0;";
string dbSource = "Data Source = D:/programming/sample.mdb";
con.ConnectionString = dbProvider + dbSource;
OleDbCommand cmd = new OleDbCommand();
cmd.Connection = con;
cmd.CommandText = "INSERT INTO tblBooks (Title, Price, Tag, Author) VALUES (@title, @price, @tag, @author)";
cmd.Parameters.AddWithValue("@title", "Dummy Text 1");
cmd.Parameters.AddWithValue("@price", 10);
cmd.Parameters.AddWithValue("@tag", "Dummy Text 2");
cmd.Parameters.AddWithValue("@author", "Dummy Text 3");
con.Open();

for (int i = 0; i < 100000; i++)
{
    cmd.ExecuteNonQuery();    
}

 con.Close();

This code takes around a minute to run. Is this normal? What is the proper way to do this faster?

Answer 1

If you happen to already have a "numbers table" available (with at least 100,000 rows) then Remou's answer will almost certainly get the job done fastest. I tried a quick test in VBA and the query

Dim t0 As Single
t0 = Timer
CurrentDb.Execute _
        "INSERT INTO tblBooks (Title, Price, Tag, Author) " & _
        "SELECT 'Dummy Text 1', 10, 'Dummy Text 2', 'Dummy Text 3' FROM Numbers", _
        dbFailOnError
Debug.Print Format(Timer - t0, "0.0") & " seconds"

created the 100,000 rows in less than 2 seconds.

However, if you don't already have a numbers table then you would need to create that table first, so if this is a one-time requirement then you might be better off just optimizing your code.

The code as posted in your question took 45 seconds on my machine. Two enhancements that significantly reduced the execution time were:

1. Use .Prepare(): that alone reduced the elapsed time to 16 seconds

2. Use an OleDbTransaction: Wrapping the inserts in a transaction (in addition to using .Prepare()) further reduced the elapsed time to 10 seconds.

The modified code looks like this:

var sw = new System.Diagnostics.Stopwatch();
sw.Start();
OleDbConnection con = new OleDbConnection();
string dbProvider = "PROVIDER=Microsoft.Jet.OLEDB.4.0;";
string dbSource = "Data Source = C:/Users/Gord/Desktop/speed.mdb";
con.ConnectionString = dbProvider + dbSource;
con.Open();
OleDbCommand cmd = new OleDbCommand();
cmd.Connection = con;
cmd.CommandText = "INSERT INTO tblBooks (Title, Price, Tag, Author) VALUES (?,?,?,?)";
cmd.Parameters.Add("?", OleDbType.VarWChar, 255);
cmd.Parameters.Add("?", OleDbType.Currency);
cmd.Parameters.Add("?", OleDbType.VarWChar, 255);
cmd.Parameters.Add("?", OleDbType.VarWChar, 255);
cmd.Prepare();
cmd.Parameters[0].Value = "Dummy Text 1";
cmd.Parameters[1].Value = 10;
cmd.Parameters[2].Value = "Dummy Text 2";
cmd.Parameters[3].Value = "Dummy Text 3";
OleDbTransaction trn = con.BeginTransaction();
cmd.Transaction = trn;
for (int i = 0; i < 100000; i++)
{
    cmd.ExecuteNonQuery();
}
trn.Commit();
con.Close();
sw.Stop();
Console.WriteLine(String.Format("{0:0.0} seconds", sw.ElapsedMilliseconds / 1000.0));

Answer 2

(Bonus answer:)

In case anyone is wondering whether an OleDbDataAdapter can insert the rows faster, it seems not. The following code does create the 100,000 records...

var da = new OleDbDataAdapter("SELECT [ID], [Title], [Price], [Tag], [Author] FROM [tblBooks] WHERE False", con);
var cb = new OleDbCommandBuilder(da);
cb.QuotePrefix = "["; cb.QuoteSuffix = "]";
var dt = new System.Data.DataTable();
da.Fill(dt);
for (int i = 0; i < 100000; i++)
{
    System.Data.DataRow dr = dt.NewRow();
    dr["Title"] = "Dummy Text 1";
    dr["Price"] = 10;
    dr["Tag"] = "Dummy Text 2";
    dr["Author"] = "Dummy Text 3";
    dt.Rows.Add(dr);
}
da.Update(dt);

...but it takes about 30% longer to run than the original code in the question.

Answer 3

You can use a numbers table to add multiple identical rows, for example:

INSERT INTO aTable ( aText, aNumber )
SELECT @param1 , @param2 
FROM Numbers
WHERE Numbers.Number<1000

The Numbers table is:

Number
0
1
2
<...>