Is there a faster way to use SqlBulkCopy than using a DataTable?

Question

I load a large amount of records into my application (1 million+) and do a ton of processing on them. The processing requires them all to be in the memory.

Afterwards, I want to dump all of the (now modified) records into an empty table.

Loading the records takes mere seconds, and I end up with a large array of MyRecord items.

Saving using SqlBulkCopy takes mere seconds as well.

However SqlBulkCopy requires (I believe) a DataTable - and loading my records into a DataTable is slow - approximately 7500 records per minute using

dataTable.Rows.Add(myRecord.Name, myRecord.Age, ....)

Is there a faster way of performing this middle step?

Answer 1

The delay is caused because you have to buffer everything into a DataTable before sending it to the server. To get better performance you should send the records to SqlBulkCopy immediatelly, and let the class use its own buffering and batching.

SqlBulkCopy can work with an IDataReader. All ADO.NET data readers implement this interface, so you can push data that you read from any data reader to SqlBulkCopy.

In other cases, assuming you have an IEnumerable of your objects, you can use Marc Gravel's ObjectReader from the FastMember package to create an IDataReader on top of the IEnumerable. This data reader does not load everything at once, so no data is cached until SqlBulkCopy asks for it :

Copying Marc Gravel's example:

IEnumerable<SomeType> data = ... 

using(var bcp = new SqlBulkCopy(connection)) 
using(var reader = ObjectReader.Create(data, "Id", "Name", "Description")) 
{ 
  bcp.DestinationTableName = "SomeTable"; 
  bcp.WriteToServer(reader); 
}

Answer 2

I don't know what the issue is. Program below runs under a second. I suspect the slow speed is due to reading data and not writing to DataTable.

       static void Main(string[] args)
        {
            DataTable dt = new DataTable();
            dt.Columns.Add("Col A", typeof(int));
            dt.Columns.Add("Col B", typeof(string));
            dt.Columns.Add("Col C", typeof(int));
            dt.Columns.Add("Col D", typeof(string));
            dt.Columns.Add("Col E", typeof(int));
            dt.Columns.Add("Col F", typeof(string));
            dt.Columns.Add("Col G", typeof(int));
            dt.Columns.Add("Col H", typeof(string));
            dt.Columns.Add("Col I", typeof(int));
            dt.Columns.Add("Col J", typeof(string));

            DateTime begin = DateTime.Now;

            for (int i = 0; i < 7500; i++)
            {
                dt.Rows.Add(new object[] {
                    i + 10000, "b", i + 20000, "d", i + 30000, "f", i + 40000, "h", i + 50000, "i"
                });
            }

            DateTime end = DateTime.Now;

            Console.WriteLine((end - begin).ToString());

            Console.ReadLine();
        }