What's the fastest way to bulk insert a lot of data in SQL Server (C# client)

Question

I am hitting some performance bottlenecks with my C# client inserting bulk data into a SQL Server 2005 database and I\'m looking for ways in which to speed up the process.

Answer 1

I'm not really a bright guy and I don't have a lot of experience with the SqlClient.SqlBulkCopy method but here's my 2 cents for what it's worth. I hope it helps you and others (or at least causes people to call out my ignorance ;).

You will never match a raw file copy speed unless your database data file (mdf) is on a separate physical disk from your transaction log file (ldf). Additionally, any clustered indexes would also need to be on a separate physical disk for a fairer comparison.

Your raw copy is not logging or maintaining a sort order of select fields (columns) for indexing purposes.

I agree with Portman on creating a nonclustered identity seed and changing your existing nonclustered index to a clustered index.

As far as what construct you're using on the clients...(data adapter, dataset, datatable, etc). If your disk io on the server is at 100%, I don't think your time is best spent analyzing client constructs as they appear to be faster than the server can currently handle.

If you follow Portman's links about minimal logging, I wouldn't think surrounding your bulk copies in transactions would help a lot if any but I've been wrong many times in my life ;)

This won't necessarily help you right now but if you figure out your current issue, this next comment might help with the next bottleneck (network throughput) - especially if it's over the Internet...

Chopeen asked an interesting question too. How did you determine to use 300 record count chunks to insert? SQL Server has a default packet size (I believe it is 4096 bytes) and it would make sense to me to derive the size of your records and ensure that you are making efficient use of the packets transmitting between client and server. (Note, you can change your packet size on your client code as opposed to the server option which would obviously change it for all server communications - probably not a good idea.) For instance, if your record size results in 300 record batches requiring 4500 bytes, you will send 2 packets with the second packet being mostly wasted. If batch record count was arbitrarily assigned, it might make sense to do some quick easy math.

From what I can tell (and remember about data type sizes) you have exactly 20 bytes for each record (if int=4 bytes and smallint=2 bytes). If you are using 300 record count batches, then you are trying to send 300 x 20 = 6,000 bytes (plus I'm guessing a little overhead for the connection, etc). You might be more efficient to send these up in 200 record count batches (200 x 20 = 4,000 + room for overhead) = 1 packet. Then again, your bottleneck still appears to be the server's disk io.

I realize you're comparing a raw data transfer to the SqlBulkCopy with the same hardware/configuration but here's where I would go also if the challenge was mine:

This post probably won't help you anymore as it's rather old but I would next ask what your disk's RAID configuration is and what speed of disk are you using? Try putting the log file on a drive that uses RAID 10 with a RAID 5 (ideally 1) on your data file. This can help reduce a lot of spindle movement to different sectors on the disk and result in more time reading/writing instead of the unproductive "moving" state. If you already separate your data and log files, do you have your index on a different physical disk drive from your data file (you can only do this with clustered indexes). That would allow for not only concurrently updating logging information with data inserting but would allow index inserting (and any costly index page operations) to occur concurrently.