Upload CSV file to SQL server

Question

What is the best way to upload a large csv data file into SQL server using C# ? The file contains about 30,000 rows and 25 columns.

Answer 1

The best way I found to import large CSV files into SQL Server is by using SqlBulkCopy along with IDataReader implementation. The good thing about it is that you're not reading the entire file into memory (which is the case using DataTable approach) and you can control the size of the batch which gets sent to SQL Server. The bad thing about it is that you have to implement IDataReader which is one of the longest MS interfaces I've seen.

I wrote a nuget package that does the trick for you. It uses the awesome CsvHelper package so there's very little config required. The simplest scenario would look like this:

//Instantiate the reader, providing the list of columns which matches 1 to 1 the data table structure.
var dataReader = new CsvDataReader(filePath,
    new List(5)
    {
        TypeCode.String,
        TypeCode.Decimal,
        TypeCode.String,
        TypeCode.Boolean,
        TypeCode.DateTime
    });

bulkCopyUtility.BulkCopy("TableName", dataReader);

There are also additional configuration options for more complex scenarios (flexible column mapping, additional static column values which are not present in the csv file, value transformation). If you're interested, the project is on Github and available as a nuget package.

For reference, here's how to use SqlBulkCopy with IDataReader:

public void BulkCopy(string tableName, IDataReader dataReader, Action  configureSqlBulkCopy)
{
    using (SqlConnection dbConnection = new SqlConnection(connectionString))
    {
        dbConnection.Open();

        using (SqlBulkCopy bulkCopy = new SqlBulkCopy(dbConnection))
        {
            bulkCopy.BatchSize = 3000; //Data will be sent to SQL Server in batches of this size
            bulkCopy.EnableStreaming = true;
            bulkCopy.DestinationTableName = tableName;

            //This will ensure mapping based on names rather than column position
            foreach (DataColumn column in dataReader.GetSchemaTable().Columns)
            {
                bulkCopy.ColumnMappings.Add(column.ColumnName, column.ColumnName);
            }

            //If additional, custom configuration is required, invoke the action
            configureSqlBulkCopy?.Invoke(bulkCopy);

            try
            {
                // Write from the source to the destination.
                bulkCopy.WriteToServer(dataReader);
            }
            finally
            {
                dataReader.Close();
            }
        }
    }
}