EPPlus save two million rows with 200+ columns datatable to multiple excel files

Question

I have function that saves all records from SQL table to excel worksheet using EPPlus. If I export small amount of data everything works fine, but with 200+ columns and 500

Answer 1

Unfortunately, there is no easy way to merge that much data with Epplus in a single file. Basically, the entire file is loaded into memory when open - its either all or nothing. In theory, you could generate the XML files that XLSX contains (they are zip files renamed) and manually insert it since it would have a smaller memory footprint but that is no small feat.

For you current code, you could always just call .dispose() manually if you want to avoid the using statement. But I understand you wanting to avoid duplicate code. What about something like this (but watch for memory usage when copying all the object data):

const int max = 10;
var loop = 0;

using (var sdr = cmd.ExecuteReader())
{
    var fieldcount = sdr.FieldCount;

    var getfi = new Func(i =>
    {
        var fi = new FileInfo(String.Format(@"c:\temp\Multi_Files{0}.xlsx", i));
        if (fi.Exists) fi.Delete();
        return fi;
    });

    var savefile = new Action>((info, rows) =>
    {
        using (var pck = new ExcelPackage(info))
        {
            var wb = pck.Workbook;
            var ws = wb.Worksheets.Add("RESULTS");
            for (var row = 0; row < rows.Count; row++)
                for (var col = 0; col < fieldcount; col++)
                    ws.SetValue(row + 1, col + 1, rows[row][col]);
            pck.Save();
        }
    });

    var rowlist = new List();

    while (sdr.Read())
    {
        var rowdata = new Object[sdr.FieldCount];
        sdr.GetValues(rowdata);
        rowlist.Add(rowdata);

        if (rowlist.Count == max)
        {
            savefile(getfi(++loop), rowlist);
            rowlist.Clear();
        }
    }
    if (rowlist.Count > 0)
        savefile(getfi(++loop), rowlist);
}