Entity framework large data set, out of memory exception

Question

I am working the a very large data set, roughly 2 million records. I have the code below but get an out of memory exception after it has process around three batches, about

Answer 1

I wrote a migration routine that reads from one DB and writes (with minor changes in layout) into another DB (of a different type) and in this case, renewing the connection for each batch and using AsNoTracking() did not cut it for me.

Note that this problem occurs using a '97 version of JET. It may work flawlessly with other DBs.

However, the following algorithm did solve the Out-of-memory issue:

• use one connection for reading and one for writing/updating
• Read with AsNoTracking()

• every 50 rows or so written/updated, check the memory usage, recover memory + reset output DB context (and connected tables) as needed:

  var before = System.Diagnostics.Process.GetCurrentProcess().VirtualMemorySize64;
  if (before > 800000000)
  {
      dbcontextOut.SaveChanges();
      dbcontextOut.Dispose();
      GC.Collect();
      GC.WaitForPendingFinalizers();
      dbcontextOut = dbcontextOutFunc();
      tableOut = Dynamic.InvokeGet(dbcontextOut, outputTableName);
  }