Entity framework large data set, out of memory exception

Question

I am working the a very large data set, roughly 2 million records. I have the code below but get an out of memory exception after it has process around three batches, about

Answer 1

The issue is that when you get data from EF there are actually two copies of the data created, one which is returned to the user and a second which EF holds onto and uses for change detection (so that it can persist changes to the database). EF holds this second set for the lifetime of the context and its this set thats running you out of memory.

You have 2 options to deal with this

1. renew your context each batch

2. Use .AsNoTracking() in your query eg:

   IEnumerable> towns = dbContext.Towns.AsNoTracking().OrderBy(t => t.TownID).Batch(200000);
   

this tells EF not to keep a copy for change detection. You can read a little more about what AsNoTracking does and the performance impacts of this on my blog: http://blog.staticvoid.co.nz/2012/4/2/entity_framework_and_asnotracking