OutOfMemoryException in Regex Matches when processing large files

Question

I\'ve got an exception log from one of production code releases.

System.OutOfMemoryException: Exception of type \'System.OutOfMemoryException\' was thrown.
          


        

Answer 1

Without seeing your Regex, I don't know for sure but sometimes you can get problems like this because your matches are Greedy instead of Lazy.

The Regex engine has to store lots of information internally and Greedy matches can end up causing the Regex to select large sections of your 800k string, many times over.

There's some good information about this over here.

Answer 2

The first thing I would try, if it is possible for your application, would be to split up the input.

Would it be possible to read the file (if the input is a file) line-by-line, applying the Regular Expression that way?

You should take a look with CLR Profiler. It can take a little time to learn how to use, but it's worth it. It will help you visualize how much memory your objects use.

Answer 3

Based on your edit, it sounds like your code may be creating strings which take up large amounts of memory. This would mean that even though the out of memory exception is generated from within the Regex code, it's not actually because the Regex itself is taking up too much memory. Therefore, if using StringBuilder in your own code resolves the issue, then that's what you should do.