Strategies For Tracking Down Memory Leaks When You've Done Everything Wrong

Question

My program, alas, has a memory leak somewhere, but I\'ll be damned if I know what it is.

Its job is to read in a bunch of ~2MB files, do some parsing and string repl

Answer 1

I like the CLR Profiler from Microsoft. It provides some great tools for visualizing the managed heap and tracking down leaks.

Answer 2

The best memory profiling tool for .Net is this:

http://memprofiler.com

Also, while I'm here, the best performance profiler for .Net is this:

http://www.yourkit.com/dotnet/download/index.jsp

They are also great value for money, have low overhead and are easy to use. Anyone serious about .Net development should consider both of these a personal investment and purchase immediately. Both of them have a free trial.

I work on a real time game engine with over 700k lines of code written in C# and have spent hundreds of hours using both these tools. I have used the Sci Tech product since 2002 and YourKit! for the last three years. Although I've tried quite a few of the others I have always returned to these.

IMHO, they are both absolutely brilliant.

Answer 3

One technique I would try is to systematically reduce the amount of code you need to demonstrate the problem without making the problem go away. This is informally known as "divide and conquer" and is a powerful debugging technique. Once you have a small example that demonstrates the same problem, it will be much easier for you to understand. Perhaps the memory problem will become clearer at that point.

Answer 4

1. Add code to the constructor of the unamanaged object to log when it's onstructed, and sort a unique ID. Use that unique ID when the object is destroyed again, and you can at least tell which ones are going astray.
2. Grep the code for every place you construct a new object; follow that code path to see if you have a matching destroy.
3. Add chaining pointers to the constructed objects, so you have a link to the object constructed before and after the current one. Then you can sweep through them later.
4. Add reference counters.
5. Is there a "debug malloc" available?

Answer 5

How do you know for a fact that you actually have a memory leak?

One other thing: You write that your processing classes are using events. If you have registered an event handler it will keep the object that owns the event alive - i.e. the GC cannot collect it. Make sure you de-register all event handlers if you want your objects to be garbage collected.

Answer 6

Be careful how you define "leak". "Uses more memory" or even "uses too much memory" is not the same as "memory leak". This is especially true in a garbage-collected environment. It may simply be that GC hasn't needed to collect the extra memory you're seeing used. Also be careful about the difference between virtual memory use and physical memory use.

Finally not all "memory leaks" are caused by "memory" sorts of issues. I was once told (not asked) to fix an urgent memory leak that was causing IIS to restart frequently. In fact, I did profiling and found I was using a lot of strings through the StringBuilder class. I implemented an object pool (from an MSDN article) for the StringBuilders, and memory usage went down substantially.

IIS still restarted just as frequently. This was because there was no memory leak. Instead, there was unmanaged code that claimed to be thread-safe but was not. Using it in a web service (multiple threads) caused it to write all over the C Runtime Library heap. Since nobody was looking for unmanaged exceptions, nobody saw this until I happened to do some profiling with AQtime from Automated QA. It happens to have an events window, that happened to display the cries of pain from the C Runtime Library.

Placed locks around the calls to the unmanaged code, and the "memory leak" went away.