.NET Garbagecollector trouble. Blocks for 15-40 mins

Question

Some facts: We have developed wcf service that acts as a layer between clients and the database. It\'s selfhosted and runs as a windows service.

The service keeps s

Answer 1

I have a similar situation. Large database data cache in a service using protobuf with WCF for client communication. The cache is not purely just for clients, the business layer uses the cache to perform operations. The memory footprint of the service can be anywhere between 2 and 10 GB. I release a segment of the cache after 8 hours of inactivity. The machine has 8 virtual cores and 32 GB of memory. I am using .Net 4.5.1.

The GC would consume 98% of the CPU for an hour as soon as I loaded the cache from the database. The interesting point here in both our cases there is no memory pressure what so ever.

I think the GC is performed regardless because something was changed where the GC tries to keep available memory for all threads. Since one thread allocated a large amount of memory when loading the cache, the GC kicked in. I had to do several things to fix it.

1) Removed Tuples from the cache. I was using them as dictionary keys and their implementation of StructuralEquality is horrible. It compares all properties as objects so there is a lot of boxing going on for properties that are values and these will have to be garbage collected at some point.

2) When replacing Tuples used as keys I could not simply replace them with structures without implementing Equals as the value comparison uses reflection and it is too expensive so I ended up creating a Generic Pair structure. I decided to use structures to remove the number of objects when they were in arrays.

3) To remove the Tuples I had to create my own Pair structure that compares the properties using default equals for property types. Eseentially the same thing that PowerCollections created.

Answer 2

Excessive garbage collection is often caused by code issues. You either create too many objects in a short time, or you keep allocating memory without releasing it.

There is actually an extensive checklist available on MSDN that should help you diagnose the problem.

A very large GC2 means that the objects in there survived multiple garbage collections, which means they are kept in memory for a longer period of time. That could be the root cause of your issue. Maybe there is a caching mechanism that could use some tuning / retention policy (remove data that isn't used for a long time).