How to maxmise the largest contiguous block of memory in the Large Object Heap

Question

The situation is that I am making a WCF call to a remote server which is returns an XML document as a string.

Most of the time this return value is a few K, sometimes a few dozen K, very occasionally a few hundred K, but very rarely it could be several megabytes (first problem is that there is no way for me to know).

It's these rare occasions that are causing grief. I get a stack trace that starts:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Xml.BufferBuilder.AddBuffer()
   at System.Xml.BufferBuilder.AppendHelper(Char* pSource, Int32 count)
   at System.Xml.BufferBuilder.Append(Char[] value, Int32 start, Int32 count)
   at System.Xml.XmlTextReaderImpl.ParseText()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.XmlTextReader.Read()
   at System.Xml.XmlReader.ReadElementString()
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderMDRQuery.Read2_getMarketDataResponse()
   at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer2.Deserialize(XmlSerializationReader reader)
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

I've read around and it is because the Large Object Heap is just getting too fragmented, so even preceding the call with a quick check to StringBuilder.EnsureCapacity just causes the OutOfMemoryException to be thrown earlier (and because I'm guessing at what's needed, it might not actually need that much so my check is causing more problems than it is solving). Some opinions are that there's not much I can do about it.

Some of the questions I've asked myself:

• Use less memory - have you checked for leaks? Yes. The memory usage goes up and down, but there's no fundamental growth that guarantees this to happen. Some of the times it fails, it succeeded at that stage previously.
• Transfer smaller amounts Not an option, this is a third party web service over which I have no control (or at least it would take a long time to resolve, in the meantime I still have a problem)
• Can you do something to the LOH to make it less likely to fail? ... now this is most fruitful course. It's a 32-bit process (it has to be for various political, technical and boring reasons) but there's normally hundreds of meg free (multiples of the largest amount for which we've seen failures).
• Can we monitor the LOH? Using perfmon I can track the size of the heaps, but I don't think there's a way to monitor the largest available contiguous block of memory.

Question is: any advice or suggestions for things to try?

Answer 1

You might review the TransferMode property of your binding to see if you meet the requirements to change it from its default value of "Buffered", to "Streamed" or "StreamedResponse".

Also, review the values for maxBufferPoolSize and maxBufferSize. Increasing the size of the internal buffers used can help with memory utilization, especially with processing large messages.

maxReceivedMessageSize is also likely already set if your receiving large messages, but I would review that value as well.

I've seen one of the values above, if your exceeding the threshold, fail with an obscure, memory related message. The original exception was actually hidden by the message that was surfaced to my application. Enabling WCF Tracing helped diagnose the problem and see the real error - I needed to increase the value of one, or more, of the binding properties above.

I didn't get the feel for the binding your using from your post, but I believe that these settings are common across major ones. Check out MSDN documentation on basicHttpBinding for example.

If it is truly LOH fragmentation there isn't anything to be done about it once tuning efforts have been exhausted. Rolling recycle of the application might be required to mitigate it (I hate recommending that) but if you've exhausted other efforts you might be left with that.

Answer 2

I cannot address any of the WCF specific issues, but if you need to maximize LOH space for a 32 bit process, you should make the application large address aware and run it on 64 bit. A large address aware 32 bit process will be able to address the entire 4 GB address space when run on 64 bit Windows. This will give you a sizable chunk of memory above the address space normally used by the process.

Answer 3

I think your problem Might be an Assembly Leak caused by using XmlSerializer and not using one of two constructors as indicated in this MSDN article:

To increase performance, the XML serialization infrastructure dynamically generates assemblies to serialize and deserialize specified types. The infrastructure finds and reuses those assemblies. This behavior occurs only when using the following constructors:

XmlSerializer.XmlSerializer(Type)

XmlSerializer.XmlSerializer(Type, String)

If you use any of the other constructors, multiple versions of the same assembly are generated and never unloaded, which results in a memory leak and poor performance.

Nice, huh. The answer is to cache your XmlSerializer (assuming you even create it).

To really figure it out you need to do what Tess tells you to do. She's a freakin genius.

Answer 4

If possible I would go for a stream based approach and use a forward only Xml parser in combination, which should give you better performance as well.

If you don't absolutely have to use WCF, you can write your own HttpRequest and then pass the response to the XmlDeserializer and then parse the response like that. It might give you more control and insight into where the problem actually occurs. You can also experiment with a mock service that returns very large documents of the type you are looking for. We had a lot of headaches with the LOH fragmentation as well so I really feel your pain.

A problem I noticed when building buffers .NET tends to double capacity every time the buffer is filled up, which causes memory fragmentation since for a document of size 10mb, memory needs to be allocated in many steps. If you know the needed buffer size in advance, it is more efficient to allocate it at once. So if you know how big the incoming document will be, you can create a StringBuilder with exactly that size.