Transferring large payloads of data (Serialized Objects) using wsHttp in WCF with message security

Question

I have a case where I need to transfer large amounts of serialized object graphs (via NetDataContractSerializer) using WCF using wsHttp. I\'m using message security and woul

Answer 1

If you still want to use Message Security, I would recommend you to use MTOM to optimize the network bandwidth that needs be used to transfer the messages, and also the chunking channel for using smaller memory buffers when security is applied. Otherwise, WCF will try to load the whole message in memory to apply security, and therefore you will get the Insufficient memory exception.

Answer 2

i used to implement kinda passing big text to/from wcf. my trig is convert it to stream and use GZipStream to compress it then send it as byte[], luckily its never exceed 10 MB.

In your case i recommend do fragmentation. Convert Serialized object to byte[] and then merg it and decompress

psudo

int transferSize = 5000000; // 5MB
byte[] compressed = ...;
var mem = new System.IO.MemoryStream(compressed);

for(int i = 0; i < compressed .length; i+= transferSize )
{
    byte[] buffer = new byte[transferSize];
    mem.Read(buffer, i, compressed);
    mem.Flush();
    sendFragmentToWCF(buffer);
}

edit 08 Dec 2010

ok based on my understanding, the situation is client is download some large serialize object throught WCF. i didn't particularly test on this solution, but guess it should work. The key is save serialized object to file and use Response transmit that file.

[WebMethod]
public void GetSerializedObject()
{
    string path = @"X:\temp.tmp";

    var serializer = new  System.Runtime.Serialization.NetDataContractSerializer();
    var file = new System.IO.FileStream(path, System.IO.FileMode.CreateNew);

    try
    {
        serializer.Serialize(file, ...);
        this.Context.Response.TransmitFile(path);
        this.Context.Response.Flush();
    }
    finally
    {
        file.Flush();
        file.Close();
        file.Dispose();
        System.IO.File.Delete(path);
    }
}

WCF shoud do file streaming automatically and u dont ahve to worry about serialized object size since we use file transmit. Dont forget to the config response limit.

Answer 3

protobuf-net generally has a significant space-saving (as in: order-of-magnitude) on most data, and can attach to WCF. Unfortunately at the moment it doesn't support full graphs, only trees. However, I have plans there that I simply haven't had time to implement. I can't promise anything, but I could try to bump that work a bit sooner.

Otherwise; there may be ways to tweak your existing code to work with a tree instead of a graph.

Answer 4

Some lighter, but not guaranteed solutions, would be to

• use the DataContractSerializer instead since you own both sides. This does not require embedded type information, which is significantly large.
• use [DataMember(EmitDefaultValue = false)] which is discussed in a question I asked - again because you own both sides; doing so will cut down some on the message size (how much depends of course on how many fields in the graph are defaults).
• use [DataContract(IsReference=true)], especially if you have many repeated value objects or reference data
• use some sort of throttling on the server to reduce memory pressure of simultaneous results

These are, of course trade-offs, for example with readability.

Answer 5

Since nobody has put it out there, perhaps using WebSockets or long polling techniques may be able to solve this issue. I've only looked into these solutions briefly and haven't developed a solution around them but I wanted to propose these concepts for the record and I'll expand upon my answer at a later point if time permits.

The underlying idea would be to achieve something similar to how the idea of how the ChunkingChannel example works but while not requiring a full duplex channel which typically breaks the port 80 web based request/response model that is desirable to avoid having to make firewall and other related configurations for clients.

Other related material:

• WCF HTTP Long Polling
• WebSockets with WCF

Update: After researching more on this it appears that by using WebSockets, which is known known as NetHttpBinding, that I would inherently not be solving the original request which is to use wsHttp in WCF with message security. I'm going to keep my answer here however as information for others who may be looking into an alternative.