WCF Concurrent requests piling up on the server when using WSHttpBinding

Question

I have a WCF client/server app which is communicating over HTTP using the WSHttpBinding.

Server Setup: self-hosting, using the standard WCF ServiceHost. My actual service class is attributed as:

[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple, 
 InstanceContextMode = InstanceContextMode.PerSession, 
 UseSynchronizationContext = false)]

Client Setup: using a visual-studio generated client proxy using synchronous service calls (proxy.call_server_method blocks until such time as the server has responded in full.)

Scenario: I have one particular method call which takes 20 seconds to execute on the server. The client calls this method in a separate thread, so it is not being held up, and the ConcurrencyMode.Multiple means WCF should execute it in a separate thread on the server as well.

This theory is supported by the fact that when I configure my app to use NetTcpBinding, everything works fine.

Problem:
If I configure the app to use WSHttpBinding, then this long method call causes the http requests to 'back up'. I have verified this behaviour both from inspecting my logs, and by debugging the HTTP requests using fiddler.

Example:

• Client initiates 20-second long request on a background thread
• Client initiates request B and C on foreground thread
• Requests B and C get sent to the server, which doesn't process them until it is done with the 20-second long request

But sometimes:

• Requests B and C do not get sent (they don't even appear in fiddler) until the 20-second request comes back (this is rare).
  • Note: setting <add address="*" maxconnection="100"/> in the client's app.config made this (appear to) stop happening.
• Request B gets sent and receives a response immediately, while request C is held back until the 20-second one completes (this is rare)

Here's a timeline from fiddler demonstrating the problem: (click for bigger version)

As you can see, the requests are all getting backed up at the server. Once the 20-second request completes, the responses all come flooding through, but note that there are some requests which aren't getting held up...

So, Questions:

• What the heck is going on here? Why does it work fine using NetTcpBinding and not work using WSHttpBinding?
• Why the inconsistent behaviour?
• What can I do to fix it?

Notes:

• It's not locking on the server. I've set breakpoints and used !syncblk and it consistently reports no locks are being held.
• It's not my threading (NetTcpBinding shouldn't work otherwise)
• I have <serviceThrottling maxConcurrentCalls="1000" maxConcurrentInstances="1000" maxConcurrentSessions="1000" /> set in the server's app.config
• The 20-second call is just waiting on a timer, it's not thrashing the CPU or disk or network
• I'd prefer a solution that didn't involve re-architecting the application to use asynchronous calls... it's a big bunch of legacy code and I really don't want to be messing with stuff I don't understand.

Answer 1

There is some throttle outside of WCF (a .Net or Windows thing) that defaults to only letting a maximum of two simultaneous outbound HTTP connections. Unfortunately I can't remember for the life of me the name of the thing (and what you'd put in app.config or your app to override it). Given that you're not seeing the requests leave the client, and that it's only HTTP, I think you're hitting "that thing". I'll keep looking for its name.

Update: Found it - try this on the client (but change the '2' to a bigger number):

<configuration>
  <system.net>
    <connectionManagement>
      <add address = "*" maxconnection = "2" />
    </connectionManagement>
  </system.net>
</configuration>

Answer 2

We saw exactly the same symptoms with a JSON service hosted in IIS/ASP.NET.

The root cause ended up being ASP.NET synchronizing the requests - not WCF. We had to disable session state (at the application level) to get concurrent WCF methods.

Web.config: <system.web> <sessionState mode="Off" /> </system.web>

Please note that our service uses webHttpBinding, not wsHttpBinding. So, I'm not sure if this also solves Orion's problem.

Answer 3

[Self-answer to show other users what our eventual solution was]

In the end, I never managed to solve this.
Our final solution was to switch our app away from WSHttpBinding and onto NetTcpBinding in production - we had been planning on doing this eventually anyway for performance reasons.

This is rather unfortunate though, as it leaves a black mark on WSHttpBinding which may or may not be warranted. If anyone does ever come up with a solution which does not involve ditching WSHttpBinding, I'd love to know about it

Answer 4

I think you hit the protocol limit and to work around it you need to modify the standard settings on the client machine:

http://support.microsoft.com/kb/183110

http://support.microsoft.com/kb/282402

I guess WSHttpBinding uses WinINET settings when issuing the requests.

Answer 5

If you change to BasicHttpBinding, does it work?

Is so, it sounds like this is your problem, Session throttling, something that bit me in the ass.

Answer 6

Consider using ConcurrencyMode.Multiple on per-call services to allow concurrent calls.

Answer 7

I forget - could it be ordering? I think maybe RM over http preserves order but maybe Tcp sessions don't (unless you explicitly request it)? Is there an attribute on the Service Contract that described ordered/unordered sessions (I forget).

Answer 8

Not sure but sometimes the problem with the concurrent calls from silverlight application are related with browser connection management. For me the solution was to put that in our App.xaml.cs, Application_Startup method as descrive here: http://weblogs.asp.net/olakarlsson/simultaneously-calling-multiple-methods-on-a-wcf-service-from-silverlight

WebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);