WCF service routing, a bottleneck?

Question

Our application server architecture is setup so that every service call goes through a custom built WCF service router - a single service that distributes the incoming requests to an appropriate service using information embedded in the message header of requests.

We are experiencing performance problems using this WCF service router (timeouts when load-testing with concurrent users). We wonder if this is because of a bug in the router, that we configured the services/IIS wrong or if it is to be expected - every call going through a single service sounds like a potential bottle-neck.

Without routing we can handle about 120 concurrent users before getting timeout errors and although we get timeouts the IIS keeps handling requests. With the router the IIS stops handling requests with about 20 concurrent users and never resumes handling any requests throughout the rest of the load-testing.

Our main question is if this is to be expected when using a service router or should the IIS be able to handle this load the way we have it setup?

---

The routing service looks like this:

/// <summary>
/// Generic service contract which can accept any WCF message.
/// </summary>
[ServiceContract]
public interface IServiceRouter
{
    [OperationContract(Action = "*", ReplyAction = "*", AsyncPattern=false)]
    Message ProcessMessage(Message requestMessage);
}

The implementation:

[ServiceBehavior( 
    InstanceContextMode = InstanceContextMode.PerCall,
    ConcurrencyMode = ConcurrencyMode.Multiple,
    AddressFilterMode = AddressFilterMode.Any, 
    ValidateMustUnderstand = false)]
public sealed class ServiceRouter : IServiceRouter

The ProcessMessage operation:

    public Message ProcessMessage(Message requestMessage)
    {            
        //Figure out the url of the downstream service             
        string serviceName = requestMessage.Headers.GetHeader<String>(ServiceNameMessageHeader, String.Empty);
        string url = String.Format(_destinationUrlFormat, _destinationAppName, _destinationAppVersion, serviceName);
        EndpointAddress endpointAddress = new EndpointAddress(url);

        using (ChannelFactory<IServiceRouter> factory = new ChannelFactory<IServiceRouter>(_binding, endpointAddress))
        {
            factory.Endpoint.Behaviors.Add(new MustUnderstandBehavior(false));
            IServiceRouter proxy = factory.CreateChannel();

            using (proxy as IDisposable)
            {
                try
                {
                    IClientChannel clientChannel = proxy as IClientChannel;

                    // invoke service
                    Message responseMessage = proxy.ProcessMessage(requestMessage);

                    return responseMessage;
                }
                catch (Exception ex)
                {
                    // ...
                }
            }
        }
    }

Answer 1

No a WCF service shouldn't be expected to cause you such a major bottle neck, but it's difficult to give an exact answer when we don't know exactly what your WCF service does, nor how it is configured, but as you said:

Without routing we can handle about 120 concurrent users before getting timeout errors and although we get timeouts the IIS keeps handling requests. With the router the IIS stops handling requests with about 20 concurrent users and never resumes handling any requests throughout the rest of the load-testing.

I think that you've answered your own question with regard to the WCF service causing problems. You'll obviously need to check how it is configured, run and what it does with the "routing".

edit

Have a look here at some issues that can affect WCF performance.

Answer 2

Looks like our problem was with application pools on the IIS, since the routing service and the services it routed to was using the same application pool the following happened: the IIS created threads for the requests going to the router-service.. the routing-service made requests to the services routed to that the IIS created new threads for - worked fine until there where no more threads available and all the threads where routing-service requests waiting for the IIS to handle their request, i.e. the IIS got deadlocked.