Server 2008 RC2, IIS 7.5, ASP.NET and Requests Queued Poor Performance

Question

I already know the answer to this, but wanted to share with the community since it is NOT documented from Microsoft.

The scenario: A surge of traffic hits your IIS 7.5 A

Answer 1

After searching around the internet, I found the following articles from Microsoft that relate to the problem:

KB 821268: Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications

This article gives some great performance-tweaking tips, however it fails to mention some some VERY important ceilings that we ran in to.

The solution for us was to modify our machine.config, and populate the following XML nodes:

I purposefully set some of these numbers to "xxx" since they are dependent upon your hardware.

From the KB article above, Microsoft suggests some equations for figuring out these values. However, they fail to mention that the MAXIMUM value for these numbers is the size of an INT, or 32767.

So, the CORRECT equations for figuring these out are as follows:

• maxWorkerThreads: 32767 / #Cores
  • In our case, we have a 24-core server. So, our maxWorkerThreads value is correctly set to: 1365. Any number that results in an integer LARGER than 32767, the server will set the maxWorkerThreads to 32767.
• maxIoThreads: Same as maxWorkerThreads (32767 / #Cores)
• minWorkerThreads: maxWorkerThreads / 2
  • This was a tricky one. If one exceeded an integer value LARGER than 32767 (and despite what the KB article says, this number IS multiplied by the number of cores you have), and unlike the "max" value, this defaults to the number of cores on your machine! In our case, this was getting set to 24 (because we set an arbitrarily high value for the min), and that was KILLING the performance on our server.
• minIoThreads: Same as minWorkerThreads
• minFreeThreads: 88 * #Cores (taken directly from the KB article)
• minLocalRequestFreeThreads: 76 * #Cores (taken directly from the KB article)

This solution is not for everyone, and should only be used if you meet the criteria in the KB article.

Another tool we used in helping us diagnose this was an .ASPX page with no code-behind that we could throw out on any server (without resetting the Application Pool). This page uses reflection to tell you what is actually going on in the thread pool, and what the values of these settings render to on your server.

<%@ Page Language="C#" %>









<%
Response.Cache.SetCacheability(HttpCacheability.NoCache);

int worker, workerMIN, workerMAX;
int port, portMIN, portMAX;
System.Threading.ThreadPool.GetAvailableThreads(out worker, out port);
System.Threading.ThreadPool.GetMinThreads(out workerMIN, out portMIN);
System.Threading.ThreadPool.GetMaxThreads(out workerMAX, out portMAX);

 %>


Thread Information


    
Worker Threads
    

        

            
Min: <%=workerMIN %>
            
Current: <%=workerMAX - worker %>
            
Max: <%=workerMAX %>
        
    



    
Completion Port Threads
    

        

            
Min: <%=portMIN %>
            
Current: <%=portMAX - port %>
            
Max: <%=portMAX %>
        
    



    
Request Queue Information
    


<%


var fi = typeof(HttpRuntime).GetField("_theRuntime", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.Static).GetValue(null);
var rq = typeof(HttpRuntime).GetField("_requestQueue", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance).GetValue(fi);
var fields = rq.GetType().GetFields(System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);

foreach (var field in fields)
{
    string name = field.Name;
    string value = "";

    switch (name)
    {
        case "_localQueue":
        case "_externQueue":
            System.Collections.Queue queue = field.GetValue(rq) as System.Collections.Queue;
            value = queue.Count.ToString();
            break;

        default:
            value = field.GetValue(rq).ToString();
            break;
    }

    %>
        

            
<%=name %>
            
<%=value %>
        
    <%
    //Response.Write(string.Format("{0}={1}
", name, value));
}   

%>