Recommendations for executing .NET HttpWebRequests in parallel in ASP.NET

Question

I have an ASP.NET MVC web application that makes REST style web service calls to other servers. I have a scenario where I am making two HttpWebRequest calls to two separate

Answer 1

I like to do this kind of thing a little more manually, rather than relying on asynchronous web requests or the thread pool's automatic sizing (25 threads by default). Of course, those are perfectly fine ways to solve your problem, but I think the following code is a bit more readable (in the example below, _links would contain a list of your links before processing occurs...):

private static IList _links = new List();
private const int NumberOfThreads = 2;

public void SpawnWebRequests()
{
    IList threadList = new List();

    for (int i = 0; i < NumberOfThreads; i++)
    {
        var thread = new Thread(ProcessWebRequests);
        threadList.Add(thread);
        thread.Start();
    }

    for (int i = 0; i < NumberOfThreads; i++)
    {
        threadList[i].Join();
    }
}

private static void ProcessWebRequests()
{
    String link;

    while (true)
    {
        lock(_links)
        {
            if (_links.Count == 0)
                break;

            link = _links.RemoveAt(0);
        }

        ProcessWebRequest(link);
    }
}

private static void ProcessWebRequest(String link)
{
    try
    {
        var request = (HttpWebRequest)WebRequest.Create(link);
        request.Method = "HEAD"; // or "GET", since some sites (Amazon) don't allow HEAD
        request.Timeout = DefaultTimeoutSeconds * 1000;

        // Get the response (throws an exception if status != 200)
        using (var response = (HttpWebResponse)request.GetResponse())
        {
            if (response.StatusCode == HttpStatusCode.OK)
                Log.Debug("Working link: {0}", request.RequestUri);
        }
    }
    catch (WebException ex)
    {
        var response = ((HttpWebResponse)ex.Response);
        var status = response != null
                         ? response.StatusCode
                         : HttpStatusCode.RequestTimeout;

        Log.WarnException(String.Format("Broken link ({0}): {1}", status, link), ex);

        // Don't rethrow, as this is an expected exception in many cases
    }
    catch (Exception ex)
    {
        Log.ErrorException(String.Format("Error processing link {0}", link), ex);

        // Rethrow, something went wrong
        throw;
    }
}

If you just want to manage the size of the thread pool (if you are using ThreadPool.QueueUserWorkItem()), you can use ThreadPool.SetMaxThreads = 2).

Of course, if you want to use the Microsoft-sanctioned async approach, check out this example: http://msdn.microsoft.com/en-us/library/86wf6409.aspx. Just be sure you clean up each response (via a "using" block or by closing the response object)!

Hope that helps, Noah