C# - how to do multiple web requests at the same time

问题

I wrote a code to check urls, however, ir works really slow.. I want to try to make it work on few urls at the same time, for example 10 urls or at least make it as fast as possible.

my Code:

Parallel.ForEach(urls, new ParallelOptions {
  MaxDegreeOfParallelism = 10
}, s => {
  try {
    using(HttpRequest httpRequest = new HttpRequest()) {
      httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0";
      httpRequest.Cookies = new CookieDictionary(false);
      httpRequest.ConnectTimeout = 10000;
      httpRequest.ReadWriteTimeout = 10000;
      httpRequest.KeepAlive = true;
      httpRequest.IgnoreProtocolErrors = true;
      string check = httpRequest.Get(s + "'", null).ToString();
      if (errors.Any(new Func < string, bool > (check.Contains))) {
        Valid.Add(s);
        Console.WriteLine(s);
        File.WriteAllLines(Environment.CurrentDirectory + "/Good.txt", Valid);
      }
    }
  } catch {

  }
});

回答1:

It is unlikely that your service calls are CPU-bound. So spinning up more threads to handle the load is maybe not the best approach-- you will get better throughput if you use async and await instead, if you can, using the more modern HttpClient instead of HttpRequest or HttpWebRequest.

Here is an example of how to do it:

var client = new HttpClient();

//Start with a list of URLs
var urls = new string[]
    {
        "http://www.google.com",
        "http://www.bing.com"
    };

//Start requests for all of them
var requests  = urls.Select
    (
        url => client.GetAsync(url)
    );

//Wait for all the requests to finish
await Task.WhenAll(requests);

//Get the responses
var responses = requests.Select
    (
        task => task.Result
    );

foreach (var r in responses)
{
    var s = await r.Content.ReadAsStringAsync();
    Console.WriteLine(r);
}

回答2:

Try doing as below.

Parallel.ForEach(urls, new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount - 1 }

At least it makes sure that all the cores are used by leaving 1 so that your machine will not run out of memory.

Also, consider @KSib comment.

来源：https://stackoverflow.com/questions/52338042/c-sharp-how-to-do-multiple-web-requests-at-the-same-time

标签

web

xnet