Why is this WebRequest code slow?

问题

I requested 100 pages that all 404. I wrote

    {
    var s = DateTime.Now;
    for(int i=0; i < 100;i++)
        DL.CheckExist("http://google.com/lol" + i.ToString() + ".jpg");
    var e = DateTime.Now;
    var d = e-s;
        d=d;
        Console.WriteLine(d);
    }

static public bool CheckExist(string url)
{
    HttpWebRequest wreq = null;
    HttpWebResponse wresp = null;
    bool ret = false;

    try
    {
        wreq = (HttpWebRequest)WebRequest.Create(url);
        wreq.KeepAlive = true;
        wreq.Method = "HEAD";
        wresp = (HttpWebResponse)wreq.GetResponse();
        ret = true;
    }
    catch (System.Net.WebException)
    {
    }
    finally
    {
        if (wresp != null)
            wresp.Close();
    }
    return ret;
}

Two runs show it takes 00:00:30.7968750 and 00:00:26.8750000. Then i tried firefox and use the following code

<html>
<body>
<script type="text/javascript">
for(var i=0; i<100; i++)
    document.write("<img src=http://google.com/lol" + i + ".jpg><br>");
</script>

</body>
</html>

Using my comp time and counting it was roughly 4 seconds. 4 seconds is 6.5-7.5faster then my app. I plan to scan through a thousands of files so taking 3.75hours instead of 30mins would be a big problem. How can i make this code faster? I know someone will say firefox caches the images but i want to say 1) it still needs to check the headers from the remote server to see if it has been updated (which is what i want my app to do) 2) I am not receiving the body, my code should only be requesting the header. So, how do i solve this?

回答1:

Probably Firefox issues multiple requests at once whereas your code does them one by one. Perhaps adding threads will speed up your program.

回答2:

I noticed that an HttpWebRequest hangs on the first request. I did some research and what seems to be happening is that the request is configuring or auto-detecting proxies. If you set

request.Proxy = null;

on the web request object, you might be able to avoid an initial delay.

With proxy auto-detect:

using (var response = (HttpWebResponse)request.GetResponse()) //6,956 ms
{
}

Without proxy auto-detect:

request.Proxy = null;
using (var response = (HttpWebResponse)request.GetResponse()) //154 ms
{
}

回答3:

change your code to asynchronous getresponse

public override WebResponse GetResponse() {
    •••
    IAsyncResult asyncResult = BeginGetResponse(null, null);
    •••
    return EndGetResponse(asyncResult);
}

Async Get

回答4:

The answer is changing HttpWebRequest/HttpWebResponse to WebRequest/WebResponse only. That fixed the problem.

回答5:

Have you tried opening the same URL in IE on the machine that your code is deployed to? If it is a Windows Server machine then sometimes it's because the url you're requesting is not in IE's (which HttpWebRequest works off) list of secure sites. You'll just need to add it.

Do you have more info you could post? I've doing something similar and have run into tons of problems with HttpWebRequest before. All unique. So more info would help.

BTW, calling it using the async methods won't really help in this case. It doesn't shorten the download time. It just doesn't block your calling thread that's all.

回答6:

close the response stream when you are done, so in your checkExist(), add wresp.Close() after wresp = (HttpWebResponse)wreq.GetResponse();

回答7:

OK if you are getting status code 404 for all webpages then it is due to not specifying credentials. So you need to add

wreq.Credentials = CredentialCache.DefaultCredentials;

Then you may also come across status code= 500 for that you need to specify User Agent. Which looks something like the below line

wreq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0";

"A WebClient instance does not send optional HTTP headers by default. If your request requires an optional header, you must add the header to the Headers collection. For example, to retain queries in the response, you must add a user-agent header. Also, servers may return 500 (Internal Server Error) if the user agent header is missing."

reference: https://msdn.microsoft.com/en-us/library/system.net.webclient(v=vs.110).aspx

To improve the Performance of the HttpWebrequest you need to add

wreq.Proxy=null

now the code will look like:

 static public bool CheckExist(string url)
{
    HttpWebRequest wreq = null;
    HttpWebResponse wresp = null;
    bool ret = false;

try
{
    wreq = (HttpWebRequest)WebRequest.Create(url);
    wreq.Credentials = CredentialCache.DefaultCredentials;
    wreq.Proxy=null;
    wreq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0";
    wreq.KeepAlive = true;
    wreq.Method = "HEAD";
    wresp = (HttpWebResponse)wreq.GetResponse();
    ret = true;
}
catch (System.Net.WebException)
{
}
finally
{
    if (wresp != null)
        wresp.Close();
}
return ret;

}

来源：https://stackoverflow.com/questions/754333/why-is-this-webrequest-code-slow

标签

httpwebrequest

httpwebresponse