Why does HttpWebRequest throw an exception instead returning HttpStatusCode.NotFound?

Question

I'm trying to verify the existence of a Url using HttpWebRequest. I found a few examples that do basically this:

HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(Url);
request.Method = "HEAD";
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
    return response.StatusCode;
}

However, if the url is indeed broken, it's not returning a response, it's instead throwing an exception.

I modified my code to this:

try
{
    HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(Url);
    request.Method = "HEAD";
    using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
    {
        return response.StatusCode;
    }
}
catch (System.Net.WebException ex)
{
    var response = ex.Response as HttpWebResponse;
    return response == null ? HttpStatusCode.InternalServerError : response.StatusCode;
}

which seems to finally do what I want.

But I would like to know, why is the request throwing an exception instead of returning the response with a NotFound status code?

Answer 1

Ya this can be quite annoying when web pages use status codes heavily and not all of them are errors. Which can make processing the body quite a pain. Personally I use this extension method for getting the response.

public static class HttpWebResponseExt
{
    public static HttpWebResponse GetResponseNoException(this HttpWebRequest req)
    {
        try
        {
            return (HttpWebResponse)req.GetResponse();
        }
        catch (WebException we)
        {
            var resp = we.Response as HttpWebResponse;
            if (resp == null)
                throw;
            return resp;
        }
    }
}

Answer 2

Why not? They're both valid design options, and HttpWebRequest was just designed to work this way.