HttpWebRequest returns “ (403) Forbidden” Error

Question

I wrote a xml grabber to receive/decode xml files from website. It works fine mostly but it always return error:

\"The remote server returned an e

Answer 1

It could be that your request is missing a header that is required by the server. I requested the page in a browser, recorded the exact request using Fiddler and then removed the User-Agent header and reissued the request. This resulted in a 403 response.

This is often used by servers in an attempt to prevent scripting of their sites just like you are doing ;o)

In this case, the server header in the 403 response is "AkamaiGHost" which indicates an edge node from some cloud security solution from Akamai. Maybe a WAF rule to prevent bots is triggering the 403.

It seems like adding any value to the User-Agent header will work for this site. For example I set it to "definitely-not-a-screen-scraper" and that seems to work fine.

In general, when you have this kind of problem it very often helps to look at the actual HTTP requests and responses using browser tools or a proxy like Fiddler. As Scott Hanselman says

The internet is not a black box

http://www.hanselman.com/blog/TheInternetIsNotABlackBoxLookInside.aspx