using a proxy with htmlagilitypack

Question

I searched this question but didn't find anything that I was looking for, basically I want to use a proxy with htmlagilitypack, I had the code to do it before but lost it, here is the code I have so far, which is working. but I timed my self out on a program I was making and need to enable proxies.

    private void button1_Click(object sender, EventArgs e)
    {
        StringBuilder output = new StringBuilder();
        string raw = "http://www.google.com";
        HtmlWeb webGet = new HtmlWeb();
        webGet.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6";
        var document = webGet.Load(raw);


    }

Answer 1

Use an overload of HtmlWeb.Load() that uses proxies. There are two overload signatures:

HtmlDocument Load(string url, string method, WebProxy proxy, NetworkCredential credentials);
HtmlDocument Load(string url, string proxyHost, int proxyPort, string userId, string password);

I don't have any first-hand experience using proxies in my code but I'd expect this to work.

Answer 2

HtmlAgilityPack doesn't download data from url. Use a class to download the page that supports Proxy.

For example

WebClient wc = new WebClient();
wc.Proxy = new WebProxy(host,port);
var page = wc.DownloadString(url);

HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(page);

EDIT

Assuming you read something like 11.22.33.44:5678 from text file, it is also possible to create the proxy as

wc.Proxy = new WebProxy("11.22.33.44:5678");

Answer 3

Within our corporate setup, adding this to app.config works for me without the need for any code changes

<system.net>
   <defaultProxy useDefaultCredentials="true" />
</system.net>