HtmlAgilityPack.HtmlDocument Cookies

Question

This pertains to cookies set inside a script (maybe inside a script tag).

System.Windows.Forms.HtmlDocument executes those scripts and the cookies set (

Answer 1

When I need to use Cookies and HtmlAgilityPack together, or just create custom requests (for example, set the User-Agent property, etc), here is what I do:

• Create a class that encapsulates the request/response. Let's call this class WebQuery
• Have a private CookieCollection (in your case public) property inside that class
• Create a method inside the class that does manually the request. The signature could be:

...

public HtmlAgilityPack.HtmlDocument GetSource(string url);

What do we need to do inside this method?

Well, using HttpWebRequest and HttpWebResponse, generate the http request manually (there are several examples of how to do this on Internet), create an instance of a HtmlDocument class using the constructor that receives an stream.

What stream do we have to use? Well, the one returned by:

httpResponse.GetResponseStream();

If you use HttpWebRequest to make the query, you can easily set the CookieContainer property of it to the variable you declared before everytime you access a new page, and that way all cookies set by the sites you access will be properly stored in the CookieContainer variable you declared in your WebQuery class, taking in count you're using only one instance of the WebQuery class.

Hope you find useful this explanation. Take in count that using this, you can do whatever you want, no matter if HtmlAgilityPack supports it or not.