HtmlAgilityPack HtmlWeb.Load returning empty Document

Question

I have been using HtmlAgilityPack for the last 2 months in a Web Crawler Application with no issues loading a webpage.

Now when I try to load a this particular webpa

Answer 1

It seems this website requires cookies to be enabled. So creating a cookie container for your web request should solve the issue:

var url = "http://www.prettygreen.com/";
var htmlWeb = new HtmlWeb();
htmlWeb.PreRequest += request =>
    {
        request.CookieContainer = new System.Net.CookieContainer();
        return true;
    };
var htmlDoc = htmlWeb.Load(url);
var outerHtml = htmlDoc.DocumentNode.OuterHtml;
Assert.AreNotEqual("", outerHtml);