HtmlAgilityPack Post Login

Question

I'm trying to login to a site using HtmlAgilityPack (site:http://html-agility-pack.net).

Now, I can't exactly figure out how to go about this.

I've tried setting the Html form values via

m_HtmlDoc.DocumentNode.SelectSingleNode("//input[@name='EMAIL']").SetAttributeValue("value", "myemail.com");

I then submit the form with

m_HtmlWeb.Load("http://example.com/", "POST");

This isn't working though. It's not logging in or anything. Does anyone else have any other insight?

Thank you

Answer 1

The HTML Agility Pack is used to parse HTML - you cannot use it to submit forms. Your first line of code changes the parsed nodes in memory. The second line does not post the page to the server - it loads the DOM again, but using the POST method instead of the default GET.

It doesn't look like you need to parse the page at all at this point, since you already know the name of the control. Use the HttpWebRequest class to send a post request to the server, with the string email=acb#example.com in the request.

Here's a sample I wrote when I needed something similar:

/// <summary>
/// Append a url parameter to a string builder, url-encodes the value
/// </summary>
/// <param name="sb"></param>
/// <param name="name"></param>
/// <param name="value"></param>
protected void AppendParameter(StringBuilder sb, string name, string value)
{
    string encodedValue = HttpUtility.UrlEncode(value);
    sb.AppendFormat("{0}={1}&", name, encodedValue);
}

private void SendDataToService()
{
    StringBuilder sb = new StringBuilder();
    AppendParameter(sb, "email", "hello@example.com");

    byte[] byteArray = Encoding.UTF8.GetBytes(sb.ToString());

    string url = "http://example.com/"; //or: check where the form goes

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = "POST";
    request.ContentType = "application/x-www-form-urlencoded";
    //request.Credentials = CredentialCache.DefaultNetworkCredentials; // ??

    using (Stream requestStream = request.GetRequestStream())
    {
        requestStream.Write(byteArray, 0, byteArray.Length);
    }

    HttpWebResponse response = (HttpWebResponse)request.GetResponse();

    // do something with response
}

Answer 2

If you want to do this with Html Agility Pack. Here's the code.

CookieCollection Cookies = new CookieCollection();
            var web = new HtmlWeb();
            web.OverrideEncoding = Encoding.Default;
            web.UseCookies = true;
            web.PreRequest += (request) =>
            {
                if (request.Method == "POST")
                {
                    string payload = request.Address.Query;
                    byte[] buff = Encoding.UTF8.GetBytes(payload.ToCharArray());
                    request.ContentLength = buff.Length;
                    request.ContentType = "application/x-www-form-urlencoded";
                    System.IO.Stream reqStream = request.GetRequestStream();
                    reqStream.Write(buff, 0, buff.Length);
                }

                request.CookieContainer.Add(Cookies);

                return true;
            };

            web.PostResponse += (request, response) =>
            {
                if (request.CookieContainer.Count > 0 || response.Cookies.Count > 0)
                {
                    Cookies.Add(response.Cookies);
                }
            };

            string baseUrl = "Your Website URL";
            string urlToHit = baseUrl + "?QueryString with Login Credentials";
            HtmlDocument doc = web.Load(urlToHit, "POST");

Answer 3

I spent some hours about this topic and found a very simple solution actually.

I have :

.net core 1.1.2

HttmlAgilityPack 1.4.9.5

login urlLogin: "www.url.com/login".

url for urlData: "www.url.com/data/3" => to get this you should be connected.

Here what I did and it just did work:

HttpClient hc = new HttpClient();

HttpResponseMessage resultLogin = await hc.PostAsync(urlLogin, new StringContent("login=myUserName&password=myPaswordValue", Encoding.UTF8, "application/x-www-form-urlencoded"));

HttpResponseMessage resultPlaylist = await hc.GetAsync(urlData);

Stream stream = await resultPlaylist.Content.ReadAsStreamAsync();

HtmlDocument doc = new HtmlDocument();

doc.Load(stream);

string webContent = doc.DocumentNode.InnerHtml;  => it works

I think It needs to login first your HttpClient then you can send the request you want.

Enjoy