Using WebClient or WebRequest to login to a website and access data

Question

I'm trying to access restricted data on a website using WebClient/WebRequest. There is no official API in that website, so what I'm trying to do is simply fill the HTML form and post the values to the server, so I'm logged in.

I tried this and this, but it doesn't look like the upcoming requests are logged in.

The latter example is much more appealing since I obviously prefer WebClient, but legacy WebRequest will do.

Anyway, in the first example I think it did login, but the upcoming requests that access the private data return a page with a message "This is member only content".

How to make a WebClient permanently logged in?

Answer 1

Update:

See my comment below.

---

Here's what I did and it works (credit).

Add this class first:

namespace System.Net
{
  using System.Collections.Specialized;
  using System.Linq;
  using System.Text;

  public class CookieAwareWebClient : WebClient
  {
    public void Login(string loginPageAddress, NameValueCollection loginData)
    {
      CookieContainer container;

      var request = (HttpWebRequest)WebRequest.Create(loginPageAddress);

      request.Method = "POST";
      request.ContentType = "application/x-www-form-urlencoded";

      var query = string.Join("&", 
        loginData.Cast<string>().Select(key => $"{key}={loginData[key]}"));

      var buffer = Encoding.ASCII.GetBytes(query);
      request.ContentLength = buffer.Length;
      var requestStream = request.GetRequestStream();
      requestStream.Write(buffer, 0, buffer.Length);
      requestStream.Close();

      container = request.CookieContainer = new CookieContainer();

      var response = request.GetResponse();
      response.Close();
      CookieContainer = container;
    }

    public CookieAwareWebClient(CookieContainer container)
    {
      CookieContainer = container;
    }

    public CookieAwareWebClient()
      : this(new CookieContainer())
    { }

    public CookieContainer CookieContainer { get; private set; }

    protected override WebRequest GetWebRequest(Uri address)
    {
      var request = (HttpWebRequest)base.GetWebRequest(address);
      request.CookieContainer = CookieContainer;
      return request;
    }
  }
}

Usage:

public static void Main()
{
  var loginAddress = "www.mywebsite.com/login";
  var loginData = new NameValueCollection
    {
      { "username", "shimmy" },
      { "password", "mypassword" }
    };

  var client = new CookieAwareWebClient();
  client.Login(loginAddress, loginData);
}

Answer 2

HTTP is stateless. So, you cannot WebClient permanently logged in. The concept of a session does not exist in HTTP. The server-side technologies such as ASP.NET simulate a stateful behavior through the concept of session using cookie or a query string parameter that gets sent back and forth in every request. Having said that, it is possible to emulate what a browser does from WebClient. If you have access to the website, connect to it using the right credentials and capture the traffic using Fiddler. Then, make sure WebClient sends out the right cookies, request headers, query strings, etc exactly same as the browser.