Using WebClient or WebRequest to login to a website and access data

Question

I\'m trying to access restricted data on a website using WebClient/WebRequest. There is no official API in that website, so what I\'m trying to do

Answer 1

HTTP is stateless. So, you cannot WebClient permanently logged in. The concept of a session does not exist in HTTP. The server-side technologies such as ASP.NET simulate a stateful behavior through the concept of session using cookie or a query string parameter that gets sent back and forth in every request. Having said that, it is possible to emulate what a browser does from WebClient. If you have access to the website, connect to it using the right credentials and capture the traffic using Fiddler. Then, make sure WebClient sends out the right cookies, request headers, query strings, etc exactly same as the browser.

Answer 2

Update:

See my comment below.

---

Here's what I did and it works (credit).

Add this class first:

namespace System.Net
{
  using System.Collections.Specialized;
  using System.Linq;
  using System.Text;

  public class CookieAwareWebClient : WebClient
  {
    public void Login(string loginPageAddress, NameValueCollection loginData)
    {
      CookieContainer container;

      var request = (HttpWebRequest)WebRequest.Create(loginPageAddress);

      request.Method = "POST";
      request.ContentType = "application/x-www-form-urlencoded";

      var query = string.Join("&", 
        loginData.Cast<string>().Select(key => $"{key}={loginData[key]}"));

      var buffer = Encoding.ASCII.GetBytes(query);
      request.ContentLength = buffer.Length;
      var requestStream = request.GetRequestStream();
      requestStream.Write(buffer, 0, buffer.Length);
      requestStream.Close();

      container = request.CookieContainer = new CookieContainer();

      var response = request.GetResponse();
      response.Close();
      CookieContainer = container;
    }

    public CookieAwareWebClient(CookieContainer container)
    {
      CookieContainer = container;
    }

    public CookieAwareWebClient()
      : this(new CookieContainer())
    { }

    public CookieContainer CookieContainer { get; private set; }

    protected override WebRequest GetWebRequest(Uri address)
    {
      var request = (HttpWebRequest)base.GetWebRequest(address);
      request.CookieContainer = CookieContainer;
      return request;
    }
  }
}

Usage:

public static void Main()
{
  var loginAddress = "www.mywebsite.com/login";
  var loginData = new NameValueCollection
    {
      { "username", "shimmy" },
      { "password", "mypassword" }
    };

  var client = new CookieAwareWebClient();
  client.Login(loginAddress, loginData);
}