Python: requests can't login to a website

Question

I need to scrape website that requires login. I\'m trying to create a session and login as I have to scrape different pages after logging in. But can\'t find out wh

Answer 1

You might be missing some headers. I would intercept a request made by a web browser to see the things you are missing, then add these headers to your request.

You will find informations on how to do it on the official documentation, right here : http://docs.python-requests.org/en/master/user/quickstart/#custom-headers

Answer 2

I know that this question was made long ago, but anyway, I'll propose a solution for those who are still having trouble with this: I recommend to check if the form you are trying to post takes some kind of hidden input, which the example of the question does. This is very frequent, and does sometimes prevent us from logging to a site if we do not notice it. So, let's suppose in the site there is a form like this:

<form method='post' id='signin-form' class='big-form'>
 <input type="hidden" id="whatever" name="foo" value="check">
 <input type="text" id="u" name="user">
 <input type="password" id="pwd" name="pass">
</form>

In that case, the variable login_data should be like this:

login_data = {
       "foo":"check",
       "user":"your username",
       "pass":"your password",           
}

Having done this, and provided the website does not check the headers, you should have no trouble logging to a website via the requests module.

Answer 3

Try doing a get on the login page first. Perhaps it's setting some cookies that it expects to be present on the post.