Python - Login and download specific file from website

Question

My attempt to log into a website and download a specific file has hit a fall.

Specifically, I am logging into this website http://www.gaez.iiasa.ac.at/w/ctrl?_flow=

Answer 1

Website that you linked uses HTTP POST based login from. In your code you have:

resp = requests.get(url, auth=(user, password))

which will use basic http authentication http://docs.python-requests.org/en/master/user/authentication/#basic-authentication

To login to this site you need two things:

• persistent session cookie
• HTTP POST request to login form URL

First of all let's create session object that will be holding cookies form server http://docs.python-requests.org/en/master/user/advanced/#session-objects

s = requests.Session()

Next you need to visit site using GET request. This will generate cookie for you (server will send cookie for your session).

s.get(site_url)

Final step will be to login to site. You can use Firebug or Chrome Developer Console (depending of what browser you use) to examine what fields needs to be send (Go to Network tab).

s.post(site_url, data={'_username': 'user', '_password': 'pass'})

This two fields (_username, _password) seems to be valid for your site, but as I examine data which was send during POST request, there were more fields. I don't know if they are necessary.

After that you will be authenticated. Next thing will be to visit URL for file you would like to download.

s.get(file_url)

The link you provided contains query string with various options that are related probably to options you want to be highlighted. You can use it to download file with desired options.

Warning Note

Note that this site is not using HTTPS secure connection. Any credentials you will provide will go through the internet unencrypted and can be potentially see by someone who should not see them.