Login on a site using urllib

Question

I\'m trying to get information from this site http://cheese.formice.com/maps/@5865339 , but when i request using urllib.urlopen, its says that i need to login, i was using t

Answer 1

It's not using urllib directly, but you may find it easier working with the requests package. requests has a session object see this answer

import requests

url = 'http://cheese.formice.com/forum/login/login'
login_data = dict(login='Cfmaccount', password='tfmdev321')
session = requests.session()

r = session.post(url, data=login_data)

That will log you in to the site. You can verify with:

print r.text #prints the  response.

Once logged in, you can call the specific url you want.

r2 = session.get('http://cheese.formice.com/maps/@5865339')
print r2.content #prints the raw html you can now parse and scrape