How to click a link that has [removed]__doPostBack in href?

Question

I am writing a screen scraper script in python with module \'mechanize\' and I would like to use the mechanize.click_link() method on a link that has javascript:__doPostBack

Answer 1

I don't think mechanize supports Javascript; to scrape pages which intrinsically rely on Javascript execution for their functionality, you may need to use a different tool, such as Selenium RC.

Answer 2

I don't use mechanize, but I do a lot of web scraping myself with python.

When I run into a javascript function like __doPostBack, I do the following:
I access the web site in Firefox, and use the HttpFox extension to see the parameters of the POST request the browser sent to the web server when clicking the relevant link.
I then build the same request in python using urllib.parse.urlencode to build the query strings and POST data I need.
Sometimes the website uses cookies as well, so I just use python's http.cookiejar.

I have used this technique successfully several times.

Answer 3

>>> next_link.__class__.__name__
'Link'
>>> next_link
Link(base_url='http://www.citius.mj.pt/Portal/consultas/ConsultasDistribuicao.aspx', url="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext','')", text='2', tag='a', attrs=[('id', 'ctl00_ContentPlaceHolder1_Pager1_lnkNext'), ('title', 'P\xc3\xa1gina seguinte: 2'), ('href', "javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext','')")])
>>> req = mech.click_link(next_link)
>>> req
<urllib2.Request instance at 0x025BEE40>
>>> req.has_data()
False