I am scrapping a site using scrapy framework and having trouble clicking on a javascript link for opening another page.
I can identify the code on the page as:
As far as I know, scrappy crawler implemented over urrlib2 and urllib obviously dont work with js. For working with js you can use qt webkit or selenium for example. Or you could find all ajax links on page and see how implemented a data exchange with the server and send response to server api indirectly.