Scrapy crawling not working on ASPX website

Question

I\'m scraping the Madrid Assembly\'s website, built in aspx, and I have no idea how to simulate clicks on the links where I need to get the corresponding politicians from. I

Answer 1

Do agree with ELRuLL - Firebug is your best friend while scraping. If you want to avoid JS simulation then you need reproduce carefully all the params/headers that are being sent.

For example from what I see for __EVENTTARGET you're sending just id="ctl00_m_g_36ea0310_893d_4a19_9ed1_88a133d06423_ctl00_Repeater2_ctl01_lnk_Diputado")

and via Firebug we see that:

__EVENTTARGET=ctl00$m$g_36ea0310_893d_4a19_9ed1_88a133d06423$ctl00$Repeater2$ctl01$lnk_Diputado

It maybe the reason and maybe not, just repeat and test.

Firebug link just in case.