Parse a .Net Page with Postbacks

Question

I need to read data from an online database that\'s displayed using an aspx page from the UN. I\'ve done HTML parsing before, but it was always by manipulating query-string

Answer 1

You may still only need to send one request, but that one request can be rather complicated. ASP.Net is notoriously difficult (though not impossible) to screen scrape. Between event validation and the ViewState, it's tricky to get your requests just right. The simplest way to do it is often to use a sniffer tool like fiddler to see exactly what the http request looks like, and then just mimic that request.

If you do still need to send two requests, it's because the first request also places some state in a session somewhere, and that means whatever you use to send those requests needs to be able to send them with the same session. This often means supporting cookies.

Answer 2

I'd look at HtmlAgilityPack with the FormProcessor addon.

Answer 3

Watin would be my first choice. You would code the selecting and clicking, then parse the HTML after.