web scraping dynamic content with python

Question

I\'d like to use Python to scrape the contents of the \"Were you looking for these authors:\" box on web pages like this one: http://academic.research.microsoft.com/Search?q

Answer 1

For scraping dynamic content, you need not a simple scraper but a full-fledged headless browser.

dhamaniasad/HeadlessBrowsers: A list of (almost) all headless web browsers in existence is the fullest list of these that I've seen; it lists which languages each has bindings for.

(Note that more than a few of the listed projects are abandoned!)