web scraping dynamic content with python

Question

I\'d like to use Python to scrape the contents of the \"Were you looking for these authors:\" box on web pages like this one: http://academic.research.microsoft.com/Search?q

Answer 1

Instead of trying to reverse engineer it, you can use ghost.py to directly interact with JavaScript on the page.

If you run the following query in a chrome console, you'll see it returns everything you want.

document.getElementsByClassName('inline-text-org');

Returns

[
​University of Manchester​
, 
 
​University of California ...​
​
  etc...

You can run JavaScript through python in a real life DOM using ghost.py.

This is really cool:

from ghost import Ghost
ghost = Ghost()
page, resources = ghost.open('http://academic.research.microsoft.com/Search?query=lander')
result, resources = ghost.evaluate(
    "document.getElementsByClassName('inline-text-org');")