empty result set beautiful soup

问题

Scraping article from New York Times site and getting an empty result set. My aim is to get the urls and the text of the h3 items. When I run this I get an empty set. Printing the section scrape shows I'm on the right path... target url - http://query.nytimes.com/search/sitesearch/?action=click&contentCollection&region=TopBar&WT.nav=searchWidget&module=SearchSubmit&pgtype=sectionfront#/san+diego/24hours

url = "http://query.nytimes.com/search/sitesearch/?action=click&contentCollection&region=TopBar&WT.nav=searchWidget&module=SearchSubmit&pgtype=sectionfront{data}"
html = urlopen(url.format(data="#"+'/san+diego/24hours'))

soup = BeautifulSoup(html.read().decode('utf-8'),"lxml")
section = soup.find("ol",class_='searchResultsList flush')
items = section.find_all('li', class_="story")
print items

回答1:

The HTML indeed doesn't contain the data. Looking at the Network tab in Chrome Developer Tools, you can see that the search results are fetched by an AJAX query to this URL: http://query.nytimes.com/svc/add/v1/sitesearch.json?q=san%20diego&begin_date=24hoursago&facet=true

Here's a screenshot of finding that:

You have to open the Developer Tools (try the View menu), choose the Network tab, reload the page, and look around. XHR = XmlHttpRequest which these days is referred to as an AJAX request. It means some Javascript asked the server for data.

This is JSON, so you're actually lucky, because this will be much better than parsing HTML.

来源：https://stackoverflow.com/questions/37335219/empty-result-set-beautiful-soup