问题
I am trying to scrape listings from Airbnb. Every listing has its own ID. However, the output of the code below is None:
import requests, bs4
response = requests.get('https://www.airbnb.pl/s/Girona--Hiszpania/homes?refinement_paths%5B%5D=%2Fhomes&query=Girona%2C%20Hiszpania&checkin=2018-07-04&checkout=2018-07-25&allow_override%5B%5D=&ne_lat=42.40450221314142&ne_lng=3.3245690859736214&sw_lat=41.97668610374056&sw_lng=1.7960961855829964&zoom=10&search_by_map=true&s_tag=nrGiXgWC')
soup = bs4.BeautifulSoup(response.text, "html.parser")
element = soup.find(id="listing-18354577")
print(element)
Why does the soup does not see this element, even though it is already loaded on the page?
Is it in a container of some type I need to scrape differently?
回答1:
The element with id listing-18354577 is created via javascript after the initial HTML page has loaded into your browser. Requests is just an HTTP client, not a full-fledged browser engine, so it doesn't execute the Javascript that ends up fetching that element. The response from Requests is just the initial HTML of the page (which does not include listing-18354577).
回答2:
requests don't wait for js, you can use selenium to load all page and after this use bs4 for example this works:
import requests, bs4
from selenium import webdriver
# put the path to chromedriver
driver = webdriver.Chrome('path/to/chromedriver')
website = "https://www.airbnb.pl/s/Girona--Hiszpania/homes?refinement_paths%5B%5D=%2Fhomes&query=Girona%2C%20Hiszpania&checkin=2018-07-04&checkout=2018-07-25&allow_override%5B%5D=&ne_lat=42.40450221314142&ne_lng=3.3245690859736214&sw_lat=41.97668610374056&sw_lng=1.7960961855829964&zoom=10&search_by_map=true&s_tag=nrGiXgWC"
driver.get(website)
html = driver.page_source
soup = bs4.BeautifulSoup(html, "html.parser")
element = soup.find(id="listing-18354577")
print(element)
Output
<div class="_1wq3lj" id="listing-18354577"> ... #and many other data
来源：https://stackoverflow.com/questions/51117692/beautifulsoup-does-not-see-element-even-though-it-is-present-on-a-page