Beautiful Soup loop over div element in HTML

问题

I am attempting to use Beautiful Soup to extract some values out of a web page (not very much wisdom here..) which are hourly values from a weatherbug forecast. In Chrome developer mode I can see the values are nested within the div classes as shown in the snip below:

In Python I can attempt to mimic a web browser and find these values:

import requests
import bs4 as BeautifulSoup
import pandas as pd
from bs4 import BeautifulSoup

url = 'https://www.weatherbug.com/weather-forecast/hourly/san-francisco-ca-94103'

header = {
  "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36",
  "X-Requested-With": "XMLHttpRequest"
}

page = requests.get(url, headers=header)

soup = BeautifulSoup(page.text, 'html.parser')

With the code below, I can find 12 of these hour-card_mobile_cond div classes which seems about correct as when searching for hourly forecast I can see 12 hours/variables of future data. Im not sure why I am picking up a mobile device method to view...(?)

temp_containers = soup.find_all('div', class_ = 'hour-card__mobile__cond')
print(type(temp_containers))
print(len(temp_containers))

Output:

<class 'bs4.element.ResultSet'>
12

I am doing something incorrect below if I attempt to make up some code to loop thru all these div classes to dive down a little further.. I can 12 empty lists returned.. Would anyone have a tip at all where I can improve? Ultimately I am looking to put all 12 future hourly forecasted values into a pandas dataframe.

for div in temp_containers:
    a = div.find_all('div', class_ = 'temp ng-binding')
    print(a)

EDIT, complete code based on answer with pandas dataframe

import requests
from bs4 import BeautifulSoup
import pandas as pd


r = requests.get(
    "https://www.weatherbug.com/weather-forecast/hourly/san-francisco-ca-94103")
soup = BeautifulSoup(r.text, 'html.parser')

stuff = []

for item in soup.select("div.hour-card__mobile__cond"):
    item = int(item.contents[1].get_text(strip=True)[:-1])
    print(item)
    stuff.append(item)


df = pd.DataFrame(stuff)
df.columns = ['temp']

回答1:

The website is loaded via JavaScript dynamically once the page loads. so you can use requests-html or selenium.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.add_argument('--headless')
driver = webdriver.Firefox(options=options)

driver.get(
    "https://www.weatherbug.com/weather-forecast/hourly/san-francisco-ca-94103")


data = driver.find_elements_by_css_selector("div.temp.ng-binding")

for item in data:
    print(item.text)

driver.quit()

Output:

51°

52°

53°

54°

53°

53°

52°

51°

51°

50°

50°

49°

Updated per user-request:

import requests
from bs4 import BeautifulSoup

r = requests.get(
    "https://www.weatherbug.com/weather-forecast/hourly/san-francisco-ca-94103")
soup = BeautifulSoup(r.text, 'html.parser')

for item in soup.select("div.hour-card__mobile__cond"):
    item = int(item.contents[1].get_text(strip=True)[:-1])
    print(item, type(item))

Output:

51 <class 'int'>
52 <class 'int'>
53 <class 'int'>
53 <class 'int'>
53 <class 'int'>
53 <class 'int'>
52 <class 'int'>
51 <class 'int'>
51 <class 'int'>
50 <class 'int'>
50 <class 'int'>
50 <class 'int'>

回答2:

When you see class = "temp ng-binding" this means the div has class of "temp" and of "ng-binding" so looking for both won't work. Also, when I ran your script, the html of the temp containers, looked like this:

print(temp_containers[0])

<div class="temp">
                    51°
</div>

so I ran this and got results

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = 'https://www.weatherbug.com/weather-forecast/hourly/san-francisco-ca-94103'

header = {
  "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36",
  "X-Requested-With": "XMLHttpRequest"
}

page = requests.get(url, headers=header)

soup = BeautifulSoup(page.text, 'html.parser')

temp_containers = soup.find_all('div', class_ = 'hour-card__mobile__cond')
print(type(temp_containers))
print(len(temp_containers))

for div in temp_containers:
    a = div.find('div', class_ = 'temp')
    print(a.text)

来源：https://stackoverflow.com/questions/60728681/beautiful-soup-loop-over-div-element-in-html

标签

python

html

pandas

beautifulsoup