Empty value from web scraping with python beautiful soup

问题

I am trying to scrap this website but having issues extracting the right values. The website is on the prices of Sliver, gold, palladium and platinum. http://www.lbma.org.uk/precious-metal-prices The html of the website is below.

      <div id="header-tabs-content" data-tabs-content="header-tabs">
        <div class="tabs-panel is-active" id="header-tabs-panel1" 
     role="tabpanel" aria-hidden="false" aria-labelledby="header-tabs-
     panel1-label">
          <a href="/precious-metal-prices">
          <p>Gold Price</p>
          <p>AM: 
              <strong>$
              <span id="daily_gold_am_usd">1325.40</span>
              </strong> <br>
            <em class="update">Updated: <span 
          id="daily_gold_am_timestamp">08/03 10:31:00</span></em> </p>
          <p>PM: 
              <strong>$
              <span id="daily_gold_pm_usd">1321.00</span>
              </strong> <br>
            <em class="update">Updated: <span 
          id="daily_gold_pm_timestamp">08/03 15:02:00</span></em> </p>
            </a>

I am interested to obtain the daily_gold_am_usd of 1325.40 and the daily_gold_pm_usd of 1321.00 from the html data structure below. However the code I attempted after consulting from past posts can't seems to return these values.

#Import packages

import pandas as pd
import numpy as np
import requests
from bs4 import BeautifulSoup

#define url and get html

url = "http://www.lbma.org.uk/precious-metal-prices"
r=requests.get(url)
data=r.text
soup = BeautifulSoup(data,"html.parser")

#Find the object of interest

gold_am_price = soup.find("span", {"id": "daily_gold_am_usd"})
Au_price_am = gold_am_price.text.strip()

gold_pm_price = soup.find("span", {"id": "daily_gold_pm_usd"})
Au_price_pm = gold_pm_price.text.strip()

Appreciate any help. Thanks guys.

回答1:

Those values comes from XHR to http://lbma.oblive.co.uk/api/today/both.json, so you can get them as:

import requests
url = "http://lbma.oblive.co.uk/api/today/both.json"
response = requests.get(url).json()

Output of print(response):

{'gold': {'am': {'usd': '1325.40', 'gbp': '955.080', 'eur': '1070.390', 'timesta
mp': '08/03 10:31:00'}, 'pm': {'usd': '1321.00', 'gbp': '953.370', 'eur': '1069.
750', 'timestamp': '08/03 15:02:00'}}, 'silver': {'usd': '16.48000', 'usdc': '16
48', 'gbp': '11.89000', 'gbpp': '1189', 'eur': '13.31000', 'eurc': '1331', 'time
stamp': '08/03 12:01:00'}, 'platinum': {'am': {'usd': '949.00', 'gbp': '683.960'
, 'eur': '766.250', 'timestamp': '08/03 09:49:00'}, 'pm': {'usd': '954.00', 'gbp
': '687.570', 'eur': '769.670', 'timestamp': '08/03 14:09:00'}}, 'palladium': {'
am': {'usd': '970.00', 'gbp': '699.100', 'eur': '783.210', 'timestamp': '08/03 0
9:49:00'}, 'pm': {'usd': '985.00', 'gbp': '709.910', 'eur': '794.680', 'timestam
p': '08/03 14:09:00'}}}

Then you can extract required as:

response['gold']['am']['usd']  #  1325.40
response['gold']['pm']['usd']  #  1321.00

来源：https://stackoverflow.com/questions/49191954/empty-value-from-web-scraping-with-python-beautiful-soup