Not able to do webscrapping using beautifulsoup and requests

问题

I am trying to scrape the first two sections values i.e 1*2 and DOUBLECHANCE sections values using bs4 and requests from this website https://web.bet9ja.com/Sport/SubEventDetail?SubEventID=76512106 The code which I written is:

import bs4 as bs
import urllib.request

source = urllib.request.urlopen('https://web.bet9ja.com/Sport/SubEventDetail?SubEventID=76512106')
soup = bs.BeautifulSoup(source,'lxml')

for div in soup.find_all('div', class_='SEItem ng-scope'):
    print(div.text)

when I run I am not getting anything please help me anyone

回答1:

The page is loaded via JavaScript, so you have 2 option. or to use selenium or to call the Direct API.

Instead of using Selenium, I've called the API directly and got the required info.

Further explanation about XHR & API < can be found once you click here.

import requests

data = {
    'IDGruppoQuota': '0',
    'IDSottoEvento': '76512106'
}


def main(url):
    r = requests.post(url, json=data).json()
    count = 0
    for item in r['d']['ClassiQuotaList']:
        count += 1
        print(item['ClasseQuota'], [x['Quota']
                                    for x in item['QuoteList']])
        if count == 2:
            break


main("https://web.bet9ja.com/Controls/ControlsWS.asmx/GetSubEventDetails")

Output:

1X2 ['3.60', '4.20', '1.87']
Double Chance ['1.83', '1.19', '1.25']

回答2:

Try:

import bs4 as bs
import urllib.request
import lxml
source = urllib.request.urlopen('https://web.bet9ja.com/Sport/SubEventDetail?SubEventID=76512106')
soup = bs.BeautifulSoup(source,'lxml')
a = soup.find_all('div')
for i in a:
    try:
        print(i['class'])
    except:
        pass
        try:
            sp = i.find_all('div')
            for j in sp:
                print(j['class'])
        except:
            pass

This helps you find available classes in the <div> tag.
You get nothing when the class you give doesn't exist. This happens as many of the sites are dynamically generated and requests can't get them. In these cases, we need to use selenium.

来源：https://stackoverflow.com/questions/61139207/not-able-to-do-webscrapping-using-beautifulsoup-and-requests

标签

python

web-scraping

beautifulsoup