Generating URL for Yahoo and Bing Scrapping for multiple pages with Python and BeautifulSoup

问题

I want to scrape news from different sources. I found a way to generate URL for scrapping multiple pages from google, but I think that there is a way to generate much shorter link.

Can you please tell me how to generate the URL for scrapping multiple pages for Bing and Yahoo news, and also, is there a way to make google url shorter.

This is the code for google:

from bs4 import BeautifulSoup
import requests

headers = {'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

term = 'usa'
page=0

for page in range(1,5):

    page = page*10

    url = 'https://www.google.com/search?q={}&tbm=nws&sxsrf=ACYBGNTx2Ew_5d5HsCvjwDoo5SC4U6JBVg:1574261023484&ei=H1HVXf-fHfiU1fAP65K6uAU&start={}&sa=N&ved=0ahUKEwi_q9qog_nlAhV4ShUIHWuJDlcQ8tMDCF8&biw=1280&bih=561&dpr=1.5'.format(term,page)
    print(url)

    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')

These are the URL-s for yahoo and bing, but for 1 page:

yahoo: url = 'https://news.search.yahoo.com/search?q={}'.format(term) bing: url = 'https://www.bing.com/news/search?q={}'.format(term)

回答1:

I am not sure are you looking after this shorten url for news.

from bs4 import BeautifulSoup
import requests

headers = {'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

term = 'usa'
page=0

for page in range(1,5):

    page = page*10

    url = 'https://www.google.com/search?q={}&tbm=nws&start={}'.format(term,page)
    print(url)

    response = requests.get(url, headers=headers,verify=False)
    soup = BeautifulSoup(response.text, 'html.parser')

#Yahoo:

from bs4 import BeautifulSoup
import requests

headers = {'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

term = 'usa'
page=1
while True:

    url ='https://news.search.yahoo.com/search?q={}&pz=10&b={}'.format(term,page)
    print(url)
    page = page + 10
    response = requests.get(url, headers=headers,verify=False)
    if response.status_code !=200:
        break
    soup = BeautifulSoup(response.text, 'html.parser')

来源：https://stackoverflow.com/questions/59047342/generating-url-for-yahoo-and-bing-scrapping-for-multiple-pages-with-python-and-b

标签

python

beautifulsoup