soup.select('.r a') in 'https://www.google.com/#q=vigilante+mic' gives empty list in python BeautifulSoup

问题

I am using BeautifulSoup to extract all links from google search results page. here's the snippet of the code:

    import requests,bs4

    res = requests.get('https://www.google.com/#q=vigilante+mic')

    soup = bs4.BeautifulSoup(res.text)

    linkElem = soup.select('.r a')

Now soup.select('.r a') is returning an empty list

thankyou

回答1:

That's because of the url you are using:

https://www.google.com/#q=vigilante+mic

Is a javascript version of the search. If you curl it you will see there are no answers in the html. This happens because the results are fetched through javascript and requests doesn't handle that.

Try this other url (that is not javascript based):

https://www.google.com/search?q=vigilante+mic

Now it works:

import requests,bs4

res = requests.get('https://www.google.com/search?q=vigilante+mic')

soup = bs4.BeautifulSoup(res.text)

linkElem = soup.select('.r a')

来源：https://stackoverflow.com/questions/33587097/soup-select-r-a-in-https-www-google-com-q-vigilantemic-gives-empty-lis