Why request.get() returning wrong page content?

问题

I have been trying to parse a webpage using BeautifulSoup. When I import urlopen fromm urllib.request and open https://pbejobbers.com it returns following instead of webpage itself:

<html>
  <body>
    <script src="/aes.min.js" type="text/javascript"></script>
    <script>
         function toNumbers(d){var e=[];d.replace(/(..)/g,function(d){e.push(parseInt(d,16))});return e}function toHex(){for(var d=[],d=1==arguments.length&&arguments[
      0].constructor==Array?arguments[0]:arguments,e="",f=0;f<d.length;f++)e+=(16>d[f]?"0":"")+d[f].toString(16);return e.toLowerCase()}var a=toNumbers("0181cdf0013bf7
      0f89e91be7ef0d00c2"),b=toNumbers("a168ceeade18bccc1cdd77af68ef1753"),c=toNumbers("200a38f39b6a3fe3564acf9bd88c25da");document.cookie="OCXS="+toHex(slowAES.decryp
      t(c,2,a,b))+"; expires=Thu, 31-Dec-37 23:55:55 GMT; path=/";document.location.href="http://pbejobbers.com/product/search?search=USC4215&81e93addddb02a10cd0652f09
      370ae96=1";
    </script>
  </body>
</html>

I have array of UPC codes that I use to find products that I am looking for. I pass the array to a function and parse the html to find necessary tags but I can get to the actual html. Here is my code:

from urllib.request import urlopen
from bs4 import BeautifulSoup

upc_codes = ['USC4215', 'USC4225', 'USC12050']

def retrunh1(upc):
    html = urlopen('https://pbejobbers.com/product/search?search={}'.format(upc))
    soup = BeautifulSoup(html, 'html.parser')
    print(soup.prettify())

if __name__=='__main__':
    for upc in upc_codes:
        retrunh1(upc)

I think the problem is with request function. I isolated it to see what it is return and I am getting the same html back as above when I do this:

import requests

r = requests.get('https://pbejobbers.com')

print(r.text)

I am quite new to web parsing and I need some suggestion on how to resolve this. Thanks

回答1:

Please try this.

Python code:

from urllib.request import urlopen
from bs4 import BeautifulSoup
import requests
import re

upc_codes = ['USC4215', 'USC4225', 'USC12050']

def retrunh1(upc):
    payload = {'search': upc }
    r = requests.get('https://pbejobbers.com/product', params=payload)
    matches = re.search(r'document\.location\.href=\"(:?.*)=1\";', str(r.text), re.M|re.S)
    url = matches[1]

    response = requests.get(url)

    for resp in response.history:
      r = requests.post(resp.headers['Location'])
      soup = BeautifulSoup(r.content, 'html.parser')
      print(soup.prettify())

if __name__=='__main__':
    for upc in upc_codes:
        retrunh1(upc)

Output:

<div class="page-area-container">
    <div class=" middlebar">
        <div class=" middlebar__left">
            <a class=" logo" href="/">
                <img alt="PBE Jobbers" class=" logo-img" src="/bundles/pjfrontend/pbejobbers/images/logo/pbe-logo.svg?version=9d4c5d60"/>
            </a>
        </div>
        ...
    </div>
    ...
</div>

回答2:

The javascript probably populates the html portion of the page dynamically when the browser starts executing it, so urllib can't download the complete source.

Your python script needs to use a headless browser framework like Selenium to load the page as a browser would and then extract what you need.

As others mentioned, please do not violate their terms of service, especially if the data is private/behind a login page

回答3:

when i manually search USC4215, the url is https://pbejobbers.com/product/search?search=USC4215&_rand=0.35863039778309025

The website is appending a random secret _rand to prevent robot web-crawling. u need to make a request with a valid random secret to receive response.

In fact, usually the secret is generated with a set of cookies, if u click Inspect ==> Network ==> Doc and Ctrl + R for refreshing the website, you would find more about the network traffic as you make another request, precisely what is your http request and response content.

来源：https://stackoverflow.com/questions/59727663/why-request-get-returning-wrong-page-content

标签

python

web-scripting