How to wait for page load to complete?

醉酒当歌 提交于 2019-12-11 13:02:54

问题


I'm trying to get available boot size (under $('option.addedOption')) from http://www.neimanmarcus.com/Stuart-Weitzman-Reserve-Suede-Over-the-Knee-Boot-Black/prod179890262/p.prod

I tried below code, but it always returned before the size is got.

# config.url = 'http://www.neimanmarcus.com/Stuart-Weitzman-Reserve-Suede-Over-the-Knee-Boot-Black/prod179890262/p.prod'
import urllib2
import requests
import config
import time
from lxml.cssselect import CSSSelector
from lxml.html import fromstring

print config.url
headers = {
    "Host": "www.neimanmarcus.com",
    "Connection": "keep-alive",
    "Content-Length": 106,
    "Pragma": "no-cache",
    "Cache-Control": "no-cache",
    "Accept": "*/*",
    "Origin": "http://www.neimanmarcus.com",
    "X-Requested-With": "XMLHttpRequest",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36",
    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
    "Referer": "http://www.neimanmarcus.com/Stuart-Weitzman-Reserve-Suede-Over-the-Knee-Boot-Black/prod179890262/p.prod",
    "Accept-Language": "en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,fr;q=0.2,cs;q=0.2,zh-TW;q=0.2"
}
request = urllib2.Request(config.url, headers=headers)
html = urllib2.urlopen(request)
time.sleep(10)
html = html.read()
print html
html = fromstring(html)
sel = CSSSelector('option.addedOption')
try:
    options = sel(html)
    print options
except Exception as e:
    print e

I found size is got in a request 'http://www.neimanmarcus.com/product.service' (actually the Header is created according to the request header of this request).

How can I get the whole page information (especially with the boot size)?

I also tried to request http://www.neimanmarcus.com/product.service directly but failed as well.


回答1:


As I understand correctly: no matter how long the code sleeps it still hasn't loaded the shoe size?

Since you are not using a headless browser you do not execute javascript on the requested page. Try using a headless browser like PhantomJS. Here a list of more headless browsers.

Here one way how to use PhantomJS in Python.




回答2:


Use it like:

with urllib2.urlopen(request) as response:
   html = response.read()
   print html
   html = fromstring(html)
   sel = CSSSelector('option.addedOption')
   try:
       options = sel(html)
       print options
   except Exception as e:
       print e

instead of

html = urllib2.urlopen(request)
time.sleep(10)
html = html.read()
...


来源:https://stackoverflow.com/questions/37138085/how-to-wait-for-page-load-to-complete

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!