Reliably detect page load or time out, Selenium 2

Question

I am writing a generic web-scraper using Selenium 2 (version 2.33 Python bindings, Firefox driver). It is supposed to take an arbitrary URL, load the page, and rep

Answer 1

As far as i know, your readystate_complete is not doing anything as driver.get() is already checking for that condition. Anyway, i have seen it not working in many cases. One thing you could try is to route your traffic thru a proxy and use that for pinging for any network traffic. Ie browsermob has wait_for_traffic_to_stop method:

def wait_for_traffic_to_stop(self, quiet_period, timeout):
"""
Waits for the network to be quiet
:Args:
- quiet_period - number of seconds the network needs to be quiet for
- timeout - max number of seconds to wait
"""
    r = requests.put('%s/proxy/%s/wait' % (self.host, self.port),
        {'quietPeriodInMs': quiet_period, 'timeoutInMs': timeout})
    return r.status_code