selenium implicitly wait doesn't work

Question

This is the first time I use selenium and headless browser as I want to crawl some web page using ajax tech.

The effect is great, but for some case it takes too much

Answer 1

As you mentioned in your question it takes too much time to load the whole page(especially when some resource is unavailable) is pretty much possible if the Application Under Test (AUT) uses JavaScript or AJAX Calls.

• In your first scenario you have induced both set_page_load_timeout(5) and set_script_timeout(5)
  • set_page_load_timeout(time_to_wait) : Sets the amount of time to wait for a page load to complete before throwing an exception.
  • set_script_timeout(time_to_wait) : Sets the amount of time that the script should wait during an execute_async_script call before throwing an exception.

Hence the Application Under Test being dependent on JavaScript or AJAX Calls in presence of both the conditions raises TimeoutException.

• In your second scenario you have induced both implicitly_wait(2) and WebDriverWait(driver, 2, 0.5).

  • implicitly_wait(time_to_wait) : Sets the timeout to implicitly wait for an element to be found or a command to complete.
  • WebDriverWait(driver, timeout, poll_frequency=0.5, ignored_exceptions=None) : Sets the timeout in-conjunction with different expected_conditions
  • But you are experiancing a very long timeout(40+ seconds) as it is clearly mentioned in the docs Do not mix implicit and explicit waits which can cause unpredictable wait times

WARNING : Do not mix implicit and explicit waits. Doing so can cause unpredictable wait times. For example setting an implicit wait of 10 seconds and an explicit wait of 15 seconds, could cause a timeout to occur after 20 seconds.

Solution :

The best solution would be to remove all the instance of implicitly_wait(time_to_wait) and replace with WebDriverWait() for a stable behavior of the Application Under Test (AUT).

---

Update

As per your counter question, the current code block looks perfect. The measurement of time which you are seeing as time used: 44.6049938202 s is the time required for the Web Page to load completely and functionally that is the time required for the Client (i.e. the Web Browser) to return back the control to the WebDriver instance once 'document.readyState' equals to "complete" is achieved. Selenium or as an user you have no control on this rendering process. However for a better performance you may follow the best practices as follows :

• Keep your JDK version updated currently Java SE Development Kit 8u162
• Keep your Selenium Client version updated currently selenium 3.9.0
• Keep your WebDriver version updated.
• Keep your Web Browser version updated.
• Clean you Project Workspace within your IDE regularly to build your project with required dependencies only.
• Use CCleaner tool to wipe away the OS chores before and after your Test Suite execution.
• If your Web Browser base version is too old uninstall the Web Browser through Revo Uninstaller and install a recent GA released version of the Web Browser.
• Execute your Test.