Selenium headless browser webdriver [Errno 104] Connection reset by peer

Question

I am trying to scrape data from the URLs below. But selenium fails when driver.get(url) Some times the error is [Errno 104] Connection reset by peer

Answer 1

After days of jingling around this issue, finally found the cause. Writing it here for the benefit of the community. The headless browser was failing due to lack of RAM on the server, strange error messages from webdriver were real pita.

The server was running straight up for 60 days without reboot, Rebooting it did the trick. After increasing the swap by 3 times, has not faced issue for past few days. Also scheduled a task to cleanup page file caches (http://www.yourownlinux.com/2013/10/how-to-free-up-release-unused-cached-memory-in-linux.html).

Answer 2

Found this question while looking for similar error. Look's like it's a selenium 3.8.1 and 3.9.0 bug.

https://github.com/SeleniumHQ/selenium/issues/5296

Downgrade to 3.8.0 solves this problem

Answer 3

I have been using Selenium and chromedriver (python3) for scraping purposes for some time now. With the latest Google Chrome update I had to deal with two issues.

1) Error on webdriver launch:

Solution: I had to add "no-sandbox" argument.

chrome_options.add_argument('--no-sandbox')

2) [Errno 104] Connection reset by peer:

Solution. There seems to be a problem with sockets and http requests. Either the webpage content is too big or you don't give the page enough time to load. At least that's what I thought. I set the maximum page load time to 60 seconds and it seems to be working fine.

driver.set_page_load_timeout(60)

I added a small delay between webdrivers initialisations which also seems to help.

time.sleep(0.5)