Selenium headless browser webdriver [Errno 104] Connection reset by peer

Question

I am trying to scrape data from the URLs below. But selenium fails when driver.get(url) Some times the error is [Errno 104] Connection reset by peer

Answer 1

I have been using Selenium and chromedriver (python3) for scraping purposes for some time now. With the latest Google Chrome update I had to deal with two issues.

1) Error on webdriver launch:

Solution: I had to add "no-sandbox" argument.

chrome_options.add_argument('--no-sandbox')

2) [Errno 104] Connection reset by peer:

Solution. There seems to be a problem with sockets and http requests. Either the webpage content is too big or you don't give the page enough time to load. At least that's what I thought. I set the maximum page load time to 60 seconds and it seems to be working fine.

driver.set_page_load_timeout(60)

I added a small delay between webdrivers initialisations which also seems to help.

time.sleep(0.5)