unable to call firefox from selenium in python on AWS machine

Question

I am trying to use selenium from python to scrape some dynamics pages with javascript. However, I cannot call firefox after I followed the instruction of selenium on the pyp

Answer 1

For Debian 10 and Ubuntu 18.04 this is a complete running example:

1. Download the Chrome driver in ~/Downloads:
   $ wget https://chromedriver.storage.googleapis.com/80.0.3987.16/chromedriver_linux64.zip

2. Unpack it with unzip chromedriver_linux64.zip

3. Move the file to an executable folder (already with a path):
   $ sudo mv chromedriver /usr/local/bin

Then run this code in a notebook with Jupyter or within a a script:

from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.set_headless(headless=True)


browser = Chrome(chrome_options=chrome_options)
browser.get('http://www.linkedin.com/')
print(browser.page_source)

This will print the whole source HTML in the page.