Can i use selenium with Scrapy without actual browser opening with python

问题

I want to do some web crawling with scrapy and python. I have found few code examples from internet where they use selenium with scrapy.

I don't know much about selenium but only knows that it automates some web tasks. and browser actually opens and do stuff. but i don't want the actual browser to open but i want everything to happen from command line.

Can i do that in selenium and scrapy

回答1:

You can use selenium with PyVirtualDisplay, at least on linux.

from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(1024, 768))
display.start()
browser = webdriver.Chrome()

回答2:

Updated: PhantomJS is abandoned, and you can use headless browsers directly now, such like Firefox and Chrome!

---

Use PhantomJS instead.

You can do browser = webdriver.PhantomJS() in selenium v2.32.0.

来源：https://stackoverflow.com/questions/13468755/can-i-use-selenium-with-scrapy-without-actual-browser-opening-with-python