scrapy-splash

I'm writing a scrapy spider where I need to render some of the responses with splash. My spider is based on CrawlSpider. I need to render my start_url responses to feed my crawl spider. Unfortunately my crawl spider stops after rendering of the first responds. Any idea what is going wrong? class VideoSpider(CrawlSpider): start_urls = ['https://juke.com/de/de/search?q=1+Mord+f%C3%BCr+2'] rules = ( Rule(LinkExtractor(allow=()), callback='parse_items',process_request = "use_splash",), ) def use_splash(self, request): request.meta['splash'] = { 'endpoint':'render.html', 'args':{ 'wait':0.5, } }

I've read through many of the related questions but am still unclear how to do this as there are many software combinations available and many solutions seem outdated. What is the best way to install the following on my virtual environment on OSX: python-gtk2 python-webkit python-jswebkit Do I also have to install GTK+ and Webkit? If so, how? Would also appreciate a simple explanation on how these pieces of software work together. (I'm trying to use scrapyjs which requires these libraries) You should try using pip (A tool for installing and managing Python packages.) https://pypi.python.org

So far, I have been using just scrapy and writing custom classes to deal with websites using ajax. But if I were to use scrapy-splash, which from what I understand, scrapes the rendered html after javascript, will the speed of my crawler be affected significantly? What would be the comparison between time it takes to scrape a vanilla html page with scrapy vs javascript rendered html with scrapy-splash? And lastly, how do scrapy-splash and Selenium compare? It depends on the amount of javascript present on the page. You must know that to render all the javascript the splash takes some time and

I have scrapy and scrapy-splash set up on a AWS Ubuntu server. It works fine for a while, but after a few hours I'll start getting error messages like this; Traceback (most recent call last): File "/home/ubuntu/.local/lib/python3.5/site- packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/home/ubuntu/.local/lib/python3.5/site- packages/twisted/python/failure.py", line 393, in throwExceptionIntoGe nerator return g.throw(self.type, self.value, self.tb) File "/home/ubuntu/.local/lib/python3.5/site- packages/scrapy/core