How to combine scrapy and htmlunit to crawl urls with javascript

Question

I\'m working on Scrapy to crawl pages,however,I can\'t handle the pages with javascript. People suggest me to use htmlunit, so I got it installed,but I don\'t know how to us

Answer 1

To handle the pages with javascript you can use Webkit or Selenium.

Here some snippets from snippets.scrapy.org:

Rendered/interactive javascript with gtk/webkit/jswebkit

Rendered Javascript Crawler With Scrapy and Selenium RC