Using one Scrapy spider for several websites

Question

I need to create a user configurable web spider/crawler, and I\'m thinking about using Scrapy. But, I can\'t hard-code the domains and allowed URL regex:es -- this will inst

Answer 1

Shameless self promotion on domo! you'll need to instantiate the crawler as given in the examples, for your project.

Also you'll need to make the crawler configurable on runtime, which is simply passing the configuration to crawler, and overriding the settings on runtime, when configuration changed.