How to generate the start_urls dynamically in crawling?

Question

I am crawling a site which may contain a lot of start_urls, like:

http://www.a.com/list_1_2_3.htm
         


        

Answer 1

The best way to generate URLs dynamically is to override the start_requests method of the spider:

from scrapy.http.request import Request

def start_requests(self):
      with open('urls.txt', 'rb') as urls:
          for url in urls:
              yield Request(url, self.parse)