Recording the total time taken for running a spider in scrapy

Question

I am using scrapy to scrap a site

I had written a spider and fetched all the items from the page and saved to a csv file, and now i want to save the total exec

Answer 1

I'm quite a beginner but I did it in a bit simpler method and I hope it makes sense.

import datetime

then declare two global variables i.e self.starting_time and self.ending_time.

Inside the constructor of the spider class, set the starting time as

def __init__(self, name=None, **kwargs):
        self.start_time = datetime.datetime.now()

After that, use the closed method to find the difference between the ending and the starting. i.e

def closed(self, response):
   self.ending_time = datetime.datetime.now()
   duration = self.ending_time - self.starting_time
   print(duration)

That's pretty much of it. The closed method is called soon after the spider has ended the process. See the documentation here.