How can scrapy export items to separate csv files per item

后端 未结 3 623
隐瞒了意图╮
隐瞒了意图╮ 2020-12-08 05:02

I am scraping a soccer site and the spider (a single spider) gets several kinds of items from the site\'s pages: Team, Match, Club etc. I am trying to use the CSVItemExporte

3条回答
  •  伪装坚强ぢ
    2020-12-08 05:57

    I have tried the answer. It seems do not work in the latest version (2.21).

    I have included my code for your reference:

    class MultiCSVItemPipeline(object):
        SaveTypes = ['CentalineTransactionsItem','CentalineTransactionsDetailItem','CentalineBuildingInfo']
    
        def open_spider(self, spider):
            self.files = dict([ (name, open(name+'.csv','w+b')) for name in self.SaveTypes ])
            self.exporters = dict([ (name,CsvItemExporter(self.files[name])) for name in self.SaveTypes])
            [e.start_exporting() for e in self.exporters.values()]
    
        def close_spider(self, spider):
            [e.finish_exporting() for e in self.exporters.values()]
            [f.close() for f in self.files.values()]
    
        def process_item(self, item, spider):
            what = type(item).__name__
            if what in set(self.SaveTypes):
                self.exporters[what].export_item(item)
            return item
        
    

提交回复
热议问题