Order a json by field using scrapy

后端 未结 2 1743
后悔当初
后悔当初 2021-01-26 02:22

I have created a spider to scrape problems from projecteuler.net. Here I have concluded my answer to a related question with

I launch this with the comma

2条回答
  •  青春惊慌失措
    2021-01-26 02:52

    By now I've found a working solution using pipeline:

    import json
    
    class JsonWriterPipeline(object):
    
        def open_spider(self, spider):
            self.list_items = []
            self.file = open('euler.json', 'w')
    
        def close_spider(self, spider):
            ordered_list = [None for i in range(len(self.list_items))]
    
            self.file.write("[\n")
    
            for i in self.list_items:
                ordered_list[int(i['id']-1)] = json.dumps(dict(i))
    
            for i in ordered_list:
                self.file.write(str(i)+",\n")
    
            self.file.write("]\n")
            self.file.close()
    
        def process_item(self, item, spider):
            self.list_items.append(item)
            return item
    

    Though it may be non optimal, because the guide suggests in another example:

    The purpose of JsonWriterPipeline is just to introduce how to write item pipelines. If you really want to store all scraped items into a JSON file you should use the Feed exports.

提交回复
热议问题