Order a json by field using scrapy

后端 未结 2 1748
后悔当初
后悔当初 2021-01-26 02:22

I have created a spider to scrape problems from projecteuler.net. Here I have concluded my answer to a related question with

I launch this with the comma

2条回答
  •  萌比男神i
    2021-01-26 03:00

    If I needed my output file to be sorted (I will assume you have a valid reason to want this), I'd probably write a custom exporter.

    This is how Scrapy's built-in JsonItemExporter is implemented.
    With a few simple changes, you can modify it to add the items to a list in export_item(), and then sort the items and write out the file in finish_exporting().

    Since you're only scraping a few hundred items, the downsides of storing a list of them and not writing to a file until the crawl is done shouldn't be a problem to you.

提交回复
热议问题