Should I create pipeline to save files with scrapy?

Question

I need to save a file (.pdf) but I\'m unsure how to do it. I need to save .pdfs and store them in such a way that they are organized in a directories much like they are stor

Answer 1

It's a perfect tool for the job. The way Scrapy works is that you have spiders that transform web pages into structured data(items). Pipelines are postprocessors, but they use same asynchronous infrastructure as spiders so it's perfect for fetching media files.

In your case, you'd first extract location of PDFs in spider, fetch them in pipeline and have another pipeline to save items.