How to access scrapy settings from item Pipeline

后端 未结 4 1207
攒了一身酷
攒了一身酷 2020-12-07 20:48

How do I access the scrapy settings in settings.py from the item pipeline. The documentation mentions it can be accessed through the crawler in extensions, but I don\'t see

4条回答
  •  执笔经年
    2020-12-07 21:32

    UPDATE (2020-10-28)
    Please note that this answer is now ~ 7years old, so it's validity can no longer be ensured.

    The way to access your Scrapy settings (as defined in settings.py) from within your_spider.py is simple. All other answers are way too complicated. The reason for this is the very poor maintenance of the Scrapy documentation, combined with many recent updates & changes. Neither in the "Settings" documentation "How to access settings", nor in the "Settings API" have they bothered giving any workable example. Here's an example, how to get your current USER_AGENT string.

    Just add the following lines to your_spider.py:

    # To get your settings from (settings.py):
    from scrapy.utils.project import get_project_settings
    ...
    class YourSpider(BaseSpider):
        ...
        def parse(self, response):
            ...
            settings = get_project_settings()
            print "Your USER_AGENT is:\n%s" % (settings.get('USER_AGENT'))
            ...
    

    As you can see, there's no need to use @classmethod or re-define the from_crawler() or __init__() functions. Hope this helps.

    PS. I'm still not sure why using from scrapy.settings import Settings doesn't work the same way, since it would be the more obvious choice of import?

提交回复
热议问题