scrapy how to set referer url

Question

I need to set the referer url, before scraping a site, the site uses refering url based Authentication, so it does not allow me to login if the referer is not valid.

Answer 1

If you want to change the referer in your spider's request, you can change DEFAULT_REQUEST_HEADERS in the settings.py file:

DEFAULT_REQUEST_HEADERS = {
    'Referer': 'http://www.google.com' 
}