how to scrapy handle dns lookup failed

问题

I am looking to handle a DNS error when scraping domains Scrapy.

Here's the error that I am seeing:

ERROR: Error downloading <GET http://domain.com>: DNS lookup failed: address 'domain.com' not found [Errno 8] nodename nor servname provided, or not known.

How could I be notified when I get an error like this, so that I can handle it myself without Scrapy just throwing an error and moving on.

回答1:

Use errback along with callback:

Request(url, callback=your_callback, errback=your_errorback)

and errback:

def your_errorback(self, response):
    //your logic will be here

来源：https://stackoverflow.com/questions/25840049/how-to-scrapy-handle-dns-lookup-failed