Scrapy: effective way to test inline requests

问题

I wrote a spider using scrapy-inline-requests library. So the parse method in my spider looks something like this:

@inline_requests
def parse(self, response1):
    item = MyItem()
    loader = ItemLoader(item=item, response=response1)

    #extracting some data from the response1

    try:
        response 2 = yield Request(some_url)
        #extracting some other data from response2
    except Exception:
            self.logger.warning("Failed request to: %s", some_url)

    yield loader.load_item()

I want to effectively test this method. I can easily write a test, in which a create a fake mock response1 and pass it to the function. However, I've got no idea how to mock response2 and get the complete item with the data from both the fake responses. Do you have any suggestions?

回答1:

Might be a bit late but check out the tests in scrapy-inline-requests' github repo: https://github.com/rmax/scrapy-inline-requests/blob/master/tests/test_inline_requests.py.

Basically, feed the callback a list of Response objects in the order that your callback will yield Requests for these responses.

来源：https://stackoverflow.com/questions/39733655/scrapy-effective-way-to-test-inline-requests