Python Scrapy: Convert relative paths to absolute paths

Question

I have amended the code based on solutions offered below by the great folks here; I get the error shown below the code here.

from scrapy.spider import BaseSp         


        

Answer 1

From Scrapy docs:

def parse(self, response):
    # ... code ommited
    next_page = response.urljoin(next_page)
    yield scrapy.Request(next_page, self.parse)

that is, response object has a method to do exactly this.