Scrapy crawl with next page

Question

I have this code for scrapy framework:

# -*- coding: utf-8 -*-
import scrapy
from scrapy.contrib.spiders import Rule
from scrapy.linkextractors import LinkEx         


        

Answer 1

try this

  next_page_url = response.xpath('//a[@class="button next"]').extract_first()

  if next_page_url is not None:
        yield scrapy.Request(response.urljoin(next_page_url))