Download a full page with scrapy

Question

I want to download the content a whole page using scrapy.

With selenium this is quite easy:

import os,sys
reload(sys)  
sys.setdefaultencoding(\'utf         


        

Answer 1

Code will download this page and save it in file download-a-full-page-with-scrapy.html

test_scr.py

import scrapy
class TestSpider(scrapy.Spider):
    name = "test"

    start_urls = [
        "http://stackoverflow.com/questions/38233614/download-a-full-page-with-scrapy",
    ]

    def parse(self, response):
        filename = response.url.split("/")[-1] + '.html'
        with open(filename, 'wb') as f:
            f.write(response.body)

run scrapy by this command

scrapy runspider test_scr.py