Python Scrapy on offline (local) data

Question

I have a 270MB dataset (10000 html files) on my computer. Can I use Scrapy to crawl this dataset locally? How?

Answer 1

SimpleHTTP Server Hosting

If you truly want to host it locally and use scrapy, you could serve it by navigating to the directory it's stored in and run the SimpleHTTPServer (port 8000 shown below):

python -m SimpleHTTPServer 8000

Then just point scrapy at 127.0.0.1:8000

$ scrapy crawl 127.0.0.1:8000

file://

An alternative is to just have scrapy point to the set of files directly:

$ scrapy crawl file:///home/sagi/html_files # Assuming you're on a *nix system

Wrapping up

Once you've set up your scraper for scrapy (see example dirbot), just run the crawler:

$ scrapy crawl 127.0.0.1:8000

If links in the html files are absolute rather than relative though, these may not work well. You would need to adjust the files yourself.

Answer 2

Go to your Dataset folder :

import os
files = os.listdir(os.getcwd())
for file in files:
    with open(file,"r") as f:
        page_content = f.read()
        #do here watever you want to do with page_content. I guess parsing with lxml or Beautiful soup.

No need to go for Scrapy !