Using a pickle file as a substitute for in-memory list for large dataset

Question

I have to crawl a locally running version of Wikipedia and have to do the following -

1. Keep count of external (non-wikipedia) and internal links.
2. Keep c