Loading a large dictionary using python pickle

Question

I have a full inverted index in form of nested python dictionary. Its structure is :

{word : { doc_name : [location_list] } }

For example l

Answer 1

Have you tried using an alternative storage format such as YAML or JSON? Python supports JSON natively from Python 2.6 using the json module I think, and there are third party modules for YAML.

You may also try the shelve module.