Huge memory usage of loading large dictionaries in memory

Question

I have a file on disk that\'s only 168MB. It\'s just a comma separated list of word,id. The word can be 1-5 characters long. There\'s 6.5 million lines.

I created a

Answer 1

I have the same problem though I'm later. The others has answered this question well. And I offer an easy to use(maybe not so easy :-) ) and rather efficient alternative, that's pandas.DataFrame. It performs well in memory usage when saving large data.