C# dictionary - how to solve limit on number of items?

Question

I am using Dictionary and I need to store almost 13 000 000 keys in it. Unfortunatelly, after adding 11 950 000th key I got an exception \"System out of memory\". Is there a

Answer 1

I think that you need a new approach to your processing.

I must assume that you obtain the data from a file or a database, either way that is where it should remain.

There is no way that you may actually increase the limit on the number of values stored within a Dictionary, other than increasing system memory, but eitherway it is an extremely inefficient means of processing such a alarge amount of data.

You should rethink your algorithm so that you can process the data in more manageable portions. It will mean processing it in stages until you get your result. This may mean many hundreeds of passes through the data, but it's the only way to do it.

I would also suggest that you look at using generics to help speed up this repetitive processing and cut down on memory usage.

Remember that there will still be a balancing act between system performance and access to externally stored data (be it external disk store or database).