Need memory efficient way to store tons of strings (was: HAT-Trie implementation in java)

Question

I am working with a large set (5-20 million) of String keys (average length 10 chars) which I need to store in an in memory data structure that supports th

Answer 1

For space efficiency, O(log(n)) lookup, and simple code, try binary search over an array of characters. 20 million keys of average length 10 makes 200 million characters: 400MB if you need 2 bytes/char; 200MB if you can get away with 1. On top of this you need to somehow represent the boundaries between the keys in the array. If you can reserve a separator character, that's one way; otherwise you might use a parallel array of int offsets.

The simplest variant would use an array of Strings, at a high space cost from per-object overhead. It ought to still beat a hashtable in space efficiency, though not as impressively.