I\'ve currently got a spreadsheet type program that keeps its data in an ArrayList of HashMaps. You\'ll no doubt be shocked when I tell you that this hasn\'t proven ideal.
Trove collections should have a particular care about space occupied (I think they also have tailored data structures if you stick to primitive types).. take a look here.
Otherwise you can try with Apache collections.. just do your benchmarks!
In anycase, if you've got many references around to same elements try to design some suited pattern (like flyweight)