I\'ve currently got a spreadsheet type program that keeps its data in an ArrayList of HashMaps. You\'ll no doubt be shocked when I tell you that this hasn\'t proven ideal.
So I'm assuming that you have a map of Map, where the column is actually something like ArrayList.
A few possibilities -
Are you completely sure that memory is an issue? If you're just generally worried about size it'd be worth confirming that this will really be an issue in a running program. It takes an awful lot of rows and maps to fill up a JVM.
You could test your data set with different types of maps in the collections. Depending on your data, you can also initialize maps with preset size/load factor combinations that may help. I've messed around with this in the past, you might get a 30% reduction in memory if you're lucky.
What about storing your data in a single matrix-like data structure (an existing library implementation or something like a wrapper around a List of Lists), with a single map that maps column keys to matrix columns?