The serialized size is definitely not the way to go, for two reasons:
- In the standard java serialization there can be quite a lot of overhead which would add to the size.
- It would not be any quicker than using the getObjectSize() method which we can presume will iterate over all the references, and use some kind of lookup to determine the size of the primitive values/references of an object.
If you need better performance then that really will depend on the distribution of your objects. One possiblility would be to do some random sampling of the values in your map, determine an average and calculate an estimate from this value.
For advice on how to look up a random value in a hashmap, see this question.