When I first load my object I calculate the initial AABB with the max and min (x,y,z) points. But this is in object space and the object moves around the world and more impo
To do that you have to loop over every vertex, calculate it's position in the world (multiply by modelview) and find minimum / maximum vertex coordinates within every object (just like when you compute it for the first time).
You can scale a bit your AABB so that you don't have to recalculate it - it is enough to enlarge it by factor sqrt(2) - your rotated object then always fits in AABB.
There is also a quesion in which direction you rotate? If always in one then you can enlarge AABB only in that direction.
Optionally you can use bounding spheres instead of AABBs.Then you don't care about rotation and scaling is not a problem.
At the end I must ask if you are sure that this is a bottleneck in your application. I believe it's not and in that case I would use first option I mentioned (iterate over all vertices).