Sometimes I start a MATLAB script and realize too late that it is going to output way too many figures. Eventually I get an
Exception in thread \"AWT
You can check the free memory, if there is no enough trigger a GC and check again. If there is still not enough, fail. You might want to allow 1-10 MB head room.
You can use Runtime.gc() and Runtime.freeMemory();
If you don't set the maximum memory it will make it a percentage of the available memory.