-Xmx just defines the maximum size of the heap. It makes no guarantee on wether there is so much memory or not. It only ensures that the heap will never be bigger then the given value. That said, Option B.) will happen, an outOfMemoryError will be thrown.