Allocation always depends on your OS. If you allocate too much memory, you could end up having loaded portions into swap, which indeed is slow.
Whether your program runs slower or faster depends on the references the VM has to handle and to clean. The GC doesn't have to sweep through the allocated memory to find abandoned objects. It knows it's objects and the amount of memory they allocate by reference mapping. So sweeping just depends on the size of your objects. If your program behaves the same in both cases, the only performance impact should be on VM startup, when the VM tries to allocate memory provided by your OS and if you use the swap (which again leads to 1.)