I just executed a program that mallocs 13 MB in a 12 MB machine (QEMU Emulated!) . Not just that, i even browsed through the memory and filled junk in it...
It's called virtual memory which is allocated for your program. It's not real memory which you call RAM.
There is a max limit for virtual memory as well, but it's higher than RAM. It's implemented (and defined) by your operating system.