Does malloc lazily create the backing pages for an allocation on Linux (and other platforms)?

Question

On Linux if I were to malloc(1024 * 1024 * 1024), what does malloc actually do?

I\'m sure it assigns a virtual address to the allocation (by walking the

Answer 1

I gave this answer to a similar post on the same subject:

Are some allocators lazy?

This starts a little off subject (and then I'll tie it in to your question), but what's happening is similar to what happens when you fork a process in Linux. When forking there is a mechanism called copy on write which only copies the memory space for the new process when the memory is written too. This way if the forked process exec's a new program right away then you've saved the overhead of copying the original programs memory.

Getting back to your question, the idea is similar. As others have pointed out, requesting the memory gets you the virtual memory space immediately, but the actual pages are only allocated when write to them.

What's the purpose of this? It basically makes mallocing memory a more or less constant time operation Big O(1) instead of a Big O(n) operation (similar to the way the Linux scheduler spreads it's work out instead of doing it in one big chunk).

To demonstrate what I mean I did the following experiment:

rbarnes@rbarnes-desktop:~/test_code$ time ./bigmalloc

real    0m0.005s
user    0m0.000s
sys 0m0.004s
rbarnes@rbarnes-desktop:~/test_code$ time ./deadbeef

real    0m0.558s
user    0m0.000s
sys 0m0.492s
rbarnes@rbarnes-desktop:~/test_code$ time ./justwrites

real    0m0.006s
user    0m0.000s
sys 0m0.008s

The bigmalloc program allocates 20 million ints, but doesn't do anything with them. deadbeef writes one int to each page resulting in 19531 writes and justwrites allocates 19531 ints and zeros them out. As you can see deadbeef takes about 100 times longer to execute than bigmalloc and about 50 times longer than justwrites.

#include 

int main(int argc, char **argv) {

    int *big = malloc(sizeof(int)*20000000); // Allocate 80 million bytes

    return 0;
}

.

#include 

int main(int argc, char **argv) {

    int *big = malloc(sizeof(int)*20000000); // Allocate 80 million bytes

    // Immediately write to each page to simulate an all-at-once allocation
    // assuming 4k page size on a 32-bit machine.

    for (int* end = big + 20000000; big < end; big += 1024)
        *big = 0xDEADBEEF;

    return 0;
}

.

#include 

int main(int argc, char **argv) {

    int *big = calloc(sizeof(int), 19531); // Number of writes

    return 0;
}