Make calloc opportunistic

问题

On linux malloc behaves opportunistically, only backing virtual memory by real memory when it is first accessed. Would it be possible to modify calloc so that it also behaves this way (allocating and zeroing pages when they are first accessed)?

回答1:

It is not a feature of malloc() that makes it "opportunistic". It's a feature of the kernel with which malloc() has nothing to do whatsoever.

malloc() asks the kernel for a slap of memory everytime it needs more memory to fulfill a request, and it's the kernel that says "Yeah, sure, you have it" everytime without actually supplying memory. It is also the kernel that handles the subsequent page faults by supplying zero'ed memory pages. Note that any memory that the kernel supplies will already be zero'ed out due to safety considerations, so it is equally well suited for malloc() and for calloc().

That is, unless the calloc() implementation spoils this by unconditionally zeroing out the pages itself (generating the page faults that prompt the kernel to actually supply memory), it will have the same "opportunistic" behavior as malloc().

Update:

On my system, the following program successfully allocates 1 TiB (!) on a system with only 2 GiB of memory:

#include <stdlib.h>
#include <stdio.h>

int main() {
    size_t allocationCount = 1024, successfullAllocations = 0;
    char* allocations[allocationCount];
    for(int i = allocationCount; i--; ) {
        if((allocations[i] = calloc(1, 1024*1024*1024))) successfullAllocations++;
    }
    if(successfullAllocations == allocationCount) {
        printf("all %zd allocations were successfull\n", successfullAllocations);
    } else {
        printf("there were %zd failed allocations\n", allocationCount - successfullAllocations);
    }
}

I think, its safe to say that at least the calloc() implementation on my box behaves "opportunistically".

回答2:

From the related /proc/sys/vm/overcommit_memory section in proc:

The amount of memory presently allocated on the system. The committed memory is a sum of all of the memory which has been allocated by processes, even if it has not been "used" by them as of yet. A process which allocates 1GB of memory (using malloc(3) or similar), but only touches 300MB of that memory will only show up as using 300MB of memory even if it has the address space allocated for the entire 1GB. This 1GB is memory which has been "committed" to by the VM and can be used at any time by the allocating application. With strict overcommit enabled on the system (mode 2 /proc/sys/vm/overcommit_memory), allocations which would exceed the CommitLimit (detailed above) will not be permitted. This is useful if one needs to guarantee that processes will not fail due to lack of memory once that memory has been successfully allocated.

Though not explicitly said, I think similar here means calloc and realloc. So calloc already behaves opportunistically as malloc.

来源：https://stackoverflow.com/questions/29451172/make-calloc-opportunistic

标签

Linux

malloc

glibc

calloc