Malloc on linux without overcommitting

Question

How can I allocate memory on Linux without overcommitting, so that malloc actually returns NULL if no memory is available and the process doesn\'t randomly cras

Answer 1

From the discussion in the comments, it appears calling

mlockall( MCL_CURRENT | MCL_FUTURE );

upon process start would satisfy the requirement for malloc() to return NULL when the system can not actually provide memory.

Per the Linux mlockall() man page:

mlockall() and munlockall()

mlockall() locks all pages mapped into the address space of the calling process. This includes the pages of the code, data and stack segment, as well as shared libraries, user space kernel data, shared memory, and memory-mapped files. All mapped pages are guaranteed to be resident in RAM when the call returns successfully; the pages are guaranteed to stay in RAM until later unlocked.

The flags argument is constructed as the bitwise OR of one or more of the following constants:

   MCL_CURRENT Lock all pages which are currently mapped into the
               address space of the process.

   MCL_FUTURE  Lock all pages which will become mapped into the address
               space of the process in the future.  These could be, for
               instance, new pages required by a growing heap and stack
               as well as new memory-mapped files or shared memory
               regions.

   MCL_ONFAULT (since Linux 4.4)
               Used together with MCL_CURRENT, MCL_FUTURE, or both.
               Mark all current (with MCL_CURRENT) or future (with
               MCL_FUTURE) mappings to lock pages when they are faulted
               in.  When used with MCL_CURRENT, all present pages are
               locked, but mlockall() will not fault in non-present
               pages.  When used with MCL_FUTURE, all future mappings
               will be marked to lock pages when they are faulted in,
               but they will not be populated by the lock when the
               mapping is created.  MCL_ONFAULT must be used with either
               MCL_CURRENT or MCL_FUTURE or both.

If MCL_FUTURE has been specified, then a later system call (e.g., mmap(2), sbrk(2), malloc(3)), may fail if it would cause the number of locked bytes to exceed the permitted maximum (see below). In the same circumstances, stack growth may likewise fail: the kernel will deny stack expansion and deliver a SIGSEGV signal to the process.

Note that using mlockall() in this manner might have other, unexpected consequences. Linux has been developed assuming memory overcommit is available, so something as simple as calling fork() after mlockall() might run into issues.