c++ system() raises ENOMEM

Question

This question is a M(not)WE of this question. I wrote a code that reproduces the error:

#include 
#include 
#include 

        

Answer 1

The system() function works by first creating a new copy of the process with fork() or similar (in Linux, this ends up in the clone() system call, as you show) and then, in the child process, calling exec to create a shell running the desired command.

The fork() call can fail if there is insufficient virtual memory for the new process (even though you intend to immediately replace it with a much smaller footprint, the kernel can't know that). Some systems allow you to trade the ability to fork large processes for reduced guarantees that page faults may fail, with copy-on-write (vfork()) or memory overcommit (/proc/sys/vm/overcommit_memory and /proc/sys/vm/overcommit_ratio).

Note that the above applies equally to any library function that may create new processes - e.g. popen(). Though not exec(), as that replaces the process and doesn't clone it.

If the provided mechanisms are inadequate for your use case, then you may need to implement your own system() replacement. I recommend starting a child process early on (before you allocate lots of memory) whose sole job is to accept NUL-separated command lines on stdin and report exit status on stdout.

An outline of the latter solution in pseudo-code looks something like:

int request_fd[2];
int reply_fd[2];

pipe(request_fd);
pipe(reply_fd);

if (fork()) {
    /* in parent */
    close(request_fd[0]);
    close(reply_fd[1]);
} else {
    /* in child */
    close(request_fd[1]);
    close(reply_fd[0]);
    while (read(request_fd[0], command)) {
        int result = system(command);
        write(reply_fd[1], result);
    }
    exit();
}

// Important: don't allocate until after the fork()
std::vector a(7e8,1);  // allocate a big chunk of memory

int my_system_replacement(const char* command) {
    write(request_fd[1], command);
    read(reply_fd[0], result);
    return result;
}

You'll want to add appropriate error checks throughout, by reference to the man pages. And you might want to make it more object-oriented, and perhaps use iostreams for your read and write operations, etc.