This question is a M(not)WE of this question. I wrote a code that reproduces the error:
#include
#include
#include
Your line
std::vector a(7e8,1);
is probably wrong. You are calling a constructor for std::vector which takes a vector size and an initializing element. The 7e8 is converted to a huge size (i.e. to 700000000 elements).
You might want to construct a two-element vector, so use
std::vector a{7e8,1};
And with your huge vector, system(3) library function will call fork(2) system call which is failing with:
ENOMEM
fork()failed to allocate the necessary kernel structures because memory is tight.
Maybe you reached some limit, e.g. set by setrlimit(2) somewhere else.
Try cat /proc/self/limits to find them (on Linux).
Use strace(1) (e.g. as strace -f yourprogram) to find out what is happening; look around the fork or clone line...
BTW, system(3) should return an error code on failure. You should test it. And you might want to call system("echo here pid $$"); instead of system(NULL);