What happens to malloc'ed memory after exec() changes the program image?

Question

I know that when I call one of the exec() system calls in Linux that it will replace the currently running process with a new image. So when I fork a new proces

Answer 1

When you call fork(), a copy of the calling process is created. This child process is (almost) exactly the same as the parent, i.e. memory allocated by malloc() is preserved and you're free to read or modify it. The modifications will not be visible to the parent process, though, as the parent and child processes are completely separate.

When you call exec() in the child, the child process is replaced by a new process. From execve(2):

execve() does not return on success, and the text, data, bss, and stack
of the calling process are overwritten by that of the program loaded.

By overwriting the data segment, the exec() call effectively reclaims the memory that was allocated before by malloc().

The parent process is unaffected by all this. Assuming that you allocated the memory in the parent process before calling fork(), the memory is still available in the parent process.

EDIT: Modern implementations of malloc() use anonymous memory mappings, see mmap(2). According to execve(2), memory mappings are not preserved over an exec() call, so this memory is also reclaimed.

Answer 2

The entire heap -- allocated memory, and all of the logic malloc uses to manage it -- is part of the process image which gets replaced. It simply disappears, as far as your process is concerned. The system, of course, recovers it and recycles it.