R and shared memory for parallel::mclapply

Question

I am trying to take advantage of a quad-core machine by parallelizing a costly operation that is performed on a list of about 1000 items.

I am using R\'s parallel::m

Answer 1

Linux and macosx have a copy-on-write mechanism when forking, it means that the pages of memory are not actually copied, but shared until the first write. mclapply is based on fork(), so probably (unless you write to your big shared data), the memory that you see reported by your process lister is not actual memory.

But, when collecting the results, the master process will have to allocate memory for each returned result of the mclapply.

To help you further, we would need to know more about your fun function.