R and shared memory for parallel::mclapply

Question

I am trying to take advantage of a quad-core machine by parallelizing a costly operation that is performed on a list of about 1000 items.

I am using R\'s parallel::m

Answer 1

I guess I would have thought this would not have used extra memory because of the copy-on-write functionality. I take it the elements of the list are large? Perhaps when R passes the elements to fun() it is actually making a copy of the list item instead of using copy on write. If so, the following might work better:

fun <- function(itemNumber){
  myitem <- lst[[itemNumber]]
  # now do your computations
}
res = rbind.fill(parallel::mclapply(1:length(lst), fun, mc.cores=3, mc.preschedule=T))

Or use lst[[itemNumber]] directly in your function. If R/Linux/macos isn't smart enough to use copy-on-write as you wrote the function, it may with this modified approach.

Edit: I assume you are not modifying the items in the list. If you do, R is going to make copies of the data.