Can rbind be parallelized in R?

Question

As I am sitting here waiting for some R scripts to run...I was wondering... is there any way to parallelize rbind in R?

I sitting waiting for this call to complete

Answer 1

This is expanding on @Dominik 's answer.

We can use the mclapply from parallel package to increase the speed further. Also rbind.fill does a better job than rbind so here's the improved code. NOTE: this will only work on mac/linux. mclapply is not supported on Windows. EDIT: if you want to see progress, uncomment the print(i) line, and make sure you run from a terminal, not from RStudio. Printing to RStudio from a parallel process, kind of messes RStudio up.

library(parallel)
rbind.fill.parallel <- function(list){
  while(length(list) > 1) {
    idxlst <- seq(from=1, to=length(list), by=2)

    list <- mclapply(idxlst, function(i) {
      #print(i) #uncomment this if you want to see progress
      if(i==length(list)) { return(list[[i]]) }
      return(rbind.fill(list[[i]], list[[i+1]]))
    })
  }
}