Can rbind be parallelized in R?

Question

As I am sitting here waiting for some R scripts to run...I was wondering... is there any way to parallelize rbind in R?

I sitting waiting for this call to complete

Answer 1

Looks like this has already been well answered by a number of people, but if it comes up for some one, here is a version of a parallel rbind for non-data.table/data.frame-esque objects:

rbind.parallel <- function(list,ncore)
  {
  library(parallel)
  do.call.rbind<-function(x){do.call(rbind,x)}
  cl<-makeCluster(ncore)
  list.split<-split(list,rep(1:ncore,length(list)+1)[1:length(list)])
  list.join<-parLapply(cl,list.split,do.call.rbind)
  stopCluster(cl)
  list.out<-do.call(rbind,list.join)
  return(list.out)
  }

This works effectively on sf type objects. For example, if you read in a list of shapefiles from a directory using lapply(.,st_read), obviously rbind.fill and its variants not going to work to join all the features.