should I pre-install cran r packages on worker nodes when using sparkr

Question

I want to use r packages on cran such as forecast etc with sparkr and meet following two problems.

1. Should I pre-install all those packages on w

Answer 1

Add libraries works with spark 2.0+. For example, I am adding the package forecast in all node of cluster. The code works with Spark 2.0+ and databricks environment.

schema <- structType(structField("out", "string"))
out <- gapply(
  df,
  c("p", "q"),
  function(key, x) 
  if (!all(c("forecast") %in% (.packages()))){
     if (!require("forecast")) {
        install.packages("forecast", repos ="http://cran.us.r-project.org", INSTALL_opts = c('--no-lock'))
     }
  }  
  #use forecast
  #dataframe out
  data.frame(out = x$column, stringAsFactor = FALSE)
}, 
schema)

Answer 2

It is boring to repeat this but you shouldn't use internal RDD API in the first place. It's been removed in the first official SparkR release and it is simply not suitable for general usage.

Until new low level API* is ready (see for example SPARK-12922 SPARK-12919, SPARK-12792) I wouldn't consider Spark as a platform for running plain R code. Even when it changes adding native (Java / Scala) code with R wrappers can be a better choice.

That being said lets start with your question:

1. RPackageUtils are designed to handle packages create with Spark Packages in mind. It doesn't handle standard R libraries.

2. Yes, you need packages to be installed on every node. From includePackage docstring:

   The package is assumed to be installed on every node in the Spark cluster.

---

* If you use Spark 2.0+ you can use dapply, gapply and lapply functions.

Answer 3

a better choice is to pass your local R package by spark-submit archive option, which means you do not need install R package in each worker and do not install and compile R package while running SparkR::dapply for time consuming waiting. for example:

Sys.setenv("SPARKR_SUBMIT_ARGS"="--master yarn-client --num-executors 40 --executor-cores 10 --executor-memory 8G --driver-memory 512M --jars /usr/lib/hadoop/lib/hadoop-lzo-0.4.15-cdh5.11.1.jar --files /etc/hive/conf/hive-site.xml --archives /your_R_packages/3.5.zip --files xgboost.model sparkr-shell")

when call SparkR::dapply function, let it call .libPaths("./3.5.zip/3.5") first. And you need notice that the server version R version must be equal your zip file R version.