Can sparklyr be used with spark deployed on yarn-managed hadoop cluster?

Question

Is the sparklyr R package able to connect to YARN-managed hadoop clusters? This doesn\'t seem to be documented in the cluster deployment documentation. Using the Spark

Answer 1

Yes, sparklyr can be used against a yarn-managed cluster. In order to connect to yarn-managed clusters one needs to:

1. Set SPARK_HOME environment variable to point to the right spark home directory.
2. Connect to the spark cluster using the appropriate master location, for instance: sc <- spark_connect(master = "yarn-client")

See also: http://spark.rstudio.com/deployment.html