What is yarn-client mode in Spark?

Question

Apache Spark has recently updated the version to 0.8.1, in which yarn-client mode is available. My question is, what does yarn-client mode really mean? In the d

Answer 1

A Spark application running in

yarn-client mode:

1. driver program runs in client machine or local machine where the application has been launched.

2. Resource allocation is done by YARN resource manager based on data locality on data nodes and driver program from local machine will control the executors on spark cluster (Node managers).

Please refer this cloudera article for more info.

The difference between standalone mode and yarn deployment mode,

1. Resource optimization won't be efficient in standalone mode.
2. In standalone mode, driver program launch an executor in every node of a cluster irrespective of data locality.
3. standalone is good for use case, where only your spark application is being executed and the cluster do not need to allocate resources for other jobs in efficient manner.