What happens when Spark master fails?

Question

Does the driver need constant access to the master node? Or is it only required to get initial resource allocation? What happens if master is not available after Spark cont

Answer 1

Below are the steps spark application does, when it starts,

1. Starts the Spark Driver
2. Spark Driver, connects to spark master for resource allocation.
3. Spark Driver, sends the jar attached in spark context to master server.
4. Spark Driver, keeps polling master server to get the job status.
5. If there is a shuffling or broadcast in code, data is routed via spark driver. That is why, it is required for spark driver to have sufficient memory.
6. If there is any operation like take, takeOrdered, or collect, data is accumulater on driver.

So, yes, failing on master will result in executors not able to communicate with it. So, they will stop working. Failing of master will make driver unable to communicate with it for job status. So, your application will fail.