What happens when Spark master fails?

Question

Does the driver need constant access to the master node? Or is it only required to get initial resource allocation? What happens if master is not available after Spark cont

Answer 1

Yes, the driver and master communicate constantly throughout the SparkContext's lifetime. That allows driver to:

• Display detailed status of jobs / stages / tasks on its Web Interface and REST API
• Listen on job start and end events (you can add your own listeners)
• Wait for jobs to end (via synchronous API - e.g. rdd.count() won't terminate until job is completed) and get their result

A disconnect between driver and master will fail the job.