spark submit “Service 'Driver' could not bind on port” error

Question

I used the following command to run the spark java example of wordcount:-

time spark-submit --deploy-mode cluster --master spark://192.168.0.7:6066 --class o         


        

Answer 1

This issue is related to IP address alone. Error messages in the log file are not informative. check with following 3 steps:

1. check your IP address - can be checked with ifconfig or ip commands. If your service is not a Public service. IP addresses with 192.168 should be good enough. 127.0.0.1 cannot be used if you are planning a cluster.

2. check your environment variable SPARK_MASTER_HOST - check there are no typos in the name of the variable or actual IP address.

   env | grep SPARK_

3. check the port you are planning to use for sparkMaster is free with command netstat. Do not use a port below 1024. For example:

   netstat -a | 9123

After your sparkmaster starts running if you are not able see webui from a different machine, then open the webui port with command iptables.