EMR Spark - TransportClient: Failed to send RPC

Question

I\'m getting this error, I tried to increase memory on cluster instances and in the executor and driver parameters without success.

17/05/07 23:17:07 ERROR T         


        

Answer 1

When I setup hadoop and spark in my laptop and try to launch spark as "spark-shell --master yarn" I got the same error message.

Solution:

sudo vim /usr/local/hadoop/etc/hadoop/yarn-site.xml

Add the following property:

       
yarn.nodemanager.vmem-pmem-ratio      
5  

Then restart hadoop service

stop-all.sh 
start-all.sh