java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Question

I have Hadoop 2.7.1 and apache-hive-1.2.1 versions installed on ubuntu 14.0.

1. Why this error is occurring ?
2. Is any metastore installation required?

Answer 1

I solved this problem by removing --deploy-mode cluster from spark-submit code. By default , spark submit takes client mode which has following advantage :

1. It opens up Netty HTTP server and distributes all jars to the worker nodes.
2. Driver program runs on master node , which means dedicated resources to driver process.

While in cluster mode :

 1.  It runs on worker node.
 2. All the jars need to be placed in a common folder of the cluster so that it is accessible to all the worker nodes or in folder of each worker node.

Here it's not able to access hive metastore due to unavailability of hive jar to any of the nodes in cluster.