Spark 2.1 - Error While instantiating HiveSessionState

Question

With a fresh install of Spark 2.1, I am getting an error when executing the pyspark command.

Traceback (most recent call last):
File \"/usr/local/spark/pytho         


        

Answer 1

The issue for me was solved by disabling HADOOP_CONF_DIR environment variable. It was pointing to hadoop configuration directory and while starting pyspark shell, the variable caused spark to initiate hadoop cluster which wasn't initiated.

So if you have HADOOP_CONF_DIR variable enabled, then you have to start hadoop cluster started before using spark shells

Or you need to disable the variable.