Spark 2.1 - Error While instantiating HiveSessionState

Question

With a fresh install of Spark 2.1, I am getting an error when executing the pyspark command.

Traceback (most recent call last):
File \"/usr/local/spark/pytho         


        

Answer 1

I was getting same error in windows environment and Below trick worked for me.

in shell.py the spark session is defined with .enableHiveSupport()

 spark = SparkSession.builder\
            .enableHiveSupport()\
            .getOrCreate()

Remove hive support and redefine spark session as below:

spark = SparkSession.builder\
        .getOrCreate()

you can find shell.py in your spark installation folder. for me it's in "C:\spark-2.1.1-bin-hadoop2.7\python\pyspark"

Hope this helps