Why spark-shell fails with NullPointerException?

Question

I try to execute spark-shell on Windows 10, but I keep getting this error every time I run it.

I used both latest and spark-1.5.0-bin-hadoop2.4 versions

Answer 1

My issue was having other .exe's/Jars inside the winutils/bin folder. So I cleared all the others and was left with winutils.exe alone. Was using spark 2.1.1

Answer 2

Or perhaps this link here below be easier to follow,

https://wiki.apache.org/hadoop/WindowsProblems

Basically download and copy winutils.exe to your spark\bin folder. Re-run spark-shell

If you have not set your /tmp/hive to a writable state, please do so.

Answer 3

Issue was resolved after installing correct Java version in my case its java 8 and setting the environmental variables. Make sure you run the winutils.exe to create a temporary directory as below.

c:\winutils\bin\winutils.exe chmod 777 \tmp\hive

Above should not return any error. Use java -version to verify the version of java you are using before invoking spark-shell.

Answer 4

In Windows, you need to clone "winutils"

git clone https://github.com/steveloughran/winutils.git

And

set var HADOOP_HOME to DIR_CLONED\hadoop-{version}

Remember to choose the version of your hadoop.

Answer 5

You need to give permission to /tmp/hive directory to resolve this exception.

Hope you already have winutils.exe and set HADOOP_HOME environment variable. Then open the command prompt and run following command as administrator:

If winutils.exe is present in D:\winutils\bin location and \tmp\hive is also in D drive:

D:\winutils\bin\winutils.exe chmod 777 D:\tmp\hive

For more details,you can refer the following links :

Frequent Issues occurred during Spark Development
How to run Apache Spark on Windows7 in standalone mode

Answer 6

For Python - Create a SparkSession in your python (This config section is only for Windows)

spark = SparkSession.builder.config("spark.sql.warehouse.dir", "C:/temp").appName("SparkSQL").getOrCreate()

Copy winutils.exe and keep in C:\winutils\bin and execute the bellow commands

C:\Windows\system32>C:\winutils\bin\winutils.exe chmod 777 C:/temp

Run command prompt in ADMIN mode ( Run as Administrator)