Why does ./bin/spark-shell give WARN NativeCodeLoader: Unable to load native-hadoop library for your platform?

Question

On Mac OS X, I compiled Spark from the sources using the following command:

jacek:~/oss/spark
$ SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true SP         


        

Answer 1

Supported Platforms of the Native Libraries Guide documentation in Apache Hadoop reads:

The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.

The native hadoop library is mainly used on the GNU/Linus platform and has been tested on these distributions:

• RHEL4/Fedora
• Ubuntu
• Gentoo

On all the above distributions a 32/64 bit native hadoop library will work with a respective 32/64 bit jvm.

It appears that the WARN message should be disregarded on Mac OS X as the native library doesn't simply exist for the platform.

Answer 2

In my experience, if you cd into the /sparkDir/conf and rename the spark-env.sh.template to spark-env.sh, and then set the JAVA_OPTSand hadoop_DIR, it works.

You will also have to edit this /etc/profile line:

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/:$LD_LIBRARY_PATH