UnsatisfiedLinkError: no snappyjava in java.library.path when running Spark MLLib Unit test within Intellij

Question

The following exception is occurring when running a spark unit test that requires snappy compression:

java.lang.reflect.InvocationTargetException
    at sun.         


        

Answer 1

Faced this problem with clean standalone installation of Spark 1.6.1. To solve it I had to:

1) manually add libsnappyjava.jnilib (it's in the jar) to java.library.path (which includes multiple locations, ~/Library/Java/Extensions/ is fine)

2) add snappy-java-1.1.2.4.jar to Spark's classpath (in spark-env.sh add "export SPARK_CLASSPATH=.../snappy-java-1.1.2.4.jar"