Why do Scala 2.11 and Spark with scallop lead to “java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror”?

Question

I am using Scala 2.11, Spark, and Scallop (https://github.com/scallop/scallop). I used sbt to build an application fat jar without Spark provided dependencies (this is at: <

Answer 1

The issue is that you've used incompatible Scala versions, i.e. Spark was compiled with Scala 2.10 and you were trying to use Scala 2.11.

Move everything to Scala 2.10 version and make sure you update your SBT as well.

You may also try to compile Spark sources for Scala 2.11.7 and use it instead.