Exception in thread “main” java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)

Question

Any reason why I get this error ? Initially the IDE plugin for Scala was 2.12.3. But since I\'m working with Spark 2.2.0, I manually changed it to Scala 2.11.11.

<         


        

Answer 1

Make sure Spark is compatible with corresponding Scala version

The error is common when using Scala version 2.12 series with any version of Spark offering Scala 2.11.

You can try using the 2.11 series of Scala with Spark . i.e.

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0"

As you can see in this dependency spark-core_2.11 is associated with scala version 2.11.

That's why it's safer (more compatible) to use %% and avoid hardcoding the version of Scala in Spark dependencies. Let the tool resolve the required Scala version for you automatically as follows:

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0"

The above declaration will automatically infer the scala version.