Spark2.1.0 incompatible Jackson versions 2.7.6

Question

I am trying to run a simple spark example in intellij, but I get the error like that:

Exception in thread \"main\" java.lang.ExceptionInInitializerError
at o         


        

Answer 1

Spark 2.1.0 contains com.fasterxml.jackson.core as transitive dependency. So, we do not need to include then in libraryDependencies.

But if you want to add a different com.fasterxml.jackson.core dependencies' version then you have to override them. Like this:

name := "testSpark2"

version := "1.0"

scalaVersion := "2.11.8"


dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"

So, change your build.sbt like the one above and it will work as expected.

I hope it helps!