问题
I am trying to run a spark scala program where in I have used "import org.apache.spark.internal.Logging"
.
Program was running fine in local until I try to build a fat jar and added assembly.sbt
using build.sbt
Here's my build.sbt
lazy val root = (project in file(".")).
settings(
name := "TestingSparkApplicationProject",
version := "0.0.1-SNAPSHOT",
scalaVersion := "2.11.12",
mainClass in Compile := Some("com.test.spark.job.TestSparkJob.scala")
)
val sparkVersion = "2.4.3"
libraryDependencies ++= Seq(
"net.jcazevedo" %% "moultingyaml" % "0.4.2",
"net.liftweb" %% "lift-json" % "3.4.1",
"com.github.scopt" %% "scopt" % "4.0.0-RC2",
"com.google.cloud" % "google-cloud-storage" % "1.23.0",
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided"
)
scalaSource in Compile := baseDirectory.value / "src"
My assembly.sbt
file as:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10")
Though it has not build successfully and still showing a error message in the right corner of assembly.sbt file
"Expression type def.settings[Seq[ModuleID]] must conform to Dsl entry in sbt file
"
I am getting below error when running my main module 'TestSparkJob'
java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main"
Update- I got rid of assembly.sbt
issue when I recreated my build.sbt
with the same code. Code is running fine when deployed to dataproc cluster but I am still getting same error when trying to run my code in local environment.
来源:https://stackoverflow.com/questions/63090249/java-lang-noclassdeffounderror-org-slf4j-logger-class-not-found-with-intellij-a