Spark fails with NoClassDefFoundError for org.apache.kafka.common.serialization.StringDeserializer

Question

I am developing a generic Spark application that listens to a Kafka stream using Spark and Java.

I am using kafka_2.11-0.10.2.2, spark-2.3.2-bin-hadoop2.7 - I also t

Answer 1

Spark runs the program as by running an instance of a JVM. So if the libraries (JARs) are not in the classpath of that JVM we run into this runtime exception. The solution is to package all the dependent JARs along with main JAR. The following build script will work for that.

Also, as mentioned in https://stackoverflow.com/a/54583941/1224075 the scope of the spark-core and spark-streaming libraries need to be declared as provided. This is because some of the libraries are implicitly provided by the Spark JVM.

The build section of the POM which worked for me -

    
        
            org.apache.maven.plugins
            maven-assembly-plugin
            2.2.1
            
                
                    jar-with-dependencies
                
            
            
                
                    make-assembly
                    package
                    
                        single