Why does Spark application fail with “Exception in thread ”main“ java.lang.NoClassDefFoundError: …StringDeserializer”?

Question

I am developing a Spark application that listens to a Kafka stream using Spark and Java.

I use kafka_2.10-0.10.2.1.

I have set various parameters for Kafka prope

Answer 1

Turned out issue was with uber jar not building correctly. In case you would like to assemble the application and package an uber jar.

Create an assembly file in src/assembly/assembly.xml

    bin
    
        jar
    
    false
    
        
            true
            provided
        
    

And add the maven-assembly-plugin to the pom.xml

        
            
                maven-assembly-plugin
                3.3.0
                
                    
                        src/assembly/assembly.xml
                    
                
                
                    
                        make-assembly
                        package
                        
                            single
                        
                    
                
            
        

If you would like to add a dependency to the uber jar, just add provided scope to it.
In your case, it will be like this:

    
        org.apache.spark
        spark-core_2.11
        2.1.1
    
    
        org.apache.spark
        spark-streaming_2.10
        2.1.1
    

    
        org.apache.spark
        spark-streaming-kafka-0-10_2.11
        2.1.1
        provided
    

    
        org.apache.kafka
        kafka-clients
        0.10.2.1
        provided
    

$spark-submit --class Main application-bin.jar