How to fix “java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord” in Spark Streaming Kafka Consumer?

Question

• Spark 2.0.0
• Apache Kafka 0.10.1.0
• scala 2.11.8

---

When I use spark streaming and kafka integration with kafka broker version 0.10.1

Answer 1

ConsumerRecord does not implement serialization, when performing operations that require serialization, ie persist or window, print. You need to add the below config to avoid the error.

    sparkConf.set("spark.serializer","org.apache.spark.serializer.KryoSerialize");
    sparkConf.registerKryoClasses((Class[] )Arrays.asList(ConsumerRecord.class).toArray());