How to Deserialising Kafka AVRO messages using Apache Beam

Question

The main goal is the aggregate two Kafka topics, one compacted slow moving data and the other fast moving data which is received every second.

I have been able to c

Answer 1

You can use KafkaAvroDeserializer as following:

PCollection> input = p.apply(KafkaIO.read()
.withKeyDeserializer(LongDeserializer.class)
  .withValueDeserializerAndCoder(KafkaAvroDeserializer.class, AvroCoder.of(MyClass.class))

Where MyClass is the POJO class generated Avro Schema.

Make sure your POJO class has annotation AvroCoder as in below example :

@DefaultCoder(AvroCoder.class)
   public class MyClass{
      String name;
      String age;

      MyClass(){}
      MyClass(String n, String a) {
         this.name= n;
         this.age= a;
      }
  }