Send Custom Java Objects to Kafka Topic

Question

I have my custom Java Object and wish to leverage JVM\'s in built serialization to send it to a Kafka topic, but serialization fails with below error

Answer 1

Since you are using ByteArraySerializer,you need to instantiate a byte[] producer.

Producer<byte[],byte[]> producer = new KafkaProducer<>(props);

and then while producing pass the byte[] after serializing or some other method,for instance,

producer.send(new ProducerRecord<byte[],byte[]>("test", new Payload().toString().getBytes()));

If you are passing just a Payload Object to the producer then it will be better to have key serializer and value serializer as whatever you intend to pass and while reading you need to read from that data.

It is good practice to use Serializable and ByteArraySerializer/ByteArrayDeserializer.

Answer 2

We have 2 Options as listed below

1) If we intend to send custom java objects to producer, We need to create a serializer which implements org.apache.kafka.common.serialization.Serializer and pass that Serializer class during creation of your producer

Code Reference below

public class PayloadSerializer implements org.apache.kafka.common.serialization.Serializer {

    public void configure(Map map, boolean b) {

    }

    public byte[] serialize(String s, Object o) {

       try {
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            ObjectOutputStream oos = new ObjectOutputStream(baos);
            oos.writeObject(o);
            oos.close();
            byte[] b = baos.toByteArray();
            return b;
        } catch (IOException e) {
            return new byte[0];
        }
    }

    public void close() {

    }
}

And set the value serializer accordingly

<entry key="value.serializer"
                       value="com.spring.kafka.PayloadSerializer" />

2) No need to create custom serializer class. Use the existing ByteArraySerializer, but during send follow the process

Java Object -> String (Preferrably JSON represenation instead of toString)->byteArray