How to create Custom serializer in kafka?

Question

There is only few serializer available like,

org.apache.kafka.common.serialization.StringSerializer
org.apache.kafka.common.serialization.StringSerializer 
<         


        

Answer 1

No words, only code

1. Some object, which you sent to Kafka

   import lombok.AllArgsConstructor;
   import lombok.Data;
   import lombok.NoArgsConstructor;
   import lombok.ToString;
   
   @Data
   @AllArgsConstructor
   @NoArgsConstructor
   @ToString
   public class TestDto {
   
       private String name;
       private String version;
   
   }
   

2. Create Serializer, which will be used by Producer

   @Slf4j
   public class KafkaValueSerializer implements Serializer {
   
       private ObjectMapper objectMapper = new ObjectMapper();
   
       @Override
       public void configure(Map configs, boolean isKey) {
       }
   
       @Override
       public byte[] serialize(String topic, TestDto data) {
           try {
               return objectMapper.writeValueAsBytes(data);
           } catch (JsonProcessingException e) {
               log.error("Unable to serialize object {}", data, e);
               return null;
           }
       }
   
       @Override
       public void close() {
       }
   }
   

3. Of couser, don't foget about Deserialiser for Consumer

   @Slf4j
   public class KafkaValueDeserializer implements Deserializer {
   
       private ObjectMapper objectMapper = new ObjectMapper();
   
       @Override
       public void configure(Map configs, boolean isKey) {
       }
   
       @Override
       public TestDto deserialize(String topic, byte[] data) {
           try {
               return objectMapper.readValue(new String(data, "UTF-8"), TestDto.class);
           } catch (Exception e) {
               log.error("Unable to deserialize message {}", data, e);
               return null;
           }
       }
   
       @Override
       public void close() {
       }
   }
   

4. Last moment, add serializer/deserializer to application.yml

   spring:
       kafka:
         bootstrap-servers:  192.168.192.168:9092
         producer:
             value-serializer: com.package.service.kafka.KafkaValueSerializer
         consumer:
             group-id: groupId
             value-deserializer: com.package.service.kafka.KafkaValueDeserializer
   

That's all. It's not necessary any configuration file or dancing with a tamboirine :)

5. Send

   KafkaTemplate kafkaTemplate;
   
   TestDto test = new TestDto("test name", "test-version");
   kafkaTemplate.send(topic, testDto);
   

6. Listen

   @KafkaListener(topics = "${ktp-agent.kafka.request-topic}", groupId = "${spring.kafka.consumer.group-id}")
   public void listen(TestDto message) {
   
       log.info("Received message '{}' from Kafka.", message.toString());
   }