Infinite retries with SeekToCurrentErrorHandler in kafka consumer

问题

I've configured a kafka consumer with SeekToCurrentErrorHandler in Spring boot application using spring-kafka. My consumer configuration is :

     @Bean
  public ConsumerFactory<String, String> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafkaserver");
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "group-id");

    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer2.class);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer2.class);
    props.put(ErrorHandlingDeserializer2.KEY_DESERIALIZER_CLASS, StringDeserializer.class);
    props.put(ErrorHandlingDeserializer2.VALUE_DESERIALIZER_CLASS, StringDeserializer.class.getName());
    props.put(JsonDeserializer.KEY_DEFAULT_TYPE, "java.lang.String");
    props.put(JsonDeserializer.VALUE_DEFAULT_TYPE, "java.lang.String");

    return new DefaultKafkaConsumerFactory<>(props);
  }


@Bean
  public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
    SeekToCurrentErrorHandler seekToCurrentErrorHandler = new SeekToCurrentErrorHandler(5);
    seekToCurrentErrorHandler.setCommitRecovered(true);

    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());
    factory.getContainerProperties().setAckOnError(false);
    factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
    factory.setErrorHandler(seekToCurrentErrorHandler);

    return factory;
  }

To test SeekToCurrentErrorHandler config, I pushed a record in kafka with incorrect format so that it fails with deserialization exception. As per my understanding the error handler should try to handle the failed record 5 times and after that it should log and move on to the next record. But it keeps on reading the failed record infinite number of times.

Please tell me where am I going wrong.

回答1:

I have exactly the same problem and the only fix I do is make sure the concurrency level is same as the number of partition for the topic. Otherwise it will keeps retrying infinitely.

Sounds like a bug of spring kafka.

来源：https://stackoverflow.com/questions/57859158/infinite-retries-with-seektocurrenterrorhandler-in-kafka-consumer