Stop KafkaListener ( Spring Kafka Consumer) after it has read all messages till some specific time

问题

I am trying to schedule my consumption process from a single partition topic. I can start it using endpointlistenerregistry.start() but I want to stop it after I have consumed all the messages in current partition i.e. when I reach to last offset in current partition. Production into the topic is done after I have finished the consumption and close it. How should I achieve the assurance that I have read all the messages till the time I started scheduler and stop my consumer ? I am using @Kafkalistener for consumer.

回答1:

Set the idleEventInterval container property and add an @EventListener method to listen for ListenerContainerIdleEvents.

Then stop the container.

回答2:

1. To read till the last offset, you simply poll till you are getting empty records.

2. You can invoke kafkaConsumer.pause() at the end of consumption. During next schedule it is required to invoke kafkaConsumer.resume().

Suspend fetching from the requested partitions. Future calls to poll(Duration) will not return any records from these partitions until they have been resumed using resume(Collection). Note that this method does not affect partition subscription. In particular, it does not cause a group rebalance when automatic assignment is used.

Something like this,

List<TopicPartition> topicPartitions = new ArrayList<>();
void scheduleProcess() {
    topicPartitions = ... // assign partition info for this
    kafkaConsumer.resume(topicPartitions)
    while(true) {
       ConsumerRecords<String, Object> events = kafkaConsumer.poll(Duration.ofMillis(1000));
       if(!events.isEmpty()) {
          // processing logic
       } else {
          kafkaConsumer.pause(List.of(topicPartition));
          break;
       }
    }
}

来源：https://stackoverflow.com/questions/62339968/stop-kafkalistener-spring-kafka-consumer-after-it-has-read-all-messages-till