Spring @KafkaListener execute and poll records after certain interval

问题

We wanted to consume the records after a certain interval (e.g. every 5 minutes). Consumer properties are standard:

@Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<Integer, String>> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<Integer, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());
    factory.setConcurrency(1);
    factory.setBatchListener(true);
    factory.getContainerProperties().setPollTimeout(300000);
    factory.getContainerProperties().setAckMode(AbstractMessageListenerContainer.AckMode.BATCH);
    return factory;
}

Even though when i change the property setPollTimeout it doesnot poll after defined interval (5 minutes), it continuously polls after 30 seconds, here are my logs:

2018-01-23 18:07:26.875 INFO 60905 --- [        2-0-C-1] c.t.k.s.consumer.FavoriteEventConsumer   : Consumed: san@1516710960000->1516711080000 2

2018-01-23 18:07:56.901 INFO 60905 --- [        2-0-C-1] c.t.k.s.consumer.FavoriteEventConsumer   : Consumed: san@1516710960000->1516711080000 4

We were trying to build a kafka stream application with windowed aggregations and planning to consume the window x after y interval.

I can see that in the class: KafkaMessageListenerContainer, setConsumerTaskExecutor is set:

if (containerProperties.getConsumerTaskExecutor() == null) {
        SimpleAsyncTaskExecutor consumerExecutor = new SimpleAsyncTaskExecutor(
                (getBeanName() == null ? "" : getBeanName()) + "-C-");
        containerProperties.setConsumerTaskExecutor(consumerExecutor);
    }

But how do we configure when this (frequency) thread pool polls records. Any help appreciated.

回答1:

You cannot control the rate at which the consumer polls, the pollTimeout is how long the poll() will wait for new records to arrive. If new records arrive more often, it will not wait that long.

If you wish to control the rate at which you receive records, simply use the DefatulKafkaConsumerFactory to create a consumer and poll it whenever you want.

You can't use that with a @KafkaListener though - you have to deal with the record yourself.

回答2:

If you want to control rate at which Kafka consumer using Spring @KafkaListener, please autowire KafkaListenerEndpointRegistry bean use in following way and access the required MessageListenerContainer. thereafter, you can use the pause() and resume() functionalities to control the required behaviour.

@Autowired
private KafkaListenerEndpointRegistry listener;

@Autowired
private Map<String, Set<String>> getTopicListenerMap(){
    List<String> ids = new ArrayList<>(listener.getListenerContainerIds());
    Map<String, Set<String>> topicListenerMap = new HashMap<>();
    for(String topic: topics){
        topicListenerMap.put(topic, new HashSet<>());
    }
    for(String key: ids){
        for (String topic : listener.getListenerContainer(key).getContainerProperties().getTopics()){
            topicListenerMap.get(topic).add(key);
        }
    }
    return topicListenerMap;
}

@KafkaListener(topics = "topic", containerFactory = "smsListener")
public void listenWithHeaders(@Payload List<String> messageList, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitionList,
                              @Header(KafkaHeaders.OFFSET) List<Integer> offsetList) {
    try{
        LOG.info("Received message count: "+(messageList!=null ? messageList.size(): 0)+", offset start: "+offsetList.get(0)+", end: "+offsetList.get(offsetList.size()-1));
        pauseIfRequired(topic);
        for(int i=0; i<messageList.size(); i++){
            // process the messages
        }
    }catch (Exception e){
        LOG.error("", e);
    }finally {
        resumeIfPaused(topic);
    }
}

private void pauseIfRequired(String topic){
    try{ 
        boolean flag = pausingCondition;
        if(flag){
            LOG.info("pausing topic: "+topic);
            for(String listenerKey: getTopicListenerMap().get(topic)){
                listener.getListenerContainer(listenerKey).pause();
            }
            LOG.info("topic paused: "+topic);
        }
    } catch (Exception e){
        LOG.error("", e);
    }
}

private void resumeIfPaused(String topic){
    try {
        for (String listenerKey : getTopicListenerMap().get(topic)) {
            LOG.info("topic: "+topic+", containerPauseRequested: "+listener.getListenerContainer(listenerKey).isPauseRequested());
            if (listener.getListenerContainer(listenerKey).isPauseRequested()) {
                LOG.info("waiting to resume topic: " + topic + ", listener key: " + listenerKey);
                // wait while the condition to resume is fulfilled
                LOG.info("resuming topic: " + topic + ", listener key: " + listenerKey);
                listener.getListenerContainer(listenerKey).resume();
                LOG.info("topic resumed: " + topic + ", listener key: " + listenerKey);
            }
        }
    } catch (Exception e){
        LOG.error("", e);
    }
}

来源：https://stackoverflow.com/questions/48402355/spring-kafkalistener-execute-and-poll-records-after-certain-interval

标签

Spring

spring-kafka