How can I effectively bind my @KafkaListener to ConcurrentKafkaListenerContainerFactory?

问题

I hit this scenario which appears strange to me:

So basically I have defined two @KafkaListener in one class:

@KafkaListener(id = "listener1", idIsGroup = false, topics = "data1", containerFactory = "kafkaListenerContainerFactory")
    public void receive(){}

@KafkaListener(id = "listener2", idIsGroup = false, topics = "data2", containerFactory = "kafkaListenerContainerFactory2")
    public void receive(){}

Their id, topics, containerFactory are different, and each of them relies on a different ConcurrentKafkaListenerContainerFactory as defined in another class:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> factory = new ConcurrentKafkaListenerContainerFactory();
    factory.setConsumerFactory(consumerFactory("group1", "earliest"));
    factory.setAutoStartup(false);
    return factory;
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> kafkaListenerContainerFactory2() {
    ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> factory = new ConcurrentKafkaListenerContainerFactory();
    factory.setConsumerFactory(consumerFactory("group2", "latest"));
    factory.setAutoStartup(true);
    return factory;
}

@Bean
public ConsumerFactory<String, ConsumerRecord> consumerFactory(String groupId, String offset) {
    Map<String, Object> config = new HashMap<>();
    // dt is current timestamp in millisecond (epoch)
    config.put(ConsumerConfig.GROUP_ID_CONFIG, groupId + "-" + dt);
    config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, offset);
    // other config omitted
    return new DefaultKafkaConsumerFactory<>(config);
}

So what I expect to see (and what I want to achieve) are:

Only listener2 will auto-start because factory.setAutoStartup(true)
Listener2 will start with group.id "group2" and auto.offset.reset "latest"
Later when listener1 starts via some event listener, it will start with group.id "group1" and auto.offset.reset "earlist"

However, only the 1st is actually guaranteed. Listener2 can start with either {group2 + latest} or {group1 + earliest}. And later when listener1 starts to consume data, it will just reuse the config of listener2 (I can see the same group id which contains a timestamp is printed twice in my log)

My question is, why the group ID and offset config for listener2 are randomly picked while autoStartup is not? And why listener1 will reuse the config of listener2?

回答1:

It's because consumerFactory is a singleton @Bean and the arguments are ignored on the second call.

Add @Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE) to the factory get a new bean each time.

However, you don't need any of this, you can simply set the groupId property on the annotations and avoid all this extra definition.

You can also control autoStartup on the annotation (since 2.2).

EDIT

To answer the question in the comment below...

groupId = "#{'${group.id}' + T(java.time.Instant).now().toEpochMilli()}"

however, if you want a unique group id; this is more reliable...

groupId = "#{'${group.id}' + T(java.util.UUID).randomUUID()}"

来源：https://stackoverflow.com/questions/62091106/how-can-i-effectively-bind-my-kafkalistener-to-concurrentkafkalistenercontainer

标签

spring-boot

apache-kafka

spring-kafka