Multiple consumers using spring kafka

问题

I am looking to setup multiple listeners on a kafka topic inside my application. Below is my setup. it is supposed to be consumed by both the groups, but it is consumed by only one listener. What am i missing here?

@Bean
public Map<String, Object> consumerConfigs() {
    Map<String, Object> props = new HashMap<String, Object>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.GROUP_ID_CONFIG, groupName);

    return props;
}

@Bean
public ConsumerFactory<String, String> consumerFactory() {
    ConsumerFactory<String, String> consumerFactory = new DefaultKafkaConsumerFactory(consumerConfigs());
    return consumerFactory;
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConcurrency(100);
    factory.setConsumerFactory(consumerFactory());
    return factory;
}

@Bean("notificationFactory")
public ConcurrentKafkaListenerContainerFactory<String, String> notificationFactory() {
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConcurrency(100);
    factory.setConsumerFactory(consumerFactory());
    return factory;
}

@Bean("insertContainerFactory")
public ConcurrentKafkaListenerContainerFactory<String, String> insertContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConcurrency(100);

    factory.setConsumerFactory(consumerFactory());
    return factory;
}

@KafkaListener(id = "insert_listener", topics = "${kafka.topic.readlocation}", group = "insert_listener", containerFactory = "insertContainerFactory")
public void receiveForInsert(String message) {
    locationProcessor.insertLocationData(message);
}

@KafkaListener(id = "notification_listener", topics = "${kafka.topic.readlocation}", group = "notification_listener",containerFactory="notificationFactory")
public void receiveForNotification(String message) {
    locationProcessor.processNotificationMessage(message);
}

Edit: Below is the code that worked

@KafkaListener(id = "insert_listener", topics = "${kafka.topic.readlocation}", groupId = "insert_listener")
public void receiveForInsert(String message) {
    locationProcessor.insertLocationData(message);
}

回答1:

You need a different group.id for each; the group property is not the group.id - see the javadocs. In the upcoming 1.3 release, there is a new groupId property and we also can use the id as a group if present.

For earlier version you need a different consumer factory for each.

来源：https://stackoverflow.com/questions/45805408/multiple-consumers-using-spring-kafka

标签

spring-kafka