1. 技术文章
  2. Spring Kafka学习

Spring Kafka学习

由 二次信任 提交于 2019-11-26 09:16:38

Spring Kafka学习

使用参考

版本

2.1.6.RELEASE

配置主题

通过在ioc容器中配置KafkaAdmin实例，我们可以通过配置NewTopic实例和@bean轻松配置主题

@Component
public class KafkaAdminConfig {

    @Value("${kafka-admin.servers}")
    private String servers;

    @Bean
    public KafkaAdmin admin() {
        Map<String, Object> configs = new HashMap<>();
        configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
        return new KafkaAdmin(configs);
    }

    @Bean
    public NewTopic topic1() {
        //主题名，分区数量，副本因子
        return new NewTopic("foo", 10, (short) 2);
    }

    @Bean
    public NewTopic topic2() {
        return new NewTopic("bar", 10, (short) 2);
    }
}

还可以通过AdminClient来设置分区数量等

发送消息

KafkaTemplate

KafkaTemplate包含了生产者，提供了很方便的向主题生产消息的方法

//设置默认主题后可直接调用sendDefault方法
ListenableFuture<SendResult<K, V>> sendDefault(V data);

ListenableFuture<SendResult<K, V>> sendDefault(K key, V data);

ListenableFuture<SendResult<K, V>> sendDefault(Integer partition, K key, V data);

ListenableFuture<SendResult<K, V>> sendDefault(Integer partition, Long timestamp, K key, V data);

ListenableFuture<SendResult<K, V>> send(String topic, V data);

ListenableFuture<SendResult<K, V>> send(String topic, K key, V data);

ListenableFuture<SendResult<K, V>> send(String topic, Integer partition, K key, V data);

ListenableFuture<SendResult<K, V>> send(String topic, Integer partition, Long timestamp, K key, V data);

ListenableFuture<SendResult<K, V>> send(ProducerRecord<K, V> record);

ListenableFuture<SendResult<K, V>> send(Message<?> message);

要使用KafkaTemplate，需配置一个producer factory并提供一种构造器

@Bean
public ProducerFactory<Integer, String> producerFactory() {
    return new DefaultKafkaProducerFactory<>(producerConfigs());
}

@Bean
public Map<String, Object> producerConfigs() {
    Map<String, Object> props = new HashMap<>();
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    // 关于生产者的更多配置： https://kafka.apache.org/documentation/#producerconfigs
    return props;
}

@Bean
public KafkaTemplate<Integer, String> kafkaTemplate() {
    return new KafkaTemplate<Integer, String>(producerFactory());
}

利用ProducerListener异步地获得send方法的回调

ListenableFuture<SendResult<Integer, String>> future = template.send("foo");
future.addCallback(new ListenableFutureCallback<SendResult<Integer, String>>() {

    @Override
    public void onSuccess(SendResult<Integer, String> result) {
        ...
    }

    @Override
    public void onFailure(Throwable ex) {
        ...
    }

});

Transactions

spring kafka提供了三种方法来支持事务：

  1. KafkaTransactionManager
  2. KafkaMessageListenerContainer
  3. KafkaTemplate的executeIntransaction方法

接收消息

可以通过配置MessageListenerContainer并提供一个Message Listener来接收消息，也可以通过使用@KafkaListener注解

Message Listener Containers

提供了两个MessageListenerContainer的实现：

  • KafkaMessageListenerContainer：使用单线程从所有主题和分区接收消息
  • ConcurrentMessageListenerContainer：使用多线程接收

ConcurrentMessageListenerContainer

构造器：

public ConcurrentMessageListenerContainer(ConsumerFactory<K, V> consumerFactory,
                            ContainerProperties containerProperties)

通过设置concurrency属性可以设置其并发性，container.setConcurrency(3)可创建3个KafkaMessageListenerContainer

提交offsets

如果配置consumer的enable.auto.commit属性为true，则Kafka会自动提交offsets，如果是false，consumer支持以下的AckMode（consumer的poll()方法会返回一个或多个ConsumerRecords，每个record会调用MessageListener）：

  • RECORD - commit the offset when the listener returns after processing the record.
  • BATCH - commit the offset when all the records returned by the poll() have been processed.
  • TIME - commit the offset when all the records returned by the poll() have been processed as long as the ackTime since the last commit has been exceeded.
  • COUNT - commit the offset when all the records returned by the poll() have been processed as long as ackCount records have been received since the last commit.
  • COUNT_TIME - similar to TIME and COUNT but the commit is performed if either condition is true.
  • MANUAL - the message listener is responsible to acknowledge() the Acknowledgment; after which, the same semantics as BATCH are applied.
  • MANUAL_IMMEDIATE - commit the offset immediately when the Acknowledgment.acknowledge() method is called by the listener.

KafkaListener Annotation

@KafkaListener提供了简单的POJO listener的方法

public class Listener {
    @KafkaListener(id = "foo", topics = "myTopic", clientIdPrefix = "myClientId")
    public void listen(String data) {
        ...
    }
}

这种方法需要一个使用@EnableKafka注解的配置类，并配置一个listener container factory来配置ConcurrentMessageListenerContainer

@Configuration
@EnableKafka
public class KafkaConfig {

    @Bean
    KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<Integer, String>>
                        kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
                                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.setConcurrency(3);
        factory.getContainerProperties().setPollTimeout(3000);
        return factory;
    }

    @Bean
    public ConsumerFactory<Integer, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, embeddedKafka.getBrokersAsString());
        ...
        return props;
    }
}

可以用明确的主题、分区或初始偏移量来配置这个POJO listener

@KafkaListener(id = "bar", topicPartitions =
        { @TopicPartition(topic = "topic1", partitions = { "0", "1" }),
          @TopicPartition(topic = "topic2", partitions = "0",
             partitionOffsets = @PartitionOffset(partition = "1", initialOffset = "100"))
        })
public void listen(ConsumerRecord<?, ?> record) {
    ...
}

当使用manual AckMode时，listener可以被提供Acknowledgment

@KafkaListener(id = "baz", topics = "myTopic",
          containerFactory = "kafkaManualAckListenerContainerFactory")
public void listen(String data, Acknowledgment ack) {
    ...
    ack.acknowledge();
}

消息的元数据可以从消息的headers获得

@KafkaListener(id = "qux", topicPattern = "myTopic1")
public void listen(@Payload String foo,
        @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) Integer key,
        @Header(KafkaHeaders.RECEIVED_PARTITION_ID) int partition,
        @Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
        @Header(KafkaHeaders.RECEIVED_TIMESTAMP) long ts
        ) {
    ...
}

从版本1.1开始，@KafkaListener方法可以被配置为接收整个从poll()方法接收的consumer records，前提是需在设置container factory的时候，将batchListener属性设置为true

@Bean
public KafkaListenerContainerFactory<?> batchFactory() {
    ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
            new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());
    //设置batchListener属性
    factory.setBatchListener(true);
    return factory;
}

接收简单的list of payloads：

@KafkaListener(id = "list", topics = "myTopic", containerFactory = "batchFactory")
public void listen(List<String> list) {
    ...
}

主题、分区、偏移量等：

@KafkaListener(id = "list", topics = "myTopic", containerFactory = "batchFactory")
public void listen(List<String> list,
        @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) List<Integer> keys,
        @Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions,
        @Header(KafkaHeaders.RECEIVED_TOPIC) List<String> topics,
        @Header(KafkaHeaders.OFFSET) List<Long> offsets) {
    ...
}

你可以接收Message<?>对象的列表，但是必须是唯一参数（与使用manual commits的区分开，后者会有一个Acknowledgment参数）

@KafkaListener(id = "listMsg", topics = "myTopic", containerFactory = "batchFactory")
public void listen14(List<Message<?>> list) {
    ...
}

@KafkaListener(id = "listMsgAck", topics = "myTopic", containerFactory = "batchFactory")
public void listen15(List<Message<?>> list, Acknowledgment ack) {
    ...
}

你也可以接收ConsumerRecord<?, ?>对象的列表，但是必须是唯一参数（与使用manual commits的区分开，后者会有一个Acknowledgment参数）

@KafkaListener(id = "listCRs", topics = "myTopic", containerFactory = "batchFactory")
public void listen(List<ConsumerRecord<Integer, String>> list) {
    ...
}

@KafkaListener(id = "listCRsAck", topics = "myTopic", containerFactory = "batchFactory")
public void listen(List<ConsumerRecord<Integer, String>> list, Acknowledgment ack) {
    ...
}

从2.0版本以来，consumer的id属性就被用作Kafka的group.id属性了，会覆盖consumer factory中配置的group.id。如果提供了id属性又不想覆盖group.id，你可以另外指定group.id或者设置idIsGroup属性为false

在类上使用@KafkaListener

在类上使用@KafkaListener的时候，你可以在方法上使用@KafkaHandler，当消息传递进来时，消息负载的类型将决定调用哪个方法

@KafkaListener(id = "multi", topics = "myTopic")
static class MultiListenerBean {

    @KafkaHandler
    public void listen(String foo) {
        ...
    }

    @KafkaHandler
    public void listen(Integer bar) {
        ...
    }

    @KafkaHandler(isDefault = true`)
    public void listenDefault(Object object) {
        ...
    }

}

@KafkaListener生命周期管理

由@KafkaListener注解创建的listener containers不是ioc容器中的对象，它是被KafkaListenerEndpointRegistry类注册的，后者管理者container的生命周期。如果factory的autoStartup设置为true，它将自动启动。你可以通过使用registry来通过编程的方式管理Container的生命周期。

@Autowired
private KafkaListenerEndpointRegistry registry;

...

@KafkaListener(id = "myContainer", topics = "myTopic", autoStartup = "false")
public void listen(...) { ... }

...

    registry.getListenerContainer("myContainer").start();

使用@SendTo注解发送Listener Results

从2.0版本开始，你可以将@SendTo注释在@KafkaListener的方法上，方法的返回结果将会被发送到@SendTo指定的主题

方法返回结果必须为代表主题名的字符串（或字符串集合）

@KafkaListener(topics = "annotated21")
@SendTo("!{request.value()}") // runtime SpEL
public String replyingListener(String in) {
    ...
}

@KafkaListener(topics = "annotated22")
@SendTo("#{myBean.replyTopic}") // config time SpEL
public Collection<String> replyingBatchListener(List<String> in) {
    ...
}

@KafkaListener(topics = "annotated23", errorHandler = "replyErrorHandler")
@SendTo("annotated23reply") // static reply topic definition
public String replyingListenerWithErrorHandler(String in) {
    ...
}
...
@KafkaListener(topics = "annotated25")
@SendTo("annotated25reply1")
public class MultiListenerSendTo {

    @KafkaHandler
    public String foo(String in) {
        ...
    }

    @KafkaHandler
    @SendTo("!{'annotated25reply2'}")
    public String bar(@Payload(required = false) KafkaNull nul,
            @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) int key) {
        ...
    }

}

在使用@SendTo注解时，ConcurrentKafkaListenerContainerFactory必须被配置为replyTemplate属性的KafkaTemplate

@Bean
public KafkaTemplate<String, String> myReplyingTemplate() {
    return new KafkaTemplate<Integer, String>(producerFactory()) {

        @Override
        public ListenableFuture<SendResult<String, String>> send(String topic, String data) {
            return super.send(topic, partitionForData(data), keyForData(data), data);
        }

        ...

    };
}

序列化 / 反序列化与消息通信

我们可以通过配置生产者和消费者的属性来指定简单的序列化类：

props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
...
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
标签
string
topic
Kafka
bean
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!