kafka-consumer-api

问题 How to use li-apache-kafka-clients in spring boot app to send large message (above 1MB) from Kafka producer to Kafka Consumer? Below is the GitHub link of li-apache-kafka-clients: https://github.com/linkedin/li-apache-kafka-clients I have imported .jar file of li-apache-kafka-clients and put the below configuration for producer: props.put("large.message.enabled", "true"); props.put("max.message.segment.bytes", 1000 * 1024); props.put("segment.serializer", DefaultSegmentSerializer.class

问题 I'm using kafka consumer group management for processing my messages. The processing time for my messages vary from one another. So I have set the max poll interval to 20 min for max records of 20. And I'm using 5 partition and 5 consumer instances with default config values apart from the above two. But still I'm getting the following error intermittently: [Consumer clientId=consumer-3, groupId=amc_dashboard_analytics] Attempt to heartbeat failed since group is rebalancing The understanding

问题 I'm using kafka consumer group management for processing my messages. The processing time for my messages vary from one another. So I have set the max poll interval to 20 min for max records of 20. And I'm using 5 partition and 5 consumer instances with default config values apart from the above two. But still I'm getting the following error intermittently: [Consumer clientId=consumer-3, groupId=amc_dashboard_analytics] Attempt to heartbeat failed since group is rebalancing The understanding

问题 I am trying to poll data from a specific topic like kafka is receiving 100 records/s but most of the time it does not fetch all records. I am using timeout as 5000ms and I am calling this method every 100ms Note : I am subscribing to the specific topic too @Scheduled(fixedDelayString = "100") public void pollRecords() { ConsumerRecords<String, String> records = leadConsumer.poll("5000"); How can I fetch all the data from kafka ? 回答1: Maximum number of records returned from poll() is specified

问题 I am trying to poll data from a specific topic like kafka is receiving 100 records/s but most of the time it does not fetch all records. I am using timeout as 5000ms and I am calling this method every 100ms Note : I am subscribing to the specific topic too @Scheduled(fixedDelayString = "100") public void pollRecords() { ConsumerRecords<String, String> records = leadConsumer.poll("5000"); How can I fetch all the data from kafka ? 回答1: Maximum number of records returned from poll() is specified

问题 I would like to have Exactly-once semantics, but I don't want to read message with Consumer. I'd rather read messages with Kafka Streams AP. If I add processing.guarantee=exactly_once to Stream configuration, will exactly-once semantics be kept? 回答1: Exactly-once processing is based on a read-process-write pattern. Kafka Streams uses this pattern and thus, if you write a regular Kafka Streams application that writes the result back to a Kafka topic, you will get exactly-once processing

问题 I would like to have Exactly-once semantics, but I don't want to read message with Consumer. I'd rather read messages with Kafka Streams AP. If I add processing.guarantee=exactly_once to Stream configuration, will exactly-once semantics be kept? 回答1: Exactly-once processing is based on a read-process-write pattern. Kafka Streams uses this pattern and thus, if you write a regular Kafka Streams application that writes the result back to a Kafka topic, you will get exactly-once processing

问题 AFAIK, max.poll.interval.ms is introduced in Kafka 0.10.1. However it is still unclear that when we can use both session.timeout.ms and max.poll.interval.ms Consider the use casein which heartbeat thread is not responding, but my processing thread as it has higher value set, it still is processing the record. But as heartbeat thread is down then after crossing session.timeout.ms, what exactly happens. Because I've observed in POC that consumer re-balance doesn't happen until it reaches max

问题 AFAIK, max.poll.interval.ms is introduced in Kafka 0.10.1. However it is still unclear that when we can use both session.timeout.ms and max.poll.interval.ms Consider the use casein which heartbeat thread is not responding, but my processing thread as it has higher value set, it still is processing the record. But as heartbeat thread is down then after crossing session.timeout.ms, what exactly happens. Because I've observed in POC that consumer re-balance doesn't happen until it reaches max

问题 How can write a consumer that joins multiple Kafka topics in a scalable way? I have a topic that published events with a key and a second topic that publishes other events related to a subset of the first with the same key. I would like to write a consumer that subscribes to both topics and performs some additional actions for the subset that appears in both topics. I can do this easily with a single consumer: read everything from both topics, maintaining state locally and perform the actions