kafka-consumer-api

问题 I am having trouble with KafaConsumer to make it read from the beginning, or from any other explicit offset. Running the command line tools for the consumer for the same topic , I do see messages with the --from-beginning option and it hangs otherwise $ ./kafka-console-consumer.sh --zookeeper {localhost:port} --topic {topic_name} --from-beginning If I run it through python, it hangs, which I suspect to be caused by incorrect consumer configs consumer = KafkaConsumer(topic_name, bootstrap

问题 I am having trouble with KafaConsumer to make it read from the beginning, or from any other explicit offset. Running the command line tools for the consumer for the same topic , I do see messages with the --from-beginning option and it hangs otherwise $ ./kafka-console-consumer.sh --zookeeper {localhost:port} --topic {topic_name} --from-beginning If I run it through python, it hangs, which I suspect to be caused by incorrect consumer configs consumer = KafkaConsumer(topic_name, bootstrap

问题 What is the meaning of this kafka error ? [2018-08-22 11:40:49,429] WARN [Consumer clientId=consumer-1, groupId=console-consumer-62114] 1 partitions have leader brokers without a matching listener, including [topicname-0] (org.apache.kafka.clients.NetworkClient) I'm getting it when running: ./kafka-console-consumer.sh --topic topicname --bootstrap-server localhost:9094 And I'm getting some errors inside a golang program when trying to read this topic: 2018/08/22 11:44:12

问题 The document https://www.safaribooksonline.com/library/view/kafka-the-definitive/9781491936153/ch04.html says that "Note that with auto-commit enabled, a call to poll will always commit the last offset returned by the previous poll. It doesn’t know which events were actually processed, so it is critical to always process all the events returned by poll before calling poll again (or before calling close(), it will also automatically commit offsets)" . If that's the case how does it work if

问题 I'm trying to create a consumer client in Java. I realized the poll() function is depreciated. What are the alternatives to listen to a topic of Kafka? My code: KafkaConsumer< String, UserSegmentPayload > kc = new KafkaConsumer<>(props2); kc.subscribe(Collections.singletonList(topicName)); while (true) { ConsumerRecords<String, UserSegmentPayload> records = kc.poll(100); for (ConsumerRecord<String, UserSegmentPayload> record : records) { System.out.printf("offset = %d, key = %s, value = %s\n"

问题 I have a cluster of 3 Kafka brokers with replication factor of 3 for all topics. Since last few days I am facing this issue, suddenly(few times in a day) consumers and producers are stuck while getting response even when Kafka is running on all 3 servers which gets resolved until I check brokers logs ("Connection to 0 was disconnected before the response was read")and find out the culprit node which is 0 (1st node in this case)and restarts zookeeper and broker on that node. As per logs it is

问题 This is a followup question to "Where do zookeeper store Kafka cluster and related information?" based on the answer provided by Armando Ballaci. Now it's clear that consumer offsets are stored in the Kafka cluster in a special topic called __consumer_offsets . That's fine, I am just wondering how does the retrieval of these offsets work. Topics are not like RDBS over which we can query for arbitrary data based on a certain predicate. Ex - if the data is stored in an RDBMS, probably a query

问题 By saying cluster info, I am referring to information like subscribed consumers/consumer groups read and committed offsets leaders and followers of a partition topics on the server etc. Does zookeeper keep this info in its own db (though I never heard of zookeeper having any db of its own till date) or it stores this info in the Kafka cluster on some topics etc? EDIT: and a follow up Q: How does Zookeeper retrive the consumer offsets from __consumer_offsets topic? 回答1: The ZooKeeper Data

问题 I have a Kafka application from where I have been consuming messages using kafka-console-consumer.sh as following: $./kafka-console-consumer.sh --zookeeper zookeeperhost:2181 --topic myTopic which gives all the messages which I write to Kafka broker through a Kafka consumer without any miss. Recently I deployed the application in a different environment where zookeeperhost is not accessible (due to some reason). So I am using kafka-simple-consumer-shell.sh instead as below: $./kafka-simple

问题 I am trying to create a Kafka topology and break it down into more readable. I have a stream that I group by keys, and then I am trying to window it like so: SessionWindowedKStream<byte[], byte[]> windowedTable = groupedStream.windowedBy(SessionWindows.with(Duration.ofSeconds(config.joinWindowSeconds)).grace(Duration.ZERO)); KTable<Windowed<byte[]>, byte[]> mergedTable = windowedTable .reduce((aggregateValue, newValue) -> { try { Map<String, String> recentMap = MAPPER.readValue(new String