kafka-consumer-api

What is the difference between kafka earliest and latest offset values

由 喜欢而已 提交于 2019-12-03 11:21:41
问题 producer sends messages 1, 2, 3, 4 consumer receives messages 1, 2, 3, 4 consumer crashes/disconnects producer sends messages 5, 6, 7 consumer comes back up and should receive messages starting from 5 instead of 7 For this kind of result, which offset value I have to use and what are the other changes/configurations need to do 回答1: When a consumer joins a consumer group it will fetch the last committed offset so it will restart to read from 5, 6, 7 if before crashing it committed the latest

Kafka consumer for multiple topic

由 爱⌒轻易说出口 提交于 2019-12-03 11:03:05
I have a list of topics (for now it's 10) whose size can increase in future. I know we can spawn multiple threads (per topic) to consume from each topic, but in my case if the number of topics increases, then the number of threads consuming from the topics increases, which I do not want, since the topics are not going to get data too frequently, so the threads will sit ideal. Is there any way to have a single consumer to consume from all topics? If yes, then how can we achieve it? Also how will the offset be maintained by Kafka? Please suggest answers. Subrata Saha We can subscribe for

What is the difference in Kafka between a Consumer Group Coordinator and a Consumer Group Leader?

由 谁都会走 提交于 2019-12-03 09:49:46
问题 I see references to Kafka Consumer Group Coordinators and Consumer Group Leaders... What is the difference? What is the benefit from separating group management into two different sets of responsibilities? 回答1: The consumer group coordinator is one of the brokers while the group leader is one of the consumer in a consumer group. The group coordinator is nothing but one of the brokers which receives heartbeats (or polling for messages) from all consumers of a consumer group. Every consumer

Error UNKNOWN_MEMBER_ID occurred while committing offsets for group xxx

由 ≡放荡痞女 提交于 2019-12-03 09:23:12
问题 With Kafka client Java library, consuming logs has worked for some time but with the following errors it doesn't work any more: 2016-07-15 19:37:54.609 INFO 4342 --- [main] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead. 2016-07-15 19:37:54.933 ERROR 4342 --- [main] o.a.k.c.c.internals.ConsumerCoordinator : Error UNKNOWN_MEMBER_ID occurred while committing offsets for group logstash 2016-07-15 19:37:54.933 WARN 4342 --- [main] o.a.k.c.c.internals

Is it possible to create a kafka topic with dynamic partition count?

由 十年热恋 提交于 2019-12-03 08:44:34
问题 I am using kafka to stream the events of page visits by the website users to an analytics service. Each event will contain the following details for the consumer: user id IP address of the user I need very high throughput, so I decided to partition the topic with partition key as userId-ipAddress ie For a userId 1000 and ip address 10.0.0.1, the event will have partition key as "1000-10.0.0.1" In this use case the partition key is dynamic, so specifying the number of partitions upfront while

Kafka Consumer Marking the coordinator 2147483647 dead

由 让人想犯罪 __ 提交于 2019-12-03 08:27:59
问题 I am using Kafka Server 0.9 with consumer kafka-client version 0.9 and kafka-producer 0.8.2. Every thing is working great except i am getting lot of info that the coordinator is dead on the consumer 2016-02-25 19:30:45.046 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead. 2016-02-25 19:30:45.048 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead. 2016-02-25 19:30:45.049 INFO

How does Kafka store offsets for each topic?

由 Deadly 提交于 2019-12-03 05:54:53
问题 While polling Kafka, I have subscribed to multiple topics using the subscribe() function. Now, I want to set the offset from which I want to read from each topic, without resubscribing after every seek() and poll() from a topic. Will calling seek() iteratively over each of the topic names, before polling for data achieve the result? How are the offsets exactly stored in Kafka? I have one partition per topic and just one consumer to read from all topics. 回答1: How does Kafka store offsets for

Counting Number of messages stored in a kafka topic

由 不想你离开。 提交于 2019-12-03 05:41:56
问题 I'm using 0.9.0.0 version of Kafka and I want to count the number of messages in a topic without using the admin script kafka-console-consumer.sh. I have tried all the commands in the answer Java, How to get number of messages in a topic in apache kafka but none are yielding the result. Can anyone help me out here? 回答1: You could try to execute the command below: bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092,localhost:9093,localhost:9094 --topic test-topic -

Consumer not receiving messages, kafka console, new consumer api, Kafka 0.9

由 落花浮王杯 提交于 2019-12-03 05:33:50
问题 I am doing the Kafka Quickstart for Kafka 0.9.0.0. I have zookeeper listening at localhost:2181 because I ran bin/zookeeper-server-start.sh config/zookeeper.properties I have a single broker listening at localhost:9092 because I ran bin/kafka-server-start.sh config/server.properties I have a producer posting to topic "test" because I ran bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test yello is this thing on? let's try another gimme more When I run the old API consumer,

how to set group name when consuming messages in kafka using command line

由 左心房为你撑大大i 提交于 2019-12-03 05:12:44
Any idea how to set group name when consuming messages in kafka using command line. I tried with the following command : bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic nil_RF2_P2 --from-beginning --config group.id=test1 'config' is not a recognized option The goal is to find the offset of consumed messages with the following command: bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper localhost:2181 --group test1 Can somebody help in this regards!! Thanks in advance !! The simplest solution is: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic
订阅 kafka-consumer-api