kafka-consumer-api

问题 I am working with Kafka and trying to setup consumer group by follwing this article. The only difference is I have created my own abstract class, handler to make design simpler. Below is my abstract class: public abstract class Consumer implements Runnable { private final Properties consumerProps; private final String consumerName; public Consumer(String consumerName, Properties consumerProps) { this.consumerName = consumerName; this.consumerProps = consumerProps; } protected abstract void

问题 Closed. This question is off-topic. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow. Closed 6 months ago . I want to debug some Kafka topics so I know if the consumer or producer is at fault here. Is there a UI for Kafka where I can see what messages a topic contain? A dumper would also be nice so I can search for stuff on my own. 回答1: We use Landoop's Kafka Topics UI, which is pretty good. You can see topic

问题 Following to the solution mentioned here kafka-mirror-maker-failing-to-replicate-consumer-offset-topic. I was able to start mirror maker without any error across DC1(Live Kafka cluster) and DC2(Backup Kafka cluster) clusters. Looks like it is also able to sync __consumer_offsets topic across DC2 cluster form DC1 cluster. Issue If I close down consumer for DC1 and point same consumer(same group_id) to DC2 it reads the same messages again even though mirror maker is able sync offsets for this

问题 One of my services stopped working a few hours ago. I have restarted the server but this consumer process just isn't registering with Kafka. This is what shows up when the consumer is started : 2017-04-21 10:22:54.887 INFO 18036 --- [afka-consumer-1] o.a.k.c.c.internals.ConsumerCoordinator : Revoking previously assigned partitions [] for group mysql-conversions-group 2017-04-21 10:22:54.890 INFO 18036 --- [afka-consumer-1] o.s.k.l.KafkaMessageListenerContainer : partitions revoked:[] 2017-04

问题 I need data from kafka brokers,but for fast access I am using multiple consumers with same group id known as consumer groups.But after reading by each consumer,how can we combine data from multiple consumers? Is there any logic? 回答1: By design, different consumers in the same consumer group process data independently from each other. (This behavior is what allows applications to scale well.) But after reading by each consumer,how can we combine data from multiple consumers? Is there any logic

问题 I have a topic consisting of n partitions. To have distributed processing I create two processes running on different machines. They subscribe to the topic with same groupd id and allocate n/2 threads, each of which processes single stream(n/2 partitions per process). With this I will have achieved load distribution, but now if process 1 crashes, than process 2 cannot consume messages from partitions allocated to process 1, as it listened only on n/2 streams at the start. Or else, if I

问题 I created a topic and i put a simple-producer to publish some message in that topic bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic streams-file-input bin/kafka-console-producer.sh --broker-list localhost:9092 --streams-file-input I am running the below simple example in kafka streams and i got a weird exception which i cannot handle Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-pipe"); props

问题 Closed . This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Update the question so it focuses on one problem only by editing this post. Closed 3 months ago . I have a couple questions in Kafka. 1) Does Kafka have a default web UI? 2) How can we gracefully shutdown a standalone kafka server, kafka console- consumer/console-producer. Any solutions will be highly appreciated. Thank you. 回答1: 1) No Kafka does not have a default UI. There

问题 I am doing Python Kafka consumer (trying to use kafka.consumer.SimpleConsumer or kafka.consumer.simple.SimpleConsumer in http://kafka-python.readthedocs.org/en/latest/apidoc/kafka.consumer.html). When I run the following piece of code, it will run all the time, even if all messages consumed. I hope the consumer will stop if it consume all the messages. How to do it? Also I have no idea how to use stop() function (which is in base class kafka.consumer.base.Consumer). UPDATE I used signal

问题 So I'm trying the kafka quickstart as per the main documentation. Got the multi-cluster example all setup and test per the instructions and it works. For example, bringing down one broker and the producer and consumer can still send and receive. However, as per the example, we setup 3 brokers and we bring down broker 2 (with broker id = 1). Now if I bring up all brokers again, but I bring down broker 1 (with broker id = 0), the consumer just hangs. This only happens with broker 1 (id = 0),