kafka-consumer-api

As it can be guest from the title, is there a way to get the consumer list on a specific topic in java? Untill now Im able to get the list of topics like this final ListTopicsResult listTopicsResult = adminClient.listTopics(); KafkaFuture<Set<String>> kafkaFuture = listTopicsResult.names(); Set<String> map = kafkaFuture.get(); but I havent found a way to get the list of consumers on each topic I was recently solving the same problem for my kafka client tool. It is not easy, but the only way, which I found from the code is the following: Properties props = ...//here you put your properties

My kafka topic has 10 records/messages in total and 2 partitions having 5 messages each. My consumer group has 2 consumers and each of the consumer has already read 5 messages from their assigned partition respectively. Now, I want to re-process/read messages from my topic from start/beginning (offset 0). I stopped my kafka consumers and ran following command to reset consumer group offset to 0. ./kafka-consumer-groups.sh --group cg1 --reset-offsets --to-offset 0 --topic t1 --execute --bootstrap-server "..." My expectation was that once I restart my kafka consumers they will start reading

At first all the brokers in cluster can start and work just fine, but sometimes one of the broker will meet problem. And there are some phenomenon will show up: whole cluster is hang, nor producer and consumer are not work, hence the network flow is down to zero from monitor; use kafka-topic.sh describe the topic message, every replica is just fine, even the exceptional brokerid, and information in zk also normal; the file-description number increase gradually on abnormal broker, which is read from /proc/sys/fs/file-nr netstat the broker listen port 9092 display lots of "CLOSE_WAIT" status

I am writing test cases for kafka consumer components and mocking kafkaConsumer.poll() which returns instance of ConsumerRecords<String,String> . I want to initialize ConsumerRecords and use that in mock but the constructors of ConsumerRecords expect actual kafka topic which I don't have in tests. One way I think for this is by keeping a serialized copy of object and deserialize to initialize ConsumerRecords . Is there any other way to achieve the same. Here is some example code (Kafka clients lib version 0.10.1.1): import java.util.ArrayList; import java.util.Collection; import java.util

Is it possible to consume messages from Kafka based on a time period in which the messages were ingested? Example : I want all messages ingested to a topic between 0900-1000 today (and now it's 1200). If there is only a way to specify a start time, that's fine - my consumer can stop processing messages once it reaches the end time. I can see methods for requesting messages from a given offset, and for getting the first available offset, and for the earliest available offset, but not all messages after a given time. You could use the offsetsForTimes method which returns you offset whose

I am using kafka 0.8.1.1. I have a 3 node kafka cluster with some topics having around 5 partitions. I planned to increase the number of nodes to 5 in cluster and moving some partitions from existing topics to the new brokers. Previous partition state: broker1 : topic1 { partition 0 } broker2 : topic1 { partition 1,2} broker3 : topic1 { partition 3,4} New intended state: broker1 : topic1 { partition 0} broker2 : topic1 { partition 1} broker3 : topic1 { partition 3} broker4 : topic1 { partition 4} broker5 : topic1 { partition 2} command which I used: bin/kafka-reassign-partitions.sh -

According to the documentation on kafka javadocs if I: Subscribe to a pattern Create a topic that matches the pattern A rebalance should occur, which makes the consumer read from that new topic. But that's not happening. If I stop and start the consumer, it does pick up the new topic. So I know the new topic matches the pattern. There's a possible duplicate of this question in https://stackoverflow.com/questions/37120537/whitelist-filter-in-kafka-doesnt-pick-up-new-topics but that question got nowhere. I'm seeing the kafka logs and there are no errors, it just doesn't trigger a rebalance. The