kafka-consumer-api

问题 I am trying to find ways to get current usage statistics for my kafka cluster. I am looking to collect following information: Number of topics in kafka cluster Number of partitions per kafka broker Number of active consumers and producers Number of client connections per kafka broker Number of messages on each partition, size of disk etc. Lagging replicas, consumer lag etc. Active consumer groups Any other statistics that can and should be collected, currently I am looking at collecting the

问题 I have scenario where am seeing different behavior. Like total of 3 different services First service will listen from Solace queue and produce it to kafka topic-1 (where transaction are enabled) Second Service will listen from above kafka topic-1 and write it to another kafka topic-2 (where we have no manual commits, transactions enabled to produce to other topic, auto commit offset as false & isolation.level is set to read_commited) ago Delete Third Service will listen from kafka topic-2 and

问题 I have manually created topic test with this command: bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test and using this command: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test I inserted these records: This is a message This is another message This is a message2 First, I consume messages through the command line like this: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from

问题 I have a kafka consumer from which I am consuming data from a particular topic and I am seeing below exception. I am using 0.10.0.0 kafka version. LoggingCommitCallback.onComplete: Commit failed for offsets= {....}, eventType= some_type, time taken= 19ms, error= org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was

问题 I am working on java interface which would take user input of topic name, replication and partition to create a kafka topic in KAFKA-2.1.1-1.2.1.1. This is code that I have used from other sources but it seems to be for previous version of kafka import kafka.admin.AdminOperationException; import org.I0Itec.zkclient.ZkClient; import org.I0Itec.zkclient.ZkConnection; import java.util.Properties; import java.util.concurrent.TimeUnit; import kafka.admin.AdminUtils; import kafka.utils

问题 I have already seen a similar question as clickhere But still I want to know if streaming data from a particular partition not possible? I have used Kafka Consumer Strategies in Spark Streaming subscribe method . ConsumerStrategies.Subscribe[String, String](topics, kafkaParams, offsets) This is the code snippet I tried out for subscribing to topic and partition, val topics = Array("cdc-classic") val topic="cdc-classic" val partition=2; val offsets= Map(new TopicPartition(topic, partition) ->

问题 I want to start a consumer in kafka for a particular topic in a small delay. In detail , I want the consumer to start consuming the messages from the topic after a particular time delay from the time of producing the messages . Can anyone say is there any property or option in kafka to enable it . Thanks in advance. 回答1: We did the same stuff for spark-streaming. I hope, the approach can suits you also. The idea is very simple - use Thread.sleep . When you receive new message from kafka, you

问题 I am new to kafka and using Apache kafka consumer to read messages from producer. But when I stop and start for certain time. All the produced messages between are lost. how to handle this scenario. I am using these properties "auto.offset.reset", "latest" and "enable.auto.commit", "false" . This is the code I am using.Any help is appreciated. Properties props = new Properties(); props.put("bootstrap.servers", localhost:9092); props.put("group.id", "service"); props.put("enable.auto.commit",

问题 I am using Kafka 0.9.0 with the native Java consumer client. If I have 1 Topic with 1 Partition Can someone tell me if I do: seekToEnd(MyTopic); poll(x); I will only get the last record, hence I will know that I am in the last position? 回答1: Yes, you will only get the last (newest) record, because the seekToEnd() method " evaluates lazily " so the end is not calculated until poll() is called. Of course, by the time the poll() method returns, more messages could have been added; so there is no

问题 Say if auto-commit interval time is 30 seconds, consumer for some reasons could not process the message and hold it longer than 30 seconds then crash. does the auto-commit offset mechanism commits this offset anyway right before consumer crash? If my assumption is correct, the message is lost as its offset committed but the message itself has not been processed? 回答1: Lets consider your Consumer group name is Test and you have a single consumer in the Consumer Group. When Auto-Commit is