Kafka simple consumer intermittently missing messages

问题

I have a Kafka application from where I have been consuming messages using kafka-console-consumer.sh as following:

$./kafka-console-consumer.sh --zookeeper zookeeperhost:2181 --topic myTopic

which gives all the messages which I write to Kafka broker through a Kafka consumer without any miss.

Recently I deployed the application in a different environment where zookeeperhost is not accessible (due to some reason). So I am using kafka-simple-consumer-shell.sh instead as below:

$./kafka-simple-consumer-shell.sh --broker-list brokerhost:9092 --topic myTopic --partition 0 --max-messages 1

But with this I see few messages (around 2-4 in 5000) go missed. Could someone please explain how kafka-simple-consumer-shell.sh reads messages.

I am doubting that probably some messages are going to some different partition and as I am just reading from partition 0 so I am not getting all the messages every time. But I do not know how to check how many partitions are there? and what are the ids for other partitions? I tried with 1 but it does not work.

Could someone please help.

回答1:

kafka-simple-consumer.sh simply creates a consumer that reads messages from one partition. So your command simply reads a single message in partition 0 of myTopic from brokerhost:9092. If partition 1 is not in the same broker, it will not work as what you did. (For more information, check Code from GitHub)

If you can access to the Zookeeper host, you can simply check how partitions are distributed in a cluster with

bin/kafka-topics.sh --describe --zookeeper zookeeperhost:2181 --topic myTopic

but if you can't access to the Zookeeper host, there are two ways as I can think of.

1. Provide a list having all brokers as a parameter and try partition numbers from 0 to N. You can provide multiple brokers to --broker-list in a format of broker1:port2,broker2:port2,broker3:port3. Then you can figure out how many partitions exist in the entire cluster, but still you don't know which broker has which partitions.
2. Manually check a log directory of each broker. Check /tmp/kafka-logs (if you are using a default log directory). You will find directories like myTopic-0, myTopic-1, ... which are in a format of topic-partition#. You can check which broker has which partitions manually with this.

来源：https://stackoverflow.com/questions/29216369/kafka-simple-consumer-intermittently-missing-messages