kafka get partition count for a topic

问题:

How can I get number of partitions for any kafka topic from the code. I have researched many links but none seem to work.

Mentioning a few:

http://grokbase.com/t/kafka/users/148132gdzk/find-topic-partition-count-through-simpleclient-api

http://grokbase.com/t/kafka/users/151cv3htga/get-replication-and-partition-count-of-a-topic

http://qnalist.com/questions/5809219/get-replication-and-partition-count-of-a-topic

which look like similar discussions.

Also there are similar links on SO which do not have a working solution to this.

回答1:

Go to your kafka/bin directory.

Then run this:

./kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic_name

You should see what you need under PartitionCount.

Topic:topic_name        PartitionCount:5        ReplicationFactor:1     Configs:         Topic: topic_name       Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001         Topic: topic_name       Partition: 1    Leader: 1001    Replicas: 1001  Isr: 1001         Topic: topic_name       Partition: 2    Leader: 1001    Replicas: 1001  Isr: 1001         Topic: topic_name       Partition: 3    Leader: 1001    Replicas: 1001  Isr: 1001         Topic: topic_name       Partition: 4    Leader: 1001    Replicas: 1001  Isr: 1001 

回答2:

In the 0.82 Producer API and 0.9 Consumer api you can use something like

Properties configProperties = new Properties(); configProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092"); configProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.ByteArraySerializer"); configProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");  org.apache.kafka.clients.producer.Producer producer = new KafkaProducer(configProperties); producer.partitionsFor("test") 

回答3:

Here's how I do it:

  /**    * Retrieves list of all partitions IDs of the given {@code topic}.    *     * @param topic    * @param seedBrokers List of known brokers of a Kafka cluster    * @return list of partitions or empty list if none found    */   public static List<Integer> getPartitionsForTopic(String topic, List<BrokerInfo> seedBrokers) {     List<Integer> partitions = new ArrayList<>();     for (BrokerInfo seed : seedBrokers) {       SimpleConsumer consumer = null;       try {         consumer = new SimpleConsumer(seed.getHost(), seed.getPort(), 20000, 128 * 1024, "partitionLookup");         List<String> topics = Collections.singletonList(topic);         TopicMetadataRequest req = new TopicMetadataRequest(topics);         kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);          // find our partition's metadata         List<TopicMetadata> metaData = resp.topicsMetadata();         for (TopicMetadata item : metaData) {           for (PartitionMetadata part : item.partitionsMetadata()) {             partitions.add(part.partitionId());           }         }         break;  // leave on first successful broker (every broker has this info)       } catch (Exception e) {         // try all available brokers, so just report error and go to next one         LOG.error("Error communicating with broker [" + seed + "] to find list of partitions for [" + topic + "]. Reason: " + e);       } finally {         if (consumer != null)           consumer.close();       }     }     return partitions;   } 

Note that I just needed to pull out partition IDs, but you can additionally retrieve any other partition metadata, like leader, isr, replicas, ...
And BrokerInfo is just a simple POJO that has host and port fields.

回答4:

So the following approach works for kafka 0.10 and it does not use any producer or consumer APIs. It uses some classes from the scala API in kafka such as ZkConnection and ZkUtils.

    ZkConnection zkConnection = new ZkConnection(zkConnect);     ZkUtils zkUtils = new ZkUtils(zkClient,zkConnection,false);     System.out.println(JavaConversions.mapAsJavaMap(zkUtils.getPartitionAssignmentForTopics(          JavaConversions.asScalaBuffer(topicList))).get("bidlogs_kafka10").size()); 

回答5:

@Sunil-patil answer stopped short of answering the count piece of it. You have to get the size of the List

producer.partitionsFor("test").size()

@vish4071 no point butting Sunil, you did not mention that you are using ConsumerConnector in the question.

回答6:

I have had the same issue, where I needed to get the partitions for a topic.

With the help of the answer here I was able to get the information from Zookeeper.

Here is my code in Scala (but could be easily translated into Java)

import org.apache.zookeeper.ZooKeeper  def extractPartitionNumberForTopic(topicName: String, zookeeperQurom: String): Int = {   val zk = new ZooKeeper(zookeeperQurom, 10000, null);   val zkNodeName = s"/brokers/topics/$topicName/partitions"   val numPartitions = zk.getChildren(zkNodeName, false).size   zk.close()   numPartitions } 

Using this approach allowed me to access the information about Kafka topics as well as other information about Kafka brokers ...

From Zookeeper you could check for the number of partitions for a topic by browsing to /brokers/topics/MY_TOPIC_NAME/partitions

Using zookeeper-client.sh to connect to your zookeeper:

[zk: ZkServer:2181(CONNECTED) 5] ls /brokers/topics/MY_TOPIC_NAME/partitions [0, 1, 2] 

That shows us that there are 3 partitions for the topic MY_TOPIC_NAME

回答7:

You can explore the kafka.utils.ZkUtils which has many methods aimed to help extract metadata about the cluster. The answers here are nice so I'm just adding for the sake of diversity:

import kafka.utils.ZkUtils import org.I0Itec.zkclient.ZkClient  def getTopicPartitionCount(zookeeperQuorum: String, topic: String): Int = {   val client = new ZkClient(zookeeperQuorum)   val partitionCount = ZkUtils.getAllPartitions(client)     .count(topicPartitionPair => topicPartitionPair.topic == topic)    client.close   partitionCount } 

回答8:

below shell cmd can print the number of partitions. you should be in kafka bin directory before executing the cmd

sh kafka-topics.sh --describe --zookeeper localhost:2181 --topic TopicName | awk '{print $2}' | uniq -c |awk 'NR==2{print "count of partitions=" $1}'

please change the topic name according to your need

you can further validate this using if condition as well sh kafka-topics.sh --describe --zookeeper localhost:2181 --topic TopicName | awk '{print $2}' | uniq -c |awk 'NR==2{if ($1=="16") print "valid partitions"}'

the above cmd prints valid partitions if count is 16 . you can change count depending upon your requirement.

回答9:

cluster.availablePartitionsForTopic(topicName).size() 

回答10:

//create the kafka producer def getKafkaProducer: KafkaProducer[String, String] = { val kafkaProps: Properties = new Properties() kafkaProps.put("bootstrap.servers", "localhost:9092") kafkaProps.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer") kafkaProps.put("value.serializer",  "org.apache.kafka.common.serialization.StringSerializer")  new KafkaProducer[String, String](kafkaProps) } val kafkaProducer = getKafkaProducer val noOfPartition = kafkaProducer.partitionsFor("TopicName")  println(noOfPartition) //it will print the number of partiton for the given  //topic 

文章来源: kafka get partition count for a topic