How to get topic list from kafka server in Java

问题:

I am using kafka 0.8 version and very much new to it.

I want to know the list of topics created in kafka server along with it's metadata. Is there any API available to find out this?

Basically, I need to write a Java consumer that should auto-discover any topic in kafka server.There is API to fetch TopicMetadata, but this needs name of topic as input parameters.I need information for all topics present in server.

回答1:

A good place to start would be the sample shell scripts shipped with Kafka. In the /bin directory of the distribution there's some shell scripts you can use, one of which is ./kafka-topic-list.sh If you run that without specifying a topic, it will return all topics with their metadata. See: https://github.com/apache/kafka/blob/0.8/bin/kafka-list-topic.sh

That shell script in turn runs: https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/admin/ListTopicCommand.scala

The above are both references to the 0.8 Kafka version, so if you're using a different version (even a point difference), be sure to use the appropriate branch/tag on github

回答2:

with Kafka 0.9.0

you can list the topics in the server with the provided consumer method listTopics();

eg.

Map > topics;  Properties props = new Properties(); props.put("bootstrap.servers", "1.2.3.4:9092"); props.put("group.id", "test-consumer-group"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");  KafkaConsumer consumer = new KafkaConsumer(props); topics = consumer.listTopics(); consumer.close();,>,>,>

回答3:

If you want to pull broker or other-kafka information from Zookeeper then kafka.utils.ZkUtils provides a nice interface. Here is the code I have to list all zookeeper brokers (there are a ton of other methods there):

List listBrokers() {          final ZkConnection zkConnection = new ZkConnection(connectionString);         final int sessionTimeoutMs = 10 * 1000;         final int connectionTimeoutMs = 20 * 1000;         final ZkClient zkClient = new ZkClient(connectionString,                                                sessionTimeoutMs,                                                connectionTimeoutMs,                                                ZKStringSerializer$.MODULE$);          final ZkUtils zkUtils = new ZkUtils(zkClient, zkConnection, false);          scala.collection.JavaConversions.seqAsJavaList(zkUtils.getAllBrokersInCluster()); }

回答4:

I think this is the best way:

ZkClient zkClient = new ZkClient("zkHost:zkPort"); List topics = JavaConversions.asJavaList(ZkUtils.getAllTopics(zkClient));

回答5:

Using Scala:

import java.util.{Properties} import org.apache.kafka.clients.consumer.KafkaConsumer  object KafkaTest {   def main(args: Array[String]): Unit = {      val brokers = args(0)     val props = new Properties();     props.put("bootstrap.servers", brokers);     props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");     props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");      val consumer = new KafkaConsumer[String, String](props);     val topics = consumer.listTopics().keySet();      println(topics)   } }

回答6:

You can use zookeeper API to get the list of brokers as mentioned below:

    ZooKeeper zk = new ZooKeeper("zookeeperhost, 10000, null);     List ids = zk.getChildren("/brokers/ids", false);     List

Use this broker list to get all the topic using the following link

https://cwiki.apache.org/confluence/display/KAFKA/Finding+Topic+and+Partition+Leader

转载请标明出处:How to get