How to get consumers to work in Kafka 0.8 API

问题

I am about to write a prototype for publishing and consuming kafka messages. We do have a Cloudera infrastructure set up already (zookeepers, brokers, etc.), and I have played with the Kafka command-line tools successfully already, to produce and consume messages.

I am using [org.apache.kafka/kafka_2.10 "0.8.2.1"] as dependency, and have already been able to use the client API to set up a KafkaProducer which publishes messages with plain String content, and can be successfully read by the command-line consumer at the other side.

My question is: Is there a single code example on the internets to show how to initialize a KafkaConsumer, and read that message on the other side, because I have been searching for it for days and none of the code examples seem to be working:

They use classes or methods which are not even existing in he API itself (for example they seemingly pass the property-map into the constructor of org.apache.kafka.clients.consumer.ConsumerConfig, but no such constructor exists;
calling createJavaConsumerConnector static method on the class kafka.consumer.Consumer... in which universe these things exist?).

And usually every example looks extremely over-complicated. I would expect a messaging framework to need a few lines of configuration for connecting to brokers, and some function to put and take to/from a queue or topic. Setting up the Producer for Kafka wasn't something extremely complicated, and I was expecting the Consumer to be similar.

It also seems I am not alone with this.

回答1:

First I want to mention, that there are a couple of API changes between Kafka 0.8.0, 0.8.1, and 0.8.2 (a mayor rewrite and simplification happened for 0.9.0 and 0.10.0) -- thus, your question is a little open just asking for 0.8.

To write a Java consumer for 0.8.2.2 you need to include dependency:

This is for Scala 2.11 -- there are other Scala version available, too.

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.11</artifactId>
  <version>0.8.2.2</version>
</dependency>

Do not use kafka-clients as artifactId for 0.8.x.

A minimum example for a consumer receiving <String,String> key-value pair messages and prints them to stdout looks as follows:

import java.util.HashMap;
import java.util.Map;
import java.util.Properties;

import kafka.consumer.Consumer;
import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;

public class ConsumerExample {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("zookeeper.connect", "localhost:2181");
        props.put("group.id", "myGroup");

        final String topic = "test";

        ConsumerConnector consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(props));

        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put(topic, new Integer(1)); // number of consumer threads

        KafkaStream<byte[], byte[]> stream = consumer.createMessageStreams(topicCountMap).get(topic).get(0);

        ConsumerIterator<byte[], byte[]> it = stream.iterator();

        // infinite loop
        while(it.hasNext()) {
            System.out.println(new String(it.next().message()));
        }

        // non-reachable code...
        consumer.shutdown();
    }
}

A full example -- using multiple consumer thread, including proper shutdown -- can be found here: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example

To test this, follow the quickstart guide and send messages via Kafka's console-producer.

来源：https://stackoverflow.com/questions/37321599/how-to-get-consumers-to-work-in-kafka-0-8-api

标签

apache-kafka

Cloudera