Reset consumer offset to the beginning from Kafka Streams

问题

I am using Kafka streams and want to reset some consumer offset from Java to the beginning. KafkaConsumer.seekToBeginning(...) sounds like the right thing to do, but I work with Kafka Streams:

KafkaStreams streams = new KafkaStreams(builder, props);
...
streams.start();

I guess that depending on the concrete streams pipeline I define this would create several consumers under the hood. Can I get access to those? Or is there some other way to reset offsets programmatically?

回答1:

Since you are using Kafka Streams you will want to reset not only the consumer offsets but also the Streams internal state store.

Fortunately there is a Streams Application Reset Tool provided with Kafka.

See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Application+Reset+Tool

回答2:

Building on Hans Jespersens answer, I successfully used this code to do what the script does in Java code:

import kafka.tools.StreamsResetter;

StreamsResetter resetter = new StreamsResetter();
String[] args = {"--application-id", APP_ID, "--bootstrap-servers", KAFKA_SERVERS, "--input-topics", TEST_TOPIC_NAME, "--zookeeper", ZOOKEEPER};
resetter.run(args);

The class is part of the kafka core library that I imported in maven using:

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.12</artifactId>
        <version>${kafka.version}</version>
    </dependency>

来源：https://stackoverflow.com/questions/44995313/reset-consumer-offset-to-the-beginning-from-kafka-streams

标签

java

stream

apache-kafka

offset

consumer