How to read data using Kafka Consumer API from beginning?

Question

Please can anyone tell me how to read messages using the Kafka Consumer API from the beginning every time when I run the consumer.

Answer 1

Another option is to leave your Consumer code simple and steer the offset management from outside using the command line tool kafka-consumer-groups that comes with Kafka.

Each time, before starting the consumer, you would call

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
 --execute --reset-offsets \
 --group myConsumerGroup \
 --topic myTopic \
 --to-earliest

Depending on your requirement you can reset the offsets for each partition of the topic with that tool. The help function or documentation explain the options:

--reset-offsets also has following scenarios to choose from (atleast one scenario must be selected):

--to-datetime  : Reset offsets to offsets from datetime. Format: 'YYYY-MM-DDTHH:mm:SS.sss'
--to-earliest : Reset offsets to earliest offset.
--to-latest : Reset offsets to latest offset.
--shift-by  : Reset offsets shifting current offset by 'n', where 'n' can be positive or negative.
--from-file : Reset offsets to values defined in CSV file.
--to-current : Resets offsets to current offset.
--by-duration  : Reset offsets to offset by duration from current timestamp. Format: 'PnDTnHnMnS'
--to-offset : Reset offsets to a specific offset.