Kafka High Level Consumer Fetch All Messages From Topic Using Java API (Equivalent to --from-beginning)

Question

I am testing the Kafka High Level Consumer using the ConsumerGroupExample code from the Kafka site. I would like to retrieve all the existing messages on the topic called \

Answer 1

Looks like you need to use the "low level SimpleConsumer API"

For most applications, the high level consumer Api is good enough. Some applications want features not exposed to the high level consumer yet (e.g., set initial offset when restarting the consumer). They can instead use our low level SimpleConsumer Api. The logic will be a bit more complicated and you can follow the example in here.

This example worked for getting all messages from a topic with the following arguments: (note that the port is the Kafka port, not the ZooKeeper port, topics set up from this example):

10 my-replicated-topic 0 localhost 9092

Specifically, there is a method to get readOffset which takes kafka.api.OffsetRequest.EarliestTime():

long readOffset = getLastOffset(consumer,a_topic, a_partition, kafka.api.OffsetRequest.EarliestTime(), clientName);

---

Here is another post may provide some alternate ideas on how to sort this out: How to get data from old offset point in Kafka?