start at a given offset with KakfaIO

问题

I'm using KafkaIO.read() and I'd like to start consuming from a specific offset.

At some point there used to be a KafkaIO.read().withStartFromCheckpointMark() method to do that.

I see from the documentation that there is a way via:

KafkaCheckpointMark provided by runner;

How can I do that ?

Thanks

回答1:

There is no direct support, but there are couple of options:

• withStartReadTime() might be better suited.
• You can create a group.id and commit offsets in that group. When you set group.id in KafkaConsumer config, KafkaIO defaults to resuming from offsets committed for the group. You can also chose to enable auto.commit in KafkaConsumer. See 'Advanced Kafka Configuration' in JavaDoc.

来源：https://stackoverflow.com/questions/50725043/start-at-a-given-offset-with-kakfaio