camus

Camus Migration - Kafka HDFS Connect does not start from the set offset

允我心安 提交于 2019-12-01 11:07:31
问题 I am currently using Confluent HDFS Sink Connector (v4.0.0) to replace Camus. We are dealing with sensitive data so we need to maintain consistency in offset during cutover to connectors. Cutover plan: We created hdfs sink connector and subscribed to a topic which writes to a temporary hdfs file. This creates a consumer group with name connect- Stopped the connector using DELETE request. Using /usr/bin/kafka-consumer-groups script, I am able to set the connector consumer group kafka topic