What is most efficient way to write from kafka to hdfs with files partitioning into dates

Question

I\'m working on project that should write via kafka to hdfs. Suppose there is online server that writes messages into the kafka. Each message includes timestamp in it. I w

Answer 1

if you are using Apache Kafka 0.9 or above, you can use the Kafka Connect API.

check out https://github.com/confluentinc/kafka-connect-hdfs

This is a Kafka connector for copying data between Kafka and HDFS.