How to write Spark Streaming output to HDFS without overwriting

Question

After some processing I have a DStream[String , ArrayList[String]] , so when I am writing it to hdfs using saveAsTextFile and after every batch it overwrites the data , so h

Answer 1

Here I solve the issue without dataframe

import java.time.format.DateTimeFormatter
import java.time.LocalDateTime

 messages.foreachRDD{ rdd =>
    rdd.repartition(1)
    val eachRdd = rdd.map(record => record.value)
    if(!eachRdd.isEmpty) {
      eachRdd.saveAsTextFile(hdfs_storage + DateTimeFormatter.ofPattern("yyyyMMddHHmmss").format(LocalDateTime.now) + "/")
    }
  }