How do I upsert into HDFS with spark?

Question

I have partitioned data in the HDFS. At some point I decide to update it. The algorithm is:

• Read the new data from a kafka topic.
• Find out new data\'s

Answer 1

In the end I just decided to delete that "green" subset of partitions from HDFS, and use SaveMode.Append instead. I think this is a bug in spark.