Structured streaming won't write DF to file sink citing /_spark_metadata/9.compact doesn't exist

Question

I\'m building a Kafka ingest module in EMR 5.11.1, Spark 2.2.1. My intention is to use Structured Streaming to consume from a Kafka topic, do some processing, and store to E

Answer 1

I solved this question by clearing my checkpoint path:

1. remove your checkpoint path:

   sudo -u hdfs hdfs dfs -rmr ${your_checkpoint_path}

2. resubmit your spark job.