Structured streaming won't write DF to file sink citing /_spark_metadata/9.compact doesn't exist

Question

I\'m building a Kafka ingest module in EMR 5.11.1, Spark 2.2.1. My intention is to use Structured Streaming to consume from a Kafka topic, do some processing, and store to E

Answer 1

It turns out that S3 does not support the read-after-write semantics needed by Spark checkpointing.

This article suggests using AWS EFS for checkpointing.

S3 remains a good place to ingest data from, or egest data to.

Answer 2

I solved this question by clearing my checkpoint path:

1. remove your checkpoint path:

   sudo -u hdfs hdfs dfs -rmr ${your_checkpoint_path}

2. resubmit your spark job.