Not able to write Data in Parquet File using Spark Structured Streaming

Question

I have a Spark Structured Streaming:

val df = spark
      .readStream
      .format(\"kafka\")
      .option(\"kafka.bootstrap.servers\", \"localhost:9092\")
            


        

Answer 1

I had a similar problem but for different reasons, posting here in case someone has the same issue. When writing your output stream to file in append mode with watermarking, structured streaming has an interesting behavior where it won't actually write any data until a time bucket is older than the watermark time. If you're testing structured streaming and have an hour long water mark, you won't see any output for at least an hour.