Multiple aggregations in Spark Structured Streaming

Question

I would like to do multiple aggregations in Spark Structured Streaming.

Something like this:

• Read a stream of input files (from a folder)
• Per

Answer 1

Multiple aggregates in Spark Structured streaming is not supported as of Spark 2.4. Supporting this can be tricky esp. with event time in "update" mode since the aggregate output could change with late events. Its much straightforward to support this in "append" mode however spark does not support true watermarks yet.

Heres a proposal to add it in "append" mode - https://github.com/apache/spark/pull/23576

If interested you can watch the PR and post your votes there.