Multiple aggregations in Spark Structured Streaming

Question

I would like to do multiple aggregations in Spark Structured Streaming.

Something like this:

• Read a stream of input files (from a folder)
• Per

Answer 1

As of spark structured streaming 2.4.5, multiple aggregations are not supported in stateless processing. But it is possible to aggregate multiple times if you need stateful processing.

With append mode, you can use flatMapGroupWithState API on a grouped dataset (obtained by using groupByKey API) multiple times.