Multiple aggregations in Spark Structured Streaming

Question

I would like to do multiple aggregations in Spark Structured Streaming.

Something like this:

• Read a stream of input files (from a folder)
• Per

Answer 1

For spark 2.2 and above (not sure about earlier version), if you can design the aggregation to use flatMapGroupWithState with append mode, you can do as many aggregations as you want. The restriction is mentioned here Spark structured streaming - Output mode