How can I use foreach in Python Spark structured streaming to trigger ops on output.
query = wordCounts\
.writeStream\
.outputMode('update')\
.foreach(func)\
.start()
def func():
ops(wordCounts)
TL;DR It is not possible to use foreach method in pyspark.
Quoting the official documentation of Spark Structured Streaming (highlighting mine):
The foreach operation allows arbitrary operations to be computed on the output data. As of Spark 2.1, this is available only for Scala and Java.
Support for the foreach sink in Python has been added in Spark 2.4.0 and the documentation has been updated: http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch
Make sure that you have that version and you can now do:
def process_row(row):
# Process row
pass
query = streamingDF.writeStream.foreach(process_row).start()
It's impossible to use foreach in pyspark using any simple tricks now, besides, in pyspark, the update output mode is only ready for debugging.
I'd recommand you to use spark in scala, it's not hard to learn.
You can use DataFrame.foreach(f) instead.
来源：https://stackoverflow.com/questions/48201647/how-to-use-foreach-sink-in-pyspark