Apache Spark - foreach Vs foreachPartitions When to use What?

Question

I would like to know if the foreachPartitions will results in better performance, due to an higher level of parallelism, compared to the foreach m

Answer 1

The foreachPartition does not mean it is per node activity rather it is executed for each partition and it is possible you may have large number of partition compared to number of nodes in that case your performance may be degraded. If you intend to do a activity at node level the solution explained here may be useful although it is not tested by me