Why does foreach not bring anything to the driver program?

Question

I wrote this program in spark shell

val array = sc.parallelize(List(1, 2, 3, 4))
array.foreach(x => println(x))

this prints some debug s

Answer 1

You can use RDD.toLocalIterator() to bring the data to the driver (one RDD partition at a time):

val array = sc.parallelize(List(1, 2, 3, 4))
for(rec <- array.toLocalIterator) { println(rec) }

See also

• Spark: Best practice for retrieving big data from RDD to local machine
• this blog post about toLocalIterator