How to flatten a collection with Spark/Scala?

Question

In Scala I can flatten a collection using :

val array = Array(List(\"1,2,3\").iterator,List(\"1,4,5\").iterator)
                                                    


        

Answer 1

Try flatMap with an identity map function (y => y):

scala> val x = sc.parallelize(List(List("a"), List("b"), List("c", "d")))
x: org.apache.spark.rdd.RDD[List[String]] = ParallelCollectionRDD[1] at parallelize at <console>:12

scala> x.collect()
res0: Array[List[String]] = Array(List(a), List(b), List(c, d))

scala> x.flatMap(y => y)
res3: org.apache.spark.rdd.RDD[String] = FlatMappedRDD[3] at flatMap at <console>:15

scala> x.flatMap(y => y).collect()
res4: Array[String] = Array(a, b, c, d)

Answer 2

Use flatMap and the identity Predef, this is more readable than using x => x, e.g.

myRdd.flatMap(identity)