How to flatten list inside RDD?

Question

Is it possible to flatten list inside RDD? For example convert:

 val xxx: org.apache.spark.rdd.RDD[List[Foo]]

to:

 val yyy:         


        

Answer 1

You could pimp the RDD class to attach a .flatten method (in order to follow the List api):

object SparkHelper {
  implicit class SeqRDDExtensions[T: ClassTag](val rdd: RDD[Seq[T]]) {
    def flatten: RDD[T] = rdd.flatMap(identity)
  }
}

which can then simply be used as such:

rdd.flatten