Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]

Question

Is it possible in Spark to implement \'.combinations\' function from scala collections?

   /** Iterates over combinations.
   *
   *  @return   An Iterator w         


        

Answer 1

This is supported natively by a Spark RDD with the cartesian transformation.

e.g.:

val rdd = sc.parallelize(1 to 5)
val cartesian = rdd.cartesian(rdd)
cartesian.collect

Array[(Int, Int)] = Array((1,1), (1,2), (1,3), (1,4), (1,5), 
(2,1), (2,2), (2,3), (2,4), (2,5), 
(3,1), (3,2), (3,3), (3,4), (3,5), 
(4,1), (4,2), (4,3), (4,4), (4,5), 
(5,1), (5,2), (5,3), (5,4), (5,5))