Using reduceByKey in Apache Spark (Scala)

Question

I have a list of Tuples of type : (user id, name, count).

For example,

val x = sc.parallelize(List(
    (\"a\", \"b\", 1),
    (\"a\", \"b\", 1),
           


        

Answer 1

Your origin data structure is: RDD[(String, String, Int)], and reduceByKey can only be used if data structure is RDD[(K, V)].

val kv = x.map(e => e._1 -> e._2 -> e._3) // kv is RDD[((String, String), Int)]
val reduced = kv.reduceByKey(_ + _)       // reduced is RDD[((String, String), Int)]
val kv2 = reduced.map(e => e._1._1 -> (e._1._2 -> e._2)) // kv2 is RDD[(String, (String, Int))]
val grouped = kv2.groupByKey()            // grouped is RDD[(String, Iterable[(String, Int)])]
grouped.foreach(println)