How to reverse ordering for RDD.takeOrdered()?

问题

What is the syntax to reverse the ordering for the takeOrdered() method of an RDD in Spark?

For bonus points, what is the syntax for custom-ordering for an RDD in Spark?

回答1:

Reverse Order

val seq = Seq(3,9,2,3,5,4)
val rdd = sc.parallelize(seq,2)
rdd.takeOrdered(2)(Ordering[Int].reverse)

Result will be Array(9,5)

Custom Order

We will sort people by age.

case class Person(name:String, age:Int)
val people = Array(Person("bob", 30), Person("ann", 32), Person("carl", 19))
val rdd = sc.parallelize(people,2)
rdd.takeOrdered(1)(Ordering[Int].reverse.on(x=>x.age))

Result will be Array(Person(ann,32))

回答2:

val rdd1 = sc.parallelize(List(("Hadoop PIG Hive"), ("Hive PIG PIG Hadoop"), ("Hadoop Hadoop Hadoop")))

val rdd2 = rdd1.flatMap(x => x.split(" ")).map(x => (x,1))

val rdd3 = rdd2.reduceByKey((x,y) => (x+y))

//Reverse Order (Descending Order)

rdd3.takeOrdered(3)(Ordering[Int].reverse.on(x=>x._2))

Output:

res0: Array[(String, Int)] = Array((Hadoop,5), (PIG,3), (Hive,2))

//Ascending Order

rdd3.takeOrdered(3)(Ordering[Int].on(x=>x._2))

Output:

res1: Array[(String, Int)] = Array((Hive,2), (PIG,3), (Hadoop,5))

来源：https://stackoverflow.com/questions/26387753/how-to-reverse-ordering-for-rdd-takeordered