Window spec/function perform optimum way or any alternative should be preferred

问题

I am using spark-sql-2.4.1v. In my use-case using window spec/feature to find the latest records using the rank() function. I have to find a latest record on certain partitioning keys and order by insertion_date.

It is extremely slow. Can this window-spec rank() can be used in production-grade code? Or is there any alternative way recommended? Specifically to improve performance.

Please advice.

I'm currently using the below code:

Dataset<Row> data =  sqlContext.read.format("org.apache.spark.sql.cassandra")
     .option("spark.cassandra.connection.host", hosts)
     .options(Map( "table" -> "source_table", "keyspace" -> "calc")).load()
     .where(col("error").equalTo(lit(200)))
     .filter(col("insert_date").gt(lit("2015-01-01")))
     .filter(col("insert_date").lt(lit("2016-01-01")))
     .where(col("id").equalTo(lit(mId)))

Explained plan

== Physical Plan ==
*(1) Project [cast(cast(unix_timestamp(insert_date#199, yyyy-MM-dd, Some(America/New_York)) as timestamp) as date) AS insert_date#500, 3301 AS id#399, create_date#201, company_id#202, ... 76 more fields]
+- *(1) Filter (((((((cast(error#263 as int) = 200) && (cast(insert_date#199 as string) >= 2018-01-01)) && (cast(insert_date#199 as string) <= 2018-01-01)) && isnotnull(id#200)) && isnotnull(insert_date#199)) && isnotnull(error#263)) && (id#200 = 3301))
   +- *(1) Scan org.apache.spark.sql.cassandra.CassandraSourceRelation@261d7ee2 [... 76 more fields] PushedFilters: [IsNotNull(id), IsNotNull(insert_date), IsNotNull(error), EqualTo(id,3301)], ReadSchema: struct<...

To read read

+-----------+------+
|partitionId| count|
+-----------+------+
|       1829|    29|
|       1959| 16684|
|        496|  3795|
|       2659|   524|
|       1591|    87|
|       2811|  2436|
|       2235|   620|
|       2563|   252|
|       1721|    12|
|        737|  1695|
|        858|   182|
|       2580| 73106|
|       3179|   694|
|       1460|    13|
|       1990|    66|
|       1522|   951|
|        540|    11|
|       1127|823084|
|       2999|     9|
|        623|  6629|
+-----------+------+
only showing top 20 rows

20/05/19 06:32:53 WARN ReaderCassandra:  Processed  started : 1589864993863  ended : 1589869973496 timetaken : 4979 s



val ws = Window.partitionBy("id").orderBy(desc("insert_date"),desc("update_date"));

Dataset<Row> ranked_data =  data.withColumn("rank",rank().over(ws))
    .where($"rank".===(lit(1)))
    .select("*")

来源：https://stackoverflow.com/questions/61870530/window-spec-function-perform-optimum-way-or-any-alternative-should-be-preferred

标签

scala

apache-spark

cassandra

apache-spark-sql