Spark SQL Row_number() PartitionBy Sort Desc

Question

I\'ve successfully create a row_number() partitionBy by in Spark using Window, but would like to sort this by descending, instead of the default ascend

Answer 1

Or you can use the SQL code in Spark-SQL:

from pyspark.sql import SparkSession

spark = SparkSession\
    .builder\
    .master('local[*]')\
    .appName('Test')\
    .getOrCreate()

spark.sql("""
    select driver
        ,also_item
        ,unit_count
        ,ROW_NUMBER() OVER (PARTITION BY driver ORDER BY unit_count DESC) AS rowNum
    from data_cooccur
""").show()