How to calculate Median in spark sqlContext for column of data type double

Question

I have given the sample table. I want to get the median from \"value\" column for each group \"source\" column. Where source column is of String DataType value column is of

Answer 1

Here is how it can be done using Spark Scala dataframe functions. This is based on how Imputer is implemented for median strategy in Spark>=2.2 - https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/Imputer.scala -

  df.select(colName)
        .stat
        .approxQuantile(colName, Array(0.5), 0.001) //median
        .head