Spark DataFrame: Computing row-wise mean (or any aggregate operation)

后端 未结 2 1410
逝去的感伤
逝去的感伤 2020-11-27 06:44

I have a Spark DataFrame loaded up in memory, and I want to take the mean (or any aggregate operation) over the columns. How would I do that? (In numpy, this is

2条回答
  •  暗喜
    暗喜 (楼主)
    2020-11-27 07:32

    in Scala something like this would do it

    val cols = Seq("US","UK","Can")
    f.map(r => (r.getAs[Int]("id"),r.getValuesMap(cols).values.fold(0.0)(_+_)/cols.length)).toDF
    

提交回复
热议问题