Distinct window functions are not supported: count(distinct _w0# ) windowspecdefinition

  方案一、可以使用approx_count_distinct，但是它是概率统计，非精确统计，其是使用HyperLogLog实现的

import org.apache.spark.sql.functions._

val windowSpec = Window.partitionBy($"site_id",$"pxSessionId").orderBy($"timeMs").rowsBetween(Long.MinValue,Long.MaxValue)

res.withColumn("pageViewNumber",approx_count_distinct(when($"event_name" === "pageview",$"pxSessionId")).over(windowSpec))

  方案二、collect_set与size结合使用（函数大全见这里），示例如下

import org.apache.spark.sql.functions._

val windowSpec = Window.partitionBy($"site_id",$"pxSessionId").orderBy($"timeMs").rowsBetween(Long.MinValue,Long.MaxValue)

res.withColumn("pageViewNumber",size(collect_set(when($"event_name" === "pageview",$"pxSessionId")).over(windowSpec)))

来源：oschina

链接：https://my.oschina.net/u/4352688/blog/4690458