How to get the count for each row before half hour period having the value of 1 in a column?

问题

I have a Data frame as below:

id        time                  type     day
___       _____                 _____    ____
 1   2016-10-12 01:45:01          1       3
 1   2016-10-12 01:48:01          0       3
 1   2016-10-12 01:50:01          1       3
 1   2016-10-12 01:52:01          1       3
 2   2016-10-12 01:53:01          1       3
 2   2016-10-12 02:10:01          1       3
 3   2016-10-12 01:45:01          1       3
 3   2016-10-12 01:48:01          1       3

From this data frame I want to calculate to the occurences of type 1 in that id before half hour for each row. For example if we take the first row

1   2016-10-12 01:45:01          1       3

From this I want to count the type 1 occurences from 2016-10-12 01:45:01 to 2016-10-12 01:15:01 in that id which is eventually 0 since it is the first record.

       id        time                  type     day     count_of_type1
      ___       _____                 _____    ____    ______________   
        1   2016-10-12 01:45:01         1       3              0

If we take the thirdrow

 1   2016-10-12 01:50:01          1       3

From this I want to count the type 1 occurences from 2016-10-12 01:50:01 to 2016-10-12 01:20:01 in that id which is eventually 2.

   id        time                  type     day     count_of_type1
  ___       _____                 _____    ____    ______________   
  1   2016-10-12 01:50:01           1       3              2

I read the data frame as below and also know how to take count but the part i am not sure is how to append the column for each rows individually:

   val df  = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("hdfs:///user/rkr/datafile.csv")

Any help is appreciated.

回答1:

You can use self-join to obtain rows with timestamp-based condition.

val ds = Seq(( 1,   "2016-10-12 01:45:01",          1,       3),
  ( 1,   "2016-10-12 01:48:01",          0,       3),
  ( 1,   "2016-10-12 01:50:01",          1,       3),
  ( 1,   "2016-10-12 01:52:01",          1,       3),
  ( 2,   "2016-10-12 01:53:01",          1,       3),
  ( 2,   "2016-10-12 02:10:01",          1,       3),
  ( 3,   "2016-10-12 01:45:01",          1,       3),
  ( 3,   "2016-10-12 01:45:01",          1,       3)).
  toDF("id", "time", "type", "day")
  .withColumn("timestamp", unix_timestamp($"time", "yyyy-MM-dd HH:mm:ss"))

val happenBeforeHalfHour = ds.as("left").join(ds.as("right"), $"left.id" === $"right.id" && $"right.type" === 1 &&
  $"left.timestamp" > $"right.timestamp" && $"left.timestamp" - $"right.timestamp" <= 1800)
  .select($"left.id", $"left.time", $"left.type", $"left.day",$"left.timestamp")
happenBeforeHalfHour.groupBy("id", "time", "type", "day", "timestamp").count.show(false)


+---+-------------------+----+---+----------+-----+
|id |time               |type|day|timestamp |count|
+---+-------------------+----+---+----------+-----+
|1  |2016-10-12 01:48:01|0   |3  |1476211681|1    |
|2  |2016-10-12 02:10:01|1   |3  |1476213001|1    |
|1  |2016-10-12 01:52:01|1   |3  |1476211921|2    |
|1  |2016-10-12 01:50:01|1   |3  |1476211801|1    |
+---+-------------------+----+---+----------+-----+

来源：https://stackoverflow.com/questions/47170867/how-to-get-the-count-for-each-row-before-half-hour-period-having-the-value-of-1

标签

scala

dataframe

databricks