Spark Scala: DateDiff of two columns by hour or minute

Question

I have two timestamp columns in a dataframe that I\'d like to get the minute difference of, or alternatively, the hour difference of. Currently I\'m able to get the day diff

Answer 1

The answer given by Daniel de Paula works, but that solution does not work in the case where the difference is needed for every row in your table. Here is a solution that will do that for each row:

import org.apache.spark.sql.functions

val df2 = df1.selectExpr("(unix_timestamp(ts1) - unix_timestamp(ts2))/3600")

This first converts the data in the columns to a unix timestamp in seconds, subtracts them and then converts the difference to hours.

A useful list of functions can be found at: http://spark.apache.org/docs/latest/api/scala/#org.apache.spark.sql.functions$