How to convert unix timestamp to date in Spark

问题:

I have a data frame with a column of unix timestamp(eg.1435655706000), and I want to convert it to data with format 'yyyy-MM-DD', I've tried nscala-time but it doesn't work.

val time_col = sqlc.sql("select ts from mr").map(_(0).toString.toDateTime) time_col.collect().foreach(println) 

and I got error: java.lang.IllegalArgumentException: Invalid format: "1435655706000" is malformed at "6000"

回答1:

Since spark1.5 , there is a builtin UDF for doing that.

val df = sqlContext.sql("select from_unixtime(ts,'YYYY-MM-dd') as `ts` from mr") 

Please check Spark 1.5.2 API Doc for more info.

回答2:

import org.joda.time.{DateTimeZone} import org.joda.time.format.DateTimeFormat 

You need to import the following libraries.

val stri = new DateTime(timeInMillisec).toDateTime.toString("yyyy/MM/dd") 

Or adjusting to your case :

 val time_col = sqlContext.sql("select ts from mr")                      .map(line => new DateTime(line(0).toInt).toDateTime.toString("yyyy/MM/dd")) 

There could be another way :

  import com.github.nscala_time.time.Imports._    val date = (new DateTime() + ((threshold.toDouble)/1000).toInt.seconds )              .toString("yyyy/MM/dd") 

Hope this helps :)

回答3:

Here it is using Scala DataFrame functions: from_unix_time and to_date

// NOTE: divide by 1000 required if milliseconds // e.g. 1446846655609 -> 2015-11-06 21:50:55 -> 2015-11-06  mr.select(to_date(from_unixtime($"ts" / 1000)))  

回答4:

I have solved this issue using the joda-time library by mapping on the DataFrame and converting the DateTime into a String :

import org.joda.time._ val time_col = sqlContext.sql("select ts from mr")                          .map(line => new DateTime(line(0)).toString("yyyy-MM-dd")) 

回答5:

You needn't convert to String before applying toDataTime with nscala_time

import com.github.nscala_time.time.Imports._

scala> 1435655706000L.toDateTime res4: org.joda.time.DateTime = 2015-06-30T09:15:06.000Z 

`

文章来源: How to convert unix timestamp to date in Spark