How to convert custom datetime format to timestamp?

Question

Any Idea why I am getting the result below?

scala> val b = to_timestamp($\"DATETIME\", \"ddMMMYYYY:HH:mm:ss\")
b: org.apache.spark.sql.Column = to_timesta         


        

Answer 1

Try this UDF:

val changeDtFmt = udf{(cFormat: String,
                         rFormat: String,
                         date: String) => {
  val formatterOld = new SimpleDateFormat(cFormat)
  val formatterNew = new SimpleDateFormat(rFormat)
  formatterNew.format(formatterOld.parse(date))
}}

sourceRawData.
  withColumn("ts", 
    changeDtFmt(lit("ddMMMyyyy:HH:mm:ss"), lit("yyyy-MM-dd HH:mm:ss"), $"DATETIME")).
  show(6,false)