Creating datetime from string column in Pyspark [duplicate]

问题

Suppose I have the following datetime column as shown below. I want to convert the column in string to a datetime type so I can extract months, days and year and such.

+---+------------+
|agg|    datetime|
+---+------------+
|  A|1/2/17 12:00|
|  B|        null|
|  C|1/4/17 15:00|
+---+------------+

I have tried the following code below, but the returning values in the datetime column are nulls, which I don't understand the reason for this at the moment.

df.select(df['datetime'].cast(DateType())).show()

And I have also tried this code:

df = df.withColumn('datetime2', from_unixtime(unix_timestamp(df['datetime']), 'dd/MM/yy HH:mm'))

But, they both produce this dataframe:

+---+------------+---------+
|agg|    datetime|datetime2|
+---+------------+---------+
|  A|1/2/17 12:00|     null|
|  B|       null |     null|
|  C|1/4/17 12:00|     null|

I have already read and tried the solution as specified in this post to no avail: PySpark dataframe convert unusual string format to Timestamp

回答1:

// imports
import org.apache.spark.sql.functions.{dayofmonth,from_unixtime,month, unix_timestamp, year}

// Not sure if the datatype of the column is datetime or string
// I assume the column might be string, do the conversion
// created column datetime2 which is time stamp
val df2 = df.withColumn("datetime2", from_unixtime(unix_timestamp(df("datetime"), "dd/MM/yy HH:mm")))

+---+------------+-------------------+
|agg|    datetime|          datetime2|
+---+------------+-------------------+
|  A|1/2/17 12:00|2017-02-01 12:00:00|
|  B|        null|               null|
|  C|1/4/17 15:00|2017-04-01 15:00:00|
+---+------------+-------------------+


//extract month, year, day information
val df3 = df2.withColumn("month", month(df2("datetime2")))
  .withColumn("year", year(df2("datetime2")))
  .withColumn("day", dayofmonth(df2("datetime2")))
+---+------------+-------------------+-----+----+----+
|agg|    datetime|          datetime2|month|year| day|
+---+------------+-------------------+-----+----+----+
|  A|1/2/17 12:00|2017-02-01 12:00:00|    2|2017|   1|
|  B|        null|               null| null|null|null|
|  C|1/4/17 15:00|2017-04-01 15:00:00|    4|2017|   1|
+---+------------+-------------------+-----+----+----+

Thanks

来源：https://stackoverflow.com/questions/46779993/creating-datetime-from-string-column-in-pyspark

标签

python

apache-spark

pyspark