AWS Glue: crawler misinterprets timestamps as strings. GLUE ETL meant to convert strings to timestamps makes them NULL

Question

I have been playing around with AWS Glue for some quick analytics by following the tutorial here

While I have been able to successfully create crawlers and discover data in Athena, I've had issues with the data types created by the crawler. The date and timestamp data types get read as string data types.

I followed this up by creating an ETL job in GLUE using the data source created by the crawler as the input and a target table in Amazon S3

As part of the mapping transformation, I converted the data types of the date and timestamp as string to timestamp but unfortunately the ETL converted these column types to NULLS. I have contemplated using classifiers with GROK expressions but then decided transform them as part of ETL in GLUE.

The timestamp format is as 1/08/2010 6:15:00 PM

Answer 1

I don't know what the format of timestamp you use therefore I assumed that the format is the following:

2018-05-30T12:22:07.000Z

In this case you can set the serde parameter in the table definition as below:

'timestamp.formats'='yyyy-MM-dd\'T\'HH:mm:ss.SSSZ'

Answer 2

Following are the built-in classifiers for the timestamp data. As you can see the data you have does not match any of them. You will need to create a custom classifier and then add that to a new Crawler (updating existing crawler does not work)

 TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

 TZ (?:[PMCE][SD]T|UTC)

 DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}

 DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}

 DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}

 DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}

 CISCOTIMESTAMP %{MONTH} %{MONTHDAY} %{TIME}