Impala timestamps don't match Hive - a timezone issue?

Question

I have some eventlog data in HDFS that, in its raw format, looks like this:

2015-11-05 19:36:25.764 INFO    [...etc...]

An external table p

Answer 1

On a related note, as of Hive v1.2, you can also disable the timezone conversion behaviour with this flag:

hive.parquet.timestamp.skip.conversion

"Current Hive implementation of parquet stores timestamps to UTC, this flag allows skipping of the conversion on reading parquet files from other tools."

This was added in as part of https://issues.apache.org/jira/browse/HIVE-9482

Lastly, not timezone exactly, but for compatibility of Spark (v1.3 and up) and Impala on Parquet files, there's this flag:

spark.sql.parquet.int96AsTimestamp

https://spark.apache.org/docs/1.3.1/sql-programming-guide.html#configuration

Other: https://issues.apache.org/jira/browse/SPARK-12297