Impala timestamps don't match Hive - a timezone issue?

Question

I have some eventlog data in HDFS that, in its raw format, looks like this:

2015-11-05 19:36:25.764 INFO    [...etc...]

An external table p

Answer 1

As mentioned in https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_timestamp.html

You can use ----use_local_tz_for_unix_timestamp_conversions=true and --convert_legacy_hive_parquet_utc_timestamps=true to match Hive results.

The first one ensures it converts to local timezone when you use any datetime function. You can set it as Impala Daemon startup options as mentioned in this document.

https://docs.cloudera.com/documentation/enterprise/5-6-x/topics/impala_config_options.html