Google Cloud Dataflow to BigQuery - UDF - convert unixTimestamp to local time

自闭症网瘾萝莉.ら 提交于 2021-02-20 03:48:05

问题


What is the best way to convert unixTimestamp to local time in the following scenario?

  • I am using Pub/Sub Subscription to BigQuery Template. Dataflow fetches data in json format from PubSub, does the transformation, inserts into BigQuery
  • Preferably, I want to use UDF for data transformation setup.
  • (For simplicity,) Input data includes only unixTimestamp. Example: {"unixTimestamp": "1612325106000"}
  • Bigquery table has 3 columns:
unix_ts:INTEGER, 
iso_dt:DATETIME, 
local_dt:DATETIME 

where unix_ts will keep the input unixTimestamp as it is, iso_dt will keep the UTC datetime, local_dt will keep Europe/Frankfurt datetime. Example: 1612325106000, 2021-02-03T04:05:06, 2021-02-03T05:05:06

I was hoping that the following would work, but did not. unix_ts and iso_dt results as expected but local_dt is the part that leads to a failure. What would be your suggestion? Thanks

function transform(inJson) {
   var input = JSON.parse(inJson);

   var v_date = new Date(parseInt(input.unixTimestamp));

   var v_iso_dt = v_date.toISOString().replace('Z','');
   var v_local_dt = v_date.toLocaleString('en-US', {timeZone: 'Europe/Frankfurt'});

   var output = {
       "unix_ts": input.unixTimestamp || null,
       "iso_dt": v_iso_dt  || null,
       "local_dt": v_local_dt  || null
   };
}

Edit: (Changed) 1612325106 to 1612325106000 in the example.

来源:https://stackoverflow.com/questions/66035874/google-cloud-dataflow-to-bigquery-udf-convert-unixtimestamp-to-local-time

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!