Can we create dynamic partition in Big Query as in Hive?

流过昼夜 提交于 2020-01-06 04:51:11

问题


I wanted all my data stored in Buckets (ex: /bucket/project/odate/odate_cust.txt) to be loaded in the table in Big Query which is DAY partitioned? Do I need to import one by one file and load or I can load directly into multiple partition.

**bq mk --time_partitioning_type=DAY market.cust custid:string,grp:integer,odate:string**

回答1:


Currently, you would need to specify the partition (using the $ decorator syntax) for each load to put it in the corresponding partition. Otherwise, BigQuery will use the UTC time of the load job to select the partition. There's an upcoming feature that will allow partitioning by your own field (I assume you have either a TIMESTAMP or DATE field in your files that you can partition by). However, they have not rolled it out yet (it's going alpha soon). You can track its progress here.

If you are in a hurry, then there's a few workarounds (e.g. loading it all into a non-partitioned table, and then using SQL or Cloud Dataflow to partition it afterwards). Have a look here.

Finally, if your file names contain a date/day for the partition, then it would be easy enough to script something yourself that looks at the name of the file and runs multiple load jobs and stuffs the data into the corresponding partition in the table.



来源:https://stackoverflow.com/questions/46546718/can-we-create-dynamic-partition-in-big-query-as-in-hive

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!