问题
I have around 1500 jobs to be implemented using Dataflow. Those jobs will be scheduled on daily basis. We may get to use huge number of DML statements using Bigquery Client library within our jobs. Listing down my concerns regarding Bigquery quotas and limits.
Reference: https://cloud.google.com/bigquery/quotas
Please confirm that do we need to take the daily usage limits of Bigquery into consideration in any of the below mentioned scenarios.
- If we implement data inserts using
BigqueryIO.write() - If we use DML statements (Update/Delete) using Bigquery Client Library within the Dataflow job
Please suggest.
回答1:
You absolutely do need to take BigQuery quotas and limits into consideration - even when hooking into it from Dataflow.
Dataflow is just calling the BigQuery API on your behalf. Therefore, all quotas and limits still apply as if you were calling it directly yourself.
来源：https://stackoverflow.com/questions/47537897/does-dataflow-jobs-hit-any-bigquery-quotas-and-limits