Google Cloud Composer BigQuery Operator- Get Jobs API HTTPError 404

Question

I am trying to run a BigQueryOperator on GCC. I have already succeeded in running for BigQueryCreateEmptyTableOperator and BigQueryTableDeleteOperator.

Here is my c

Answer 1

Since your BigQuery dataset resides in asia-southeast1, BigQuery created a job in the same location by default, which is asia-southeast1. However, the Airflow in your Composer environment was trying to get the job's status without specifying location field.

Reference: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get

This has been fixed by my PR and it has been merged to master, and it will be released in v2.0.0. However, Composer's latest Airflow version is v1.10.2, so you need a work around to make it work.

To work around this, you can extend the BigQueryCursor and override the run_with_configuration() function with location support. Please refer to: https://github.com/apache/airflow/pull/4695/files#diff-ee06f8fcbc476ea65446a30160c2a2b2R1213 and check how to patch it.

Answer 2

As suggested by @RyanTheCoder, I have created custom plugin with fix. Find it here plugin code. The readme file will show you how to use it.