How to best process large query results written to intermediate table in App Engine

Question

We are running large query jobs where we hit the 128M response size and BigQuery raises the \"Response too large to return. Consider setting allowLargeResults to true in you

Answer 1

Note that BigQuery is able to export data in chunks - and you can request as many chunks as workers you have.

From https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple:

If you ask to export to:

['gs://my-bucket/file-name.json']

you will get an export in one GCS file, as long as it's less than 1GB.

If you ask to export to:

['gs://my-bucket/file-name-*.json']

you will get several files with each having a chunk of the total export. Useful when exporting more than 1GB.

If you ask to export to:

['gs://my-bucket/file-name-1-*.json',
'gs://my-bucket/file-name-2-*.json',
'gs://my-bucket/file-name-3-*.json']

you will get exports optimized for 3 workers. Each of these patterns will receive a series of exported chunks, so each worker can focus on its own chunks.