Google BigQuery: Slow streaming inserts performance

Question

We are using BigQuery as event logging platform.

The problem we faced was very slow insertAll post requests (https://cloud.google.com/bigquery/docs/reference/v2/tabl

Answer 1

To my experience any request to bigquery will take long. We've tried using it as a database for performance data but eventually are moving out due to slow response times. As far as I can see. BQ is built for handling big requests within a 1 - 10 second response time. These are the requests BQ categorizes as interactive. BQ doesn't get faster by doing less. We stream quite some records to BQ but always make sure we batch them up (per table). And run all requests asynchronously (or if you have to in another theat).

PS. I can confirm what Pentium10 sais about faillures in BQ. Make sure you retry the stuff that fails and if it fails again log it to file for retrying it another time.