BigQuery table truncation before streaming not working

Question

We are using BigQuery Python API to run some analyzes. To do so, we created the following adapter:

def stream_data(self, table, data, schema, how=None):
             


        

Answer 1

See Jordan Tigani answer and Sean Chen's comment to https://stackoverflow.com/a/36417177/132438 (both BigQuery engineers).

Summary is:

• When re-creating or truncating a table "You'll need to wait >2 minutes before streaming in order to avoid data being dropped.

So that would explain why you are getting this non-deterministic behavior.