问题
Usually, the data range I got each day should be over 100,000 records. But when I setup 2 bigquery transfer (one for each owner id), use the same suffix, so the data goes to the same table. the transfer is missing a lot of data.
The transfer status shows successful but somedays I only get a 4000-5000 records instead of 100,000+ records. But somedays I got 100,000 records.
Question 1:
Is that possible bigquery transfer only finishes transferring partial of the data? My understanding is bigquery transfer is "all or none". If it is successful, then all the transfer is done. If it fails, zero records transferred.
Although, I do see in some cases that bigquery failed but status shows complete.
Question 2:
When I set up two transfers, both transfers sending data to a same table, I can see the failure rate increases hugely.
Is it possible the two transfers somehow conflict with each other in the process?
回答1:
OK.
I confirmed.
For example, I have account 1 and account 2.
- Running account 1's transfer, I got 100,000 rows.
- Running account 2's transfer, I got 5,000 rows.
But I set up both account 1's transfer and account 2's transfer to the same table (the same suffix)
- I run account 1 transfer, I got 100,000 rows.
- Then I run account 2 transfer, I lost the the 100,000 rows that just generated by account 1 transfer, but I got 5,000 rows account 2 generated.
Advised by the bigquery expert. Yes. I cannot set up two transfers to dump data to one table. Due to bigquery transfer backfill is "delete and insert".
来源：https://stackoverflow.com/questions/46497533/setup-2-youtube-bigquery-transfer-one-for-each-owner-id-use-the-same-suffix