How to download all data in a Google BigQuery dataset?

Question

Is there an easy way to directly download all the data contained in a certain dataset on Google BigQuery? I\'m actually downloading \"as csv\", making one query after anothe

Answer 1

Detailed step-by-step to download large query output

1. enable billing

   You have to give your credit card number to Google to export the output, and you might have to pay.

   But the free quota (1TB of processed data) should suffice for many hobby projects.

2. create a project

3. associate billing to a project

4. do your query

5. create a new dataset

6. click "Show options" and enable "Allow Large Results" if the output is very large

7. export the query result to a table in the dataset

8. create a bucket on Cloud Storage.

9. export the table to the created bucked on Cloud Storage.

   • make sure to click GZIP compression

   • use a name like /prefix.gz.

     If the output is very large, the file name must have an asterisk * and the output will be split into multiple files.

10. download the table from cloud storage to your computer.

    Does not seem possible to download multiple files from the web interface if the large file got split up, but you could install gsutil and run:

    gsutil -m cp -r 'gs:///prefix_*' .
    

    See also: Download files and folders from Google Storage bucket to a local folder

    There is a gsutil in Ubuntu 16.04 but it is an unrelated package.

    You must install and setup as documented at: https://cloud.google.com/storage/docs/gsutil

11. unzip locally:

    for f in *.gz; do gunzip "$f"; done
    

Here is a sample project I needed this for which motivated this answer.