Bigquery add columns to table schema

Question

I am trying to add new column to BigQuery existing table. I have tried bq command tool and API approach. I get following error when making call to Tables.update().

Answer 1

In my case I was trying to add a REQUIRED field to a template table, and was running into this error. Changing the field to NULLABLE , let me update the table.

Also more recent version on updates for anybody stumbling from Google.

#To create table
bq mk --schema domain:string,pageType:string,source:string -t Project:Dataset.table
#Or using schema file
bq mk --schema SchemaFile.json -t Project:Dataset.table


#SchemaFile.json format
[{                                                                                                                                                                                                                                                
  "mode": "REQUIRED",
  "name": "utcTime",
  "type": "TIMESTAMP"
},    
{
  "mode": "REQUIRED",
  "name": "domain",
  "type": "STRING"
},  
{
  "mode": "NULLABLE",
  "name": "testBucket",
  "type": "STRING"
},  
{
  "mode": "REQUIRED",
  "name": "isMobile",
  "type": "BOOLEAN"                                                                                                                                                                                                                       
},
{
  "mode": "REQUIRED",
  "name": "Category",
  "type": "RECORD",
  "fields": [
    {
      "mode": "NULLABLE",
      "name": "Type",
      "type": "STRING"
     },
     {
       "mode": "REQUIRED",
       "name": "Published",
       "type": "BOOLEAN"
     }
    ]
}]

# TO update
bq update --schema UpdatedSchema.json -t Project:Dataset.table
# Updated Schema contains old and any newly added columns 

Some docs for template tables

Answer 2

You can add Schema to your table through GCP console Easier and Clear:-

Answer 3

Example using the BigQuery Node JS API:

const fieldDefinition = {
    name: 'nestedColumn',
    type: 'RECORD',
    mode: 'REPEATED',
    fields: [
        {name: 'id', type: 'INTEGER', mode: 'NULLABLE'},
        {name: 'amount', type: 'INTEGER', mode: 'NULLABLE'},
    ],
}; 

const table = bigQuery.dataset('dataset1').table('source_table_name');
const metaDataResult = await table.getMetadata();
const metaData = metaDataResult[0];

const fields = metaData.schema.fields;
fields.push(fieldDefinition);

await table.setMetadata({schema: {fields}});

Answer 4

I was stuck trying to add columns to an existing table in BigQuery using the Python client and found this post several times. I'll then let the piece of code that solved it for me, in case someone's having the same problem:

# update table schema
bigquery_client = bigquery.Client()
dataset_ref = bigquery_client.dataset(dataset_id)
table_ref = dataset_ref.table(table_id)
table = bigquery_client.get_table(table_ref)
new_schema = list(table.schema)
new_schema.append(bigquery.SchemaField('LOLWTFMAN','STRING'))
table.schema = new_schema
table = bigquery_client.update_table(table, ['schema'])  # API request

Answer 5

Try this:

bq --format=prettyjson show yourdataset.yourtable > table.json

Edit table.json and remove everything except the inside of "fields" (e.g. keep the [ { "name": "x" ... }, ... ]). Then add your new field to the schema.

Or pipe through jq

bq --format=prettyjson show yourdataset.yourtable | jq .schema.fields > table.json

Then run:

bq update yourdataset.yourtable table.json

You can add --apilog=apilog.txt to the beginning of the command line which will show exactly what is sent / returned from the bigquery server.