Parsing Nested JSON into STRUCT type BQ table

问题

I am trying to load following data into BQ to create STRUCT type table. I am uploading the file using Upload option with Auto detect schema on BigQuery web UI.

{"property": [
    {
      "NAME": "65874aca2143",
      "VALUE": [
        {
          "NAME": "time",
          "VALUE": [
            {
              "NAME": "$date",
              "VALUE": "2020-06-16T09:42:49.449Z"
            }
          ]
        },
        {
          "NAME": "type",
          "VALUE": "ACTION"
        },
        {
          "NAME": "id",
          "VALUE": "1234"
        }
      ]
    }
  ]}

But it is giving me below error.

Error while reading data, error message: Failed to parse JSON: No active field found.; ParsedString returned false; Could not parse value; Could not parse value; Could not parse value; Could not parse value; Could not parse value; Parser terminated before end of string.

Is anything wrong with my data or am i violating any BQ rules?

回答1:

Take a look at the documentation about limitations when loading JSON data.

BigQuery uses JSON Lines structure to load JSON data, in which each JSON object must be a separate new line. You need to format your JSON object into a single line:

{ "property":[ { "NAME":"65874aca2143", "VALUE":[ { "NAME":"time", "VALUE":[ { "NAME":"$date", "VALUE":"2020-06-16T09:42:49.449Z" } ] }, { "NAME":"type", "VALUE":"ACTION" }, { "NAME":"id", "VALUE":"1234" } ] } ] }

However, that will still throw an error. BigQuery tries to auto infer the schema from the provided file; arrays/lists get treated as REPEATED RECORD (STRUCT) types. Hence, it's expecting to find the same schema structure for all the elements of the array/list, which is not the case for this "VALUE" array:

"VALUE": [
    //this first element has different schema:
    {
      "NAME": "time",
      "VALUE": [
        {
          "NAME": "$date",
          "VALUE": "2020-06-16T09:42:49.449Z"
        }
      ]
    },
    {
      "NAME": "type",
      "VALUE": "ACTION"
    },
    {
      "NAME": "id",
      "VALUE": "1234"
    }
  ]

If, for example, this was changed into:

"VALUE":[
            {
               "NAME":"time",
               "VALUE": "2020-06-16T09:42:49.449Z"
            },
            {
               "NAME":"type",
               "VALUE":"ACTION"
            },
            {
               "NAME":"id",
               "VALUE":"1234"
            }
         ]

It will work (of course, after formatting it into a single line). So, you also need to restructure your data to have the same schema on all the elements of REPEATED data.

You can also consider the option of storing the entire JSON object into a single STRING column, and then querying its elements using BigQuery JSON functions.

来源：https://stackoverflow.com/questions/63792118/parsing-nested-json-into-struct-type-bq-table

标签

json

google-cloud-platform

google-bigquery