Deeply nested JSON response to pandas dataframe

Question

I am new to python/pandas and I am having some issues with converting a nested JSON to pandas dataframe. I am sending a query to a database and getting a JSON string back. <

Answer 1

A hardcoded example...

import pandas as pd

temp = [{
  "ID": "123456",
  "profile": {
    "criteria": [
      {
        "type": "type1",
        "name": "name1",
        "value": "7",
        "properties": []
      },
      {
        "type": "type2",
        "name": "name2",
        "value": "6",
        "properties": [
          {
            "type": "MAX",
            "name": "",
            "value": "100"
          },
          {
            "type": "MIN",
            "name": "",
            "value": "5"
          }
        ]
      },
      {
        "type": "type3",
        "name": "name3",
        "value": "5",
        "properties": []
      }
    ]
  }
},
{
  "ID": "456789",
  "profile": {
    "criteria": [
      {
        "type": "type4",
        "name": "name4",
        "value": "6",
        "properties": []
      }
    ]
  }
}]

cols = ['ID', 'criteria', 'type', 'name', 'value']

rows = []
for data in temp:
    data_id = data['ID']
    criteria = data['profile']['criteria']
    for d in criteria:
        rows.append([data_id, criteria.index(d)+1, *list(d.values())[:-1]])

df = pd.DataFrame(rows, columns=cols)

This is by no means elegant. It is more of a quick and dirty solution, as I don't know how the JSON data is exactly formatted - however, based on what you've provided, my code above will produce the desired DataFrame.

       ID  criteria   type   name value
0  123456         1  type1  name1     7
1  123456         2  type2  name2     6
2  123456         3  type3  name3     5
3  456789         1  type4  name4     6

Additionally, if you need to 'load' the JSON data, you can use the json library like so:

import json

temp = json.loads(json_string)

# Or from a file...
with open('some_json.json') as json_file:
    temp = json.load(json_file)

Please note the difference between json.loads and json.load.