pandas - How to load data from nested dictionary into dataframe?

问题

I'm tryping to create a dataframe with closingprices for stocks and have found a free api that returns JSON-data in the form of nested dicts with the data, looking like this:

{'name': 'AAPL',
 'history': 
           {'2019-01-04': 
                  {'open': '144.53',
                   'close': '148.26',
                   'high': '148.55',
                   'low': '143.80',
                   'volume': '58607070'},
           '2019-01-03': 
                   {'open': '143.98',
                    'close': '142.19',
                    'high': '145.72',
                    'low': '142.00',
                    'volume': '91312195'},
          '2019-01-02': 
                  {'open': '154.89',
                   'close': '157.92',
                   'high': '158.85',
                   'low': '154.23',
                   'volume': '37039737'
}}}

Since my desired key 'close' is nested in 'history' and each specific date I'm having a hard time extracting it and putting it in a dataframe.

What is the way to go/logic behind this type of situation? I have tried generating list of dates with datetime with no success. Do you have any suggestions or readings?

EDIT: CURRENT CODE, not working obviously

def make_request():
    '''Makes a request to the API that returns a JSON-response '''
    r = requests.get(url)
    sample = json.loads(r.text)
    return sample


def check_keys(data):
    '''Checks the keys in the JSON-response'''
    print(data.keys())


def check_values(data):
    '''Checks the values in the JSON-respose'''
    print(data.values())


def get_values(data):
    '''Gets the date for each day in the sample and stores it in a list'''
     for v in data.get('history'):
        values = v
        return v


def get_closeprice(data, values):
    '''Uses the dates from get_values() to iterate through the sample and get the
    closing price for each date in the sample'''
    for date in values:
        data.get('history').get(values).get('close')
        return value

回答1:

You don't need to know which key is present to access it. You can just iterate over all the keys in the dictionary.

d = <your dict>
retval = {}

for k,v in d['history'].items():
    retval[k] = v['close']

print(retval)

回答2:

If you just want to load it in dataframe:

# h = your dictionary

df = pd.DataFrame.from_dict(data=h['history'],orient='index')

cols = ['close']

df = df[cols]

# Just as an aside Quandl has been very good for free financial data to me. 
#It has a paid side with premium data but I havent used it.

回答3:

If you know your keys, and they don't change, I would use Droids answer. If the keys may change here is a different solution.

d = {'name': 'AAPL',
 'history': 
           {'2019-01-04': 
                  {'open': '144.53',
                   'close': '148.26',
                   'high': '148.55',
                   'low': '143.80',
                   'volume': '58607070'},
           '2019-01-03': 
                   {'open': '143.98',
                    'close': '142.19',
                    'high': '145.72',
                    'low': '142.00',
                    'volume': '91312195'},
          '2019-01-02': 
                  {'open': '154.89',
                   'close': '157.92',
                   'high': '158.85',
                   'low': '154.23',
                   'volume': '37039737'
}}}
def print_nested_dict(nested_dict, name, prior_keys=[]):
    for key, value in nested_dict.items():
        # current_key_path is a list of each key we used to get here
        current_key_path = prior_keys + [key]
        # Convert that key path to a string
        key_path_str = ''.join('[\'{}\']'.format(key) for key in current_key_path)

        # If the value is a dict then recurse
        if isinstance(value, dict):
            print_nested_dict(value, name, current_key_path)
        else:
            # Else lets print the key and value for this value
            # along with where it was found
            print(key, value, '{}{}'.format(name, key_path_str))
print_nested_dict(d, "d")

Output:

name AAPL d['name']
open 144.53 d['history']['2019-01-04']['open']
close 148.26 d['history']['2019-01-04']['close']
high 148.55 d['history']['2019-01-04']['high']
low 143.80 d['history']['2019-01-04']['low']
volume 58607070 d['history']['2019-01-04']['volume']
open 143.98 d['history']['2019-01-03']['open']
close 142.19 d['history']['2019-01-03']['close']
high 145.72 d['history']['2019-01-03']['high']
low 142.00 d['history']['2019-01-03']['low']
volume 91312195 d['history']['2019-01-03']['volume']
open 154.89 d['history']['2019-01-02']['open']
close 157.92 d['history']['2019-01-02']['close']
high 158.85 d['history']['2019-01-02']['high']
low 154.23 d['history']['2019-01-02']['low']
volume 37039737 d['history']['2019-01-02']['volume']

That being said, there may be a more efficient way then this using built in dataframe methods.

回答4:

You can use a regular expression:

import re

if re.match(r"^(\d+-\d+-\d+)$", key):
    # do something with it's values.

You will need to loop over the dictionary yourself, however.

来源：https://stackoverflow.com/questions/55679381/pandas-how-to-load-data-from-nested-dictionary-into-dataframe

标签

python

json

python-3.x

loops

dictionary