Using Pandas groupby to calculate many slopes

Question

Some illustrative data in a DataFrame (MultiIndex) format:

|entity| year |value| +------+------+-----+ | a | 1999 | 2 | | | 2004 | 5 | | b | 20

Answer 1

You can do this via the iterator ability of the group by object. It seems easier to do it by dropping the current index and then specifying the group by 'entity'.

A list comprehension is then an easy way to quickly work through all the groups in the iterator. Or use a dict comprehension to get the labels in the same place (you can then stick the dict into a pd.DataFrame easily).

import pandas as pd
import scipy.stats

#This is your data
test = pd.DataFrame({'entity':['a','a','b','b','b'],'year':[1999,2004,2003,2007,2014],'value':[2,5,3,2,7]}).set_index(['entity','year'])

#This creates the groups
groupby = test.reset_index().groupby(['entity'])

#Process groups by list comprehension
slopes = [scipy.stats.linregress(group.year, group.value)[0] for name, group in groupby]
#Process groups by dict comprehension
slopes = {name:[scipy.stats.linregress(group.year, group.value)[0]] for name, group in groupby}