Extending a pandas panel frame along the minor axis

Question

I would like to extend a Panel frame of data along a minor axis in pandas. I start off creating a dic of DataFrames to generate a Panel.

import pandas as pd
import numpy as np
rng = pd.date_range('1/1/2013',periods=100,freq='D')
df1 = pd.DataFrame(np.random.randn(100, 4), index = rng, columns = ['A','B','C','D'])
df2 = pd.DataFrame(np.random.randn(100, 4), index = rng, columns = ['A','B','C','D'])
df3 = pd.DataFrame(np.random.randn(100, 4), index = rng, columns = ['A','B','C','D'])
pf = pd.Panel({'df1':df1,'df2':df2,'df3':df3})

As expected I, find I have a panel with the following dimensions:

Dimensions: 3 (items) x 100 (major_axis) x 4 (minor_axis) Items axis: df1 to df3 Major_axis axis: 2013-01-01 00:00:00 to 2013-04-10 00:00:00 Minor_axis axis: A to D

I would now like to add a new data set to the Minor axis:

pf['df1']['E'] = pd.DataFrame(np.random.randn(100, 1), index = rng)
pf['df2']['E'] = pd.DataFrame(np.random.randn(100, 1), index = rng)
pf['df2']['E'] = pd.DataFrame(np.random.randn(100, 1), index = rng)

I find that after adding this new minor axis the shape of the panel array dimensions has not changed:

shape(pf)

[3,100,4]

I am able to access the data for each of the items in the major_axis:

pf.ix['df1',-10:,'E']

2013-04-01 0.168205 2013-04-02 0.677929 2013-04-03 0.845444 2013-04-04 0.431610 2013-04-05 0.501003 2013-04-06 -0.403605 2013-04-07 -0.185033 2013-04-08 0.270093 2013-04-09 1.569180 2013-04-10 -1.374779 Freq: D, Name: E

But if I extend the slicing to include more than one major axis:

pf.ix[:,:,'E']

Then I encounter an error saying that 'E' is unknown.

Can anyone suggest where I am going wrong or a better way of performing this operation?

Answer 1

This doesn't work right now see this, https://github.com/pydata/pandas/issues/2578 But you can accomplish what you want this way. This is a pretty cheap operation as nothing is copied.

In [18]: x = pf.transpose(2,0,1)

In [19]: x
Out[19]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 4 (items) x 3 (major_axis) x 100 (minor_axis)
Items axis: A to D
Major_axis axis: df1 to df3
Minor_axis axis: 2013-01-01 00:00:00 to 2013-04-10 00:00:00

In [20]: x['E'] = new_df

In [21]: x.transpose(1,2,0)
Out[21]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 3 (items) x 100 (major_axis) x 5 (minor_axis)
Items axis: df1 to df3
Major_axis axis: 2013-01-01 00:00:00 to 2013-04-10 00:00:00
Minor_axis axis: A to E

Answer 2

It seems like the bug was fixed but your question interested me.

Since you can effectively add a slice to a panel on the major and minor axis without transposing, the following 2 lines can avoid scratching your head on the size of the Dataframe...

pf.ix[:,'another major axis',:] = pd.DataFrame(np.random.randn(pf.minor_axis.shape[0],pf.items.shape[0]), index=pf.minor_axis, columns=pf.items)

pf.ix[:, :, 'another minor axis'] = pd.DataFrame(np.random.randn(pf.major_axis.shape[0],pf.items.shape[0]), index=pf.major_axis, columns=pf.items)

I wondered however if there was something simpler ?

Below the piece of code that add slices along various axes.

import pandas as pd
import numpy as np

rng = pd.date_range('25/11/2014', periods=2, freq='D')
df1 = pd.DataFrame(np.random.randn(2, 5), index=rng, columns=['A', 'B', 'C', 'D', 'E'])
df2 = pd.DataFrame(np.random.randn(2, 5), index=rng, columns=['A', 'B', 'C', 'D', 'E'])
df3 = pd.DataFrame(np.random.randn(2, 5), index=rng, columns=['A', 'B', 'C', 'D', 'E'])


pf = pd.Panel({'df1': df1, 'df2': df2, 'df3': df3})

# print("slice before adding df4:\n")
# for i in pf.items:
#     print("{}:\n{}".format(i, pf[i]))

pf['df4'] = pd.DataFrame(np.random.randn(pf.major_axis.shape[0], pf.minor_axis.shape[0]), index=pf.major_axis, columns=pf.minor_axis)
print pf

# print("slice after df4 before transposing 1:\n")
# for i in pf.items:
#     print("{}:\n{}".format(i, pf[i]))

x = pf.transpose(1, 0, 2)

x['new major axis item'] = pd.DataFrame(np.random.randn(pf.items.shape[0], pf.minor_axis.shape[0]), index=pf.items,
                                        columns=pf.minor_axis)

pf = x.transpose(1, 0, 2)

print pf
# print("slice after:\n")
# for i in pf.items:
#     print("{}:\n{}".format(i, pf[i]))

print("success on adding slice on major axis:")
print pf.major_xs(key='new major axis item')
print("trying to add major axis directly")
pf.ix[:,'another major axis',:] = pd.DataFrame(np.random.randn(pf.minor_axis.shape[0],pf.items.shape[0]), index=pf.minor_axis, columns=pf.items)

print pf.major_xs(key='another major axis')
print pf