Append Level to Column Index in python pandas

Question

I have several Dataframes with the same columns that I'd like to merge on their indices only.

print df1

out[]:               Value  ISO
       Id                      
       200001   8432000000  USD
       200230  22588186000  USD
       200247   4633000000  USD
       200291   1188880000  USD
       200418   1779776000  USD

print df2

out[]:               Value  ISO
      Id                       
      200001  1.309168e+11  USD
      200230  5.444096e+10  USD
      200247  9.499602e+09  USD
      200291  2.089603e+09  USD
      200418  3.827251e+09  USD

print df3

out[]:           Value
      Id                       
      200001  3.681908
      200230  3.408507
      200247  4.531866
      200291  0.273029
      200418  3.521822

I could use

pd.concat([df1, df2, df3], axis=1)

and get

out[]:              Value  ISO         Value  ISO     Value
      Id                                                   
      200001   8432000000  USD  1.309168e+11  USD  3.681908
      200230  22588186000  USD  5.444096e+10  USD  3.408507
      200247   4633000000  USD  9.499602e+09  USD  4.531866
      200291   1188880000  USD  2.089603e+09  USD  0.273029
      200418   1779776000  USD  3.827251e+09  USD  3.521822

But I lose the information of where each column came from. I could also do a merge on two dataframes and use the suffixes parameter

print df1.merge(df2, left_index=True, right_index=True, suffixes=('_1', '_2'))

and get

out[]:            Value_1 ISO_1       Value_2 ISO_2
      Id                                           
      200001   8432000000   USD  1.309168e+11   USD
      200230  22588186000   USD  5.444096e+10   USD
      200247   4633000000   USD  9.499602e+09   USD
      200291   1188880000   USD  2.089603e+09   USD
      200418   1779776000   USD  3.827251e+09   USD

I can then daisy chain my merges but the suffixes parameter only applies to columns that share a name. Once I've suffixed the first merge, the names will no longer be in common with the third dataframe.

I figured the solution would be to append a level to the column index of each dataframe with the relevant information necessary to distinguish those columns. Then I could run a pd.concat() and get something that looks like this:

print pd.concat([df1_, df2_, df3_], axis=1)

out[]:Source           df1                df2            df3
                     Value  ISO         Value  ISO     Value
      200001     8.432e+09  USD  1.309168e+11  USD  3.681908
      200230  2.258819e+10  USD  5.444096e+10  USD  3.408507
      200247     4.633e+09  USD  9.499602e+09  USD  4.531866
      200291   1.18888e+09  USD  2.089603e+09  USD  0.273029
      200418  1.779776e+09  USD  3.827251e+09  USD  3.521822

However, in order to get this to happen. I had to abuse the dataframes like so:

df1_ = df1.T
df1_['Source'] = 'df1'
df1_.set_index('Source', append=True, inplace=True)
df1_.index = df1_.index.swaplevel(0, 1)
df1_ = df1_.T

Ultimately, I want a result to look a lot like the last concat statement. Is there a better way to get there? Is there a better way to append a level to the column index?

Thanks, PiR

Answer 1

I you want a MultiIndex, you can do this directly in the concat function to get the same results, like:

pd.concat([df1, df2, df3], axis=1, keys=['df1', 'df2', 'df3'])

or

pd.concat({'df1':df1, 'df2':df2, 'df3':df3}, axis=1)

See also Multi-index dataframe from sequence of dataframes