Unable to save DataFrame to HDF5 (“object header message is too large”)

Question

I have a DataFrame in Pandas:

In [7]: my_df
Out[7]: 

Int64Index: 34 entries, 0 to 0
Columns: 2661 entries, airp         


        

Answer 1

HDF5 has a header limit of 64kb for all metadata of the columns. This include name, types, etc. When you go about roughly 2000 columns, you will run out of space to store all the metadata. This is a fundamental limitation of pytables. I don't think they will make workarounds on their side any time soon. You will either have to split the table up or choose another storage format.