How to concatenate multiple pandas.DataFrames without running into MemoryError

Question

I have three DataFrames that I\'m trying to concatenate.

concat_df = pd.concat([df1, df2, df3])

This results in a MemoryError. How can I re

Answer 1

I've had a similar performance issues while trying to concatenate a large number of DataFrames to a 'growing' DataFrame.

My workaround was appending all sub DataFrames to a list, and then concatenating the list of DataFrames once processing of the sub DataFrames has been completed. This will bring the runtime to almost half.