Avoiding Memory Issues For GroupBy on Large Pandas DataFrame

Question

Update:

The pandas df was created like this:

df = pd.read_sql(query, engine)
encoded = pd.get_dummies(df, columns=[\'account\'])
         


        

Answer 1

As an idea i would say, splitting the data column wise let's say four times, and use the id for each subset to perform the operations and then remerge