How to convert panda df to sparse df

问题

I have a huge sparse dataset in a dataframe and have been using df.to_sparse but it will be deprecated soon so wanted to switch to pd.Series(pd.SparseArray()) but not sure how to do that for an entire dataframe?

My final df is 100K rows and 49K columns so need an automated way.

回答1:

You could try something like this :

dtype = {key: pd.SparseDtype(df.dtypes[key].type, fill_value=df[key].value_counts().argmax()) for key in df.dtypes.keys()}

df = df.astype(dtype)

And then check the density with df.sparse.density.

This will create sparse data for each column, taking most frequent value as filling value.

(not sure if it's the best approach though)

来源：https://stackoverflow.com/questions/59578301/how-to-convert-panda-df-to-sparse-df