Create Pandas DataFrames from Unique Values in one Column

Question

I have a Pandas dataframe with 1000s of rows. and it has the Names column includes the customer names and their records. I want to create individual dataframes

Answer 1

To create a dataframe for all the unique values in a column, create a dict of dataframes, as follows.

• Creates a dict, where each key is a unique value from the column of choice and the value is a dataframe.
• Access each dataframe as you would a standard dict (e.g. df_names['Name1'])
• .groupby() creates a generator, which can be unpacked.
  • k is the unique values in the column and v is the data associated with each k.

With a for-loop and .groupby:

df_names = dict()
for k, v in df.groupby('customer name'):
    df_names[k] = v

With a Python Dictionary Comprehension

• PEP 274 -- Dict Comprehensions

Using .groupby

df_names = {k: v for (k, v) in df.groupby('customer name')}

• This comes from a conversation with rafaelc, who pointed out that using .groupby is faster than .unique.
  • With 6 unique values in the column, .groupby is faster, at 104 ms compared to 392 ms
  • With 26 unique values in the column, .groupby is faster, at 147 ms compared to 1.53 s.
• Using an a for-loop is slightly faster than a comprehension, particularly for more unique column values or lots of rows (e.g. 10M).

Using .unique:

• Use Boolean indexing to match the unique values in the column of choice.

df_names = {name: df[df['customer name'] == name] for name in df['customer name'].unique()}

Testing

• The following data was used for testing

import pandas as pd
import string
import random

random.seed(365)

# 6 unique values
data = {'class': [random.choice(['1-5', '6-25', '26-100', '100-500', '500-1000', '>1000']) for _ in range(1000000)],
        'treatment': [random.choice(['Yes', 'No']) for _ in range(1000000)]}

# 26 unique values
data = {'class': [random.choice( list(string.ascii_lowercase)) for _ in range(1000000)],
        'treatment': [random.choice(['Yes', 'No']) for _ in range(1000000)]}

df = pd.DataFrame(data)