How do I turn a dataframe into a series of lists?

问题

I have had to do this several times and I'm always frustrated. I have a dataframe:

df = pd.DataFrame([[1, 2, 3, 4], [5, 6, 7, 8]], ['a', 'b'], ['A', 'B', 'C', 'D'])

print df

   A  B  C  D
a  1  2  3  4
b  5  6  7  8

I want to turn df into:

pd.Series([[1, 2, 3, 4], [5, 6, 7, 8]], ['a', 'b'])

a    [1, 2, 3, 4]
b    [5, 6, 7, 8]
dtype: object

I've tried

df.apply(list, axis=1)

Which just gets me back the same df

What is a convenient/effective way to do this?

回答1:

You can first convert DataFrame to numpy array by values, then convert to list and last create new Series with index from df if need faster solution:

print (pd.Series(df.values.tolist(), index=df.index))
a    [1, 2, 3, 4]
b    [5, 6, 7, 8]
dtype: object

Timings with small DataFrame:

In [76]: %timeit (pd.Series(df.values.tolist(), index=df.index))
1000 loops, best of 3: 295 µs per loop

In [77]: %timeit pd.Series(df.T.to_dict('list'))
1000 loops, best of 3: 685 µs per loop

In [78]: %timeit df.T.apply(tuple).apply(list)
1000 loops, best of 3: 958 µs per loop

and with large:

from string import ascii_letters
letters = list(ascii_letters)
df = pd.DataFrame(np.random.choice(range(10), (52 ** 2, 52)),
                  pd.MultiIndex.from_product([letters, letters]),
                  letters)

In [71]: %timeit (pd.Series(df.values.tolist(), index=df.index))
100 loops, best of 3: 2.06 ms per loop

In [72]: %timeit pd.Series(df.T.to_dict('list'))
1 loop, best of 3: 203 ms per loop

In [73]: %timeit df.T.apply(tuple).apply(list)
1 loop, best of 3: 506 ms per loop

回答2:

pandas tries really hard to make making dataframes convenient. As such, it interprets lists and arrays as things you'd want to split into columns. I'm not going to complain, this is almost always helpful.

I've done this one of two ways.

Option 1:

# Only works with a non MultiIndex
# and its slow, so don't use it
df.T.apply(tuple).apply(list)

Option 2:

pd.Series(df.T.to_dict('list'))

Both give you:

a    [1, 2, 3, 4]
b    [5, 6, 7, 8]
dtype: object

However Option 2 scales better.

Timing

given df

much larger df

from string import ascii_letters
letters = list(ascii_letters)
df = pd.DataFrame(np.random.choice(range(10), (52 ** 2, 52)),
                  pd.MultiIndex.from_product([letters, letters]),
                  letters)

Results for df.T.apply(tuple).apply(list) are erroneous because that solution doesn't work over a MultiIndex.