How to encode a pandas.DataFrame column containing lists using Sklearn.preprocessing

问题

I have a pandas df and some of the columns are lists with data in them and I would like to encode the labels within the lists.

I get this error: ValueError: Expected 2D array, got 1D array instead:

from sklearn.preprocessing import OneHotEncoder
mins = pd.read_csv('recipes.csv')

enc = OneHotEncoder(handle_unknown='ignore')

X = mins['Ingredients']

'''
[[lettuce, tomatoes, ginger, vodka, tomatoes]
[lettuce, tomatoes, flour, vodka, tomatoes]
...
[flour, tomatoes, vodka, vodka, mustard]]
'''

enc.fit(X)

I hope to get a a column of lists that would have the correctly encoded information

[[lettuce, tomatoes, ginger, vodka, tomatoes]
[lettuce, tomatoes, flour, vodka, tomatoes]
...
[flour, tomatoes, vodka, vodka, mustard]

[[0, 1, 2, 3, 1]
[0, 1, 4, 3, 1]
...
[4, 1, 3, 3, 9]]

回答1:

To label encode list of lists in a DataFrame series, we first train the encoder with the unique text labels and then use apply to transform each text label to the trained integer label in the list of lists. Here is an example:

In [2]: import pandas as pd

In [3]: from sklearn import preprocessing

In [4]: df = pd.DataFrame({"Day":["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"], "Veggies&Drinks":[["lettuce"
   ...: , "tomatoes", "ginger", "vodka", "tomatoes"], ["flour", "vodka", "mustard", "lettuce", "ginger"], ["mustard", "
   ...: tomatoes", "ginger", "vodka", "tomatoes"], ["ginger", "vodka", "lettuce", "tomatoes", "flour"], ["mustard", "le
   ...: ttuce", "ginger", "flour", "tomatoes"]]})

In [5]: df
Out[5]:
         Day                                Veggies&Drinks
0     Monday  [lettuce, tomatoes, ginger, vodka, tomatoes]
1    Tuesday      [flour, vodka, mustard, lettuce, ginger]
2  Wednesday  [mustard, tomatoes, ginger, vodka, tomatoes]
3   Thursday     [ginger, vodka, lettuce, tomatoes, flour]
4     Friday   [mustard, lettuce, ginger, flour, tomatoes]

In [9]: label_encoder = preprocessing.LabelEncoder()

In [19]: list_of_veggies_drinks = ["lettuce","tomatoes","ginger","vodka","flour","mustard"]

In [20]: label_encoder.fit(list_of_veggies_drinks)
Out[20]: LabelEncoder()

In [21]: integer_encoded = df["Veggies&Drinks"].apply(lambda x:label_encoder.transform(x))

In [22]: integer_encoded
Out[22]:
0    [2, 4, 1, 5, 4]
1    [0, 5, 3, 2, 1]
2    [3, 4, 1, 5, 4]
3    [1, 5, 2, 4, 0]
4    [3, 2, 1, 0, 4]
Name: Veggies&Drinks, dtype: object

In [23]: df["Encoded"] = integer_encoded

In [24]: df
Out[24]:
         Day                                Veggies&Drinks          Encoded
0     Monday  [lettuce, tomatoes, ginger, vodka, tomatoes]  [2, 4, 1, 5, 4]
1    Tuesday      [flour, vodka, mustard, lettuce, ginger]  [0, 5, 3, 2, 1]
2  Wednesday  [mustard, tomatoes, ginger, vodka, tomatoes]  [3, 4, 1, 5, 4]
3   Thursday     [ginger, vodka, lettuce, tomatoes, flour]  [1, 5, 2, 4, 0]
4     Friday   [mustard, lettuce, ginger, flour, tomatoes]  [3, 2, 1, 0, 4]

回答2:

Since you want to apply it directly to the pandas.DataFrame:

from sklearn.preprocessing import LabelEncoder

# Get a flat list with all the ingredients
all_ingr = mins.Ingredients.apply(pd.Series).stack().values

enc = LabelEncoder()
enc.fit(all_ingr)

mins['Ingredients_enc'] = mins.Ingredients.apply(enc.transform)

来源：https://stackoverflow.com/questions/56721250/how-to-encode-a-pandas-dataframe-column-containing-lists-using-sklearn-preproces

标签

python

python-3.x

dataframe

scikit-learn

sklearn-pandas