sklearn.LabelEncoder with never seen before values

Question

If a sklearn.LabelEncoder has been fitted on a training set, it might break if it encounters new values when used on a test set.

The only solution I c

Answer 1

I ended up switching to Pandas' get_dummies due to this problem of unseen data.

• create the dummies on the training data
  dummy_train = pd.get_dummies(train)
• create the dummies in the new (unseen data)
  dummy_new = pd.get_dummies(new_data)
• re-index the new data to the columns of the training data, filling the missing values with 0
  dummy_new.reindex(columns = dummy_train.columns, fill_value=0)

Effectively any new features which are categorical will not go into the classifier, but I think that should not cause problems as it would not know what to do with them.