问题
I have a set of categorical columns (strings), that I'm parsing and converting into Vectors of features to pass to a mllib classifier (random forest).
In my input data, some columns have null values. Say, in one of those columns, I have p values + a null value : How should I build my feature Vectors, and the categoricalFeaturesInfo map of the classifier ?
- option 1 : I tell p values in categoricalFeaturesInfo, and I use Double.NaN in my input Vectors ?
- side question : How NaNs are handled by classifiers ?
- option 2 : I consider nulls as a value, so I tell (p+1) values in categoricalFeaturesInfo, and I map nulls to some double ?
Thanks for your help.
(PS : I know the the new dataframe + pipeline + vectorindexer API, but for reasons it doesn't fit well my need, so I need to do that by myself)
回答1:
Looks like option 2 is the one. If null for you is actually another level of your categorical feature, just map it into some value. Note that the categorical feature levels should be mapped into 0,1,2 .... etc before you can properly use them, see here:
response in stackoverflow
So nulls will be mapped into one of these numbers.
来源：https://stackoverflow.com/questions/32999099/handle-null-nan-values-in-spark-mllib-classifier