Apache Spark throws NullPointerException when encountering missing feature

Question

I have a bizarre issue with PySpark when indexing column of strings in features. Here is my tmp.csv file:

x0,x1,x2,x3 
asd2s,1e1e,1.1,0
asd2s,1e1e,0.1,0
,1e3         


        

Answer 1

Well, currently, the only solution is to get rid of NA's like @zero323 proposed or to convert Spark DataFrame to Pandas DataFrame using toPandas() method and impute the data using sklearn Imputer or any custom imputer, e.g., Impute categorical missing values in scikit-learn, then convert Pandas Dataframe back to Spark DataFrame and work with it. Still, the issue remains, I'll try submit a bug report if any. I'm relatively new to Spark, so there is a chance I'm missing something.