Apache Spark throws NullPointerException when encountering missing feature

Question

I have a bizarre issue with PySpark when indexing column of strings in features. Here is my tmp.csv file:

x0,x1,x2,x3 
asd2s,1e1e,1.1,0
asd2s,1e1e,0.1,0
,1e3         


        

Answer 1

It looks like module you're using converts empty strings to nulls and it is messing at some point with downstream processing. At first glance it looks like a PySpark bug.

How to fix it? A simple workaround is to either drop nulls before indexing:

features.na.drop()

or replace nulls with some placeholder:

from pyspark.sql.functions import col, when

features.withColumn(
    "x0", when(col("x0").isNull(), "__SOME_PLACEHOLDER__").otherwise(col("x0")))

Also, you could use spark-csv. It is efficient, tested and as a bonus doesn't convert empty strings to nulls.

features = (sqlContext.read
    .format('com.databricks.spark.csv')
    .option("inferSchema", "true")
    .option("header", "true")
    .load("tmp.csv"))

Answer 2

Well, currently, the only solution is to get rid of NA's like @zero323 proposed or to convert Spark DataFrame to Pandas DataFrame using toPandas() method and impute the data using sklearn Imputer or any custom imputer, e.g., Impute categorical missing values in scikit-learn, then convert Pandas Dataframe back to Spark DataFrame and work with it. Still, the issue remains, I'll try submit a bug report if any. I'm relatively new to Spark, so there is a chance I'm missing something.