Why do columns change to nullable in Apache Spark SQL?

Question

Why is nullable = true used after some functions are executed even though there are no NaN values in the DataFrame.

Answer 1

You could change schema of dataframe very quickly as well. something like this would do the job -

def setNullableStateForAllColumns( df: DataFrame, columnMap: Map[String, Boolean]) : DataFrame = {
    import org.apache.spark.sql.types.{StructField, StructType}
    // get schema
    val schema = df.schema
    val newSchema = StructType(schema.map {
    case StructField( c, d, n, m) =>
      StructField( c, d, columnMap.getOrElse(c, default = n), m)
    })
    // apply new schema
    df.sqlContext.createDataFrame( df.rdd, newSchema )
}