How to deal with Spark UDF input/output of primitive nullable type

Question

The issues:

1) Spark doesn\'t call UDF if input is column of primitive type that contains null:

inputDF.show()

+-----+
|  x  |
+-----+
         


        

Answer 1

Based on the solution provided at SparkSQL: How to deal with null values in user defined function? by @zero323, an alternative way to achieve the requested result is:

import scala.util.Try
val udfHandlingNulls udf((x: Double) => Try(2.0).toOption)
inputDF.withColumn("y", udfHandlingNulls($"x")).show()