Problems to create DataFrame from Rows containing Option[T]

Question

I\'m migrating some code from Spark 1.6 to Spark 2.1 and struggling with the following issue:

This worked perfectly in Spark 1.6

import org.apache.spark.         


        

Answer 1

There is actually an JIRA SPARK-19056 about this issue which is not actually one.

So this behavior is intentional.

Allowing Option in Row is never documented and brings a lot of troubles when we apply the encoder framework to all typed operations. Since Spark 2.0, please use Dataset for typed operation/custom objects. e.g.

val ds = Seq(1 -> None, 2 -> Some("str")).toDS
ds.toDF // schema: <_1: int, _2: string>