How to create encoder for Option type constructor, e.g. Option[Int]?

Question

Is it possible to use Option[_] member in a case class used with Dataset API? eg. Option[Int]

I tried to find an example but could not find

Answer 1

We only define implicits for a subset of the types we support in SQLImplicits. We should probably consider adding Option[T] for common T as the internal infrastructure does understand Option. You can workaround this by either creating a case class, using a Tuple or constructing the required implicit yourself (though this is using and internal API so may break in future releases).

implicit def optionalInt: org.apache.spark.sql.Encoder[Option[Int]] = org.apache.spark.sql.catalyst.encoders.ExpressionEncoder()

val ds = Seq(Some(1), None).toDS()