How to convert a simple DataFrame to a DataSet Spark Scala with case class?

Question

I am trying to convert a simple DataFrame to a DataSet from the example in Spark: https://spark.apache.org/docs/latest/sql-programming-guide.html

case class         


        

Answer 1

This is how you create dataset from case class

case class Person(name: String, age: Long) 

Keep the case class outside of the class that has below code

val primitiveDS = Seq(1,2,3).toDS()
val augmentedDS = primitiveDS.map(i => Person("var_" + i.toString, (i + 1).toLong))
augmentedDS.show()

augmentedDS.as[Person].show()

Hope this helped