Exploding nested Struct in Spark dataframe

Question

I\'m working through a Databricks example. The schema for the dataframe looks like:

> parquetDF.printSchema
root
|-- department: struct (nullable = true)
|         


        

Answer 1

You could use something like that:

var explodeDF = explodeDF.withColumn("id", explodeDF("department.id"))
explodeDeptDF = explodeDeptDF.withColumn("name", explodeDeptDF("department.name"))

which you helped me into and these questions:

• Flattening Rows in Spark
• Spark 1.4.1 DataFrame explode list of JSON objects