Spark parquet nested value flatten

问题

I have parquet file. I loaded using Spark.And one of the value is nested key,value pairs. How do I flatten?

df.printSchema
root
|-- location: string (nullable = true)
|-- properties: string (nullable = true)


texas,{"key":{"key1":"value1","key2":"value2"}}

thanks,

回答1:

You can use explode on your dataframe and pass it a function that reads the JSON column using scala4s. Scala4s has easy parsing API, for your case it will look like:

val list = for {
  JArray(keys) <- parse(json) \\ "key"
  json @ JObject(key) <- keys
  JField("key1", JString(key1)) <- key
  JField("key2", JString(key2)) <- key
} yield {
  Seq(key1, key2)
}

This flattens your dataframe.

If you also want to add column for key, you can use withColumn after explode(keep the key also in the new column).

来源：https://stackoverflow.com/questions/37003083/spark-parquet-nested-value-flatten