Apply filter condition on dataframe created from JSON

Question

I am working on the dataframe created by JSON and then I want to apply the filter condition over the dataframe.

val jsonStr = \"\"\"{ \"metadata\": [{ \"key\": 8         


        

Answer 1

First you should use explode to get an easy-to-work-with dataFrame. Then you can select both key and value of you given input:

val explodedDF = df.withColumn("metadata", explode($"metadata"))
  .select("metadata.key", "metadata.value")

Output:

+-----+-----+
|  key|value|
+-----+-----+
|84896|   54|
| 1234|   12|
+-----+-----+

This way you'll be able to perform your filtering logic as usual:

scala> explodedDF.where("key == 84896").show
+-----+-----+
|  key|value|
+-----+-----+
|84896|   54|
+-----+-----+

You can concatenate your filtering requirements, some examples below:

explodedDF.where("key == 84896 AND value == 54")
explodedDF.where("(key == 84896 AND value == 54) OR key = 1234")