how to convert json string to dataframe on spark

Question

I want to convert string variable below to dataframe on spark.

val jsonStr = \"{ \"metadata\": { \"key\": 84896, \"value\": 54 }}\"

I know

Answer 1

Here is an example how to convert Json string to Dataframe in Java (Spark 2.2+):

String str1 = "{\"_id\":\"123\",\"ITEM\":\"Item 1\",\"CUSTOMER\":\"Billy\",\"AMOUNT\":285.2}";
String str2 = "{\"_id\":\"124\",\"ITEM\":\"Item 2\",\"CUSTOMER\":\"Sam\",\"AMOUNT\":245.85}";
List jsonList = new ArrayList<>();
jsonList.add(str1);
jsonList.add(str2);
SparkContext sparkContext = new SparkContext(new SparkConf()
        .setAppName("myApp").setMaster("local"));
JavaSparkContext javaSparkContext = new JavaSparkContext(sparkContext);
SQLContext sqlContext = new SQLContext(sparkContext);
JavaRDD javaRdd = javaSparkContext.parallelize(jsonList);
Dataset data = sqlContext.read().json(javaRdd);
data.show();

Here is the result:

+------+--------+------+---+
|AMOUNT|CUSTOMER|  ITEM|_id|
+------+--------+------+---+
| 285.2|   Billy|Item 1|123|
|245.85|     Sam|Item 2|124|
+------+--------+------+---+