Read values from config in Scala

Question

In Scala, if I have the following config:

id = 777
username = stephan
password = DG#%T@RH

The idea is to open a file, transform it into a s

Answer 1

If your Spark version is less than 2.2 then first convert your JSON file content in to JSON String i.e. convert your file content to single string and load it to HDFS location.

Sample JSON:

{ 
  "planet" : "Earth",
  "continent" : "Antarctica"
}

Convert to:

{ "planet" : "Earth", "continent" : "Antarctica"}

Next, to access data create a data frame:

val dataDF = spark.read.format("json").load("")
val planet = dataDF.select("planet").collect(0).mkString("")

Hope this helps Spark 2.1 and less users.