How should you end a Spark job inside an if statement?

Question

What is the recommended way to end a spark job inside a conditional statement?

I am doing validation on my data, and if false, I want to end the spark job gracefull

Answer 1

Once you stop the SparkSession means, your SparkContext is killed on the JVM. sc is no longer active now.

So You can't call any sparkContext related objects/functions for creating RDD/Dataframe or anything else. If you call the same sparksession again in the flow of program.. you should find the above Exception. For example.

`    val rdd=sc.parallelize(Seq(Row("RAMA","DAS","25"),Row("smritu","ranjan","26")))
    val df=spark.createDataFrame(rdd,schema)
    df.show()   //It works fine
    if(df.select("fname").collect()(0).getAs[String]("fname")=="MAA"){
    println("continue")
    }
    else{
    spark.stop()   //stopping sparkSession
    println("inside Stopiing condition")
    }
    println("code continues")
    val rdd1=sc.parallelize(Seq(Row("afdaf","DAS","56"),Row("sadfeafe","adsadaf","27")))
    //Throws Exception...
    val df1=spark.createDataFrame(rdd1,schema)
    df1.show()
`