Spark jobs finishes but application takes time to close

Question

Running spark job using scala, as expected all jobs are finishing up on time , but somehow some INFO logs are printed for 20-25 minutes before job stops.

Posting few

Answer 1

I had the same kind of problem when writing files to S3. I use the spark 2.0 version, just to give you a updated code for the verified answer

In Spark 2.0 you can use,

val spark = SparkSession.builder().master("local[*]").appName("App_name").getOrCreate()

spark.conf.set("spark.hadoop.mapred.output.committer.class","com.appsflyer.spark.DirectOutputCommitter")
spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")

This solved my problem of job getting struck