Apache Spark does not delete temporary directories

Question

After a spark program completes, there are 3 temporary directories remain in the temp directory. The directory names are like this: spark-2e389487-40cc-4a82-a5c7-353c0feefbb

Answer 1

I assume you are using the "local" mode only for testing purposes. I solved this issue by creating a custom temp folder before running a test and then I delete it manually (in my case I use local mode in JUnit so the temp folder is deleted automatically).

You can change the path to the temp folder for Spark by spark.local.dir property.

SparkConf conf = new SparkConf().setMaster("local")
                                .setAppName("test")
                                .set("spark.local.dir", "/tmp/spark-temp");

After the test is completed I would delete the /tmp/spark-temp folder manually.