Apache Spark does not delete temporary directories

Question

After a spark program completes, there are 3 temporary directories remain in the temp directory. The directory names are like this: spark-2e389487-40cc-4a82-a5c7-353c0feefbb

Answer 1

for spark.local.dir, it will only move spark temp files, but the snappy-xxx file will still exists in /tmp dir. Though didn't find way to make spark automatically clear it, but you can set JAVA option:

JVM_EXTRA_OPTS=" -Dorg.xerial.snappy.tempdir=~/some-other-tmp-dir"

to make it move to another dir, as most system has small /tmp size.