Spark job fails: storage.DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file

Question

I have a Spark (1.4.1) application, running on Yarn, that fails with the following executor log entry:

16/07/21 23:09:08 ERROR executor.CoarseGrainedExecutor         


        

Answer 1

You can try to improve spark.yarn.executor.memoryOverhead.