Error ExecutorLostFailure when running a task in Spark

Question

when I am trying to run it on this folder it is throwing me ExecutorLostFailure everytime

Hi I am a beginner in Spark. I am trying to run a job on S

Answer 1

The Most common cause of ExecutorLostFailure as per my understanding is OOM in executor.

In order to resolve the OOM issue, one needs to figure out what exactly is causing it. Simply increasing the default parallelism or increasing the executor memory is not a strategic solution.

If you look at what increasing parallelism do is it tries to create more executors so that each executor can work on less and less data. But if your data is skewed such that the key on which data partitioning happens (for parallelism) has more data, simply increasing parallelism will be of no effect.

Similarly just by increasing Executor memory will be a very inefficient way of handing such a scenario as if only one executor is failing with ExecutorLostFailure , requesting increased memory for all the executors will make your application require much more memory then actually expected.