Ever increasing physical memory for a Spark application in YARN

Question

I am running a Spark application in YARN having two executors with Xms/Xmx as 32 GB and spark.yarn.excutor.memoryOverhead as 6 GB.

I am seeing that the app

Answer 1

Finally I was able to get rid of the issue. The issue was that the compressors created in Spark SQL's parquet write path weren't getting recycled and hence, my executors were creating a brand new compressor (from native memory) for every parquet write file and thus exhausting the physical memory limits.

I had opened the following bug in Parquet Jira and have raised the PR for same :-

https://issues.apache.org/jira/browse/PARQUET-353

This fixed the memory issue at my end.

P.S. - You will see this problem only in a Parquet write intensive application.