Why does Spark job fail with “too many open files”?

Question

I get \"too many open files\" during the shuffle phase of my Spark job. Why is my job opening so many files? What steps can I take to try to make my job succeed.

Answer 1

Another solution for this error is reducing your partitions.

check to see if you've got a lot of partitions with:

someBigSDF.rdd.getNumPartitions()

Out[]: 200

#if you need to persist the repartition, do it like this
someBigSDF = someBigSDF.repartition(20)

#if you just need it for one transformation/action, 
#you can do the repartition inline like this
someBigSDF.repartition(20).groupBy("SomeDt").agg(count("SomeQty")).orderBy("SomeDt").show()