Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)

Question

I get the following error when I add --conf spark.driver.maxResultSize=2050 to my spark-submit command.

17/12/27 18:33:19 ERROR Tra         


        

Answer 1

Cause: caused by actions like RDD's collect() that send big chunk of data to the driver

Solution: set by SparkConf: conf.set("spark.driver.maxResultSize", "4g") OR set by spark-defaults.conf: spark.driver.maxResultSize 4g OR set when calling spark-submit: --conf spark.driver.maxResultSize=4g