Spark 1.4 increase maxResultSize memory

Question

I am using Spark 1.4 for my research and struggling with the memory settings. My machine has 16GB of memory so no problem there since the size of my file is only 300MB. Alth

Answer 1

From the command line, such as with pyspark, --conf spark.driver.maxResultSize=3g can also be used to increase the max result size.