java.lang.OutOfMemoryError: Unable to acquire 100 bytes of memory, got 0

Question

I\'m invoking Pyspark with Spark 2.0 in local mode with the following command:

pyspark --executor-memory 4g --driver-memory 4g

The input da

Answer 1

In my case replacing the coalesce(1) with repartition(1) Worked.