java.lang.OutOfMemoryError: Unable to acquire 100 bytes of memory, got 0

Question

I\'m invoking Pyspark with Spark 2.0 in local mode with the following command:

pyspark --executor-memory 4g --driver-memory 4g

The input da

Answer 1

In my case the driver was smaller than the workers. Issue was resolved by making the driver larger.