问题
Could anyone give me a more precise description of this Spark parameter and how it effects program execution? I cannot tell exactly what this parameter does "under the hood" from the documentation.
回答1:
The parameter influences the memory limit for Python workers. If the RSS of a Python worker process is larger than the memory limit, then it will spill data from memory to disk, which will reduce the memory utilization but is generally an expensive operation.
Note that this value applies per Python worker, and there will be multiple workers per executor.
If you want to take a look under the hood, then look at the python/pyspark directory in the Spark source tree, e.g. the ExternalMerger implementation: https://github.com/apache/spark/blob/41afa16500e682475eaa80e31c0434b7ab66abcb/python/pyspark/shuffle.py#L280
来源：https://stackoverflow.com/questions/31565497/what-is-spark-python-worker-memory