What are the Spark transformations that causes a Shuffle?

后端 未结 4 1386
情书的邮戳
情书的邮戳 2020-11-29 22:55

I have trouble to find in the Spark documentation operations that causes a shuffle and operation that does not. In this list, which ones does cause a shuffle and which ones

4条回答
  •  萌比男神i
    2020-11-29 23:42

    This might be helpful: https://spark.apache.org/docs/latest/programming-guide.html#shuffle-operations

    or this: http://www.slideshare.net/SparkSummit/dev-ops-training, starting with slide 208

    from slide 209: "Transformations that use 'numPartitions' like distinct will probably shuffle"

提交回复
热议问题