What are the Spark transformations that causes a Shuffle?

后端 未结 4 1385
情书的邮戳
情书的邮戳 2020-11-29 22:55

I have trouble to find in the Spark documentation operations that causes a shuffle and operation that does not. In this list, which ones does cause a shuffle and which ones

4条回答
  •  执念已碎
    2020-11-29 23:32

    Here is the generalised statement on shuffling transformations.

    Transformations which can cause a shuffle include repartition operations like repartition and coalesce, ‘ByKey operations (except for counting) like groupByKey and reduceByKey, and join operations like cogroup and join.

    source

提交回复
热议问题