In spark join, does table order matter like in pig?

后端 未结 1 1585
萌比男神i
萌比男神i 2020-12-20 13:53

Related to Spark - Joining 2 PairRDD elements

When doing a regular join in pig, the last table in the join is not brought into memory but streamed through instead, s

1条回答
  •  情话喂你
    2020-12-20 14:35

    It does not make a difference, in spark the RDD will only be brought into memory if it is cached. So in spark to achieve the same effect you can cache the smaller RDD. Another thing you can do in spark which I'm not sure that pig does, is if all RDD's being joined have the same partitioner no shuffle needs to be done.

    0 讨论(0)
提交回复
热议问题