How to use COGROUP for large datasets

Question

I have two rdd\'s namely val tab_a: RDD[(String, String)] and val tab_b: RDD[(String, String)] I\'m using cogroup for tho

Answer 1

TL;DR Don't collect.

To run this code safely, without additional assumptions (on average requirements for worker nodes might be significantly smaller), every node (driver and each executor) would require memory significantly exceeding total memory requirements for all data.

If you were to run it outside Spark you would need only one node. Therefore Spark provides no benefits here.

However if you skip collect.toArray and make some assumptions about data distribution you might run it just fine.