What will spark do if I don't have enough memory?

Question

I\'m new to Spark, and I found the Documentation says Spark will will load data into memory to make the iteration algorithms faster.

But what if I have a log file of

Answer 1

It will not load the full 10G, as you don't have enough memory available. From my experience, one of three will happen depending on how you use your data:

If you are trying to cache the 10GBs:

1. You are going to get an OOME
2. Data are going to be loaded

If you are just processing the data:

1. Data are going to be swapped in/out of memory

Of course, this is highly related to your code and the transformations you are applying.