Spark Transformation - Why its lazy and what is the advantage?

后端 未结 3 940
我在风中等你
我在风中等你 2020-11-27 06:29

Spark Transformations are lazily evaluated - when we call the action it executes all the transformations based on lineage graph.

What is the advantage o

3条回答
  •  时光说笑
    2020-11-27 06:59

    From https://www.mapr.com/blog/5-minute-guide-understanding-significance-apache-spark

    Lazy evaluation means that if you tell Spark to operate on a set of data, it listens to what you ask it to do, writes down some shorthand for it so it doesn’t forget, and then does absolutely nothing. It will continue to do nothing, until you ask it for the final answer. [...]

    It waits until you’re done giving it operators, and only when you ask it to give you the final answer does it evaluate, and it always looks to limit how much work it has to do.

    It saves time and unwanted processing power.

提交回复
热议问题