What is the concept of application, job, stage and task in spark?

后端 未结 3 775
你的背包
你的背包 2020-12-13 00:08

Is my understanding right?

  1. Application: one spark submit.

  2. job: once a lazy evaluation happens, there is a job.

  3. stage: It

3条回答
  •  情歌与酒
    2020-12-13 00:45

    The main function is the application.

    When you invoke an action on an RDD, a "job" is created. Jobs are work submitted to Spark.

    Jobs are divided into "stages" based on the shuffle boundary. This can help you understand.

    Each stage is further divided into tasks based on the number of partitions in the RDD. So tasks are the smallest units of work for Spark.

提交回复
热议问题