How to add custom description to Spark Job for displaying in Spark Web UI

Question

When we submit application to Spark, and after performing any operation Spark Web UI displays Job and Stages like count at MyJob.scala:15. But in my application

Answer 1

use the sc.setJobGroup:

Examples:
python:

In [28]: sc.setJobGroup("my job group id", "job description goes here")
In [29]: lines = sc.parallelize([1,2,3,4])
In [30]: lines.count()
Out[30]: 4

Scala:

scala> sc.setJobGroup("my job group id", "job description goes here")
scala> val lines = sc.parallelize(List(1,2,3,4))
scala> lines.count()
res3: Long = 4

SparkUI:

I hope this is what you are looking for.