How can I tell if my spark job is progressing?

问题

I have a spark job running on YARN and it appears to just hang and not be doing any computation.

Here's what yarn says when I do yarn application -status <APPLICATIOM ID>:

Application Report : 
Application-Id : applicationID
Application-Name : test app
Application-Type : SPARK
User : ec2-user
Queue : default
Start-Time : 1491005660004
Finish-Time : 0
Progress : 10%
State : RUNNING
Final-State : UNDEFINED
Tracking-URL : http://<ip>:4040
RPC Port : 0
AM Host : <host ip>
Aggregate Resource Allocation : 36343926 MB-seconds, 9818 vcore-seconds
Log Aggregation Status : NOT_START
Diagnostics : 

And, when I check the yarn application -list it says that it is RUNNING. But I'm not sure I trust that. When I go to the spark webUI, I see only one stage the entire few hours I've been running it:

Also, when I click on the "Stages" tab, I see nothing running:

How do ensure that my application is actually running and that YARN is not lying to me?

I would actually prefer for this to throw an error rather than keep me waiting to see if the job is actaully runing. How do I do that?

回答1:

On the spark application UI

If you click on the link : "parquet at Nativexxxx" it would show you Details for the running stage.

On that screen there would be a column "Input Size/Records". If your job is progressing the number shown in that column would change.

It basically depicts number of records read by your executor.

回答2:

if you go to Spark UI and search for "executors" tab. There you will have the list executors that your job is running on and next to executor ID and address you will have "logs" column there you will have "stdout" & "stderr" tabs. Click on stdout and there you can see the logs those were written on your container when your job is running.

回答3:

YARN is saying that it is RUNNING because the application hasn't exited or been killed. It doesn't know if it is stuck. In your case you can run Spark job in yarn-client mode and see the logs which should suggest if it is really running.

来源：https://stackoverflow.com/questions/43152126/how-can-i-tell-if-my-spark-job-is-progressing