“Application priority” in Yarn

问题

I am using Hadoop 2.9.0. Is it possible to submit jobs with different priorities in YARN? According to some JIRA tickets it seems that application priorities have now been implemented.

I tried using the YarnClient, and setting a priority to the ApplicationSubmissionContext before submitting the job. I also tried using the CLI and using updateApplicationPriority. However, nothing seems to be changing the application priority, it always remains 0.

Have I misunderstood the concept of ApplicationPriority for YARN? I saw some documentation about setting priorities to queues, but for my use case I need all jobs in one queue.

Will appreciate any clarification on my understanding, or suggestions about what I could be doing wrong.

Thanks.

回答1:

Yes, it is possible to set priority of your applications on the yarn cluster.

Leaf Queue-level priority
You can define queues with different priority and use spark-submit to submit your application to the specific queue with the wanted priority.

Basically you can define your queues in etc/hadoop/capacity-scheduler.xml like this:

<property>
  <name>yarn.scheduler.capacity.root.prod.queues</name>
  <value>prod1,prod2</value>
  <description>Production queues.</description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.test.queues</name>
  <value>test1,test2</value>
  <description>Test queues.</description>
</property>

See documentation of queue properties here

Note: Application priority works only along with FIFO ordering policy. Default ordering policy is FIFO.

In order to set application priority you can add properties like this to the same file:

<property>
  <name>yarn.scheduler.capacity.root.test.default-application-priority</name>
  <value>10</value>
  <description>Test queues have low priority.</description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.prod.default-application-priority</name>
  <value>90</value>
  <description>Production queues have high priority.</description>
</property>

See more information about application priority here

Changing application priority at runtime:
If you want to change application priority at runtime you can also use the CLI like this:

yarn application -appId <ApplicationId> -updatePriority <Priority>

Can you share what command you execute on what node and what response you get?

See more info here

Using YarnClient
You did not share your code so it is difficult to see if you do it right. But it is possible to submit a new application with a specific priority using YarnClient

ApplicationClientProtocol.submitApplication(SubmitApplicationRequest)

See more info here

来源：https://stackoverflow.com/questions/48673972/application-priority-in-yarn

标签

Hadoop

yarn