Default queue under YARN capacity policy

Question

Using the following queue configuration under YARN capacity policy, how is the default queue chosen when no queue is specified at job launch?

<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>prod,dev</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dev.queues</name>
<value>eng,science</value>
</property>
</configuration>

I know that under the fair policy, you can choose a default queue, but I cannot find the analog in the capacity one.

This is an example for the fair policy.

<queuePlacementPolicy>
<rule name="specified" create="false" />
<rule name="primaryGroup" create="false" />
<rule name="default" queue="dev.eng" />
</queuePlacementPolicy>

Look into the mapred-site if you're asking about mapreduce jobs. Hive and Spark also have respective defaults — OneCricketeer

pradeep pradeep · Accepted Answer · 2018-03-06T20:28:29

You can refer https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_yarn-resource-management/content/default_queue_mapping_based_on_user_or_group.html which explains to set default queue mapping for Capacity Scheduler

Also, if you are looking to configure Capacity scheduler, this post may help you - https://discuss.pivotal.io/hc/en-us/articles/201623853-How-to-configure-queues-using-YARN-capacity-scheduler-xml

Default queue under YARN capacity policy

1 Answers