5
votes

My organization is currently using Hortonworks HDP to manage our Hadoop Cluster. The default YARN scheduler is the Capacity Scheduler. I would like to switch to a Fair Scheduler. I am completely new to HDP.

In the absence of a cluster management suite, this would be done by editing the yarn-site.xml and changing the yarn.resourcemanager.scheduler.class property to org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler and creating an extra fair-scheduler.xml file to specify the queue configurations as mentioned here and then referring YARN to that configuration by setting the yarn.scheduler.fair.allocation.file property in yarn-site.xml.

Now in Ambari, while it is possible to change the yarn.resourcemanager.scheduler.class property via the UI, and add a new custom property yarn.scheduler.fair.allocation.file, I cannot (for the love of god) find a way to have ambari read fair-scheduler.xml instead of capacity-scheduler.xml.

So my question is; How do I go about switching to the fair scheduler via Ambari? There's got to be an easy way, right?

Properties in capacity-scheduler.xml

1

1 Answers

2
votes

On your RM node, set yarn.scheduler.fair.allocation.file with the full path of your fair-scheduler.xml (or in custom yarn-site under ambari)

tail -n 1000 /var/log/hadoop-yarn/yarn/hadoop-yarn-resourcemanager-master.log | grep "fair-scheduler.xml"

After a restart of your ResourceManager, you should see that it is loading your file:

2019-02-19 15:49:26,358 INFO  fair.AllocationFileLoaderService (AllocationFileLoaderService.java:reloadAllocations(230)) - Loading allocation file file:/usr/hdp/current/hadoop-client/conf/fair-scheduler.xml

Works on Hdp 3.1.1, and probably 3.0.0 too