Yarn: How to make Yarn utilize more memory and vcores

Question

we have a 5 node cluster managed by Yarn and running hadoop

1 Master name node 8 vcores and 24GB memory

4 data nodes each 8 vcores and 24GB memory

When i look at Yarn configuration on the ui as highlighted in the picture below it's only using 16GB and 6 vcores

Our application is using all of the 16gb so want to increase the memory since its available (24 - 2gb for os so avialable is 22gb)

where do i need to configure this 22gb instead of 16gb ?

From research found yarn-site.xml might be the place so went ahead and updated it and restarted yarn but still its showing 16gb

Would grealty appreciate if any of the experts in the community will help out as we are new to Yarn

this is the yarn-site.xml

<configuration>

    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hdfs-name-node</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>21528</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>6</value>
    </property>

    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>21528</value>
    </property>

    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>file:///tmp/hadoop/data/nm-local-dir,file:///tmp/hadoop/data/nm-local-dir/filecache,file:///tmp/hadoop/data/nm-local-dir/usercache</value>
    </property>

    <property>
        <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
        <value>500</value>
    </property>

    <property>
         <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
         <value>512</value>
    </property>

    <property>
        <name>yarn.nodemanager.address</name>
        <value>0.0.0.0:8039</value>
    </property>

    </configuration>

BruceWayne BruceWayne · Accepted Answer · 2017-02-03T04:07:39

Your understanding is completely wrong. According to your attached image, your total memory which is used by yarn is 64GB and VCores are 24.

Highlighted part in image is scheduler memory and vcores, which designed by your admin for a tenant.

If you are part of the scheduler queue, you can use maximum 16 GB and 6 Vcores.

Yarn: How to make Yarn utilize more memory and vcores

2 Answers