How much memory and vcore allocated on hadoop YARN?

Question

I want to ask, in hadoop yarn both on yarn-site.xml and mapred-site.xml there are property like minimum and maximum memory or vcore. I'm little bit confuse, actually in real how much memory and vcore allocated because on configuration we only write minimum and maximum not the actual size.

If I have memory 16 GB and 4 vcore And the configuration like this :

yarn.scheduler.minimum-allocation-mb = 1024
yarn.scheduler.maximum-allocation-mb = 8192
yarn.scheduler.minimum-allocation-vcores = 1
yarn.scheduler.maximum-allocation-vcores = 32
yarn.nodemanager.resource.memory-mb = 8192
yarn.nodemanager.resource.cpu-vcores = 8

So actually, how much memory, and vcore that my computer allocated for a one container?

That's all, thans a lot

banjara banjara · Accepted Answer · 2015-05-05T11:32:09

YARN is general purpose resource manager, you can use it for various applications like mapreduce, spark, hive, kafka....

So In yarn we define min and max memory/cores per container an application can request, actual memory and cores required are defined in specific application.

In mapreduce we define cores and memory to allocate to map and reduce task in mapred-site.xml.

<property>
    <name>mapreduce.map.cpu.vcores</name>
    <value>1</value>
</property>

<property>
    <name>mapreduce.reduce.cpu.vcores</name>
    <value>1</value>
</property>

<property>
    <name>mapreduce.map.memory.mb</name>
    <value>6144</value>
    <description>The value specified is in MB(s)</description>
</property>

<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>6144</value>
    <description>The value specified is in MB(s)</description>
</property>

How much memory and vcore allocated on hadoop YARN?

1 Answers