Hadoop Yarn container configuration (CPU, Memory...)

0
votes

I've just setup a new Hadoop cluster with Hadoop 2.2.0 and running the MapReduce job on HBase based on Yarn framework.

I have a problem of the configuration of containers. In general, we have 8 nodes, half of which are old machines with 8 cores CPU and half of which are new machines with 24 cores CPU. I wonder if it's possible to configure separately with more containers in new machines and less in old machines. With actual setting, the number of containers are limited to 8 which means 1 core per container at least. Even though, I have resources left in new machines, it's not allocated to more containers in new machines. We use the fair scheduler.

Thanks

1
hadoopmapreduceyarn

1 Answers

0
votes

In the configuration file yarn-site.xml, there is a property named yarn.nodemanager.resource.cpu-vcores which defines the CPU cores of the node. As I set this value differently of the old and the new machines, more containers are running in new nodes.

I again answer my own question :)