Hadoop MapReduce Load Balancing

Question

Assume a hadoop cluster with 3 slave nodes and one master node with a replication factor of 2. Further assume a file F is split into 3 blocks A, B, and C and they are stored as follows: Slave 1: A, B Slave 2: A, C Slave 3: B, C

In addition, assume you kick of a map reduce job to count the number of words in F and that each block is split 3 times.

My question is how are mappers distributed so that they are optimized for maximum productivity? One possibility: Slave 1: 3 mappers to work on A Slave 2: 3 mappers to work on C Slave 3: 3 mappers to work on B

But how does hadoop avoid the following? Slave 1: 6 mappers (3 on A, 3 on B) Slave 2: idle Slave 3: 3 mappers on C

Answer 1

Map Reduce execution is controlled by YARN starting from Hadoop 2, in which there will be Resource Manager (Master) and Node Managers (on each slave). Node Managers will send the usage such as memory, CPU etc on each of their server to Resource Manager as part of heartbeat.

Now when job is submitted, client will talk to Resource Manager and create Application Master. Application Master will be created based up on the usage of the cluster, Resource Manager with the information from Node Manager will make the call.

Once Application Master is created it will talk to Namenode to get block locations associated with your input and also with Node Managers to understand the usage of the cluster. In an idle cluster, there is high probability that each of the slaves run one mapper to process the corresponding block. But it is not guaranteed in heavily used cluster, one node might process all the 3 blocks, while others are doing some other work.

By default number of mappers will be same as number of blocks. Number of mappers is determined by split size which is equal to block size. You can reset split size to use more mappers to process your data. However, load balancing will be done based up on the usage in the cluster. It can run 3 mappers on A, 3 mappers on B and no mappers on C or any other combination.