Time to complete Map and Reduce Tasks in Hadoop

Question

I would like to know the real meaning of these two counters Total time spent by all maps in occupied slots (ms) and Total time spent by all reduces in occupied slots (ms). I just wrote MR program similar to word count I got

**Total time spent by all maps in occupied slots (ms)=15667400

Total time spent by all reduces in occupied slots (ms)=158952

CPU time spent (ms)=51930

real 7m38.886s**

Why is it so?????? The first counter is having a very very high value which is actually incomparable with the other three. Kindly clear this to me.

Thank You

With Regards

Answer 1

Probably need some more context around your input data but the first two counters show how much time was spent across all map and reduce tasks. This number is larger than everything else as you probably have a multi-node hadoop cluster and a large input dataset - meaning you have lots of map tasks running in parallel. Say you have 1000 map tasks running in parallel and each takes 10 seconds to complete - in this case the total time across all mappers would be 1000*10, 10000 secs. In reality the map phase may only take 10-30 seconds to complete in parallel, but if you were to run them in serial they would take 10000 secs to complete with a single node, single map slot cluster.

The CPU time spent refers to the how much of the total time was pure CPU processing - this is smaller than the others as your job is mostly IO bound (reading from and writing to disk, or across the network).