Hadoop MR2 Job statistics

Question

I have Hadoop version 2.6.0 installed in my machine.

hduser@vagrant:/usr/local/hadoop$ hadoop version
Hadoop 2.6.0

Also, I started the hadoop cluster using bash sbin/start-dfs.sh and see the Datanode, namenode and secondarynode running.

hduser@vagrant:/usr/local/hadoop$ jps
2627 DataNode
2503 NameNode
3634 Jps
2825 SecondaryNameNode

I'm also able to submit a job and able to see the output without any issues.

hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 2 5

Question:- 1. I dont see the (nodemanager and resourcemanager) YARN not running but still the jobs are completed. Where did the MR job run and where I can see the status of the job and the number of jobs run?

Here is my netstat results:-

hduser@vagrant:/usr/local/hadoop$ netstat -tulpn|grep java (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)

tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      2503/java       
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      2627/java       
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      2627/java       
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      2627/java       
tcp        0      0 127.0.0.1:54310         0.0.0.0:*               LISTEN      2503/java       
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      2825/java

facha facha · Accepted Answer · 2016-01-08T08:49:39

You still have to configure and launch YARN services (start-yarn.sh script) and configure your mapreduce jobs to use it:

etc/hadoop/mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

Currently your jobs are being launched in "local" mode (the job runs inside the JVM you've launched with "hadoop jar"), not in "yarn" mode. It works for debugging, but since there is only one JVM involved, you are not doing parallell/distributed computing in "local" mode.

Hadoop MR2 Job statistics

1 Answers