Running Mapreduce issues

0
votes

I'm trying to run a wordcount jar on a cluster hadoop 2.7.1 (one master and 4 slaves), but the MapReduce job was blocked at:

$ hadoop jar wc.jar WordCount /input /output_hocine 
17/03/13 09:41:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
17/03/13 09:41:43 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 
17/03/13 09:41:43 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 
17/03/13 09:41:44 INFO input.FileInputFormat: Total input paths to process : 3 
17/03/13 09:41:44 INFO mapreduce.JobSubmitter: number of splits:3 
17/03/13 09:41:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489393376058_0003 
17/03/13 09:41:44 INFO impl.YarnClientImpl: Submitted application application_1489393376058_0003 
17/03/13 09:41:44 INFO mapreduce.Job: The url to track the job: http://ibnbadis21:8088/proxy/application_1489393376058_0003/ 
17/03/13 09:41:44 INFO mapreduce.Job: Running job: job_1489393376058_0003

Via The navigator, The output via the navigator is shown at this image:

Here is the content of the configuration files:

Core-site.xml:

<configuration>
<!--    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://ibnbadis21:9000</value>
    </property>-->
    <property>
        <name>fs.default.name</name>
        <value>hdfs://ibnbadis21:9000</value>
    </property>
   <property>
      <name>dfs.permissions</name>
      <value>false</value>
   </property>
 </configuration>

yarn-site.xml:

<?xml version="1.0"?> <configuration>
   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

mapred-site.xml:

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. -->
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>ibnbadis21:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>ibnbadis21:19888</value>
        </property>
        <property>
                <name>yarn.app.mapreduce.am.staging-dir</name>
                <value>/user/app</value>
        </property>
</configuration>

hdfs-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <property>
        <name>dfs.replication</name>
        <value>2</value>
 </property>
 <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
 </property>
 <property> <name>dfs.namenode.checkpoint.dir</name>
        <value>file:/usr/local/hadoop_data/hdfs/namesecondary</value>
 </property>
 <property> <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop_data/hdfs/datanode</value>
 </property>
</configuration>

Can anyone tell me how can solve this problem, please?

2
hadoopmapreducehadoop-yarn
Can you check with the jps command whether every daemon is working properly or not ? it might happen that NameNode is not listening at that port 9000. – Deepan Ram

2 Answers

0
votes

Connecting to ResourceManager at /0.0.0.0:8032

0.0.0.0 (the default) is not a valid hostname.

So, add this in yarn-site.xml

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value> YOUR VALUE HERE </value> <!-- Needs Fully Qualified Domain Name -->
</property>

There are many values that you probably didn't set.

Refer Hadoop | Configuring the Hadoop Daemons

By the way, fs.defaultFS is the correct property to use.

0
votes

Finally the problem was about access rights. The framework haven't the right to access at my yarn-site.xml file. That's why it used the default value 0.0.0.0/8030. Thus When I executed the command with privilege (sudo):

sudo hadoop jar wc.jar WordCount /input /output

My job MapReduce is executed successfully!