DataNode can't talk to NameNode in Hadoop 2.2

1
votes

I'm setting up a hadoop 2.2 cluster. I have successfully configured a master and a slave. When I enter start-dfs.sh and start-yarn.sh on the master, all the daemons start correctly.

To be specific, on the master the following are running:

  • DataNode
  • NodeManager
  • NameNode
  • ResourceManager
  • SecondaryNameNode

On the slave, the following are running:

  • DataNode
  • NodeManager

When I open http://master-host:50070 I see that there is only 1 "Live Node" and it is referring to the datanode on the master.

The datanode on the slave is started, but not being able to tell the master that it started. This is the only error I can find:

From /logs/hadoop-hduser-datanode.log on the slave: WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: ec2-xx-xxx-xx-xx.compute-1.amazonaws.com/xx.xxx.xx.xxx:9001

Things I have checked/verified:

  • 9001 is open
  • both nodes can ssh into each other
  • both nodes can ping each other

Any suggestions are greatly appreciated.

1
hadoopamazon-ec2

1 Answers

1
votes

My issue was in the hosts file:

The hosts file on the slave and master needed to be (they're identical_:

127.0.0.1 localhost
<master internal ip> master
<slave internal ip> slave

For AWS you need to use the internal ip that is something like xx.xxx.xxx.xxx (not the external ip in the ec2-xx-xx-xxx.aws.com and not the ip-xx-xx-xxx).

Also, core-site.xml should refer to the location of hdfs as http://master:9000.