Remote access to namenode is not allowed despite the services are already started.

Question

I successfully installed and run Hadoop on a single machine whose ip is 192.168.1.109 (In fact it is actually an Ubuntu instance running on virtual box ) . When typing jps it shows

2473 DataNode

2765 TaskTracker

3373 Jps

2361 NameNode

2588 SecondaryNameNode

2655 JobTracker

This should mean that the hadoop is up and running. Running commands like ./hadoop fs -ls is fine and produces the expected result.

But If I try to connect it from my windows box whose ip is 192.168.1.80 by writing Java code's HDFS API to connect it as follows:

Configuration conf = new Configuration();

FileSystem hdfs = null;

Path filenamePath = new Path(FILE_NAME);

hdfs = FileSystem.get(conf); <-- the problem occurred at this line

when I run the code, the error displayed as follows:

11/12/07 20:37:24 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 0 time(s).

11/12/07 20:37:26 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 1 time(s).

11/12/07 20:37:28 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 2 time(s).

11/12/07 20:37:30 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 3 time(s).

11/12/07 20:37:32 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 4 time(s).

11/12/07 20:37:33 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 5 time(s).

11/12/07 20:37:35 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 6 time(s).

11/12/07 20:37:37 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 7 time(s).

11/12/07 20:37:39 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 8 time(s).

11/12/07 20:37:41 INFO ipc.Client: Retrying connect to server: /192.168.1.109:9000. Already tried 9 time(s).

java.net.ConnectException: Call to /192.168.1.109:9000 failed on connection exception: java.net.ConnectException: Connection refused: no further information

To make sure if the socket is already opened and waits for the incoming connections on the hadoop serer, I netstat on the ubuntu box the result shows as follows:

tcp 0 0 localhost:51201 : LISTEN 2765/java
tcp 0 0 *:50020 : LISTEN 2473/java
tcp 0 0 localhost:9000 : LISTEN 2361/java
tcp 0 0 localhost:9001 : LISTEN 2655/java
tcp 0 0 *:mysql : LISTEN -
tcp 0 0 *:50090 : LISTEN 2588/java
tcp 0 0 *:11211 : LISTEN -
tcp 0 0 *:40843 : LISTEN 2473/java
tcp 0 0 *:58699 : LISTEN -
tcp 0 0 *:50060 : LISTEN 2765/java
tcp 0 0 *:50030 : LISTEN 2655/java
tcp 0 0 *:53966 : LISTEN 2655/java
tcp 0 0 *:www : LISTEN -
tcp 0 0 *:epmd : LISTEN -
tcp 0 0 *:55826 : LISTEN 2588/java
tcp 0 0 *:ftp : LISTEN -
tcp 0 0 *:50070 : LISTEN 2361/java
tcp 0 0 *:52822 : LISTEN 2361/java
tcp 0 0 *:ssh : LISTEN -
tcp 0 0 *:55672 : LISTEN -
tcp 0 0 *:50010 : LISTEN 2473/java
tcp 0 0 *:50075 : LISTEN 2473/java

I noticed that if the local address column is something like localhost:9000 (starts with localhost: not *:) It will not be able to be connected from remote host or even in it own box in some case. I tried telnet localhost 9000 it works, I means it can connect to the port but If I use telnet 192.168.1.109 9000 The errors displays like

$ telnet 192.168.1.109 9000 Trying 192.168.1.109... telnet: Unable to connect to remote host: Connection refused

I have spent almost a week figuring out the issue I am really exhausted now and I hope someone can help me.

Note: I am not sure if namenode by default refuses remote connection. Do I need to change some settings in order for it to allow remote connections?

Answer 1

Change the value of fs.default.name to hdfs://106.77.211.187:9000 from hdfs://localhost:9000in core-site.xml for both the client and the NameNode. Replace the IP address with the IP address of the node on which the NameNode is running or with the hostname.

Was able to telnet 106.77.211.187 9000 and here is the output of netstat -a | grep 9000

tcp6 0 0 106.77.211.187:9000 [::]:* LISTEN
tcp6 0 0 106.77.211.187:50753 106.77.211.187%819:9000 ESTABLISHED
tcp6 0 0 106.77.211.187:9000 106.77.211.187%81:50753 ESTABLISHED

As to why, the source code look like the following for fs.default.name set to localhost

ServerSocket socket = new ServerSocket(9000);
socket.bind(localhost);

Because bind address is assigned to localhost, the namenode process only can accept connection from localhost. If bind address is assigned to the name of machine name or ip address, then namenode process can accept any connection from remote machine.

Answer 2

I replaced all localhost with its ip address in all configuration file, now it is working fine.

Answer 3

Check /etc/hosts file and make sure you have the IP associated with the fully qualified name (FQN) of your node. Example:

127.0.0.1      localhost localhost.localdomain localhost4 localhost4.localdomain4
::1            localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.11 node1.mydomain.com node1
192.168.100.12 node2.mydomain.com node2 

In my case, I had line 127.0.0.1 node1.mydomain.com which was definetly wrong.

Answer 4

I faced the same issue but was able to get it fixed by doing the following. I had the hadoop master and slaves as CentOS7 VirtualBox VMs and I couldn't access web GUIs from the windows host by using the IP address and port of the Master node. Make sure you follow the steps given below to get it fixed;

1. As mentioned in the others posts, make sure the /etc/hosts file is correctly populated

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.10.2.20     hdp-master1.hadoop.cluster      hdp-master1

172.10.2.21     hdp-slave1.hadoop.cluster       hdp-slave1

172.10.2.22     hdp-slave2.hadoop.cluster       hdp-slave2

2. And in all your hadoop xml files use the fully qualified hostname or ip instead of localhost, as others have mentioned

3. Add the following entry to hdfs-site.xml to make the web gui port to run from ip instead of 0.0.0.0:9870

<property>
        <name>dfs.namenode.http-address</name>
        <value>hdp-master1.hadoop.cluster:9870</value>
</property>

4. Add the following entry to yarn-site.xml to make the resource manager web gui port to run from ip instead of 0.0.0.0:8088

<property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hdp-master1.hadoop.cluster:8088</value>
</property>

<property>
        <name>yarn.resourcemanager.webapp.https.address</name>
        <value>hdp-master1.hadoop.cluster:8090</value>
</property>

5. Stop and Start all services using start-all.sh. Just to be safe I ran hdfs namenode -format first before restarting services

6. Use netstat -tulnp on master node and make sure web ports run based on the ip

   netstat -tulnp
   
   tcp 0 0 172.16.3.20:8088 0.0.0.0:* LISTEN 14651/java
   tcp 0 0 172.16.3.20:9870 0.0.0.0:* LISTEN 14167/java
   

7. Even after all that, I still couldn't access from the windows host and the culprit was the firewall on the hadoop nodes. So stop stop the firewall on all master and slave node as below

Check status
------------
systemctl status firewalld

Stop Firewall
-------------
systemctl stop firewalld

Disable from Startup
--------------------
systemclt disable firewalld

Now you should be able to access from the windows host through a web browser. I had entries added to windows hosts file so the even the following worked

http://hdp-master1.hadoop.cluster:9870
http://hdp-master1.hadoop.cluster:8088

Hope this helps