Change Node Cluster IP Hadoop-Yarn

Question

I have the following problem: These picture describe my Resource Manager on Hadoop Cluster, in node "Node HTTP Address" there are two nodes these address

1. localhost:8042
2. localhost:8042

Instead of that I want the real ip address of machine that host the node. My cluster configuration is

1. First Machine with HadoopMaster and HadoopSlave1
2. Second Machine with HadoopSlave 2

My /etc/host configuration on first machine:

127.0.0.1       localhost vlchaosfile001 vlchaosfil001
192.168.143.91  HadoopMaster HadoopSlave2
192.168.143.92  HadoopSlave1
# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

My /etc/host configuration on second machine:

127.0.0.1 localhost vlchaosfil002
192.168.143.91  HadoopMaster HadoopSlave2
192.168.143.92  HadoopSlave1
#following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

This is my yarn-site.xml:

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>HadoopMaster:8025</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>HadoopMaster:8035</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>HadoopMaster:8050</value>
</property>

Answer 1

First off: disable ipv6 when using hadoop.

You didn't post your Yarn configs so my best guess at this point is that you have set yarn.resourcemanager.xxx settings to "localhost"

Answer 2

Well, to change the hostnames in a hadoop cluster is a pretty tough and risky job. Consider a new installation from the scratch, you are aware of it ;)

It's also strongly recommended to use FQDN instead short hostnames. That's a good time to perform such change.

Overall, you have to check/change all config files, but if you want to have a functional Ambari console you also must update ambari db metadata.

Here you are a general procedure:

1. Stop cluster.
2. Export Ambari’s MySQL database.
3. Update host names (host_name) in the MySQL export tables ambari.hosts, ambari.hoststate, ambari.ClusterHostMapping, ambari.host_role_command and ambari.hostcomponentdesiredstate
4. Update hive.DBS for proper DB_LOCATION_URI and hive.SDS for new hostnames.
5. Re-import database.
6. Grant db access to new host names into MySQL backed Hive/Metastore
7. Edit Ambari agent ini files and hue config files updating hostnames.
8. Check/change all config files like hdfs-site, yarn-site, mapred-site etc as well as less obvious ones like ambari agent, zookeeper etc...
9. Start then stop cluster to write all configs and review metadata.
10. Cross your fingers and do a final start.