HBase Error - assignment of -ROOT- failure

Question

I've just installed hadoop and hbase from cloudera (3) but when I try to go to http://localhost:60010 it just sits there continually loading.

I can get to the regionserver fine - http://localhost:60030... Looking at the master hbase server logs I can see the following.

Looks like a problem with the root region.

All of this is installed on a ext4 1TB partition running Ubuntu (Natty) 11. No cluster/other boxes).

Any help would be great!

11/05/15 19:58:27 WARN master.AssignmentManager: Failed assignment of -ROOT-,,0.70236052 to serverName=localhost,60020,1305452402149, load=(requests=0, regions=0, usedHeap=24, maxHeap=995), trying to assign elsewhere instead; retry=0
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to /127.0.0.1:60020 after attempts=1
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:957)
    at org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:606)
    at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:541)
    at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:901)
    at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:730)
    at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:710)
    at org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor.chore(AssignmentManager.java:1605)
    at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
    at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
    at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
    at $Proxy6.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
    ... 8 more
11/05/15 19:58:27 WARN master.AssignmentManager: Unable to find a viable location to assign region -ROOT-,,0.70236052

Answer 1

Fixed this issue for anyone else who finds this. Was a problem with the host file (/etc/hosts). Need to remove entries relating to 127.0.1.1 COMPNAME - just put a hash (#) in front of this line and then restart all hadoop and hbase services.

More on the solution here: http://blog.nemccarthy.me/?p=110

Answer 2

As per @Manav:

If you find yourself in a situation wherein you can't edit /etc/hosts, the following >workaround will also work:

in conf/hadoop-env.sh add the following line:

export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

I'm using Ubuntu 11.10 (Oneiric) and HBase 0.92.1. These steps fixed my issue for my single server install:

1. Edit the /etc/hosts: change the IP address associated with the hostname so that it uses your LAN IP instead of 127.0.0.1
2. Open <HBASE_DIR>/conf/hbase-env.sh

3. edit HBASE_OPTS, append -Djava.net.preferIPv4Stack=true. The line should look like this:

   export HBASE_OPTS="-XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true"

4. Restart HBase

Answer 3

If you find yourself in a situation wherein you can't edit /etc/hosts, the following workaround will also work:

in conf/hadoop-env.sh add the following line:

export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

(removed edit, moved as a separate answer)

Answer 4

your hosts file should look like this

#127.0.0.1  localhost
#127.0.1.1  ubuntu.ubuntu-domain    ubuntu
192.168.2.100   ubuntu
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

This file can be found in /etc/hosts

Regards Shuja

Answer 5

The trick with a subinterface worked for me, but i used the loopback interface rather than eth0 because eth0 is not always available on my machine (external adapter) and i want it managed by NetworkManager (which refuses to manage eth0 if eth0.1 is defined in /etc/network/interfaces on ubuntu 13.04)

Relevant snippet:

auto lo:0
iface lo:0 inet static
address 127.0.1.1
netmask 255.255.255.0

in addition to the regular

auto lo
iface lo inet loopback  

of course

Answer 6

Here's another work-around that Works For Me, if you're unwilling to alter /etc/hosts (since Ubuntu put that entry there for a reason).

As this post explains, the core problem is that the loopback interface has multiple IPs bound to it while hbase assumes there will be only one. The resulting mismatch causes the master to think a region server has one IP (127.0.0.1), when it's really listening on another (127.0.1.1, the IP bound to the host's declared FQDN.)

Removing the /etc/hosts entry is one way to restore the one-interface-one-IP assumption. Replacing 127.0.1.1 in /etc/hosts with a "real" permanent IP is another. Finally, another is to create a new interface: drop this in the bottom of /etc/network/interfaces:

# Bind an interface solely for the default host FQDN IP, to fix reverse dns
auto eth0.1
iface eth0.1 inet static
pre-up ip link add eth0.1 name eth0.1 type bridge
address 127.0.1.1
netmask 255.255.255.0

You should then be able to sudo ifup eth0.1 and see it in ifconfig. Restart hbase & you should be good to go.

If you happen to already be using eth0.1 the pick another slot (i.e. eth0.2), it shouldn't matter.

EDIT: @bcolyn's use of lo:0 also works for me, and is superior since loopback will always be available. In that case the pre-up line also appears unnecessary.

Answer 7

In your hosts file to change host address from 127.0.1.1 to 127.0.0.1