org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

17
votes

I am trying to setup the multinode cluster of Hbase. When i do the jps on slave i get

5780 Jps
5558 HQuorumPeer
5684 HRegionServer
1963 DataNode
2093 TaskTracker

similarly on master i get 

4254 SecondaryNameNode
15226 Jps
14982 HMaster
3907 NameNode
14921 HQuorumPeer
4340 JobTracker

EVerything is runnnig properly. But when i try to create table on hbase shell. It gives an error 

ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

regionserver log of my slave(where region server is running):

2013-06-11 13:09:53,119 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at localhost,60000,137093$
2013-06-11 13:10:53,190 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: localhost/127.0.0.1:60000
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:425)
        at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
        at $Proxy8.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2037)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2083)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)
        at java.lang.Thread.run(Thread.java:722)
2013-06-11 13:10:53,391 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at localhost,60000,137093$

FYI, i have also took care of /etc/hosts file on both master and slave.

127.0.0.1       localhost
127.0.0.1       naresh-PC

I again did changes in /etc/hosts file 127.0.1.1 to naresh-PC. But still getting this error

2013-06-11 14:51:17,781 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at naresh-pc,60000,137094$
2013-06-11 14:52:17,817 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.UnknownHostException: unknown host: naresh-pc
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.<init>(HBaseClient.java:276)
        at org.apache.hadoop.hbase.ipc.HBaseClient.createConnection(HBaseClient.java:255)
        at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1111)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
        at $Proxy8.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2037)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2083)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)
        at java.lang.Thread.run(Thread.java:722)
5
hadoophbasehive
Also, i am being able to do password-less ssh from master to slave and slave to master. – Naresh

5 Answers

21
votes

Try clearing all the states in Zookeeper.

  • Stop Zookeeper
  • Wipe the Zookeeper data directory
  • Start Zookeeper

I was getting the same issue and followed this approach and it worked fine.

4
votes

You need to change the configuration on the slave node to point at the master. It is currently pointing to localhost and not connecting to the actual master:

"org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: localhost/127.0.0.1:60000 at "

1
votes

I'm hosting my own cluster inside Docker. Here's what worked in my case. I grepped the HBase log file for errors and found "Master passed us a different hostname to use" 

`[root@docker-iop bin]# grep ERROR /var/log/hbase/hbase-hbase-regionserver-bi-mgmt01.local.log
2016-10-06 00:05:29,816 ERROR [regionserver/bi-mgmt01.local/111.11.2.3:16020] regionserver.HRegionServer: Master passed us a different hostname to use; was=my-host-name, but now=111.22.33.444'

I mapped my-host-name to 111.22.333.444 in my hosts file, restarted HBase and it worked.

0
votes

I also had the same issue with a fully distributed hbase cluster with the configuration below.

  1. Master Node (Node-A)
  2. Backup Masters ($HBASE_HOME/conf/backup-masters) (Node-B & Node-C)
  3. 3 Replication servers (Node-A, Node-B & Node-C)

RCA: The backup-masters nodes were attempted to be started when the cluster started.

Solution I removed the backup masters by making $HBASE_HOME/conf/backup-masters empty in all hbase nodes.

So I had a cluster running without backup masters.

I wonder if the master node and master nodes must not also function as regionservers? The HBase documentation says otherwise though.

0
votes

I came across the same issue and could not find anything, it turns out I was copy pasting from the Hbase documentation (https://hbase.apache.org/book.html#shell_exercises). I believe some character in there may be creating the error, so try to manually enter:

create 'test', 'cf'