Hbase Multiple Master issue

Question

I am evaluating HBase Multiple master utility as I require that functionality. I used a simple 4 machine hbase cluster with M1 as master. Now I started the Hmaster in M4 as well. I could see logs in M4 Master log as, This HMaster is waiting as already M1 is acting as HMaster.

After this I stopped the Master in the M1 and after few seconds, the M4 master took over. But I am not seeing any region server communicating to this new master. The region server logs show,

2011-10-31 23:21:18,834 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was: 
java.net.ConnectException: Connection refused 
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) 
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) 
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) 
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311) 
        at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865) 
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732) 
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) 
        at $Proxy3.getProtocolVersion(Unknown Source) 
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) 
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) 
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) 
        at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349) 
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1443) 
        at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737) 
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586) 
        at java.lang.Thread.run(Thread.java:619)

Also in the M4 Master log I am seeing,

2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 60010 webServer.getConnectors()[0].getLocalPort() returned 60010 
2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 60010 
2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26 
2011-10-31 23:20:48,691 INFO org.mortbay.log: Started [email protected]:60010 
2011-10-31 23:20:48,691 DEBUG org.apache.hadoop.hbase.master.HMaster: Started service threads 
2011-10-31 23:20:50,192 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin 
2011-10-31 23:20:51,695 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin 
2011-10-31 23:20:53,197 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin

Seems the region servers are only communicating to the M1 Hmaster and not to the new one!!

I am using Hbase - 0.90.0 version

Any help regarding this is highly appreciated!!

Today saw the zookeeper to make sure whether the new master is getting registered in it or not. Just used the zk_dump and the new master is present as current master node in zookeeper. But still this is not considered by region servers. I expected the region servers would update the master info in them by communicating with the zoo keeper. But doesn't seem to happen here. Any suggestions Plz !!! — sriram

sriram sriram · Accepted Answer · 2011-11-03T07:44:27

It seems this works fine with Hbase 0.90.3. Need to verify though!!

http://old.nabble.com/Hbase-Multiple-Master-issue-td32757197.html#a32758542

Hbase Multiple Master issue

1 Answers