HBase fully distributed mode [Zookeeper error while executing HBase shell]

Question

Following these two tutorials: i.e tutorial 1 and tutorial 2, I was able to set up HBase cluster in fully-distributed mode. Initially the cluster seems to work okay.

The 'jps' output in HMaster/ Name node

The jps output in DataNodes/ RegionServers

Nevertheless, when every I try to execute hbase shell, it seems that the HBase processors are interrupted due to some Zookeeper error. The error is pasted below:

2021-03-13 11:52:26,047 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper exists failed a│1951 HRegionServer
fter 4 attempts                                                                               │hduser@master-vm:~$ 
2021-03-13 11:52:26,048 WARN  [main] zookeeper.ZKUtil: hconnection-0x4375b0130x0, quorum=137.4│
3.49.59:2181,137.43.49.58:2181,137.43.49.50:2181,137.43.49.49:2181, baseZNode=/hbase Unable to│
 set watcher on znode (/hbase/hbaseid)                                                        │
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss│
 for /hbase/hbaseid                                                                           │
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)               │
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)               │
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)                         │
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.│
java:221)                                                                                     │
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:417)              │
        at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:6│

I tried several attempts to solve this issue (including trying out with different HBase/ Hadoop compatible versions). But still no progress.

Would like to have your input on this.

Shared below are other information required:

in /etc/hosts file:

(I already tried commenting the HBase related hosts in /etc/hosts/, still didn'w work)

in hbase-site.xml

Instead of using IP addresses in hbase-site.xml and regionservers files, use the host names that you speicfied in /etc/hosts. Also it seems a bit strange that your jps output on regionservers shows HMaster. That process should only be running on the master machine (unless you are co-locating the master with ONE of the region servers). Same is true fo the HQuorumPeer process (again, unless you are co-locating). — VS_FF
Noticing one more thing in your hbase-site.xml: your hbase master port is incorrect. In the hbase.master setting, I'd expect the port to be 60000. With your current setting, you might be confusing it with 16010, which is the master's port for web ui (but you don't have to configure it anywhere). — VS_FF
also I see your zookeeper dataDir is in HDFS. Never tried it that way, so not sure if it would work (I normally use some local directory on disk). — VS_FF
Not sure I know of good comprehensive posts on setting up the cluster. Yes, of course the ports need to be open. You can test whether your machines can talk to each other on relevant ports by using telnet and specifying the port. If you see that the connection gets established, then you are fine; otherwise you have a problem. Also obviously they should all be SSH'able from the master. — VS_FF

CoolCK CoolCK · Accepted Answer · 2021-03-17T00:35:49

After 5 days of hustle, I learned what went wrong. Posting my solution here. Hope it can help some of the other developers too. Also would like to thank @VV_FS for the comments.

In my scenario, I used virtual machines which I burrowed from an external party. Therefore, there were certain firewalls and other security measures. In case if you follow a similar experimental setup, these steps might help you.

To set up HBase cluster, follow the following tutorials.

Set up Hadoop in distributed mode.

Notes when setting up HBase in fully distributed-mode:

Make sure to open all the ports mentioned in the post. For example, use sudo ufw allow 9000 to open port 9000. Follow the command to open all the ports in relation to running Hadoop.

Set up Zookeeper in distributed mode.

Notes when setting up Zookeeper in fully distributed mode:

Make sure to open all the ports mentioned in the post. For example, use sudo ufw allow 3888 to open port 3888. Follow the command to open all the ports in relation to running Zookeeper.
DO NOT START ZOOKEEPER NODES AFTER INSTALLATION. ZOOKEEPER WILL BE MANAGED HBASE INTERNALLY. THEREFORE, DON'T START ZOOKEEPER AT THIS STAGE.

Set up HBase in distributed mode.

When setting up values for hbase-site.xml, use port number 60000 for hbase.master tag, not 60010. (thanks @VV_FS to point this out in the earlier discussion).
Make sure to open all the ports mentioned in the post. For example, use sudo ufw allow 60000 to open port 60000. Follow the command to open all the ports in relation to running Zookeeper.

[Important thoughts]: If encounters errors, always refer to HBase logs. In my case, hbase-mater-xxxxx.log and zookeeper-master--xxx.log helped me to track down exact errors.

HBase fully distributed mode [Zookeeper error while executing HBase shell]

1 Answers