Hive/HBase Integration - Zookeeper Session Closes Immediately

Question

We have an 8 node cluster using CDH3u2 configured using Cloudera Manager. We have a dedicated master node running our only instance of zookeeper. When I configure hive to run local hadoop, executed from the master node, I have no problem retreiving the data from HBase. When I run distributed map/reduce via hive, I am getting the following error when the slave nodes connect to zookeeper.

HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default).

We have tried setting max connections higher (we even tried removing the limit). This is a development cluster that has very few users, I know that the problem is not that there are too many connections (I am able to connect to zookeeper from the slave nodes using ./zkCli).

Server side logs indicate that the session was terminated by the client.

Client side hadoop log says: 'Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase

Any idea why I am unable to maintian a connection to zookeeper via Hive Map/Reduce?

Configs for hbase and zookeeper are:

# Autogenerated by Cloudera SCM on Wed Dec 28 08:42:23 CST 2011
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=1000
minSessionTimeout=4000
maxSessionTimeout=40000

HBase Site-XML is:

<property>
  <name>hbase.rootdir</name>
  <value>hdfs://alnnimb01:8020/hbase</value>
  <description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR</description>
</property>

<property>
  <name>hbase.master.port</name>
  <value>60000</value>
  <description>The port master should bind to.</description>
</property>

<property>
  <name>hbase.cluster.distributed</name>
  <value>true</value>
  <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)</description>
</property>


<property>
  <name>hbase.master.info.port</name>
  <value>60010</value>
  <description>The port for the hbase master web UI Set to -1 if you do not want the info server to run.</description>
</property>
















<property>
  <name>zookeeper.znode.parent</name>
  <value>/hbase</value>
  <description>Root ZNode for HBase in ZooKeeper. All of HBase's ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase's ZooKeeper file path are configured with a relative path, so they will all go under this directory unless changed.</description>
</property>

<property>
  <name>zookeeper.znode.rootserver</name>
  <value>root-region-server</value>
  <description>Path to ZNode holding root region location. This is written by the master and read by clients and region servers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, this means the root location is stored at /hbase/root-region-server.</description>
</property>


  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
    <description>The ZooKeeper client port to which HBase clients will     connect</description>
  </property>

  <property>
<name>hbase.zookeeper.quorum</name>
<value>alnnimb01.aln.experian.com</value>
<description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".</description>

Joseph McCullough Joseph McCullough · Accepted Answer · 2012-02-13T14:02:20

Turns out that the Map/Reduce submitted by Hive is trying to connect to zookeeper at 'localhost', regardless of how the zookeeper.quorom is setup in the config file. I changed /etc/hosts to have to the alias 'localhost' point to the IP of my master node and the connection to zookeeper is maintained. Still looking for a better resolution, but this will work for now.

Hive/HBase Integration - Zookeeper Session Closes Immediately

4 Answers