I'm trying to install HBase v1.1.2 (with Hadoop v2.7.2) in standalone mode, and am following the Apache HBase docs as closely as I can, however the configuration seems a bit tricky. Not sure if it's related to the fact that I installed HBase with homebrew as opposed to a direct download via one of their mirror sites. My goal is to get to the point where I can do things like create tables, list existing tables, etc.
I have Java 1.7.0 installed, and I have run "brew install hbase" to install hbase. In Homebrew's hbase directory, I've opened /libexec/conf/hbase-env.sh and I see the following on line 27:
export JAVA_HOME="$(/usr/libexec/java_home)"
When I execute /usr/libexec/java_home in the command line, it outputs /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home. So it appears that there is a valid $JAVA_HOME env variable being set.
In my /libexec/conf/hbase-site.xml configuration file, I've set the following key/value pairs:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///home/richiethomas/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/richiethomas/zookeeper</value>
</property>
</configuration>
In other words, I've made an exact copy of the configuration shown in the HBase docs, only replacing 'testuser' with 'richiethomas' as this is the username I'm using in my command line.
Then in the Homebrew hbase directory, I run `./bin/start-hbase.sh'. I see the following:
starting master, logging to /usr/local/Cellar/hbase/1.1.2/libexec/bin/../logs/hbase-richiethomas-master-MacBook-Pro-5.local.out
Then, I run hbase shell and I see the following:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/1.1.2/libexec/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2017-01-03 17:27:52,056 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.1.2, rcc2b70cf03e3378800661ec5cab11eb43fafe0fc, Wed Aug 26 20:11:27 PDT 2015
>>
After I do this, in a new CLI tab I cat the logs from the path I saw above (/usr/local/Cellar/hbase/1.1.2/libexec/bin/../logs/hbase-richiethomas-master-MacBook-Pro-5.local.out), and I see the following:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/1.1.2/libexec/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Then, back in my HBase shell, I run the following to see if everything worked, and I see the following:
>> list
TABLE
2017-01-03 17:28:05,108 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
2017-01-03 17:28:05,320 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
2017-01-03 17:28:05,625 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
2017-01-03 17:28:06,128 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
2017-01-03 17:28:07,134 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
2017-01-03 17:28:09,144 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
2017-01-03 17:28:13,158 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
ERROR: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
Here is some help for this command:
List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:
hbase> list
hbase> list 'abc.*'
hbase> list 'ns:abc.*'
hbase> list 'ns:.*'
According to the Apache docs:
You do not need to create the HBase data directory. HBase will do this for you. If you create the directory, HBase will attempt to do a migration, which is not what you want.
I interpret this to mean that I do not need to set the following property in hbase-site.xml, expecially since 'hbase' is the default for 'zookeeper.znode.parent':
<property>
<name>zookeeper.znode.parent</name>
<value>/hbase</value>
</property>
The output of jps is the following:
32146 ZooKeeperMain
1364 QuorumPeerMain
83568
34120 Jps
My zoo.cfg file is located in /usr/local/etc/zookeeper/zoo.cfg, and looks like this:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/usr/local/var/run/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
Solutions I've Tried:
I.
I tried replacing the value of zookeeper.property.dataDir in the hbase-site.xml file with /usr/local/var/run/zookeeper/data, but this didn't fix the issue.
II.
When I run zkcli in the command line and run ls /, I only see zookeeper, no hbase directory. I tried running the following command inside the zookeeper shell:
create /hbase my_data
I then see the following:
[zk: localhost:2181(CONNECTED) 4] ls /
[hbase, zookeeper]
Then when I exit and re-run the HBase shell, and try list, I see the following:
>> list
TABLE
ERROR: Can't get master address from ZooKeeper; znode data == null
III.
According to this Quora article, HBase 1.1.2 only supports Hadoop up to versions 2.5.x, so I ran brew uninstall hadoop followed by brew install hadoop25, ran the stop and start hbase scripts, and re-launched the hbase shell. Still no luck.
IV.
I tried uninstalling the Homebrew version and downloading HBase from an Apache mirror site, and repeating the configuration process in hbase-site.xml, but I got the same error (The node /hbase is not in ZooKeeper. It should have been written by the master.).
V.
I uninstalled/reinstalled HBase via Homebrew, and followed the instructions in the "Caveats" section:
==> Caveats
To have launchd start hbase now and restart at login:
brew services start hbase
Or, if you don't want/need a background service you can just run:
/usr/local/opt/hbase/bin/start-hbase.sh
I noticed that after running the above commands, the hbase-site.xml file had been populated with plausible-looking values for hbase.rootdir and hbase.zookeeper.property.dataDir (specifically, file:///usr/local/var/hbase and /usr/local/var/zookeeper, respectively). I also noticed that when I ran brew services start hbase, then ran jps, I didn't see an 'HMaster' process running, but I did see HMaster when I ran /usr/local/opt/hbase/bin/start-hbase.sh. However, I then tried launching hbase shell and typing status, and I got the same error as before (The node /hbase is not in ZooKeeper.).
VI.
I've verified my Hadoop installation is running OK:
MacBook-Pro-5:2.5.2 richiethomas$ hadoop jar /usr/local/Cellar/hadoop25/2.5.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
17/01/03 20:09:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Starting Job
...
Job Finished in 2.156 seconds
Estimated value of Pi is 3.60000000000000000000
It appears there is something wrong with my HBase configuration, but I'm not sure what else to try here.
