HBase Error IllegalStateException when starting Master: hsync

Question

I'm trying to install HBase on a hadoop cluster and can't figure out why the HMaster fails to initiate when called from start-hbase.sh. The log files indicate an issue with hsync. I have confirmed that zookeeper is running correctly in distributed mode,and I have not had any issues working with the hadoop cluster through spark. When attempting to start HBase the region servers start on all data nodes.

I have hadoop version 3.0.0, zookeeper 3.4.11 and hbase 2.0.0 beta 1.

I have cleared out the zookeeper hbase node and hdfs hbase directory and restarted with no luck. I have the myid zookeeper files stored in the same directory as the .pid files for zookeeper and hbase region servers /usr/local/zookeeper/data/

Below I have an excerpt of the hbase master log file pertaining to the hsync exception.

2018-02-08 17:26:54,786 ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Failed to become active master java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it. at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532) at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547) at java.lang.Thread.run(Thread.java:748) 2018-02-08 17:26:54,787 ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Master server abort: loaded coprocessors are: [] 2018-02-08 17:26:54,787 ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Unhandled exception. Starting shutdown. java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it. at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532) at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547) at java.lang.Thread.run(Thread.java:748)

Below is the hbase-site.xml file

<property>
    <name>hbase.master</name>
    <value>localhost:60000</value>
    <description>The host and port that the HBase master runs at.A value of 'lo$
</property>
<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
    <description>Property from ZooKeeper's config zoo.cfg.The port at which the$
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>MasterNode,Slave1,Slave2</value>
    <description>Comma separated list of servers in the ZooKeeper Quorum.For ex$
</property>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://MasterNode:9000/hbase</value>
</property>
<property>
    <name>hbase.tmp.dir</name>
    <value>/usr/local/zookeeper/data</value>
</property>
<property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/usr/local/zookeeper/data</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
</property>

Below are the alterations made to the hbase-env.sh file settings

export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HBASE_HOME=/usr/local/hadoop/HBase
export HADOOP_HOME=/usr/local/hadoop
export HBASE_MANAGES_ZK=false
export HBASE_PID_DIR=/usr/local/zookeeper/data

Below is the core-site.xml

<property>
    <name>hadoop.tmp.dir</name>
    <value>/app/hadoop/tmp</value>
</property>
<property>
    <name>fs.default.name</name>
    <value>hdfs://MasterNode:9000</value>
</property>

Below is the zoo.cfg

tickTime=2000
dataDir=/usr/local/zookeeper/data
dataLogDir=/usr/local/zookeeper/log
clientPort=2181
initLimit=5
syncLimit=2
server.1=MasterNode:2888:3888
server.2=Slave1:2888:3888
server.3=Slave2:2888:3888

Below is the .bashrc settings related to the hadoop setup

export HADOOP_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_PREFIX=/usr/local/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONFDIR=$HADOOP_HOME/etc/hadoop
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/
export PATH=$PATH:$HADOOP_HOME/bin
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HBASE_HOME=/usr/local/hadoop/HBase
export PATH=$PATH:$HBASE_HOME/bin

Answer 1

I encountered the same problem and my solution is as follows: Edit hbase-site.xml and add the following:

<property>
  <name>hbase.unsafe.stream.capability.enforce</name>
  <value>false</value>
</property>

The property controls whether HBase will check for stream capabilities (hflush/hsync).More details refer to the official website.

Answer 2

I encountered exactly the same problem as you did. I was running hbase-2.0.1 and hadoop-2.9.1 when I encountered the problem. My solution was I downloaded a later version of hbase(hbase-0.98.13-hadoop2). Probably there is version incompatibility between the hadoop and hbase version you are using.

Answer 3

I had similar issue while using HBase 2.2.1 with Hadoop 3.2.1 on a 3-N Hbase cluster while using HDFS as the base.rootdir.

Solution

1. Set the configuration to hbase.unsafe.stream.capability.enforce to false in hbase-site.xml
2. Not sure of this was necessary , also changed the hbase.wal.dir to an HDFS directory outside the hbase.rootdir

Answer 4

Ideally we should not be setting hbase.unsafe.stream.capability.enforce to false. This is risky. Instead just copy some of your hadoop binaries to hbase lib.

For ex, If your HBase is 2.2.5 build with Hadoop 2.8.5. (You can grep for hadoop binary using ls hadoop*2.8.5*.jar files under hbase libs, which will tell you Hadoop version using which HBase is built).

Curent HBase shows Hadoop version as, when hbase.unsafe.stream.capability.enforce is set to false

Hadoop Version 2.8.5, revision=826afbeae31ca687bc2f8471dc841b66ed2c6704

Now i am trying to use HBase 2.2.5 (Prebuilt with Hadoop 2.8.5) with Hadoop 2.9.2

Steps to resolve would be

1. Go to $HBASE_HOME/lib
2. List all files for hadoop ls hadoop*2.8.5*.jar and make a note.
3. Remove old hadoop libs rm hadoop*2.8.5*.jar.
4. Find noted binaries from your hadoop extracted folder, if available? otherwise download binaries from maven repository Hadoop Binary Home as per your hadoop version.
5. Go to $HBASE_HOME/lib, copy all downloaded binaries.
6. Start HBase, Based on my example HBase UI will show hadoop version as below

Hadoop Version 2.9.2, revision=826afbeae31ca687bc2f8471dc841b66ed2c6704

Overall, You have make sure HBase lib has all binaries of only the Hadoop you are trying to use with.

Answer 5

It appears you downloaded HBase-2.0-beta1 distribution from HBase mirror sites. The default compilation of HBase is done with default hadoop profile i.e 2.0. HBase lib folder contains hadoop dependencies with hadoop-2.7 versions which doesn't support hsync capabilities. So try building HBase from source by passing -Dhadoop.profile=3.0. This builds Hbase distribution with hadoop-3.0 dependencies. Use this distribution to run HBase.