hbase Regionserver start, zookeeper start but hmaster is not starting (regionserver.HRegionServer: Failed construction RegionServer)

Question

Hbase zookeeper start, regionserver start on multi node cluster but hmaster is not starting and generating below log file.

hbase-site.xml snap

<configuration>

        <property>
                <name>hbase.master</name>
                <value>namenode:60000</value>
        </property>

        <property>
                <name>hbase.rootdir</name>
                <value>hdfs://namenode:9001</value>
        </property>

        <property>
                <name>hbase.cluster.distributed</name>
                <value>true</value>
        </property>

        <property>
                <name>hbase.zookeeper.quorum</name>
                <value>datanode</value>
        </property>

        <property>
                <name>hbase.zookeeper.property.dataDir</name>
                <value>/hadoop2/zookeeper</value>
        </property>

        <property>
                <name>hbase.zookeeper.property.clientPort</name>
                <value>2181</value>
        </property>

</configuration>

Two machines datanode and namenode:

on datanode: jps commands shows

10977 HRegionServer
10810 HQuorumPeer
1675 DataNode

on namenode: jps command show

12017 ResourceManager
2353 NameNode
14904 Jps
11326 Jps

Below is the snap of hbase-root-master-namenode.log

2018-09-12 09:52:23,430 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer
java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilder
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:635)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
        at org.apache.hadoop.hbase.util.CommonFSUtils.getRootDir(CommonFSUtils.java:358)
        at org.apache.hadoop.hbase.util.CommonFSUtils.isValidWALRootDir(CommonFSUtils.java:407)
        at org.apache.hadoop.hbase.util.CommonFSUtils.getWALRootDir(CommonFSUtils.java:383)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeFileSystem(HRegionServer.java:691)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:600)
        at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:484)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2965)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2983)
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.SamplerBuilder
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 25 more
2018-09-12 09:52:23,432 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2972)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236

I am trying to connect it from last 4 days install many time hbase. Please help me out, all the setup is on Ubuntu 16.

Below is regionserver log file:

hbase-root-regionserver-datanode.log

2018-09-12 09:52:25,989 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=datanode:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1c12f3ee
2018-09-12 09:52:26,006 INFO  [main-SendThread(datanode:2181)] zookeeper.ClientCnxn: Opening socket connection to server datanode/192.168.1.134:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-12 09:52:26,023 INFO  [main-SendThread(datanode:2181)] zookeeper.ClientCnxn: Socket connection established to datanode/192.168.1.134:2181, initiating session
2018-09-12 09:52:26,079 INFO  [main-SendThread(datanode:2181)] zookeeper.ClientCnxn: Session establishment complete on server datanode/192.168.1.134:2181, sessionid = 0x165cc0408850000, negotiated timeout = 90000
2018-09-12 09:52:26,149 INFO  [main] util.log: Logging initialized @3383ms
2018-09-12 09:52:26,233 INFO  [main] http.HttpRequestLog: Http request log for http.requests.regionserver is not defined
2018-09-12 09:52:26,252 INFO  [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
2018-09-12 09:52:26,252 INFO  [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter)
2018-09-12 09:52:26,255 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver
2018-09-12 09:52:26,255 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2018-09-12 09:52:26,255 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2018-09-12 09:52:26,284 INFO  [main] http.HttpServer: Jetty bound to port 16030
2018-09-12 09:52:26,286 INFO  [main] server.Server: jetty-9.3.19.v20170502
2018-09-12 09:52:26,327 INFO  [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@675ffd1d{/logs,file:///hbase/logs/,AVAILABLE}
2018-09-12 09:52:26,328 INFO  [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@30506c0d{/static,file:///hbase/hbase-webapps/static/,AVAILABLE}
2018-09-12 09:52:26,488 INFO  [main] handler.ContextHandler: Started o.e.j.w.WebAppContext@6a0ac48e{/,file:///hbase/hbase-webapps/regionserver/,AVAILABLE}{file:/hbase/hbase-webapps/regionserver}
2018-09-12 09:52:26,498 INFO  [main] server.AbstractConnector: Started ServerConnector@f84967f{HTTP/1.1,[http/1.1]}{0.0.0.0:16030}
2018-09-12 09:52:26,498 INFO  [main] server.Server: Started @3733ms

Below is zookeeper log file:hbase-root-zookeeper-datanode.log

2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:host.name=datanode
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:java.version=1.8.0_181
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:java.home=/usr/local/jdk1.8.0_181/jre
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: l-4.0.23.Final.jar:/hbase/bin/../lib/org.eclipse.jdt.core-3.8.2.v20130121.jar:/hbase/bin/../lib/osgi-resource-locator-1.0.1.jar:/hbase/bin/../lib/paranamer-2.3.jar:/hbase/bin/../lib/protobuf-java-2.5.0.jar:/hbase/bin/../lib/snappy-java-1.0.5.jar:/hbase/bin/../lib/spymemcached-2.12.2.jar:/hbase/bin/../lib/validation-api-1.1.0.Final.jar:/hbase/bin/../lib/xmlenc-0.52.jar:/hbase/bin/../lib/xz-1.0.jar:/hbase/bin/../lib/zookeeper-3.4.10.jar:/hbase/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/hbase/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/hbase/bin/../lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/hbase/bin/../lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/hbase/bin/../lib/client-facing-thirdparty/log4j-1.2.17.jar:/hbase/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/hbase/bin/../lib/client-facing-thirdparty/htrace-core-3.1.0-incubating.jar:/hbase/bin/../lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:java.compiler=<NA>
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:os.name=Linux
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:os.arch=amd64
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:os.version=4.13.0-46-generic
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:user.name=root
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:user.home=/root
2018-09-12 09:52:21,017 INFO  [main] server.ZooKeeperServer: Server environment:user.dir=/root
2018-09-12 09:52:21,026 INFO  [main] server.ZooKeeperServer: tickTime set to 3000
2018-09-12 09:52:21,026 INFO  [main] server.ZooKeeperServer: minSessionTimeout set to -1
2018-09-12 09:52:21,026 INFO  [main] server.ZooKeeperServer: maxSessionTimeout set to 90000
2018-09-12 09:52:21,037 INFO  [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
2018-09-12 09:52:26,016 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /192.168.1.134:44004
2018-09-12 09:52:26,028 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /192.168.1.134:44004
2018-09-12 09:52:26,028 INFO  [SyncThread:0] persistence.FileTxnLog: Creating new log file: log.3
2018-09-12 09:52:26,077 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x165cc0408850000 with negotiated timeout 90000 for client /192.168.1.134:44004

I've seen this error posted here before, and it went away when they downgraded version from 2.1 to 2.0.2 — OneCricketeer

sOliver sOliver · Accepted Answer · 2018-10-17T14:28:12

In my installation I have:

Ubuntu 18.04
Hadoop 3.1.1
HBase 2.1.0

The same error was discussed in this question.

While starting HBase Master Daemon I've got the same error:

2018-10-17 17:53:17,058 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilder
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:635)  at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) ...

I've guess that the error happens because HBase master use jars from HADOOP_HOME and can't find there htrace-core library.

I've check what htrace jars are located in HBASE_HOME and HADOOP_HOME:

$ find $HBASE_HOME/ -type f -name 'htrace-core*' -ls
   689258   1472 -rw-r--r--   1 hadoop   hadoop    1506370 июл  6 18:28 /opt/hbase-2.1.0/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar
   689260   1444 -rw-r--r--   1 hadoop   hadoop    1475955 июл  6 18:33 /opt/hbase-2.1.0/lib/client-facing-thirdparty/htrace-core-3.1.0-incubating.jar

$ find $HADOOP_HOME/ -type f -name 'htrace-core*' -ls
   672815   1444 -rw-r--r--   1 hadoop   hadoop    1475955 авг  2 09:50 /opt/hadoop-3.1.1/share/hadoop/yarn/timelineservice/lib/htrace-core-3.1.0-incubating.jar
   673317   1468 -rw-r--r--   1 hadoop   hadoop    1502280 авг  2 09:28 /opt/hadoop-3.1.1/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar
   656456   1444 -rw-r--r--   1 hadoop   hadoop    1475955 июл  6 18:33 31 /opt/hadoop-3.1.1/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar

So, I put htrace-core-3.1.0-incubating.jar from HBASE_HOME into $HADDOP_HOME/share/hadoop/common/lib.

And I've got the next error:

2018-10-17 18:13:18,380 ERROR [Thread-14] master.HMaster: Failed to become active master                                                                       
java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem
 does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config
 value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.                                                                                    
        at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1044)                                              
        at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:383)                                             
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:545)                                                               
        at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1325)                                                                   
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:871)                                                           
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2109)                                                                  
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:566)                                                                               
        at java.lang.Thread.run(Thread.java:748)

As said here all what we need is to simply recompile HBase by ourselvs or to put property into hbase-site.xml:

<property>
  <name>hbase.unsafe.stream.capability.enforce</name>
  <value>false</value>
</property>

I took the second way. And Hbase was started up.

hbase Regionserver start, zookeeper start but hmaster is not starting (regionserver.HRegionServer: Failed construction RegionServer)

2 Answers