I am trying to setup HBase in a fully distributed mode: consisting of 1 master and 2 region servers. I have set HBASE_MANAGES_ZK = true in hbase-env.sh. The hadoop cluster is running on the cluster with following configurations:
Master: node-master Regionserver1: node1 Regionserver2: node2
When I am starting HBase, I can see that RegionServers are getting started and HQuorumPeer on master also, but HMaster is not showing.
Please find the logs as below: Master hbase-site.xml
<configuration>
<property>
<name>hbase.master</name>
<value>nodemaster.hbasecluster.com:60000</value>
<description>The host and port that the HBase master runs at.A value of ‘local’ runs the master and a regionserver in a single </description>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://nodemaster.hbasecluster.com:9000/hbase</value>
<description>The directory shared by region servers.</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) </description>
</property>
<property>
<name>hbase.zookeeper.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/usr/local/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
<description>Property from ZooKeeper’s config zoo.cfg. The port at which the clients will connect. </description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>nodemaster.hbasecluster.com</value>
<description>Comma separated list of servers in the ZooKeeper Quorum. </description>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>/hbase/tmp</value>
<description>Temporary directory on the local filesystem.</description>
</property>
</configuration>
/etc/hosts on master
127.0.0.1 localhost
192.168.2.154 nodemaster.hbasecluster.com node-master
192.168.2.186 node1.hbasecluster.com node1
192.168.2.187 node2.hbasecluster.com node2
Logs on regionserver1
Fri Aug 17 12:32:15 IST 2018 Starting regionserver on node1.hbasecluster.com
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15701
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 10000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15701
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
2018-08-17 12:32:15,420 INFO [main] regionserver.HRegionServer: STARTING executorService HRegionServer
2018-08-17 12:32:15,422 INFO [main] util.VersionInfo: HBase 2.1.0
2018-08-17 12:32:15,422 INFO [main] util.VersionInfo: Source code repository git://zhangduo-Gen8/home/zhangduo/hbase/code revision=e1673bb0bbfea21d6e5dba73e013b09b8b49b89b
2018-08-17 12:32:15,422 INFO [main] util.VersionInfo: Compiled by zhangduo on Tue Jul 10 17:26:48 CST 2018
2018-08-17 12:32:15,422 INFO [main] util.VersionInfo: From source with checksum c8fb98abf2988c0490954e15806337d7
2018-08-17 12:32:15,703 INFO [main] util.ServerCommandLine: hbase.tmp.dir: /tmp/hbase-root
2018-08-17 12:32:15,703 INFO [main] util.ServerCommandLine: hbase.rootdir: hdfs://nodemaster.hbasecluster.com:9000/hbase
2018-08-17 12:32:15,703 INFO [main] util.ServerCommandLine: hbase.cluster.distributed: true
2018-08-17 12:32:15,703 INFO [main] util.ServerCommandLine: hbase.zookeeper.quorum: nodemaster.hbasecluster.com
2018-08-17 12:32:15,703 INFO [main] util.ServerCommandLine: env:HBASE_LOGFILE=hbase-root-regionserver-node1.hbasecluster.com.log
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:LANG=en_US.UTF-8
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:XDG_SESSION_ID=182
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:MAIL=/var/mail/root
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:LOGNAME=root
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:HBASE_REST_OPTS=
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:PWD=/root
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:SHELL=/bin/bash
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:HBASE_IDENT_STRING=root
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/tmp/hbase-root-regionserver.znode
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:SSH_CLIENT=192.168.2.154 46760 22
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:HBASE_LOG_PREFIX=hbase-root-regionserver-node1.hbasecluster.com
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:HBASE_LOG_DIR=/root/install/hbase-2.1.0/bin/../logs
2018-08-17 12:32:15,704 INFO [main] util.ServerCommandLine: env:USER=root
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: root/install/hbase-2.1.0/bin/../lib/spymemcached-2.12.2.jar:/root/install/hbase-2.1.0/bin/../lib/validation-api-1.1.0.Final.jar:/root/install/hbase-2.1.0/bin/../lib/xmlenc-0.52.jar:/root/install/hbase-2.1.0/bin/../lib/xz-1.0.jar:/root/install/hbase-2.1.0/bin/../lib/zookeeper-3.4.10.jar:/root/install/hbase-2.1.0/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/root/install/hbase-2.1.0/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/root/install/hbase-2.1.0/bin/../lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/root/install/hbase-2.1.0/bin/../lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/root/install/hbase-2.1.0/bin/../lib/client-facing-thirdparty/log4j-1.2.17.jar:/root/install/hbase-2.1.0/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/root/install/hbase-2.1.0/bin/../lib/client-facing-thirdparty/htrace-core-3.1.0-incubating.jar:/root/install/hbase-2.1.0/bin/../lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:HBASE_MANAGES_ZK=true
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:SSH_CONNECTION=192.168.2.154 46760 192.168.2.186 22
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:HBASE_AUTOSTART_FILE=/tmp/hbase-root-regionserver.autostart
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:HBASE_NICENESS=0
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:HBASE_OPTS= -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/root/install/hbase-2.1.0/bin/../logs -Dhbase.log.file=hbase-root-regionserver-node1.hbasecluster.com.log -Dhbase.home.dir=/root/install/hbase-2.1.0/bin/.. -Dhbase.id.str=root -Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:HBASE_SECURITY_LOGGER=INFO,RFAS
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:XDG_RUNTIME_DIR=/run/user/0
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS=
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:HBASE_HOME=/root/install/hbase-2.1.0/bin/..
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:SHLVL=3
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:HOME=/root
2018-08-17 12:32:15,705 INFO [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4
2018-08-17 12:32:15,706 INFO [main] util.ServerCommandLine: vmName=OpenJDK 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.171-b11
2018-08-17 12:32:15,707 INFO [main] util.ServerCommandLine: vmInputArguments=[-Dproc_regionserver, -XX:OnOutOfMemoryError=kill -9 %p, -XX:+UseConcMarkSweepGC, -Dhbase.log.dir=/root/install/hbase-2.1.0/bin/../logs, -Dhbase.log.file=hbase-root-regionserver-node1.hbasecluster.com.log, -Dhbase.home.dir=/root/install/hbase-2.1.0/bin/.., -Dhbase.id.str=root, -Dhbase.root.logger=INFO,RFA, -Dhbase.security.logger=INFO,RFAS]
2018-08-17 12:32:21,194 INFO [main] metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2018-08-17 12:32:21,245 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-08-17 12:32:21,489 INFO [main] regionserver.RSRpcServices: regionserver/node1:16020 server-side Connection retries=45
2018-08-17 12:32:21,503 INFO [main] ipc.RpcExecutor: Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, maxQueueLength=300, handlerCount=30
2018-08-17 12:32:21,505 INFO [main] ipc.RpcExecutor: Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=300, handlerCount=20
2018-08-17 12:32:21,505 INFO [main] ipc.RpcExecutor: Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=300, handlerCount=3
2018-08-17 12:32:21,639 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService
2018-08-17 12:32:21,832 INFO [main] io.ByteBufferPool: Created with bufferSize=64 KB and maxPoolSize=1.88 KB
2018-08-17 12:32:21,937 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer
java.lang.UnsupportedOperationException: Constructor threw an exception for org.apache.hadoop.hbase.ipc.NettyRpcServer
at org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:66)
at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:45)
at org.apache.hadoop.hbase.ipc.RpcServerFactory.createRpcServer(RpcServerFactory.java:66)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.createRpcServer(RSRpcServices.java:1271)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:1238)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:1191)
at org.apache.hadoop.hbase.regionserver.HRegionServer.createRpcServices(HRegionServer.java:733)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:571)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2991)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:63)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3009)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:58)
... 17 more
Caused by: org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newIOException(Errors.java:117)
at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.bind(Socket.java:285)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel.doBind(AbstractEpollChannel.java:714)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollServerSocketChannel.doBind(EpollServerSocketChannel.java:70)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1283)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:989)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
at org.apache.hbase.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:364)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
2018-08-17 12:32:21,940 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: Failed construction of Regionserver: class org.apache.hadoop.hbase.regionserver.HRegionServer
at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2994)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:63)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3009)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2991)
... 5 more
Caused by: java.lang.UnsupportedOperationException: Constructor threw an exception for org.apache.hadoop.hbase.ipc.NettyRpcServer
at org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:66)
at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:45)
at org.apache.hadoop.hbase.ipc.RpcServerFactory.createRpcServer(RpcServerFactory.java:66)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.createRpcServer(RSRpcServices.java:1271)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:1238)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:1191)
I at org.apache.hadoop.hbase.regionserver.HRegionServer.createRpcServices(HRegionServer.java:733) at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:571) ... 10 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:58) ... 17 more Caused by: org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newIOException(Errors.java:117) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.bind(Socket.java:285) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel.doBind(AbstractEpollChannel.java:714) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollServerSocketChannel.doBind(EpollServerSocketChannel.java:70) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1283) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:989) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) at org.apache.hbase.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:364) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748)
regionserver hbase-site.xml
<configuration>
<property>
<name>hbase.master</name>
<value>nodemaster.hbasecluster.com:60000</value>
<description>The host and port that the HBase master runs at.A value of ‘local’ runs the master and a regionserver in a single </description>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://nodemaster.hbasecluster.com:9000/hbase</value>
<description>The directory shared by region servers.</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) </description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
<description>Property from ZooKeeper’s config zoo.cfg. The port at which the clients will connect. </description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>nodemaster.hbasecluster.com</value>
<description>Property from ZooKeeper’s config zoo.cfg. The port at which the clients will connect. </description>
</property>
<property>
<name>hbase.zookeeper.distributed</name>
<value>true</value>
</property>
</configuration>
/etc/hosts file in regionserver1
127.0.0.1 localhost
192.168.2.154 nodemaster.hbasecluster.com node-master
192.168.2.186 node1.hbasecluster.com node1
192.168.2.187 node2.hbasecluster.com node2
Master node jps output:
19717 SecondaryNameNode
20441 HQuorumPeer
20781 Jps
19470 NameNode
19887 ResourceManager
regionserver jps output:
28404 NodeManager
28185 DataNode
28844 Jps
28687 HRegionServer
EDIT: I was trying to run ./bin/start-hbase.sh. When I used the commands ./bin/hbase-daemon.sh start master I get the following error in my master logs.
2018-08-20 11:50:42,742 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer
java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilder
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:635)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.hadoop.hbase.util.CommonFSUtils.getRootDir(CommonFSUtils.java:358)
at org.apache.hadoop.hbase.util.CommonFSUtils.isValidWALRootDir(CommonFSUtils.java:407)
at org.apache.hadoop.hbase.util.CommonFSUtils.getWALRootDir(CommonFSUtils.java:383)
at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeFileSystem(HRegionServer.java:691)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:600)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:484)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2965)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2983)
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.SamplerBuilder
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 25 more
2018-08-20 11:50:42,744 ERROR [main] master.HMasterCommandLine: Master exiting
The zookeeper was able to create connections to the slaves and the Region servers are running on each slave.
