OpenTSDB integration with kerberized HBase

Question

I want to do some POC on OpenTSDB. I have installed OpenTSDB as per the installation instruction, but having a tough time starting it. I am using HDP environment which is Kerberos enabled, I am integrating OpenTSDB with Kerberized HBase but facing below exception. If anybody have integrated OpenTSDB with Kerberos HBase, please guide..

Exception:

 2017-06-07 14:07:14,254 INFO  [main-SendThread(ZKIP1:2181)] ClientCnxn: Opening socket connection to server ZKIP1/192.xxx.xxx.xxx:2181. Will not attempt to authenticate using SASL (unknown error)

Console O/P:

    2017-06-07 14:07:14,233 INFO  [main] ZooKeeper: Client environment:java.library.path=/usr/local/hawq/./lib:/usr/local/hawq/./ext/python/lib::/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    2017-06-07 14:07:14,233 INFO  [main] ZooKeeper: Client environment:java.io.tmpdir=/tmp
    2017-06-07 14:07:14,233 INFO  [main] ZooKeeper: Client environment:java.compiler=<NA>
    2017-06-07 14:07:14,233 INFO  [main] ZooKeeper: Client environment:os.name=Linux
    2017-06-07 14:07:14,233 INFO  [main] ZooKeeper: Client environment:os.arch=amd64
    2017-06-07 14:07:14,233 INFO  [main] ZooKeeper: Client environment:os.version=2.6.32-431.29.2.el6.x86_64
    2017-06-07 14:07:14,234 INFO  [main] ZooKeeper: Client environment:user.name=user
    2017-06-07 14:07:14,234 INFO  [main] ZooKeeper: Client environment:user.home=/home/user
    2017-06-07 14:07:14,234 INFO  [main] ZooKeeper: Client environment:user.dir=/home/user/opentsdb-2.3.0
    2017-06-07 14:07:14,235 INFO  [main] ZooKeeper: Initiating client connection, connectString=ZKIP1:2181,ZKIP3:2181,ZKIP2:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@63d6a00c
 2017-06-07 14:07:14,254 INFO  [main-SendThread(ZKIP1:2181)] ClientCnxn: Opening socket connection to server ZKIP1/192.xxx.xxx.xxx:2181. Will not attempt to authenticate using SASL (unknown error)
    2017-06-07 14:07:14,257 INFO  [main] HBaseClient: Need to find the -ROOT- region
    2017-06-07 14:07:14,263 INFO  [main-SendThread(ZKIP1:2181)] ClientCnxn: Socket connection established to ZKIP1/192.xxx.xxx.xxx:2181, initiating session
    2017-06-07 14:07:14,270 INFO  [main-SendThread(ZKIP1:2181)] ClientCnxn: Session establishment complete on server ZKIP1/192.xxx.xxx.xxx:2181, sessionid = 0x35ae21cf9870828, negotiated timeout = 5000
    2017-06-07 14:07:14,285 ERROR [main-EventThread] HBaseClient: The znode for the -ROOT- region doesn't exist!
    2017-06-07 14:07:15,310 ERROR [main-EventThread] HBaseClient: The znode for the -ROOT- region doesn't exist!
    2017-06-07 14:07:16,329 ERROR [main-EventThread] HBaseClient: The znode for the -ROOT- region doesn't exist!
    2017-06-07 14:07:17,349 ERROR [main-EventThread] HBaseClient: The znode for the -ROOT- region doesn't exist!
    2017-06-07 14:07:18,370 ERROR [main-EventThread] HBaseClient: The znode for the -ROOT- region doesn't exist!

OpenTSDB uses "AsyncHBase" client, which is not supported by Apache HBase project, and did not support Kerberos until recently -- cf. opentsdb.github.io/asynchbase/docs/build/html/… — Samson Scharfrichter
I am referring github.com/OpenTSDB/opentsdb/issues/683 ..Can anyone help me on how should I pass -Djava.security.auth.login.config=/home/user/opentsdb/opentsdb.jaas command with ./build/tsdb tsd — nilesh1212
I think my kerberos issue resolved now, I can see in my logs as successful Login. 2017-06-12 16:12:59,756 INFO [main-SendThread(ZK1:2181)] Login: successfully logged in. 2017-06-12 16:12:59,771 INFO [main-SendThread(ZK1:2181)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. 2017-06-12 16:12:59,922 INFO [Thread-1] Login: TGT valid starting at: Mon Jun 12 14:06:15 CEST 2017. But now I am facing HBaseClient: The znode for the -ROOT- region doesn't exist!. Any Idea on this error. — nilesh1212

nilesh1212 nilesh1212 · Accepted Answer · 2017-06-13T09:06:31

I am able to connect to kerberos HBase using OpenTSDB. I am listing the configuration changes I have done to connect to kerberos HBase.

1) Configuration changes in 'opentsdb.conf',please make sure this file is in path when you will run tsdb command.

tsd.network.port = 4242
tsd.storage.hbase.zk_basedir = /hbase-secure
tsd.storage.hbase.zk_quorum = ZKhostname1,ZKhostname2,ZKhostname3
hbase.security.auth.enable=true
hbase.security.authentication=kerberos
hbase.kerberos.regionserver.principal=hbase/[email protected]
hbase.sasl.clientconfig=Client

2) Inorder to avoid classpath/JVM issues, I have done all the classpath and jvm configuration in 'tsdb' file located on '/home/user/opentsdb-2.3.0/build'

# Add the src dir so we can find logback.xml 
CLASSPATH="$CLASSPATH:$abs_srcdir/src:/usr/hdp/2.4.2.0-258/zookeeper/lib/:/usr/hdp/2.4.2.0-258/zookeeper/:/etc/hadoop/2.4.2.0-258/0/:/usr/hdp/2.4.2.0-258/hbase/:/etc/hbase/2.4.2.0-258/0/:/home/user/phoenix-4.4.0-HBase-1.1-client.jar"

JVMARGS=${JVMARGS-'-Djava.security.krb5.conf=/etc/krb5.conf -Dhbase.security.authentication=kerberos -Dhbase.kerberos.regionserver.principal=hbase/[email protected] -Dhbase.rpc.protection=authentication -Dhbase.sasl.clientconfig=Client -Djava.security.auth.login.config=/home/user/opentsdb-jaas.conf -enableassertions -enablesystemassertions'}

3)opentsdb-jaas.conf file

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true;
}

4) Command used to run opentsdb tsd instance, please note we can override some of the properties configured in 'opentsdb.conf' on command line.

./build/tsdb tsd --port=4242 --staticroot=/home/user/opentsdb-2.3.0/build/staticroot --cachedir=/home/user/opentsdb-2.3.0/build/cache-dir --zkquorum=ZKhostname1:2181,ZKhostname2:2181,ZKhostname3:2181

OpenTSDB integration with kerberized HBase

2 Answers