I've setup a Hbase cluster with two nodes and I've noticed the warning "AssignmentManager: STUCK Region-In-Transition" which is not allowing the master to startup.
- Node 1: observepreserve.corp.com (Master / Zookeeper)
- Node 2: knewshoe.corp.com (Region Server)
Why it is happeneing and how to fix it?
Under Hbase UI I can see the below message.
b94eb458bf643b46deaf6b00998d1f95 hbase:namespace,,1542792846910.b94eb458bf643b46deaf6b00998d1f95. state=OPENING, ts=Wed Nov 21 09:39:46 UTC 2018 (PT18M9.696S ago), server=knewshoe.corp.com,16020,1542792833282
Logs:
2018-11-21 09:40:45,900 INFO [ReadOnlyZKClient-observepreserve.corp.com:2181@0x4068418f] zookeeper.ZooKeeper: Session: 0x167359e5ad60006 closed
2018-11-21 09:40:45,900 INFO [ReadOnlyZKClient-observepreserve.corp.com:2181@0x4068418f-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x167359e5ad60006
2018-11-21 09:40:49,266 WARN [master/observepreserve:16000:becomeActiveMaster] master.HMaster: hbase:namespace,,1542792846910.b94eb458bf643b46deaf6b00998d1f95. is NOT online; state={b94eb458bf643b46deaf6b00998d1f95 state=OPENING, ts=1542793186164, server=knewshoe.corp.com,16020,1542792833282}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2018-11-21 09:41:46,095 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=knewshoe.corp.com,16020,1542792833282, table=hbase:namespace, region=b94eb458bf643b46deaf6b00998d1f95
2018-11-21 09:41:53,267 WARN [master/observepreserve:16000:becomeActiveMaster] master.HMaster: hbase:namespace,,1542792846910.b94eb458bf643b46deaf6b00998d1f95. is NOT online; state={b94eb458bf643b46deaf6b00998d1f95 state=OPENING, ts=1542793186164, server=knewshoe.corp.com,16020,1542792833282}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
