Ghost Drillbit (One or more drillbits in cluster have different version: Undefined.)

Question

I have a kubernetes stack with some drills and zookeeper installed.

3 drillbits
1 zookeeper
The drillbits are visible to each other
The zookeeper is visible from each drill container

However, whenever I navigate to any drillbit:8047 I see this one extra "ghost" drillbit:

(screenshot of debug configuration 1 drill : 1 zookeeper)

In this case, no query works - all will silently fail with no error and there is no profile to be found after
zookeeper log is full of these errors:

WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket

What I'm doing wrong?

rudolfdobias rudolfdobias · Accepted Answer · 2019-07-22T23:32:10

At the end I found out that the drill-override.conf was misconfigured.

This error was caused by empty cluster-id in my configuration:

drill.exec: {
  cluster-id: "",
  zk.connect: "my-zookeeper-host:2181",
  ...

That's because I used an env propagation of variables:

drill.exec: {
  cluster-id: "${DRILL_CLUSTER_NAME}",
  zk.connect: "${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT}",
  ...
  ---

  envsubst < core-site.xml.template > ./conf/core-site.xml

...and I had a typo in the $DRILL_CLUSTER_NAME variable.

It took me 3 days to find it. Three bloody days. Would be nice if drill threw some exception in this case or whatever?

Well, maybe this will save someone three days of his life...

Ghost Drillbit (One or more drillbits in cluster have different version: Undefined.)

1 Answers