Can not retrieve kafka offset for a topic/group/partition from zookeeper

1
votes

We are running zookeeper 3.3 and kafka 0.8 on separate servers. We are using HL consumers and they access the data in kafka queue as expected, on a restart they pick up form where they left off. So, the consumers behave as expected. The problem is we can't see the offsets in the zookeeper when we use zkCli.sh. For now the consumer is set up for running in one partition only for a topic .
the cmd "ls /consumers/mygrpid/offsets/mytopic/0" returns []. same for "ls /consumers/mygrpid/owners/mytopic", it returns [].

Because the consumer behaves as expected when the consumer is stopped and restarted again (ie. it picks up from the offset it left off last time it ran. we can tell this by looking at the log which gives the offsets it starts with and every time it commits) we know that somewhere zookeeper should be saving the committed offsets for the consumer. My understanding is that the zookeeper keeps track for the HL consumer, and not the kafka broker. Yet the "ls" commands that are supposed to show the offsets show null instead.

Should I be looking at a different place for accessing the offsets? (ultimately, I need to have a script that reports on the offsets for all the consumers.)

Very much appreciate any help or suggestion.

1
apache-zookeeperapache-kafka

1 Answers

4
votes

You should use get instead of ls. ls gets child nodes and in your case /consumers/mygrpid/offsets/mytopic/0 does not have children. But it has a value, so running get /consumers/mygrpid/offsets/mytopic/0 should show you something like this:

47
cZxid = 0x568
ctime = Tue Feb 03 19:08:10 EET 2015
mZxid = 0x568
mtime = Tue Feb 03 19:08:10 EET 2015
pZxid = 0x568
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 2
numChildren = 0

where 47 is the offset value.