In Zookeeper client how to look up offset for a group id / topic?

4
votes

We are using Kafka 2.10 and zookeeper 3.4. Following the info on http://kafka.apache.org/08/documentation.html#distributionimpl I am trying to look up the offset for a group id, topic partition, but find the offset to be empty: ls /consumers/test6/offsets/ViewerLogs/0 ---> returns []

Any suggestion how to access that value?

Thanks

2
apache-zookeeperapache-kafka
Did you try to connect with zookeeper cli to see what the output is? Also 2.10 is not the Kafka version, it says which version of scala this kafka is build with. The latest stable version for kafka is 0.8.1.1 (probably) - user2720864
Sorry, I am not sure I understand what you mean by "to see what the output is". Can you explain pls. Thx - FZF
once the servie is up you can use the ZOO_ROOT/bin.zkCli.sh script to connect to it .. read more here zookeeper.apache.org/doc/r3.3.3/… - user2720864
Thx, but I had connected using zkCli.sh and that's how I was able to get to do "ls /consumers" and so on and so forth..... I have run a consumer with a groupid for partition 0. Trying to see the offset that zookeeper saved from the last offset commit by my consumer. But when I do /consumers/<groupid>/offsets/<topic>/<partition> the offset value shows as []. - FZF
shouldn't it be /[topic]/[broker_id-partition_id] instead [topic]/[partition_id] ? - user2720864

2 Answers

4
votes

To get the offset value, run command get /consumers/test6/offsets/ViewerLogs/0 in zookeeper

1
votes

Take a look at this document:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper

So, the offset will be stored in Zookeeper at:

/[clusterBasePath]/consumers/[groupId]/offsets/[topic]/[partitionId] -> long (offset)

If that znode is empty, it might mean that your consumer group hasn't owned the partitions yet (decided on a mapping from partitions to consumers).

To see the current partition owners in the consumer group, take a look at:

/[clusterBasePath]/consumers/[groupId]/owners/[topic]/[partitionId] -> string (consumerId)

From personal experience, the most common cause of this is that your your consumer group is having trouble owning partitions due to timeouts. On your consumer config, you might want to try increasing rebalance.max.retries to something like 50 (or higher) and rebalance.backoff.ms to something like 5000. Also, check your Zookeeper session timeouts and increase those if necessary.

Depending on what consumer you're using (are you using consumer groups at all?), there's also a chance that you're just not committing your offsets to Zookeeper (this is ok if you don't care too much about fault tolerance). In that case, you won't be able to find the offsets in Zookeeper and will need to get them from your consumer directly.