kafka 0.11 reset offset for consumer group by --to-datetime

17
votes

I want to reset the offset of kafka consumer group by timestamp. But when I am using following command:

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --to-datetime 2017-11-1907:52:43:00:000 --group <group_name> --topic <topic_name> --execute

I am getting the following error message:

Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

how to reset offset according to time

3
apache-kafkakafka-consumer-api

3 Answers

17
votes

Invoking 

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092
--group test-group --reset-offsets --all-topics --to-datetime 2017-08-04T00:00:00.000

can resets offsets to the earliest ones after the given datetime. Datetime format is yyyy-MM-ddTHH:mm:ss.xxx, 2017-08-04T00:00:00.000 for instance.

You could also reset offsets by duration. See an example below:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 
--group test-group --reset-offsets --all-topics --by-duration PT0H30M0S

--by-duration resets offsets to offset by duration from current timestamp. Format: 'PnDTnHnMnS'.

3
votes

It's not an error, but just a warning - because you specified --bootstrap-server option then changes will affect only consumers that are implemented using new Java API. If you have consumers that are built using other APIs, then you need to specify --zookeeper option instead.

2
votes

Old thread, but this link is helpful if you want to play with kafka offsets.

https://gist.github.com/marwei/cd40657c481f94ebe273ecc16601674b#file-how_to_reset_kafka_consumer_group_offset-md

Highlight:

Kafka 0.11.0.0 (Confluent 3.3.0) added support to manipulate offsets for a consumer group via cli kafka-consumer-groups command.

  1. List the topics to which the group is subscribed bash kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --describe Note the values under "CURRENT-OFFSET" and "LOG-END-OFFSET". "CURRENT-OFFSET" is the offset where this consumer group is currently at in each of the partitions.

  2. Reset the consumer offset for a topic (preview) bash kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest This will print the expected result of the reset, but not actually run it.

  3. Reset the consumer offset for a topic (execute) bash kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest --execute This will execute the reset and reset the consumer group offset for the specified topic back to 0.

  4. Repeat 1 to check if the reset is successful

Note

  • The consumer group must have no running instance when performing the reset. Otherwise the reset will be rejected.

  • There are many other resetting options, run kafka-consumer-groups for details

    • --shift-by
    • --to-current
    • --to-latest
    • --to-offset
    • --to-datetime
    • --by-duration

The command also provides an option to reset offsets for all topics the consumer group subscribes to: --all-topics