Fetch last written message offset number from kafka

Question

I'm producing messages to a topic named as default.

My producer partition is RD_KAFKA_PARTITION_UA, When I'm using unassigned partition, how can I get the offset of the last message pushed to default topic.

I don't need offset of consumer, I need offset of the last published message.

I need the offset for crash recovery, human intervention is not intended.

I need the offset using librdkafka, when my program starts back again, based on the offset, I will do some modifications to my program, so I need to fetch it through API only and when needed.

I don't think I can rely on callback,

consider I pushed my message, I received callback and my program crashed.

or

I pushed my message, I was about to poll and my program crashed.

I'm using C language librdkafka library.

Mickael Maison Mickael Maison · Accepted Answer · 2018-10-05T12:41:39

This can easily be done using the delivery report of the producer.

One of the librdkafka example, rdkafka_example.c, demonstrate this feature.

For example:

./rdkafka_example -P -b localhost:9092 -t testtopic -o report
% Type stuff and hit enter to send
hello
del: Success: offset 0
% Message delivered (5 bytes, offset 0, partition 0): hello
hello
% Sent 5 bytes to topic testtopic partition -1
del: Success: offset 1
% Message delivered (5 bytes, offset 1, partition 0): hello
^Cdel: Success: offset 2
% Message delivered (5 bytes, offset 2, partition 0): hello

You see for each message sent, the delivery report contains the offset the produced message.

Basically this example:

sets produce.offset.report=true [1]
registers a delivery report [2]
in the delivery report, access the offset field [3]

Fetch last written message offset number from kafka

1 Answers