Kafka being able to write to a topic beyond retention period

Question

What is the behavior of topics that go beyond their retention period? Is it possible to write to them after their retention policy period? I have been trying to write to them on confluent-kafka-dotnet (a .NET library) but when I run a consumer I don't see any messages streaming through. The value of the partition is just an empty string array. Is it possible to write to a topic beyond the retention time or does the topic get disabled?

using (var consumer = new Consumer(config))
{
    consumer.Assign(new List<TopicPartitionOffset> { new TopicPartitionOffset(topicName, 0, 0) });

    while (true)
    {
        Message msg;
        if (consumer.Consume(out msg))
        {
            Console.WriteLine("Topic: {0} Response: P{1},O{1} :{3}", msg.Topic, msg.Partition, msg.TopicPartitionOffset, Encoding.UTF8.GetString(msg.Value));
        }
    }
}

Gwen Shapira Gwen Shapira · Accepted Answer · 2017-07-28T14:07:28

Kafka's retention period applies to past events, not event you are writing right now. You should be able to see new messages no matter what your retention policy is.

Example: A topic with 24h retention is created at midnight, july 1st. On 23:50, July 1st, all the data you wrote during the entire day is still there. At 01:00, July 2nd, only data that was written after 1:00am, July 1st will be stored (last 24h), earlier events are "purged".

If your consumer can't see new messages, it is one of the two: 1. Are your messages being written at all? (Kafka has a file-dump utility to help check) 2. When does your consumer start reading? If your consumer starts at the end of the topic, it may miss earlier messages. If it starts at the beginning, it will get all messages in topic. "auto.offset.reset" configuration controls this. "--from-beginning" if you are using the console consumer.

Kafka being able to write to a topic beyond retention period

1 Answers