Does retention.bytes in Kafka limit the size of a partition per replica or a partition's all replica combined?

0
votes

I have a kafka topic needs to be limited based on size config: retention.bytes. This topic has 4 partitions.

Each partition has 1 leader and 2 replicas on 3 different brokers.

Say the retention.bytes=1,073,741,824

Is it that each replica will be limited to 1GB, which means each partition is limited to 3GB of logs.

or each partition (logs on 3 brokers in total) will be limited to 1GB of logs?

1
apache-kafka

1 Answers

1
votes

The retention.bytes settings applies to each partition.

For example, if set to 1073741824, Kafka will retain at least 1GB of data in each partition. So if you have 4 partitions, the total size of data retained will be 4GB.

All replicas will have the same size. So if you have a topic with 4 partitions and a replication factor of 2 and retention.bytes set to 1GB, the total size (that will be spread onto at least 2 brokers) will be 8GB.