Kafka Streams state store backup topic partitioning strategy

0
votes

Kafka guarantees that messages with same key will always go to the same partition. For instance, I have message with the string key: 2329. And two topics t1 and t2. When I perform write of this message it goes into partition 1 in both topics, as expected.

Now the problem itself: I'm using Kafka Streams 0.10.2.0 persistent state store, which automatically creates a backup topic. Now in case of this backup topic message with the key: 2329 goes into another partition (partition 0), which is strange to me.

Has anyone encountered this issue?

1
apache-kafkakafka-producer-apiapache-kafka-streams
Are you using the same serializer? Partition number is determined by the serialized value of the key. If the Kafka Streams Serde is using a different serializer than the Serializer used by your KafkaProducer, that could explain it. – Michal Borowiecki
Serializers are the same, partitioner is default. – user3735823

1 Answers

0
votes

I've found where was the issue.

To increase performance I've skipped repartitioning before write data to the state store. And used another column from the value as the state store key. And it worked until additional topic with enrichment information has been added. So I just forgot to perform repartitioning.