How to run more than 1 application instances of ktable-ktable joins kafka streams application on single partitioned kafka topics?

0
votes
KTable<Key1, GenericRecord> primaryTable = createKTable(key1, kstream, statestore-name);
KTable<Key2, GenericRecord> childTable1 = createKTable(key1, kstream, statestore-name);
KTable<Key3, GenericRecord> childTable2 = createKTable(key1, kstream, statestore-name);

primaryTable.leftJoin(childTable1, (primary, choild1) -> compositeObject)
            .leftJoin(childTable2,(compositeObject, child2) -> compositeObject, Materialized.as("compositeobject-statestore"))
.toStream().to(""composite-topics)

For my application, I am using KTable-Ktable joins, so that whenever data is received on primary or child stream, it can set it compositeObject with setters and getters for all three tables. These three incoming streams have different keys, but while creating KTable, I make the keys same for all three KTable.

I have all topics with single partition. When I run application on single instance, everything runs fine. I can see compositeObject populated with data from all three tables. All interactive queries also runs fine passing the recordID and local statestore name.

But when I run two instances of same application, I see compositeObject with primary and child1 data but child2 remains empty. Even if i try to make call to statestore using interactive query, it doesn't return anything.

I am using spring-cloud-stream-kafka-streams libraries for writing code.

Please suggest what is the reason it is not setting and what should be a right solution to handle this.

1
apache-kafka-streamsspring-cloud-streamspring-kafka

1 Answers

2
votes

Kafka Streams' scaling model is coupled to the number of input topic partitions. Thus, if your input topics are single partitioned you cannot scale-out. The number of input topic partitions determine your maximum parallelism.

Thus, you would need to create new topics with higher parallelism.