spring-cloud-stream consumer partition reassignment

Question

Scenario:

1. Running spring-boot project consuming from partition named 'test' that has 10 partitions. Partition assignment happens at 13:00:00
   
2. At ~13:00:30 adding partitions to the topic using:
   ./kafka-topics.sh --alter --zookeeper zookeeper:2181 --topic test --partitions 100
   
3. At ~13:05:30 partition reassignment is triggered.

I ran those steps few times and it looks like that reassignment happens every ~5 minutes.

1. Is there a way to change the reassignment check operation frequency?
   
2. We would like it to be every few seconds. Does this operation is heavy and this is the reason that it happens every 5 minutes? Or it's pretty negligible?

EDIT:

My use case is the following: we have integration tests which boots our microservices. When a consumer of a topic boots first, it creates the topic if it does not exists and the number of partitions it's creating equals to the configured concurrency (10 for example). Then the producer of this topic boots and his configured partitonCount (20 for example) is bigger than the number of created partitions, so spring-cloud-stream adds the missing partitions, in the mean time the consumer assigned partitions , haven't changed and it keeps consuming from the first 10 partitions (1-10). The problem is that the producer is publishing messages to all 20 partitions so messages that are sent to the last 10 partitions (11-20) will not be consumed until the consumer is assigned the new partitions.
This behavior causes problems to our tests and we cannot wait for 5 minutes until all of the partitions are assigned to consumers. Also we would not want to create the topic with the number of desired partitions in advance, and we would like that it will still be handled by spring-cloud-stream.

EDIT 2:

It seems like that the relevant property that controls the "reassignment" is metadata.max.age.ms.

The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

Answer 1

So there are couple of concerns here.

First, "spring-cloud-stream" and/or "spring-kafka" are NOT doing any type of rebalancing, partition reassignment etc. This is all done inside of Kafka. There is a client-side property in Kafka that defaults to 5 min (i believe) if a consumer didn't poll for that much time consider it dead etc. In any event I would refer you to apache-kafka channel to get more information on Kafka internals.

Also, adding partitions, reassignments and rebalancing are expensive operations and should not be attempted without serious consideration of their impact. So, I'd be curious to know what is your use case for constantly adding partitions?