Evolution of kafka-streams topology in production

2
votes

I have a complex topology including KStreams, KTables, joins, rekeying, filters, through, branching, zips etc with live clients on output topics.

All the input, intermidate and output topics are avro based topics.

How do I handle changes to the topology (adding new steps, new input-intermidate-output topics), knowing kafka-streams maintains internal state and changelogs, complying with the following business requirements:

  • no data loss
  • no data duplication (to a degree)
  • no downtime

Is the answer lying somewhere between using kafka-streams-application-reset or creating brand new consumer group (application id)?

1
scalaapache-kafkaapache-kafka-streams

1 Answers

4
votes

At the moment, there is no checkpointing or savepoint type concept in Kafka Streams.

If you need to update the existing applications, it is safe to use the new application.id or reset the existing application.id.

There are scenarios where you need to define a new application.id based on the changes in topology.

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Data+%28Re%29Processing+Scenarios