Automatically scale Axon's tracking event processors

3
votes

I am using Axon framework 4.0.3 with Spring Boot to have event sourcing, and have one tracking processor which is configured to have multiple segments/threads to process events concurrently:

axon.eventhandling.processors[my_processor].initial-segment-count = 6
axon.eventhandling.processors[my_processor].thread-count = 3

It is meant to have 2 nodes of my_processor using 3 threads each.

However, the problem with this solution is that it's not scalable. I have to know from the very beginning how many nodes and threads I must have, as it's not possible to change it later: if I increase the initial-segment-count and restart the processor, nothing happens. Even worse if I decrease the segment count: events that were meant for the "removed" segments are never processed!

Ideally it should be able to specify just the number of threads that each node should use. After that, when new nodes are added to the processor, the number of segments should scale up accordingly. Similarly if I remove nodes, the number of segments should scale down. Is this possible with Axon, or is it not designed to be scaled this way at all?

1
cqrsevent-sourcingaxon

1 Answers

3
votes

This assumption from your part is completely right - just adjusting these fields is not scalable, at runtime. This is exactly why we have introduced the Split and Merge operations, to split/merge segments at runtime of your Axon application. See this GitHub pull request for it's introduction into the framework.

This feature will be part of Axon 4.1, which will be released today. Do note that if you're only using the framework, this feature does not give you the automatic scaling. It will require implementation from your part, leveraging the provided split and merge API, to make it automatic.

Axon Server on the other hand provides you with a split/merge button in the UI, thus relieving you of the necessity to build this yourself. I am fairly certain Axon Server will also introduce an auto scaling solution eventually, but not as part of release 4.1.

Hope this gives you some background Archie!