How to limit memory consumption for Akka.net system?

Question

We have a windows service using Akka.net to process messages. Specifically, we have 10 identical private MSMQ queues and 10 coordinator actors. We assigned each actor to process one queue and send the result messages to 10 child worker actors (using round robin router) to insert into the database. As the number of messages in the private queue increases the child actors cannot keep up. The memory consumption is growing as fast as the number of messages sent. We have two problems:

1. How to limit the memory growth? Can we monitor the child actor inbox to make sure we don't overload it.

2. 1 coordinator actor creates 10 worker actors, so we have 100 child worker actors total. Is this correct?

Answer 1

What I think is a problem here is not a memory consumption, but the approach to workflow itself.

From what you're saying I can assume, that you're pulling the messages from MSMQ, forwarding them immediately to workers, and then pulling more data, without waiting for the workers to actually finish their job. This is traditional problem with push-based data flow with no backpressure mechanism attached.

The most trivial way to solve it, is to change dataflow to pull-based model - in that case coordinator actor won't try to pull data from the MSMQ unless it has been notified by one of the workers, that they finished their work and are waiting for more data (in that case you won't need to put them behind a router too). Notification can be expressed as a message send from worker, when it finishes processing current message. Downside of that model is that you'll have idle workers waiting for MSMQ messages to be picked.

More advanced concepts include demand-driven model (backpressure), which can utilize message buffering - so you can prefetch some amount of data ahead, and then refill it while it's processed. Also this way no worker will stay idle while waiting for coordinator to pick the next message from the queue. You can read more about idea behind backpressure concept here.

PS: Once Akka.Streams module will come out, you'll get build-in mechanisms which works with backpressure principles built-in.