Akka - java.util.concurrent.timeoutexception - frequent timeouts with thread pool

Question

When I run a load test on API which uses akka sync actor (uses ask pattern with 20 seconds timeout, code snippet below), the API invokes a micro service which interacts with Oracle DB for a get operation.

But during load test(100 request per second) I see lot of timeout exceptions and I don't see any error in the external microservice, nor in Oracle. When fork join executor is used, failures are quite minimal, but with thread pool executor, I see frequent failures.

Code snippet:

Timeout timeout = new Timeout(Duration.create(interaction.getActorTimeOut(), TimeUnit.SECONDS));
Future<Object> future = Patterns.ask(ipsActorSystem.synchronousActor, interactionRequest, timeout);
Object result =  Await.result(future, timeout.duration());

akka config (Fork join executor):

    "default-dispatcher": {
      "attempt-teamwork": "on",
      "default-executor": {
      "fallback": "fork-join-executor"
    },
    "executor": "fork-join-executor",
    "fork-join-executor": {
      "parallelism-factor": 50,
      "parallelism-max": 100,
      "parallelism-min": 10,
      "task-peeking-mode": "FIFO"
    },
    "mailbox-requirement": "",
    "shutdown-timeout": "1s",
    "throughput": 100,
    "throughput-deadline-time": "0ms",
    "type": "Dispatcher"
  },

Thread pool executor:

 "default-dispatcher": {
      "attempt-teamwork": "on",
      "default-executor": {
      "fallback": "thread-pool-executor"
    },
    "executor": "thread-pool-executor",
    "thread-pool-executor": {
        "fixed-pool-size" : 10
    },
    "mailbox-requirement": "",
    "shutdown-timeout": "1s",
    "throughput": 10,
    "throughput-deadline-time": "0ms",
    "type": "Dispatcher"
  },

This might help you. Check my post on the below link.. stackoverflow.com/questions/50940162/akka-actor-messaging-delay/… — VinothNair

Smalltalkguy Smalltalkguy · Accepted Answer · 2020-09-04T21:31:29

From my understanding for a blocking operation you need to have fixed thread pool with throughput as 1. If you want to do this parallel across, then you can configure a Robin Robin pool router with a number of child actors. Something like this

val synchronousActor: ActorRef =  context.actorOf(RoundRobinPool(numOfInstances).props(SynchronousActor.props(
config)), SynchronousActor.name)

You can call this like

val reqF = (synchronousActor ? InteractionRequest).mapTo[ReturnType]
val result = Await.result(reqF, timeout)

Or you can use this guys suggestion (https://github.com/alexandru/scala-best-practices/blob/master/sections/4-concurrency-parallelism.md#46-should-use-a-separate-thread-pool-for-blocking-io)

Akka - java.util.concurrent.timeoutexception - frequent timeouts with thread pool

1 Answers