Akka actor work handoff with Futures

Question

I'm trying to build a messaging process between akka actors to represent the master giving the worker a job, and keeping a close eye on it. My question is

1. Is what I'm proposing below a reasonable approach, and
2. even if it's not, I'd like to know how it could be done properly with composition of Futures, for the sake of my Future education.

The process I'd like goes something like this

1) Master sends work to Worker with an ask. It expects a reply within 5 seconds, else it considers the worker to have lost it's chance and it will have to enter bidding again.

import context.dispatcher
implicit val timeout = Timeout(5 seconds)
val workCompletedFuture = (worker ? WorkTicket(work)).mapTo[Future[WorkCompleted]]

2a) If the worker doesn't respond within the 5 seconds I'd like the master to send itself a message saying to reallocate the work.

self ! WorkAllocationFailed(work, worker)

2b) If the worker did respond then it gives us a Future[WorkCompleted]. I'd like to wait for that future to be completed for up to, say, 2 minutes.

3a) If Future[WorkCompleted] fails to complete within the timeout then reallocate the work

self ! WorkFailed(work, worker)

3b) If the Future[WorkCompleted] succeeds then collect the result

I've tried creating this logic, but I get in a mess with nested onComplete, and I don't know how to do the timeout on the Future[WorkCompleted]. I tried reading the Akka 2.10 Futures docs, but couldn't figure out a solution.

Answer 1

The general idea, that you have a master that hands off jobs to a pool of workers is a sound pattern.

On the other hand, I don't recommend using Futures when all parts of your system are already actors. Instead of submitting the work using ask, you can just send it via tell. The master then could check periodically for timed out jobs and resubmit them again.

Also, calling onComplete in the body of the actor is very dangerous, as it executes on a potentially different thread. The safe way to communicate with actors is by message passing. If you have a Future and you want something to be done in the actor when the Future is completed, it is better to use the pipe pattern.

There is also a minor bug in your snippet. If your worker actor replies with WorkCompleted, then this is the line you really wanted:

val workCompletedFuture = (worker ? WorkTicket(work)).mapTo[WorkCompleted]

Answer 2

I agree with Endre's answer - all very good points.

How about this:

1) Schedule a message to yourself for the timeout (using system.scheduler.scheduleOnce)

2) Send the work message to the worker using regular tell

3a) If the completed work comes back before the timeout message, cancel the scheduled job and reallocate the work using steps 1 and 2

3b) If the completed work comes back after the timeout message then either ignore it or cancel the reallocated work.

One place futures can be helpful is at the worker, especially if the work takes a long time or is blocking. The worker can use a future to do the work and stay available to handle more incoming messages, for example to cancel the work.