Are Akka actors overkill for doing data crunching/uploading?

Question

I'm quite new to Scala as well as Akka actors. I'm really only reading about their use and implementation now. My background is largely js and python with a bit of C#.

A new service I have to write is going to receive REST requests, then do the following:

Open a socket connection to a message broker
Query an external REST service once
Make many big, long REST requests to another internal service, do math on the responses, and send the result out. Messages are sent through the socket connection as progress updates.

Scalability is the primary concern here, as we may normally receive ~10 small requests per minute, but at unknown times receive several jaw-droppingly enormous and long running requests at once.

Using Scala Futures, the very basic implementation would be something like this:

val smallResponse = smallHttpRequest(args)

smallResponse.onComplete match {
  case Success(result) => {
    result.data.grouped(10000).toList.forEach(subList => {
      val bigResponse = getBigSlowHttpRequest(subList)
      bigResponse.onSuccess {
        case crunchableStuff =>  crunchAndDeliver(crunchableStuff)
      }
    })
  }
  case Failure(error) => handleError(error)
}

My understanding is that on a machine with many cores, letting the JVM handle all the threading underneath the above futures would allow for them all to run in parallel.

This could definitely be written using Akka actors, but I don't know what, if any, benefits I would realize in doing so. Would it be overkill to turn the above into an actor based process with a bunch of workers taking chunks of crunching?

Sounds like a case for Akka HTTP doc.akka.io/docs/akka-http/current/index.html You can configure it to be both a client and a server and (depending on how you implement it) not delve too deeply into writing actors. — CPS

RawToast RawToast · Accepted Answer · 2018-03-16T10:59:21

For such an operation, I wouldn't go near Akka Actors -- it's way too much for what looks to be a very basic chain of async requests. The Actor system gives you the ability to safely handle and/or accumulate state in an actor, whilst your task can easily be modeled as a type safe stateless flow of data.

So Futures (or preferably one of the many lazy variants such as the Twitter Future, cats.IO, fs2 Task, Monix, etc) would easily handle that.

No IDE to hand, so there's bound to be a huge mistake in here somewhere!

val smallResponse = smallHttpRequest(args)

val result: Future[List[CrunchedData]] = smallResponse.map(result => {
        result.data
              .grouped(10000)
              .toList
              // List[X] => List[Future[X]]
              .map(subList => getBigSlowHttpRequest(subList))
              // List[Future[X]] => Future[List[X]] so flatmap
              .flatMap(listOfFutures => Future.sequence(listOfFutures))
         })

Afterwards you could pass the future back via the controller if using something like Finch, Http4s, Play, Akka Http, etc. Or manually take a look like in your example code.

Are Akka actors overkill for doing data crunching/uploading?

1 Answers