How to profile Akka applications?

Question

I have a small Akka application that passes many messages between its actors and each actor does some calculations on the data it receives. What I want is to profile this application in order to see which parts of the code take up most time and so on.

I tried VisualVM but I cannot really understand what's going on. I added a picture of the profiler output.

My questions are

• What for example is this first line and why does it take up so much time? (scala.concurrent.forkjoin.ForkJoinPool.scan())
• Can Akka applications because of their asynchronous behaviour be profiled well at all?
• Can I see for instance how long one specific actor(-type) works for one specific message(-type) it receives?
• Are there other best-practices for profiling Akka applications?

Answer 1

• There are packages not profiled by default and it is their time that is accounted in the profile of scala.concurrent.forkjoin.ForkJoinPool.scan(). If all the hidden packages are allowed to be sampled, the true CPU time consumers will be revealed. For example, the following before/after illustrative profiles uncover that threads are put to sleep most of the time by sun.misc.Unsafe.park waiting to be unparked.

• Akka applications can be profiled quite well with proper instrumentation and call tracing. Google's prominent Dapper, a Large-Scale Distributed Systems Tracing Infrastructure paper contains detailed explanation of the technique. Twitter created Zipkin based on that. It is open sourced and has an extension for distributed tracing of Akka. Follow its wiki for a good explanation of how to set up a system that allows to

  • trace call hierarchies inside an actor system;
  • debug request processing pipelines (you can log to traces, annotate them with custom key-value pairs);
  • see dependencies between derived requests and their contribution to resulting response time;
  • find and analyse slowest requests in your system.

  There is also a new kid on the block, Kamon. It is a reactive-friendly toolkit for monitoring applications that run on top of the JVM, which is specially enthusiastic to applications built with the Typesafe Reactive Platform. That definitely means yes for Akka and the integration comes in the form of the kamon-akka and kamon-akka-remote modules that bring bytecode instrumentation to gather metrics and perform automatic trace context propagation on your behalf. Explore the documentation starting from Akka Integration Overview to understand what it can and how to achieve that.

Answer 2

Just a couple of days ago TypeSafe announced that TypeSafe console now is free. I don't know what can be better for profiling Scala/Akka applications. Of cause you can try JProfiler for JVM languages, I've used it with Java projects, but it's not free and for Java.

Answer 3

I was thinking about profiling/metrics in code since I also use Akka/Scala a lot for building production applications, but I also eager to hear alternative ways to make sure that application is healthy.

1. Metrics (like Dropwizard)

Very good tool for collecting metrics in the code, with good documentation and embedded support for Graphite, Ganglia, Logback, etc.

It has verbose tools for collecting in-app statistics like gauges, counter histograms, timings - information to figure out what is the current state of your app, how many actors were created, etc, if they are alive, what the current state is of majority of actors, etc.

Agree, it's a bit different from profiling but helps a lot to find roots of the problem, especially if integrated with some char building tool.

2. Profilers like (VisualVM, XRebel)

Since I'm a big fun of doing monitoring, it still answers a slightly different question - what are current insights of my application right now? But there is quite another matter may disturb us - what of my code is slow (or sloppy)?

For that reason, we have VisualVM and another answers to this question - how to profile Akka actors with VisualVM.

Also, I'd suggest trying XRebel profiler that just adds a bit more firepower to process of figuring out what code makes app slower. It's also paid but on my project it saved a lot of time dealing with sloppy code.

3. New Relic

I'd suggest it for some playground projects since you can get some monitoring/profiling solutions for free, but on more serious projects I'd go for things I highlighted above.

So I hope, that my overview was helpful.