Aggregating dropwizard timer metrics

Question

I'm currently working on monitoring system for my application that consists of several independent microservices. I'm using dropwizard-metrics for collecting data from JVMs, InfluxDB for storing and Grafana for creating nice-looking graphs.

One of my goals is creating simple dashboard that will present how my application performs. So I have to aggregate metrics from each of microservice in order to have global view.

Some metrics are easy to aggregate, e.g. sum of active database connections. I have problem with aggregating timer metric that I want to somehow simple visualize, e.g. I'm interested in aggregating percentiles of response times of systems that my application depends on from all my microservices instances.

I think this is very common case. Do you have any advice? I've read blog post about it but it doesn't cover my case.

Answer 1

A dropwizard metrics timer already does aggregation. What you want ideally is to no pre-aggregate in your metrics collecting application unless absolutely nessecary. You can do aggregation (percentiles, etc.) later in Grafana.

Answer 2

You may want to just buy an APM tool, think about the cost associated with building, upgrading, and maintaining all of that yourself. Also the storage and compute associated with it. You'll also be introducing overhead on your app as you collect all of that. It doesn't sound like the right way to solve the problem. Finally, even with dropwizard you still need to monitor the frontend and other technologies (servers, etc). This is the main reason you shouldn't do it yourself. Have a look at AppDynamics, New Relic, or Ruxit as the options which are modern that do all of this for you.