How to monitor AWS SQS with Graphite - Grafana

Question

I have a bunch of micro-services hosted on AWS. I am using StatsD, Graphite and Grafana to monitor them. Now I want to expand it to monitor the queues (SQS) through which these micro-services are talking to each other. How can I leverage Graphite/ Grafana to do this? Or a better approach if there aint any support/ plugin for the same. Thanks :)

PS : If it's gotta be Zipkin, please tell me they can co-exist or is there a catch to using multiple tracers.

I'm removing "jaeger" from the list of tags for this question, as it's not clear whether you want to do some "tracing" there. — jpkrohling

Unknown Unknown · Accepted Answer · 2019-03-14T23:07:05

Alright, so I'm going to answer this based on what you said here:

Or a better approach if there aint any support/ plugin for the same.

The way that I do it us through Prometheus, in combination with cloudwatch_exporter, and alertmanager.

The configuration for cloudwatch_exporter to monitor SQS is going to be something like (this is only two metrics, you'll need to add more based on what you're looking to monitor):

tasks:
 - name: ec2_cloudwatch
   default_region: us-west-2
   metrics:
    - aws_namespace: "AWS/SQS"
      aws_dimensions: [QueueName]
      aws_metric_name: NumberOfMessagesReceived
      aws_statistics: [Sum]
      range_seconds: 600
    - aws_namespace: "AWS/SQS"
      aws_dimensions: [QueueName]
      aws_metric_name: ApproximateNumberOfMessagesDelayed
      aws_statistics: [Sum]

You'll then need to configure prometheus to scrape the cloudwatch_exporter endpoint at an interval, for ex what I do:

  - job_name: 'somename'
    scrape_timeout: 60s
    dns_sd_configs:
    - names:
        - "some-endpoint"
    metrics_path: /scrape
    params:
      task: [ec2_cloudwatch]
      region: [us-east-1]
    relabel_configs:
      - source_labels: [__param_task]
        target_label: task
      - source_labels: [__param_region]
        target_label: region

You would then configure alertmanager to alert based on those scraped metrics; I do not alert on those metrics so I cannot give you an example. But, to give you an idea how of this architecture, a diagram is below:

If you need to use something like statsd you can use statsd_exporter. And, just in-case you were wondering, yes Grafana supports prometheus.

How to monitor AWS SQS with Graphite - Grafana

1 Answers