using plugin metrics-reporter-prometheus to monitor Gerrit internals with Prometheus

1
votes

I want to monitor Gerrit daemon with Prometheus.

How to configure Gerrit's plugin metrics-reporter-prometheus and Prometheus for white-box monitoring ?

Disclaimer: This exporter mostly exposes Gerrit daemon internal metrics, generated from Dropwizard metric. Project count, pending reviews ... are not reported

Note 2: I am not interested in using the generic exporters (JMX exporter, Javamelody...)

This exporters typical metrics looks like:

# HELP caches_disk_cached_git_tags Generated from Dropwizard metric import (metric=caches/disk_cached/git_tags, type=com.google.gerrit.metrics.dropwizard.BucketedCallback$ValueGauge)
# TYPE caches_disk_cached_git_tags gauge
caches_disk_cached_git_tags 0.0
# HELP http_server_rest_api_response_bytes_project_ListProjects Generated from Dropwizard metric import (metric=http/server/rest_api/response_bytes/project.ListProjects, type=com.codahale.metrics.Histogram)
# TYPE http_server_rest_api_response_bytes_project_ListProjects summary
http_server_rest_api_response_bytes_project_ListProjects{quantile="0.5",} 216.0
http_server_rest_api_response_bytes_project_ListProjects{quantile="0.75",} 216.0
http_server_rest_api_response_bytes_project_ListProjects{quantile="0.95",} 216.0
http_server_rest_api_response_bytes_project_ListProjects{quantile="0.98",} 216.0
http_server_rest_api_response_bytes_project_ListProjects{quantile="0.99",} 216.0
http_server_rest_api_response_bytes_project_ListProjects{quantile="0.999",} 216.0
http_server_rest_api_response_bytes_project_ListProjects_count 3.0
# HELP http_server_rest_api_response_bytes_query_change_QueryChanges Generated from Dropwizard metric import (metric=http/server/rest_api/response_bytes/query.change.QueryChanges, type=com.codahale.metrics.Histogram)
# TYPE http_server_rest_api_response_bytes_query_change_QueryChanges summary
http_server_rest_api_response_bytes_query_change_QueryChanges{quantile="0.5",} 16.0
http_server_rest_api_response_bytes_query_change_QueryChanges{quantile="0.75",} 16.0
http_server_rest_api_response_bytes_query_change_QueryChanges{quantile="0.95",} 16.0
http_server_rest_api_response_bytes_query_change_QueryChanges{quantile="0.98",} 16.0
http_server_rest_api_response_bytes_query_change_QueryChanges{quantile="0.99",} 16.0
http_server_rest_api_response_bytes_query_change_QueryChanges{quantile="0.999",} 16.0
http_server_rest_api_response_bytes_query_change_QueryChanges_count 17.0
[..]
[..]

and some usual like

# HELP proc_jvm_memory_heap_committed Generated from Dropwizard metric import (metric=proc/jvm/memory/heap_committed, type=com.google.gerrit.metrics.dropwizard.CallbackMetricImpl0$1)
# TYPE proc_jvm_memory_heap_committed gauge
proc_jvm_memory_heap_committed 5.77241088E8
# HELP proc_birth_timestamp Generated from Dropwizard metric import (metric=proc/birth_timestamp, type=com.google.gerrit.metrics.dropwizard.CallbackMetricImpl0$1)
# TYPE proc_birth_timestamp gauge
proc_birth_timestamp 1.525600508281E15
# HELP proc_cpu_usage Generated from Dropwizard metric import (metric=proc/cpu/usage, type=com.google.gerrit.metrics.dropwizard.CallbackMetricImpl0$1)
# TYPE proc_cpu_usage gauge
proc_cpu_usage 93.24
# HELP proc_num_open_fds Generated from Dropwizard metric import (metric=proc/num_open_fds, type=com.google.gerrit.metrics.dropwizard.CallbackMetricImpl0$1)
# TYPE proc_num_open_fds gauge
1
gerritmonitorprometheus

1 Answers

5
votes

Answering myself...

How it works: Prometheus will scrape

Gerrit configuration

Assumptions:

  • Gerrit is installed in the directory ~/gerrit
  • Gerrit version is >= 2.14 (the examples based on 2.14)
  • Gerrit SSH is configured with an SSH key (doc), this command works ssh -p 29418 [email protected] gerrit version

Install the Plugin

Install the Gerrit exporter plugin. The easier is to download the JAR from gerritforge. Browse https://gerrit-ci.gerritforge.com/view/Plugins-stable-2.14/job/pluin-metrics-reporter-prometheus-bazel-stable-2.14/ and download metrics-reporter-prometheus.jar, or execute the Unix commands:

cd ~/gerrit/plugins/
wget -O metrics-reporter-prometheus.jar 'https://gerrit-ci.gerritforge.com/view/Plugins-stable-2.14/job/plugin-metrics-reporter-prometheus-bazel-stable-2.14/lastSuccessfulBuild/artifact/bazel-genfiles/plugins/metrics-reporter-prometheus/metrics-reporter-prometheus.jar'

Then restart Gerrit

Connect as an admin in the Gerrit web interface, got to Plugins>Installed to verify that metrics-reporter-prometheus is listed and enabled, or run:

ssh -p 29418 [email protected] gerrit plugin ls

Authorize the account and group

Next you need to create a group and an account in Gerrit, then grant the capability View Metrics.

This procedure should be adapted to how you manage your user account in Gerrit (LDAP...). For local account (managed by SSH):

ssh -p 29418 [email protected] 'gerrit create-group "Prometheus Metrics" '

ssh -p 29418 [email protected] 'gerrit create-account prometheus  --full-name "Prometheus Monitoring" --email [email protected] '

ssh -p 29418 [email protected] 'gerrit set-members "Prometheus Metrics" --add "prometheus" '

ssh -p 29418 [email protected] 'gerrit set-account prometheus --http-password ChangeThisInsecurePassword '

Grant 'Prometheus Metrics' permissions in Gerrit

  • Log in to Gerrit web interface with an account with admin privileges, go to Projects>List>All-Projects
  • Click on the tab 'Access'
  • Click on the button [EDIT]
  • In the block Global Capabilities, click on Add Permission (that's typically the 4th drop down menu)
    • Select the permission View Metrics in the drop-down list
    • Select the group "Prometheus Metrics" we created above.
    • Click on the button [Add]
    • Go to the bottom of the page and click on the [Save Changes] button

Test !

Test authentication and authorization, there are two options: * Login the web interface with the create account (which should succeed!), then change the URL path (http://gerrit.example.com:8080/a/plugins/metrics-reporter-prometheus/metrics), you should get Prometheus metrics! * or Use wget (or curl or whatever):

wget -O - --user prometheus --password ChangeThisInsecurePassword  http://gerrit.example.com:8080/a/plugins/metrics-reporter-prometheus/metrics

IMPORTANT You may have to prefix Gerrit URL with /a/ to force HTTP authentication (read the doc)

If you get the error ERROR 403: Forbidden access., double check the procedure above (user exist? group exist ? permissions granted ?)!

Prometheus configuration

Edit the file /etc/prometheus/prometheus.yml, add a job:

  - job_name: gerrit
    metrics_path: '/a/plugins/metrics-reporter-prometheus/metrics'
    static_configs:
      - targets: ['slxhmhs1:8080']
    basic_auth:
      username: prometheus
      password: ChangeThisInsecurePassword

Check promethsu configuration file:

promtool  check-config /etc/prometheus/prometheus.yml

Let prometheus reload it's configuration, using one of:

systemctl reload prometheus
killall -HUP prometheus
docker exec -ti graphite-exporter killall -HUP prometheus

Test:

Visit Prometheus' status page (http://prom.example.com:9090/targets#job-gerrit ), then check the job/target status.

Use

Go to prometheus Expression browser (http://prom.example.com:9090 ), then type :

{job="gerrit"}

And enjoy in Grafana :-)