Gitlab metrics not being scraped by prometheus even though servicemonitor is available and appears in service discovery

2
votes

I am currently trying to work on monitoring my company's Gitlab that runs on a kubernetes cluster. We have deployed a separate prometheus (we are not using the bundled prometheus). Currently my problem is that prometheus doesn't scrape a whole lot of the metrics. I do get a lot of Gitaly but none of the exposed gitlab metrics.

As of right now I have gone to the metrics endpoint to check which metrics are exposed there. I have checked the cluster and the servicemonitor that points to the correct endpoint <url>/-/metrics is there and has been picked up by the prometheus operator/service discovery.

I have checked prometheus itself and it appears under service discovery enter image description here

I am at a complete loss right now what the issue could be. I have tried reinstalling the gitlab instance and killing the prometheus pod to maybe kickstart the scraping process but that did not seem to work.

Hopefully someone else here knows what could be the issue.

1
kubernetesgitlabmonitoringprometheusmetrics
Could you share your prometheus configuration and /etc/gitlab/gitlab.rc file? - Mariusz K.
I will edit the post, but we use helm to deploy gitlab and prometheus, so afaik we are not able to edit the gitlab.rc file only adjust settings using the helm values. - Pim Schwippert
Could you provide more information about your environment? Its cloud or local environment, how did you deployed Prometheus, are you using some own query. How did you configured Prometheus? What K8s version are you using? - PjoterS

1 Answers

0
votes

I decided to double check everything and I figured out the issue was due to a port that wasn't set right at the service (by someone else) so as much as prometheus was trying to scrape at port XYZ it doesn't get any metrics because the port was not the correct one.