prometheus consul sd “Error refreshing service”

Question

I am getting expected results in prometheus UI & target shows UP, but I am getting this error in prometheus logs,

level=error ts=2020-05-07T04:58:40.627Z caller=consul.go:487 component="discovery manager scrape" discovery=consul msg="Error refreshing service" service=TL-GRAPHQL-SERVICE tags= err="Unexpected response code: 503 ({"timestamp":"2020-05-07T04:58:40.625+0000","status":503,"error":"Service Unavailable","message":"No message available","path":"/v1/health/service/TL-GRAPHQL-SERVICE"})"

here is the response from my server. Can you please help?

http://XX.XXX.XXX.XXX:8761/v1/health/service/TL-GRAPHQL-SERVICE
[
{
"Node": {
"Node": "TL-GRAPHQL-SERVICE",
"Address": "XX.XXX.XXX.191",
"Meta": {}
},
"Service": {
"ID": "XX.XXX.XXX:tl-graphql-service:0",
"Service": "TL-GRAPHQL-SERVICE",
"Tags": [],
"Address": "XX.XXX.XXX.191",
"Meta": {},
"Port": 36126
},
"Checks": [
{
"Node": "TL-GRAPHQL-SERVICE",
"CheckID": "service:XX.XXX.XXX:tl-graphql-service:0",
"Name": "Service 'XX.XXX.XXX:tl-graphql-service:0' check",
"Status": "UP"
}
]
},
{
"Node": {
"Node": "TL-GRAPHQL-SERVICE",
"Address": "XX.XXX.XXX.192",
"Meta": {}
},
"Service": {
"ID": "XX.XXX.XXX:tl-graphql-service:0",
"Service": "TL-GRAPHQL-SERVICE",
"Tags": [],
"Address": "XX.XXX.XXX.192",
"Meta": {},
"Port": 45759
},
"Checks": [
{
"Node": "TL-GRAPHQL-SERVICE",
"CheckID": "service:XX.XXX.XXX:tl-graphql-service:0",
"Name": "Service 'XX.XXX.XXX:tl-graphql-service:0' check",
"Status": "UP"
}
]
}
]

My Prometheus config:

scrape_configs:
  - job_name: 'eureka'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 20s
    consul_sd_configs:
      - server: 'localhost:8761'

Sathish Sathish · Accepted Answer · 2020-05-07T06:04:23

it's a timeout issue in my service. Its working after increasing the async timeout. It is explained here,

https://github.com/twinformatics/eureka-consul-adapter

prometheus consul sd “Error refreshing service”

1 Answers