I'm using a Prometheus datasource from Grafana, and I'm sometimes getting 200 OK, sometimes getting 405 Method not allowed when looking at graphs, or inserting new graphs.
It is very strange that it only appears sometimes, for random graphs, sometimes only for some graphs in a single dashboard.
The datasource is set up to proxy requests through the backend.
Both Grafana and Prometheus are running in Kubernetes as StatefulSets in Google cloud.
I'm accessing Grafana at localhost:3000 through an SSH tunnel to the pod in Kubernetes, and Grafana is accessing Prometheus at http://prometheus:9090/.
I've tried changing the method from GET to POST in the datasource setup, but then I get 405 on every request.
The raw headers in the request for http://localhost:3000/api/datasources/proxy/1/api/v1/query_range?query=kafka_topic_highwater{topic="test"}&start=1541499015&end=1541499930&step=15 is
Host: localhost:3000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://localhost:3000/d/tDB6XEaiz/kafka-realtime-timeseries?orgId=1
X-Grafana-Org-Id: 1
DNT: 1
Connection: keep-alive
Cookie: grafana_user=admin; grafana_remember=asdf8a620; grafana_sess=<secret>
And the response is:
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache
Content-Length: 19
Content-Type: text/plain; charset=utf-8
Date: Tue, 06 Nov 2018 10:25:22 GMT
Expires: -1
Pragma: no-cache
X-Content-Type-Options: nosniff
Any ideas what might be causing this?
