Prometheus error - Error on ingesting samples

3
votes

I am getting following error in Prometheus on monitoring my Node JS based web application.

level=warn ts=2019-08-02T18:23:48.658364708Z caller=scrape.go:932 component="scrape manager" scrape_pool=batch_web target=https://example.com:443/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=6

Any pointers on why this could be happening and how can I solve this?

3
prometheus
Prometheus doesn't accept samples with historical timestamps. If you need storing samples with out of order timestamps, then take a look at other Prometheus-compatible systems with backfilling support such as VictoriaMetrics. – valyala

3 Answers

-1
votes

The error message appears to be quite clear|precise.

The metrics that its trying to scrape appear to be timestamped (possibly the problem!) and are either too old or too far into the future.

Generally, prometheus metrics don't include a timestamp.

  • If you can drop the timestamps, that would solve the problem.
  • If you can't drop the timestamps, then correct them so that they're ~current.

You may have solved this issue more quickly by searching Google for the error message.

1
votes

  • Enable debugging mode in Prometheus by passing --log.level=debug as an argument to prometheus

  • Check the log. In my case the metric that was dropped, was returned:

    msg="Out of order sample" series="some-metric{a="b",c="d"}

  • Check the prometheus configuration and rule files that could return a duplicated some-metric. In my case a duplicated rule was causing the issue as the same metric was scraped from the federated prometheus and calculated again by my prometheus. Removing the duplicated rule solved the issue.

0
votes

Working with prometheus on virtual machines in VMware, vagrant and Paralles virtual environments, all running Ubuntu 18.04, I had the same promblem with this message in syslog

msg="Error on ingesting out-of-order samples"
and veraious num_dropped= values

The thing that worked was to issue ALL of these command, on each image, then reboot the image. Without the reboot the num_dropped began to be lower but the error persisted until the reboot

/usr/bin/timedatectl set-ntp false
systemctl stop systemd-timesyncd
systemctl disable systemd-timesyncd
systemctl mask systemd-timesyncd