How can I delete old Jobs from Prometheus?

Question

I have 2 jobs. one is new and other one is old. Need to delete old one, so that it will get removed from Grafana dashboard too.

Thanks in Advance...

Answer 1

Expanding on evgenyl's answer, the exact command would be something like:

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="name_of_old_job"}'

Replace name_of_old_job with the name of the job you want deleted.

Reminder that you need to have started prometheus with the --web.enable-admin-api flag

Answer 2

To delete metrics from Prometheus, you need to use delete API.

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=a_bad_metric&match[]={region="mistake"}'

Check Deleting time series from Prometheus post for more details..

Answer 3

cd /var/lib/prometheus/      ###enter into prometheus storage directory
grep -irl "Old job name" .   ###Find out the directory of old job

Delete that directories which all listed by the above grep command.

service prometheus restart

It will remove the particular old job from Prometheus. But in the Grafana dashboard if you put the last 5 mins time range you cannot able to see metrics related to that job and you can see those metrics in a 1-hour time range. Don't worry, gradually it will get removed from our normal time range which we use!!