问题
i use grafana+prometheus monitor k8s pod,when my pod is removed ,i clean all metrics belongs to the removed pod,but still can see in grafana variable
for example ,i defined a variable named node ,the query express is " {instance=~"(.+)", job="node status"} ",it can catch all metrics ,and i use regex expression '/instance="([^"]+):9100"/' to match the ip of each monitor target ,when i click node label on dashboard,it display all target ip , and when one of these targets is removed ,i use http api provide by prometheus to clean all metrics belongs to this target,but when i click node label ,it still display the removed target ip ,why? and how i can delete this ip?
回答1:
It seems that Prometheus targets are not updated, even so that some of the pods are evicted. You can check it in Prometheus http://yourprometheus/targets page.
Does Prometheus run inside of the K8s cluster?
来源：https://stackoverflow.com/questions/53292892/grafana-variable-still-catch-old-metrics-info