问题
How can I monitor my python processes (say some script that gets triggered periodically by Cron daemon) using Prometheus?
Note that this is not a web application but a short-lived process that gets launched periodically by the Cron daemon. This script comes up, does its job, and terminates. The same python script gets launched multiple times a day (approximately 100k times) by cron daemon. I want to capture multiple stats from various runs of this script (for example, the time it takes to run a particular function, how much CPU and memory it consumes, etc.)
回答1:
You may want to look at Prometheus' Pushgateway: whenever your script completes, it can push the metrics it collected (e.g. a histogram of how long your function calls took, total CPU utilization, peak memory utilization etc.).
You seem to be saying your script will run approximately once a second. I am hoping that means something along the lines of "once every 5 minutes for each of 300 tenants". In a case like this, you would push your metrics with something like a tenant_id label and be able to see either per-tenant or aggregated metrics.
If your script runs once a second with the same parameters/configuration, then you'll probably lose some of the metrics because multiple scripts may terminate within the same second, all push their metrics and only the last one's metrics will get collected by Prometheus (as I believe you can't set a collection interval lower than 1 second in Prometheus).
回答2:
you can use pushgateway and prometheus_client
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
registry = CollectorRegistry()
g = Gauge('job_last_success_unixtime', 'Last time a job successfully finished', registry=registry)
g.set_to_current_time()
push_to_gateway('localhost:9091', job='job_a', registry=registry)
来源：https://stackoverflow.com/questions/54920309/monitoring-short-lived-python-batch-job-processes-using-prometheus