Monitoring and alerting on pod status or restart with Google Container Engine (GKE) and Stackdriver

Question

Is there a way to monitor the pod status and restart count of pods running in a GKE cluster with Stackdriver?

While I can see CPU, memory and disk usage metrics for all

Answer 1

You can achieve this manually with the following:

1. In Logs Viewer, creating the following filter:

   resource.labels.project_id=""
   resource.labels.cluster_name=""
   resource.labels.namespace_name=""
   jsonPayload.message:"failed liveness probe"
   

2. Create a metric by clicking on the Create Metric button above the filter input and filling up the details.

3. You may now track this metric in Stackdriver.

Would be happy to be informed of a built-in metric instead of this.