GKE - ErrImagePull pulling from Google Container Registry

Question

I have a Google Kubernetes Engine cluster which until recently was happily pulling private container images from a Google Container Registry bucket. I haven\'t changed anyth

Answer 1

Ok, this turned out to be tricky, but the cause was this:

I used Terraform to set the service account for the nodes in the GKE cluster, but instead of using the email output of the google_service_account resource to specify the service account, I used the unique_id output instead. This was accepted fine by both Terraform and the Google Cloud API.

When Kubernetes (and other things) was trying to access the internal metadata API on each node to get an token it could use, it was receiving a response of Service account is invalid/disabled and a 403 status.

Recreating the node pool with the correctly specified service account fixed the problem.