How do I force Kubernetes to re-pull an image?

Question

I have the following replication controller in Kubernetes on GKE:

apiVersion: v1
kind: ReplicationController
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 2
  selector:
    app: myapp
    deployment: initial
  template:
    metadata:
      labels:
        app: myapp
        deployment: initial
    spec:
      containers:
      - name: myapp
        image: myregistry.com/myapp:5c3dda6b
        ports:
        - containerPort: 80
      imagePullPolicy: Always
      imagePullSecrets:
        - name: myregistry.com-registry-key

Now, if I say

kubectl rolling-update myapp --image=us.gcr.io/project-107012/myapp:5c3dda6b

the rolling update is performed, but no re-pull. Why?

Answer 1

Kubernetes will pull upon Pod creation if either (see updating-images doc):

• Using images tagged :latest
• imagePullPolicy: Always is specified

This is great if you want to always pull. But what if you want to do it on demand: For example if you want to use some-public-image:latest but only want to pull a newer version manually when you ask for it. You can currently:

• Set imagePullPolicy to IfNotPresent or Never and pre-pull: Pull manually images on each cluster node so the latest is cached, then do a kubectl rolling-update or similar to restart Pods (ugly easily broken hack!)
• Temporarily change imagePullPolicy, do a kubectl apply, restart the pod (e.g. kubectl rolling-update), revert imagePullPolicy, redo a kubectl apply (ugly!)
• Pull and push some-public-image:latest to your private repository and do a kubectl rolling-update (heavy!)

No good solution for on-demande pull. If that changes, please comment; I'll update this anwser.

Answer 2

One has to group imagePullPolicy inside the container data instead of inside the spec data. However, I filed an issue about this because I find it odd. Besides, there is no error message.

So, this spec snippet works:

spec:
  containers:
  - name: myapp
    image: myregistry.com/myapp:5c3dda6b
    ports:
    - containerPort: 80
    imagePullPolicy: Always
  imagePullSecrets:
    - name: myregistry.com-registry-key

Answer 3

My hack during development is to change my Deployment manifest to add the latest tag and always pull like so

image: etoews/my-image:latest
imagePullPolicy: Always

Then I delete the pod manually

kubectl delete pod my-app-3498980157-2zxhd

Because it's a Deployment, Kubernetes will automatically recreate the pod and pull the latest image.

Answer 4

A popular workaround is to patch the deployment with a dummy annotation (or label):

kubectl patch deployment <name> -p \
  "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

Assuming your deployment meets these requirements, this will cause K8s to pull any new image and redeploy.

Answer 5

Apparently now when you run a rolling-update with the --image argument the same as the existing container image, you must also specify an --image-pull-policy. The following command should force a pull of the image when it is the same as the container image:

kubectl rolling-update myapp --image=us.gcr.io/project-107012/myapp:5c3dda6b --image-pull-policy Always

Answer 6

The rolling update command, when given an image argument, assumes that the image is different than what currently exists in the replication controller.

Answer 7

There wil be a new comand to directly do that:

Create a new kubectl rollout restart command that does a rolling restart of a deployment.

The pull request got merged. It will be part of the version 1.15 (changelog)

Answer 8

You can define imagePullPolicy: Always in your deployment file.

Answer 9

Now, the command kubectl rollout restart deploy YOUR-DEPLOYMENT combined with a imagePullPolicy: Always policy will allow you to restart all your pods with a latest version of your image.