Recommended way to persistently change kube-env variables

Question

We are using elasticsearch/kibana instead of gcp for logging (based on what is described here).

To have fluentd-elsticsearch pod's launched we've set LOGGING_DESTINATION=elasticsearch and ENABLE_NODE_LOGGING="true" in the "Compute Instance Template" -> "Custom metadata" -> "kube-env".

While this works fine when done manually it gets overwritten with every gcloud container clusters upgrade as a new Instance Template with defaults (LOGGING_DESTINATION=gcp ...) is created.

My question is: How do I persist this kind of configuration for GKE/GCE?

I thought about adding a k8s-user-startup-script but that's also defined in the Instance Template and therefore is overwritten by gcloud container clusters upgrade. I've also tried to add a k8s-user-startup-script to the project metadata but that is not taken into account.

//EDIT
Current workaround (without recreating Instance Template and Instances) for manually switching back to elasticsearch is:

for node in $(kubectl get nodes -o name | cut -f2 -d/); do
    gcloud compute ssh $node \
      --command="sudo cp -a /srv/salt/fluentd-es/fluentd-es.yaml /etc/kubernetes/manifests/; sudo rm /etc/kubernetes/manifests/fluentd-gcp.yaml";
done

kubelet will pick that up, kill fluentd-gcp and start fluentd-es.

//EDIT #2 Now running a "startup-script" DaemonSet for this:

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: startup-script
  namespace: kube-system
  labels:
    app: startup-script
spec:
  template:
    metadata:
      labels:
        app: startup-script
    spec:
      hostPID: true
      containers:
        - name: startup-script
          image: gcr.io/google-containers/startup-script:v1
          securityContext:
            privileged: true
          env:
          - name: STARTUP_SCRIPT
            value: |
              #! /bin/bash

              set -o errexit
              set -o pipefail
              set -o nounset

              # Replace Google-Cloud-Logging with EFK
              if [[ ! -f /etc/kubernetes/manifests/fluentd-es.yaml ]]; then
                if [[ -f /home/kubernetes/kube-manifests/kubernetes/fluentd-es.yaml ]]; then
                  # GCI images
                  cp -a /home/kubernetes/kube-manifests/kubernetes/fluentd-es.yaml /etc/kubernetes/manifests/
                elif [[ -f /srv/salt/fluentd-es/fluentd-es.yaml ]]; then
                  # Debian based GKE images
                  cp -a /srv/salt/fluentd-es/fluentd-es.yaml /etc/kubernetes/manifests/
                fi
                test -f /etc/kubernetes/manifests/fluentd-es.yaml && rm /etc/kubernetes/manifests/fluentd-gcp.yaml
              fi

Answer 1

There isn't a fully supported way to reconfigure the kube-env in GKE. As you've found, you can hack the instance template, but this isn't guaranteed to work across upgrades.

An alternative is to create your cluster without gcp logging enabled and then create a DaemonSet that places a fluentd-elasticsearch pod on each of your nodes. Using this technique you don't need to write a (brittle) startup script or rely on the fact that the built-in startup script happens to work when setting LOGGING_DESTINATION=elasticsearch (which may break across upgrades even if it wasn't getting overwritten).