Kubernetes Ingress (GCE) keeps returning 502 error

Question

I am trying to setup an Ingress in GCE Kubernetes. But when I visit the IP address and path combination defined in the Ingress, I keep getting the following 502 error:

Answer 1

The "Limitations" section of the kubernetes documentation states that:

All Kubernetes services must serve a 200 page on '/', or whatever custom value you've specified through GLBC's --health-check-path argument.

https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-loadbalancing/glbc#limitations

Answer 2

I was having the same issue. It turns out I had to wait a few minutes before ingress to validate the service health. If someone is going to the same and done all the steps like readinessProbe and linvenessProbe, just ensure your ingress is pointing to a service that is either a NodePort, and wait a few minutes until the yellow warning icon turns into a green one. Also, check the log on StackDriver to get a better idea of what's going on. My readinessProbe and livenessProbe is on /login, for the gce class. So I don't think it has to be on /healthz.

Answer 3

I had the same issue. I turned out that the pod itself was running ok, which I tested via port-forwarding and accessing the health-check URL.

Port-Forward can be activated in console as follows:

$    kubectl port-forward <pod-name> local-port:pod-port

So if the pod is running ok and ingress still shows unhealthy state there might be an issue with your service configuration. In my case my app-selector was incorrect, causing the selection of a non existent pod. Interestingly this isn't showed as an errors or alerts in google console.

Definition of the pods:

#pod-definition.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: <pod-name>
  namespace: <namespace>
spec:
  selector:
    matchLabels:
      app: **<pod-name>**
  template:
    metadata:
      labels:
        app: <pod-name>
    spec:
    #spec-definition follows

#service.yaml    
apiVersion: v1
    kind: Service
    metadata:
      name: <name-of-service-here>
      namespace: <namespace>
    spec:
      type: NodePort
      selector:
        app: **<pod-name>**
      ports:
      - protocol: TCP
        port: 8080
        targetPort: 8080
        name: <port-name-here>

Answer 4

Your backend k8s-be-32396--5fc40252fadea594 is showing as "UNHEALTHY".

Ingress will not forward traffic if the backend is UNHEALTHY, this will result in the 502 error you are seeing.

It will be being marked as UNHEALTHY becuase it is not passing it's health check, you can check the health check setting for k8s-be-32396--5fc40252fadea594 to see if they are appropriate for your pod, it may be polling an URI or port that is not returning a 200 response. You can find these setting under Compute Engine > Health Checks.

If they are correct then there are many steps between your browser and the container that could be passing traffic incorrectly, you could try kubectl exec -it PODID -- bash (or ash if you are using Alpine) and then try curl-ing localhost to see if the container is responding as expected, if it is and the health checks are also configured correctly then this would narrow down the issue to likely be with your service, you could then try changing the service from a NodePort type to a LoadBalancer and see if hitting the service IP directly from your browser works.

Answer 5

Issue is indeed a health check and seemed "random" for my apps where I used name-based virtual hosts to reverse proxy requests from ingress via domains to two separate backend services. Both were secured using Lets Encrypt and kube-lego. My solution was to standardize the path for health checks for all services sharing an ingress, and declare the readinessProbe and livenessProbe configs in my deployment.yml file.

I faced this issue with Google cloud cluster node version 1.7.8 and found this issue that closely-resembled what I experienced: * https://github.com/jetstack/kube-lego/issues/27

I'm using gce and kube-lego and my backend service health checks were on / and kube-lego is on /healthz. It appears differing paths for health checks with gce ingress might be the cause so it may be worth updating backend services to match the /healthz pattern so all use same (or as one commenter in Github issue stated they updated kube-lego to pass on /).

Answer 6

Log can read from Stackdriver Logging, in my case, it is backend_timeout error. After increase the default timeout (30s) via BackendConfig, it stop returning 502 even under load.

More on: https://cloud.google.com/kubernetes-engine/docs/how-to/configure-backend-service#creating_a_backendconfig