K8s Ingress, initiate ingress controller nginx error?

问题:

I have two spring boot container, I want to setup ingress service. As document here says, ingress has two parts, one is controller, the other is resources.

My two resources are two containers: gearbox-rack-eureka-server and gearbox-rack-config-server. The difference is port so that ingress could route traffic by different ports. My yaml files are listed below:

eureka_pod.yaml

apiVersion: v1 kind: Pod metadata:   name: gearbox-rack-eureka-server   labels:     app: gearbox-rack-eureka-server     purpose: platform_eureka_demo spec:   containers:   - name:  gearbox-rack-eureka-server     image: 192.168.1.229:5000/gearboxrack/gearbox-rack-eureka-server     ports:         - containerPort: 8761 

eureka_svc.yaml

apiVersion: v1 kind: Service metadata:   name: gearbox-rack-eureka-server   labels:     name: gearbox_rack_eureka_server spec:   selector:     app: gearbox-rack-eureka-server   type: NodePort   ports:     - port: 8761       nodePort: 31501       name: tcp 

config_pod.yaml

apiVersion: v1 kind: Pod metadata:   name: gearbox-rack-config-server   labels:     app: gearbox-rack-config-server     purpose: platform-demo spec:   containers:   - name:  gearbox-rack-config-server     image: 192.168.1.229:5000/gearboxrack/gearbox-rack-config-server     ports:     - containerPort: 8888     env:       - name: EUREKA_SERVER         value: http://172.16.100.83:8761 

config_svc.yaml

apiVersion: v1 kind: Service metadata:   name: gearbox-rack-config-server   labels:     name: gearbox-rack-config-server spec:   selector:     app: gearbox-rack-config-server   type: NodePort   ports:     - port: 8888       nodePort: 31502       name: tcp 

My ingress-nginx controller is mostly copied from the link above,

ingress_nginx_ctl.yaml:

kind: Service apiVersion: v1 metadata:   name: ingress-nginx spec:   type: LoadBalancer   selector:     app: ingress-nginx   ports:   - name: http     port: 80     targetPort: http   - name: https     port: 443     targetPort: https --- kind: Deployment apiVersion: extensions/v1beta1 metadata:   name: ingress-nginx spec:   replicas: 1   template:     metadata:       labels:         app: ingress-nginx     spec:       terminationGracePeriodSeconds: 60       containers:       - image: nginx:1.13.12         name: ingress-nginx         imagePullPolicy: Always         ports:           - name: http             containerPort: 80             protocol: TCP           - name: https             containerPort: 443             protocol: TCP         livenessProbe:           httpGet:             path: /healthz             port: 10254             scheme: HTTP           initialDelaySeconds: 30           timeoutSeconds: 5         env:           - name: POD_NAME             valueFrom:               fieldRef:                 fieldPath: metadata.name           - name: POD_NAMESPACE             valueFrom:               fieldRef:                 fieldPath: metadata.namespace         args:         - /nginx-ingress-controller         - --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend 

I did following commands, they are successful.

kubectl apply -f eureka_pod.yaml kubectl apply -f eureka_svc.yaml kubectl apply -f config_pod.yaml kubectl apply -f config_svc.yaml 

Then I got error from execute kubectl apply -f ingress_nginx_ctl.yaml, the pod does not start, logs are listed below:

[root@master3 nginx-ingress-controller]# kubectl get pods NAME                             READY     STATUS             RESTARTS   AGE gearbox-rack-config-server       1/1       Running            0          39m gearbox-rack-eureka-server       1/1       Running            0          40m ingress-nginx-686c9975d5-7d464   0/1       CrashLoopBackOff   6          7m [root@master3 nginx-ingress-controller]# kubectl logs -f ingress-nginx-686c9975d5-7d464 container_linux.go:247: starting container process caused "exec: \"/nginx-ingress-controller\": stat /nginx-ingress-controller: no such file or directory" 

I created a directory /nginx-ingress-controller under root, and repeat the steps again, it still said same error. Does someone could point me the problem?

I put my ingress_nginx_res.yaml as follows for reference, it may have errors also.

apiVersion: extensions/v1beta1 kind: Ingress metadata:   name: my-ingress spec:   rules:   - host: 172.16.100.83     http:       paths:       - backend:           serviceName: gearbox-rack-eureka-server           servicePort: 8761   - host: 172.16.100.83     http:       paths:       - path:         backend:           serviceName: gearbox-rack-config-server           servicePort: 8888 

==========================================

second edition

After change image link, The previous errors disappear, but still it has following permission problem:

[root@master3 ingress]# kubectl get pods NAME                             READY     STATUS             RESTARTS   AGE gearbox-rack-config-server       1/1       Running            0          15m gearbox-rack-eureka-server       1/1       Running            0          15m ingress-nginx-8679f9c8ff-5sxw7   0/1       CrashLoopBackOff   5          12m 

The log message is as follows:

It is RBAC problem. I check the install script which is downloaded from forum:

heapster-rbac.yaml:

kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata:   name: heapster roleRef:   apiGroup: rbac.authorization.k8s.io   kind: ClusterRole   name: system:heapster subjects: - kind: ServiceAccount   name: heapster   namespace: kube-system 

One of related kubelet start argument is as follows: (I do not know whether it is relevant).

Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt" 

By which way, I could grant permission to ingress controller? Just put namespace kube-system to ingress_nginx_ctl.yaml?

================================================================

Third edition

I put Kun Li's codes into ingress_nginx_role_rb.yaml, and run the following commands:

kubectl apply -f eureka_pod.yaml kubectl apply -f eureka_svc.yaml kubectl apply -f config_pod.yaml kubectl apply -f config_svc.yaml kubectl apply -f ingress_nginx_role_rb.yaml (just copy paste from Kun Li's answer) kubectl apply -f nginx_default_backend.yaml kubectl apply -f ingress_nginx_ctl.yaml 

nginx_default_backend.yaml file is listed below:

kind: Service apiVersion: v1 metadata:   name: nginx-default-backend   namespace: kube-system spec:   ports:   - port: 80     targetPort: http   selector:     app: nginx-default-backend --- kind: Deployment apiVersion: extensions/v1beta1 metadata:   name: nginx-default-backend   namespace: kube-system spec:   replicas: 1   template:     metadata:       labels:         app: nginx-default-backend     spec:       terminationGracePeriodSeconds: 60       containers:       - name: default-http-backend         image: chenliujin/defaultbackend         livenessProbe:           httpGet:             path: /healthz             port: 8080             scheme: HTTP           initialDelaySeconds: 30           timeoutSeconds: 5         resources:           limits:             cpu: 10m             memory: 20Mi           requests:             cpu: 10m             memory: 20Mi         ports:         - name: http           containerPort: 8080           protocol: TCP 

ingress_nginx_ctl.yaml is listed below:

kind: Service apiVersion: v1 metadata:   name: ingress-nginx spec:   type: LoadBalancer   selector:     app: ingress-nginx   ports:   - name: http     port: 80     targetPort: http   - name: https     port: 443     targetPort: https --- kind: Deployment apiVersion: extensions/v1beta1 metadata:   name: ingress-nginx   namespace: kube-system spec:   replicas: 1   template:     metadata:       labels:         app: ingress-nginx     spec:       terminationGracePeriodSeconds: 60       serviceAccount: lb       containers:       - image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0         name: ingress-nginx         imagePullPolicy: Always         ports:           - name: http             containerPort: 80             protocol: TCP           - name: https             containerPort: 443             protocol: TCP         livenessProbe:           httpGet:             path: /healthz             port: 10254             scheme: HTTP           initialDelaySeconds: 30           timeoutSeconds: 5         env:           - name: POD_NAME             valueFrom:               fieldRef:                 fieldPath: metadata.name           - name: POD_NAMESPACE             valueFrom:               fieldRef:                 fieldPath: metadata.namespace         args:         - /nginx-ingress-controller         - --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend 

From here, we could see service ingress-nginx namespace is default, not kube-system. But anyway, controller is up.

[root@master3 ingress]# kubectl get pods -n kube-system NAME                                      READY     STATUS    RESTARTS   AGE calico-etcd-cdn8z                         1/1       Running   0          11m calico-kube-controllers-d554689d5-tzdq5   1/1       Running   0          11m calico-node-dz4d6                         2/2       Running   1          11m coredns-65dcdb4cf-h62bh                   1/1       Running   0          11m etcd-master3                              1/1       Running   0          10m heapster-5c448886d-swp58                  1/1       Running   0          11m ingress-nginx-6ccc799fbc-hq2rm            1/1       Running   0          9m kube-apiserver-master3                    1/1       Running   0          10m 

ingress-nginx pod's namespace is kube-system (shown above), but its service's namespace is default.(shown below).

[root@master3 ingress]# kubectl get service NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE gearbox-rack-config-server   NodePort       10.97.211.136   <none>        8888:31502/TCP               43m gearbox-rack-eureka-server   NodePort       10.106.69.13    <none>        8761:31501/TCP               43m ingress-nginx                LoadBalancer   10.105.114.64   <pending>     80:30646/TCP,443:31332/TCP   42m kubernetes                   ClusterIP      10.96.0.1       <none>        443/TCP                      44m 

as mentioned in the comments, expert's response help me to move forward.

回答1:

For ingress-controller, image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0 should be used. And you need setup nginx-default-backend pod and service.

About RBAC, I think you need a seviceaccount to deploy your nginx-ingress-controller, with the following roles and bindings:

apiVersion: v1 kind: ServiceAccount metadata:   name: lb   namespace: kube-system  ---  apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata:   name: nginx-ingress-normal rules:   - apiGroups:       - ""     resources:       - configmaps       - endpoints       - nodes       - pods       - secrets     verbs:       - list       - watch   - apiGroups:       - ""     resources:       - nodes     verbs:       - get   - apiGroups:       - ""     resources:       - services     verbs:       - get       - list       - watch   - apiGroups:       - "extensions"     resources:       - ingresses     verbs:       - get       - list       - watch   - apiGroups:       - ""     resources:         - events     verbs:         - create         - patch   - apiGroups:       - "extensions"     resources:       - ingresses/status     verbs:       - update  ---  apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata:   name: nginx-ingress-minimal   namespace: kube-system rules:   - apiGroups:       - ""     resources:       - configmaps       - pods       - secrets       - namespaces     verbs:       - get   - apiGroups:       - ""     resources:       - configmaps     resourceNames:       - "ingress-controller-leader-dev"       - "ingress-controller-leader-prod"     verbs:       - get       - update   - apiGroups:       - ""     resources:       - configmaps     verbs:       - create   - apiGroups:       - ""     resources:       - endpoints     verbs:       - get  ---  apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata:   name: nginx-ingress-minimal   namespace: kube-system roleRef:   apiGroup: rbac.authorization.k8s.io   kind: Role   name: nginx-ingress-minimal subjects:   - kind: ServiceAccount     name: lb     namespace: kube-system  ---  apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata:   name: nginx-ingress-normal roleRef:   apiGroup: rbac.authorization.k8s.io   kind: ClusterRole   name: nginx-ingress-normal subjects:   - kind: ServiceAccount     name: lb     namespace: kube-system 

文章来源: K8s Ingress, initiate ingress controller nginx error?