Kubernetes pod pending when a new volume is attached (EKS)

Question

Let me describe my scenario:

TL;DR

When I create a deployment on Kubernetes with 1 attached volume, everything works perfectly. When I create the same depl

Answer 1

See this article: https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/

The gist is that you want to update your storageclass to include:

volumeBindingMode: WaitForFirstConsumer

This causes the PV to not be created until the pod is scheduled. It fixed a similar problem for me.

Answer 2

Sounds like it's trying to create a volume in an availability zone where you don't have any volumes on. You can try restricting your StorageClass to the availability zones where you have nodes.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp2
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
mountOptions:
  - debug
allowedTopologies:
- matchLabelExpressions:
  - key: failure-domain.beta.kubernetes.io/zone
    values:
    - us-east-1b
    - us-east-1c

This is very similar to this question and this answer except that the issue described is on GCP and in this case it's AWS.

Answer 3

In this case, you should check the availability zone of your worker nodes (EC2 instances).

As a Example :

worker node 1 = eu-central-1b
worker node 2 = eu-central-1c

Then create the volume in including one of an availability zone which mentioned above(do not create the volume with eu-central-1a).

after you create the volume, create your PersistentVolume and PersistentVolumeClaim by attaching a newly created volume to your cluster like below.

apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    failure-domain.beta.kubernetes.io/region: eu-central-1
    failure-domain.beta.kubernetes.io/zone: eu-central-1b
  name: mongo-pv
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 100Gi
  awsElasticBlockStore:
    fsType: ext4
    volumeID: aws://eu-central-1b/vol-063342ab9be5d2929

  storageClassName: gp2

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongo-pvc
  namespace: default
spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 100Gi
    storageClassName: gp2
    volumeName: mongo-pv