IBM Cloud Kubernetes Service

和自甴很熟 提交于 2020-07-28 07:57:54

k8s数据存储的方式很多:临时存储(emptyDir),半持久化存储(hostpath),持久化存储(文件、块、对象存存储等)。公有云各厂商几乎都支持云存储服务集成k8s托管,k8s系统本身通过两个概念PersistentVolume(PV)和 PersistentVolumeClaim(PVC)实现存储持久化的生命周期管理(申请、保留、删除、废弃等),pod根据需求向集群系统申请数据卷并使用,如果pod一旦释放,应用消失,数据卷相应分离解绑,数据会保留下来,在使用中pod以及应用对存储的后端基础架构并无感知。 这种设计很好地把应用和数据做了解藕。 至于存储资源的底层是选择ceph、portwx、弹性云存储、云对象存储等等,这就取决于业务具体需求和厂商的支持了。
在这里插入图片描述
闲话少说,切入今天的主题,既然有很多后端选择,用户在试用一款k8s集群产品,存储持久化以及应用层面的读写能力是必测一关,这里我们以ibm cloud kubernetes service为例,看看IBM云上的三种常见外挂存储的测试方法以及性能比较。

大致分为三个步骤:

  1. 创建K8S集群并通过kubectl连接使用 (部署在日本东京)
  2. 确认和安装IBM云存储k8s供应插件(CFS、CBS、COS部署在东京)
  3. dbench(fio)测试上述三种存储卷

1. k8s集群创建、连接和使用
登陆cloud.ibm.com, 在商品目录里搜索关键字“kubernetes”,进入服务创建向导,本次测试将用标准资费(免费集群的worker节点只有一个,无法满足测试要求),可以注意到的是ibm k8s的在线版本很新,选择最近的稳定版本1.17.6
在这里插入图片描述
接下来选择经典基础架构(覆盖全),vpc基础架构在六大区(东京,悉尼,法兰克福,伦敦,华盛顿,达拉斯)也已完全就绪,考虑vpc高可用区集群的worker成本,在单区开一套经典做测试性价比会高一点。
在这里插入图片描述
最后我们设置worker pool,选择三个2c*4g虚拟机作为工作节点,若考虑安全可以打开节点磁盘加密(ibm这点真的没话说),定义完集群名字,点创建
在这里插入图片描述
大约五六分钟,集群初始化完成,进入“Normal”状态
在这里插入图片描述
点击创建好的集群,进入“访问”,参考步骤在自己的笔记本上安装ibmcloud cli (k8s相关工具比如kubectl,docker等如果当前系统没有,会一并安装)
在这里插入图片描述









danws-MacBook-Pro:k8s-playground danw$ ibmcloud login --sso -a cloud.ibm.com -r jp-tok -g danw-group
danws-MacBook-Pro:k8s-playground danw$ ibmcloud ks cluster config --cluster bro62jit0fjj42dqsh7g
danws-MacBook-Pro:k8s-playground danw$ export KUBECONFIG=/Users/danw/.bluemix/plugins/container-service/clusters/bro62jit0fjj42dqsh7g/kube-config-tok05-demo-k8s-tok05.yml

也可将集群配置文件添加到~/.bash_profile中,
在这里插入图片描述
测试连接成功!

danws-MacBook-Pro:k8s-playground danw$ k get nodes -o wide
NAME           STATUS   ROLES    AGE   VERSION       INTERNAL-IP    EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
10.193.33.31   Ready    <none>   26m   v1.17.6+IKS   10.193.33.31   165.192.96.2    Ubuntu 18.04.4 LTS   4.15.0-101-generic   containerd://1.3.4
10.193.33.33   Ready    <none>   26m   v1.17.6+IKS   10.193.33.33   165.192.96.12   Ubuntu 18.04.4 LTS   4.15.0-101-generic   containerd://1.3.4
10.193.33.44   Ready    <none>   27m   v1.17.6+IKS   10.193.33.44   165.192.96.7    Ubuntu 18.04.4 LTS   4.15.0-101-generic   containerd://1.3.4

2. 安装IBM Cloud云存储插件
a. IBM Cloud File Storage
默认file stroage class(存储插件)在集群创建过程中预装好了,可直接使用

danws-MacBook-Pro:k8s-playground danw$ k get storageclasses | grep file
default                    ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-bronze           ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-bronze-gid       ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-custom           ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-gold (default)   ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-gold-gid         ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-retain-bronze    ibm.io/ibmc-file   Retain          Immediate           false                  37m
ibmc-file-retain-custom    ibm.io/ibmc-file   Retain          Immediate           false                  37m
ibmc-file-retain-gold      ibm.io/ibmc-file   Retain          Immediate           false                  37m
ibmc-file-retain-silver    ibm.io/ibmc-file   Retain          Immediate           false                  37m
ibmc-file-silver           ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-silver-gid       ibm.io/ibmc-file   Delete          Immediate           false                  37m

这里测试我们选择“silver”( 按小时计费,4 IOPS/GB)
在这里插入图片描述

danws-MacBook-Pro:k8s-playground danw$ kubectl describe storageclass ibmc-file-bronze
Name:            ibmc-file-bronze
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true"},"name":"ibmc-file-bronze"},"parameters":{"billingType":"hourly","classVersion":"2","iopsPerGB":"2","sizeRange":"[20-12000]Gi","type":"Endurance"},"provisioner":"ibm.io/ibmc-file","reclaimPolicy":"Delete"}

Provisioner:           ibm.io/ibmc-file
Parameters:            billingType=hourly,classVersion=2,iopsPerGB=2,sizeRange=[20-12000]Gi,type=Endurance
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

b. IBM Cloud Block Storage
集群默认没有预装“块存储“插件,这里需要先安装,官方在线文档可参考具体步骤。

https://cloud.ibm.com/docs/containers?topic=containers-block_storage#install_block

首先确认集群工作节点状态

danws-MacBook-Pro:k8s-playground danw$ ibmcloud ks worker ls --cluster demo-k8s-tok05
OK
ID                                                       Public IP       Private IP     Flavor              State    Status   Zone    Version
kube-bro62jit0fjj42dqsh7g-demok8stok0-default-000001b3   165.192.96.7    10.193.33.44   u3c.2x4.encrypted   normal   Ready    tok05   1.17.6_1527
kube-bro62jit0fjj42dqsh7g-demok8stok0-default-0000022e   165.192.96.2    10.193.33.31   u3c.2x4.encrypted   normal   Ready    tok05   1.17.6_1527
kube-bro62jit0fjj42dqsh7g-demok8stok0-default-000003fe   165.192.96.12   10.193.33.33   u3c.2x4.encrypted   normal   Ready    tok05   1.17.6_1527

块存储和后面的COS插件安装需要依赖helm chart,这里我们先安装helm v3,这里helm安装过程已省略,请参考以下连接:
https://github.com/helm/helm/releases/tag/v3.2.4

danws-MacBook-Pro:k8s-playground danw$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}

danws-MacBook-Pro:k8s-playground danw$ helm repo add iks-charts https://icr.io/helm/iks-charts
"iks-charts" has been added to your repositories


danws-MacBook-Pro:k8s-playground danw$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "istio.io" chart repository
...Successfully got an update from the "iks-charts" chart repository
Update Complete. ⎈ Happy Helming!⎈

danws-MacBook-Pro:k8s-playground danw$ helm search repo iks-charts
NAME                                    	CHART VERSION	APP VERSION	DESCRIPTION
iks-charts/ibm-block-storage-attacher   	1.1.4        	           	A Helm chart for installing ibmcloud block stor...
iks-charts/ibm-iks-cluster-autoscaler   	1.1.7        	           	A Helm chart for installing the IBM Cloud clust...
iks-charts/ibm-object-storage-plugin    	1.0.7        	1.0.7      	A Helm chart for installing ibmcloud object sto...
iks-charts/ibm-worker-recovery          	1.10.71      	           	IBM Autorecovery system allows automatic recove...
iks-charts/ibmcloud-alb-metrics-exporter	1.0.25       	25.0       	A Helm chart to deploy IBM Cloud ALB metrics ex...
iks-charts/ibmcloud-backup-restore      	1.0.1        	           	A Helm chart for taking backup of pvc data and ...
iks-charts/ibmcloud-block-storage-plugin	1.7.0        	           	A Helm chart for installing ibmcloud block stor...
iks-charts/ibmcloud-data-shield         	1.18.731     	1.0        	Protect data in use with IBM Cloud Data Shield
iks-charts/ibmcloud-image-enforcement   	0.2.10       	           	A Helm chart to install IBM Container Image Sec...
iks-charts/strongswan                   	2.6.3        	20.05.15   	A strongSwan IPSec VPN service to securely conn...

danws-MacBook-Pro:k8s-playground danw$ helm install 1.7.0 iks-charts/ibmcloud-block-storage-plugin -n default

danws-MacBook-Pro:k8s-playground danw$ kubectl get pods -n kube-system | grep ibmcloud-block-storage
ibmcloud-block-storage-driver-792wk                   1/1     Running   0          62s
ibmcloud-block-storage-driver-m4phr                   1/1     Running   0          62s
ibmcloud-block-storage-driver-n9pxv                   1/1     Running   0          62s
ibmcloud-block-storage-plugin-665f4b5fb5-4fn9t        1/1     Running   0          61s

danws-MacBook-Pro:k8s-playground danw$ kubectl get storageclasses | grep block
ibmc-block-bronze          ibm.io/ibmc-block   Delete          Immediate           true                   2m50s
ibmc-block-custom          ibm.io/ibmc-block   Delete          Immediate           true                   2m50s
ibmc-block-gold            ibm.io/ibmc-block   Delete          Immediate           true                   2m50s
ibmc-block-retain-bronze   ibm.io/ibmc-block   Retain          Immediate           true                   2m50s
ibmc-block-retain-custom   ibm.io/ibmc-block   Retain          Immediate           true                   2m50s
ibmc-block-retain-gold     ibm.io/ibmc-block   Retain          Immediate           true                   2m50s
ibmc-block-retain-silver   ibm.io/ibmc-block   Retain          Immediate           true                   2m50s
ibmc-block-silver          ibm.io/ibmc-block   Delete          Immediate           true                   2m50s

注:下篇我们再探讨COS pvc

3. 使用dbench对上述两种类型的pvc进行fio测试
github一个开源的容器fio测试image,可以拿来直接使用,clone repo先

danws-MacBook-Pro:dbench danw$ git clone https://github.com/leeliu/dbench.git

⚠️: 这里有个小雷,dbench image目前移动到这个repo “ndrpnt/dbench:1.0.0”

fio执行的脚本文件内容可参考下面链接,
https://github.com/leeliu/dbench/blob/master/docker-entrypoint.sh
虽然脚本已经打包到镜像中,但用户可以通过pod环境变量对部分参数进行传参调整比如快速测试,测试文件大小,数据偏移等。

编辑dbench.yaml,因为集群三个worker节点(2c4g)计算资源有限,担心三种存储同时执行fio测试,会影响到彼此的I/O效率,所以我们可以更改yaml分别挂载不同存储类型的pvc分开测试。

danws-MacBook-Pro:dbench danw$ ls -lhrt *-dbench*
-rw-r--r--  1 danw  staff   977B Jun 24 00:27 cfs-dbench.yaml
-rw-r--r--  1 danw  staff   977B Jun 24 00:27 cbs-dbench.yaml
-rw-r--r--  1 danw  staff   977B Jun 24 00:27 cos-dbench.yaml

Example: cfs-dbench.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: cfs-dbench
spec:
  template:
    spec:
      containers:
      - name: cfs-dbench
        image: ndrpnt/dbench:1.0.0
        imagePullPolicy: Always
        env:
          - name: DBENCH_MOUNTPOINT
            value: /data
          - name: DBENCH_QUICK
            value: "no"
          - name: FIO_SIZE
            value: 10G
          - name: FIO_OFFSET_INCREMENT
            value: 256M
          - name: FIO_DIRECT
            value: "1"
        volumeMounts:
        - name: cfs-vol
          mountPath: /data
      restartPolicy: Never
      volumes:
      - name: cfs-vol
        persistentVolumeClaim:
          claimName: cfs-pvc
  backoffLimit: 4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cfs-pvc
  labels:
    billingType: "hourly"
    region: jp-tok
    zone: tok05
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi
  storageClassName: ibmc-file-silver
danws-MacBook-Pro:dbench danw$ k get po
NAME               READY   STATUS    RESTARTS   AGE
cfs-dbench-m6ksp   1/1     Running   0          7m20s
danws-MacBook-Pro:dbench danw$ k top nodes
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
10.193.33.31   1223m        63%    714Mi           25%
10.193.33.33   117m         6%     1204Mi          42%
10.193.33.44   153m         7%     1102Mi          38%
danws-MacBook-Pro:dbench danw$ k logs -f job/cfs-dbench
Working dir: /data

Testing Read IOPS...
fio: posix_fallocate fails: Not supported
read_iops: (g=0): rw=randread, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process
read_iops: Laying out IO file(s) (1 file(s) / 10240MiB)
...

文件存储测试结果:

接下来看看块存储,
Example: cbs-dbench.yaml


apiVersion: batch/v1
kind: Job
metadata:
  name: cbs-dbench
spec:
  template:
    spec:
      containers:
      - name: cbs-dbench
        image: ndrpnt/dbench:1.0.0
        imagePullPolicy: Always
        env:
          - name: DBENCH_MOUNTPOINT
            value: /data
          - name: DBENCH_QUICK
            value: "no"
          - name: FIO_SIZE
            value: 10G
          - name: FIO_OFFSET_INCREMENT
            value: 256M
          - name: FIO_DIRECT
            value: "1"
        volumeMounts:
        - name: cbs-vol
          mountPath: /data
      restartPolicy: Never
      volumes:
      - name: cbs-vol
        persistentVolumeClaim:
          claimName: cbs-pvc
  backoffLimit: 4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cbs-pvc
  labels:
    billingType: "hourly"
    region: jp-tok
    zone: tok05
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Gi
  storageClassName: ibmc-block-silver
danws-MacBook-Pro:dbench danw$ k get po
NAME               READY   STATUS      RESTARTS   AGE
cbs-dbench-dv59g   1/1     Running     0          102s
cfs-dbench-m6ksp   0/1     Completed   0          20m

danws-MacBook-Pro:dbench danw$ k get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
cbs-pvc   Bound    pvc-ce517510-d529-4f66-a5b9-a0d2d9a58502   500Gi      RWO            ibmc-block-silver             11m
cfs-pvc   Bound    pvc-55ad15de-2deb-4bbd-a805-d45e14831ed1   500Gi      RWX            ibmc-file-silver              30m

danws-MacBook-Pro:dbench danw$ k top nodes
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
10.193.33.31   1173m        61%    718Mi           25%
10.193.33.33   98m          5%     1206Mi          42%
10.193.33.44   160m         8%     1104Mi          38%

块存储测试结果:
在这里插入图片描述
测试完毕,清空退掉资源

danws-MacBook-Pro:dbench danw$ k delete -f cbs-dbench.yaml
job.batch "cbs-dbench" deleted
persistentvolumeclaim "cbs-pvc" deleted
danws-MacBook-Pro:dbench danw$ k delete -f cfs-dbench.yaml
job.batch "cfs-dbench" deleted
persistentvolumeclaim "cfs-pvc" deleted

总结: 500G * 4 IOPS tier 的文件和块存储,最后测试的IOPS(read+write)都符合承诺的2000,带宽分别是128MiB/S,说明k8s容器虚拟化的I/O消耗并没有想象中那么大,值得一提的是IBM Cloud的文件和块存储在集群中供应速度很快,而且相应的storage class,包括动态PV和k8s volume yaml定义支持都不错。 当然不同的业务属性、不同结构的数据,数据冷热度、访问要求决定了持久化存储的选择,如何针对自身业务做好k8s存储规划,可以看看下面这个链接。
https://cloud.ibm.com/docs/containers?topic=containers-storage_planning

下一篇我们接着探讨COS的挂载以及fio测试。

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!