Best practices when implementing CI/CD pipeline using GitHub/Jenkins/Kubernetes

Question

This question is more advice related so I hope its not flagged for anything. Just really need help :(

Trying to implement CI/CD using GitHub/Jenkins/Kubernetes.

On a highlevel this is what should happen:

1. Build on Jenkins
2. Push to container registry
3. Deploy built image on Kubernetes development cluster
4. Once testing finished on Development cluster, deploy it on a client testing cluster and finally production cluster

So far this is what I have created a job on Jenkins which will be triggered using a Github hook. This job is responsible for the following things:

1. Checkout from GitHub
2. Run unit tests / call REST API and send unit test results
3. Build artifacts using maven / call REST API and inform if build success or fail
4. Build docker image
5. Push docker image to container registry (docker image will have incremented versions which match with the BUILD_NUMBER environment variable)

The above stated tasks are more or less completed and I dont need much assitance with it (unless anyone thinks the aforementioned steps are not best practice)

I do need help with the part where I deploy to the Kubernetes cluster.

For local testing, I have set up a local cluster using Vagrant boxes and it works. In order to deploy the built image on the development cluster, I am thinking about doing it like this: Point Jenkins build server to Kubernetes development cluster Deploy using deployment.yml and service.yml (available in my repo) This part I need help with...

Is this wrong practice? Is there a better/easier way to do it?

Also is there a way to migrate between clusters? Ex: Development cluster to client testing cluster and client testing cluster to production cluster etc

When searching on the internet, the name Helm comes up a lot but I am not sure if it will be applicable to my use case. I would test it and see but I am a bit hard pressed for time which is why I cant

Would appreciate any help y'all could provide.

Thanks a lot

Answer 1

There are countless ways of doing this. Take Helm out for now as you are just starting.

If you are already using Github and docker , then I would just recommend you to push your code/changes/config/Dockerfile to Github that will auto trigger a docker build on Dockerhub ( maybe jenkins in ur case if u dont want to use dockerhub for builds ) , it can be a multi-stage docker build where you can build code , run tests , throw away dev environmenet , and finally produce a producion docker image , once the image is produced , it will triger a web hook to your kubernetes deployment job/manifests to deploy on to test evironmenet , followed by manual triiger to deploy to production.

The docker images can be tagged based on SHA of the commits in Github/Git so that you can deploy and rollback based on commits.

Reference: https://cloud.google.com/kubernetes-engine/docs/tutorials/gitops-cloud-build

Here is my Gitlab implementation of Gtips workflow:

# Author , IjazAhmad

image: docker:latest

stages:
  - build
  - test
  - deploy

services:
  - docker:dind

variables:
  CI_REGISTRY: dockerhub.example.com
  CI_REGISTRY_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH
  DOCKER_DRIVER: overlay2

before_script:
  - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY  

docker-build:
  stage: build
  script:
     - docker pull $CI_REGISTRY_IMAGE:latest || true
     - docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest .

docker-push:
  stage: build
  script:
     - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
     - docker push $CI_REGISTRY_IMAGE:latest

unit-tests:
  stage: test
  script:
    - echo "running unit testson the image"
    - echo "running security testing on the image"
    - echo "pushing the results to build/test pipeline dashboard"


sast:
  stage: test
  script:
    - echo "running security testing on the image"
    - echo "pushing the results to build/test pipeline dashboard"


dast:
  stage: test
  script:
    - echo "running security testing on the image"
    - echo "pushing the results to build/test pipeline dashboard"


testing:
  stage: deploy
  script:
     - sed -i "s|CI_IMAGE|$CI_REGISTRY_IMAGE|g" k8s-configs/deployment.yaml
     - sed -i "s|TAG|$CI_COMMIT_SHA|g" k8s-configs/deployment.yaml
     - kubectl apply --namespace webproduction-test -f k8s-configs/
  environment:
    name: testing
    url: https://testing.example.com

  only:
    - branches


staging:
  stage: deploy
  script:
     - sed -i "s|CI_IMAGE|$CI_REGISTRY_IMAGE|g" k8s-configs/deployment.yaml
     - sed -i "s|TAG|$CI_COMMIT_SHA|g" k8s-configs/deployment.yaml
     - kubectl apply --namespace webproduction-stage -f k8s-configs/
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - master



production:
  stage: deploy
  script:
     - sed -i "s|CI_IMAGE|$CI_REGISTRY_IMAGE|g" k8s-configs/deployment.yaml
     - sed -i "s|TAG|$CI_COMMIT_SHA|g" k8s-configs/deployment.yaml
     - kubectl apply --namespace webproduction-prod -f k8s-configs/    
  environment:
    name: production
    url: https://production.example.com
  when: manual
  only:
    - master

Links:

Trigger Jenkins builds by pushing to Github

Triggering a Jenkins build from a push to Github

Jenkins: Kick off a CI Build with GitHub Push Notifications

Answer 2

Look at spinnaker for continuous delivery. After the image is built and pushed to registry, have a web hook in spinnaker trigger a deployment to required kubernetes cluster. Spinnaker works well with kubernetes and you definitely should try it out

Answer 3

I understand that you are trying to implement GitOps, my advice is to review this article where you can start to figure out a little bit more about the components you need.

https://www.weave.works/blog/managing-helm-releases-the-gitops-way

Basically, you need to implement your own helm charts for your custom services and manage it using flux, I recommend to use a different repository per environment and leave flux to manage the deployment to each environment based on the state of the master branch on the repo.