How to create a local development environment for Kubernetes?

Question

Kubernetes seems to be all about deploying containers to a cloud of clusters. What it doesn\'t seem to touch is development and staging environments (or such).

Durin

Answer 1

Another great starting point is this Vagrant setup, esp. if your host OS is Windows. The obvious advantages being

• quick and painless setup
• easy to destroy / recreate the machine
• implicit limit on resources
• ability to test horizontal scaling by creating multiple nodes

The disadvantages - you need lot of RAM, and VirtualBox is VirtualBox... for better or worse.

A mixed advantage / disadvantage is mapping files through NFS. In our setup, we created two sets of RC definitions - one that just download a docker image of our application servers; the other with 7 extra lines that set up file mapping from HostOS -> Vagrant -> VirtualBox -> CoreOS -> Kubernetes pod; overwriting the source code from the Docker image.

The downside of this is NFS file cache - with it, it's problematic, without it, it's problematically slow. Even setting mount_options: 'nolock,vers=3,udp,noac' doesn't get rid of caching problems completely, but it works most of the time. Some Gulp tasks ran in a container can take 5 minutes when they take 8 seconds on host OS. A good compromise seems to be mount_options: 'nolock,vers=3,udp,ac,hard,noatime,nodiratime,acregmin=2,acdirmin=5,acregmax=15,acdirmax=15'.

As for automatic code reload, that's language specific, but we're happy with Django's devserver for Python, and Nodemon for Node.js. For frontend projects, you can of course do a lot with something like gulp+browserSync+watch, but for many developers it's not difficult to serve from Apache and just do traditional hard refresh.

We keep 4 sets of yaml files for Kubernetes. Dev, "devstable", stage, prod. The differences between those are

• env variables explicitly setting the environment (dev/stage/prod)
• number of replicas
• devstable, stage, prod uses docker images
• dev uses docker images, and maps NFS folder with source code over them.

It's very useful to create a lot of bash aliases and autocomplete - I can just type rec users and it will do kubectl delete -f ... ; kubectl create -f .... If I want the whole set up started, I type recfo, and it recreates a dozen services, pulling the latest docker images, importing the latest db dump from Staging env and cleaning up old Docker files to save space.