How to manage secrets in a Microservice / Container / Cloud environment?

Question

Microservices and Cloud is a thing. Everyone is talking and writing about. Personally i am thinking a lot about this topics: How this can be used to benefit from? What are possible challenges? How can this speedup the daily development? And how to manage all things? One question that bothers me since a few days is "How to manage secrets in a Microservice / Cloud environment?".

Imagine a company with 150 software engineers and various teams with various products. Every team is creating a software and every service needs various amounts of secrets (API-Keys, Passwords, SSH-Keys, whatever). The "old fashion" way was to create a few configuration files in a ini / yaml / txt format and read it from. 12Factor apps say: Do it per env vars.

Env vars can be set per machine and the config files can be placed there as well. This works if you got a hand full of machines and the deployment is done by a few system admins. One of the general rules say: "Don`t store secrets in a Git repo.".

Now the new world comes in. Ever team is responsible for the application they produce itself. They should be deployed and run by the team. So our company is moving to a container and self-service way (e.g. Mesos and Marathon or Kubernetes).

Of course, Dockerfiles can set env vars as well. And yes, you can ADD your config file into the Docker container during build. But with this everyone can access the secrets (e.g. from other teams). And no one knows who uses this secrets and do something dangerous.

You want to versionize your Dockerfiles as well. And applications you want to run on Marathon should be versionized (Git or whatever) as well (and applied by REST API). So where to store and manage all the secrets for this containers / apps? Because with scheduler frameworks like Swarm and Machine (for Docker), Mesos and Marathon (usable for Docker as well) or Kubernetes you don`t know where your app will be running. This will be scheduled over several machines. And most of this tools have no authentification (by default, of course this can be added by a Nginx proxy or something).

One idea to manage secrets is using a tool like Vault. But i never saw "native" support in an app. The same applies for Blackbox. And i don`t know how configuration management can solve this. I know that Chef supports encrypted databags, but afaik it is not possible to use Chef to setup/build Docker containers.

How do you manage secrets in a multi team env with several engineers in a Microservice / Container / Cloud environment?

Answer 1

There are several solutions.

First, DO NOT put your secrets into the image. That's just a bad idea, as you've realized. If you don't add your secrets at build time, you have to do it at run-time. This leaves us with a few options:

• Use environment variables as suggested by the 12 Factor App. You will then need to write a script that will populate the config files with values of these variables when the container starts up. This works, but I don't really like it, as environment variables are easily leaked (they can be seen in linked containers and docker inspect and are often included in bug reports). Also see Summon.

• Use volumes. Just mount the config file with the secrets at run-time. This works, but does mean you have a file with the secrets lying about on the host. This gets more complicated when you don't know on which host your container will run, such as when using frameworks like Swarm and Mesos.

• Use a secure k/v store such as Vault/Keywhiz. As you point out, you will need to do some scripting to get the values into the application (as with env vars). You also need to authenticate to the k/v store somehow (you may want to look at the volume drivers for Keywhiz and Vault, or use a one-use token passed via an env var).

Kubernetes already has fairly advanced support for secrets, and I would expect to see other frameworks adopt their own solutions.