Premise to my answer:
- Containers are run with tags.
- The same tag can be pointed to different image UUID as we please/ feel appropriate.
- Updates done to an image can be committed to a new image layer
Approach
- Build all the containers in the first place with a security-patch update script
- Build an automated process for the following
- Run an existing image to new container with security patch script as the command
- Commit changes to the image as
- existing tag -> followed by restarting the containers one by one
- new version tag -> replace few containers with new tag -> validate -> move all containers to new tag
Additionally, the base image can be upgraded/ the container with a complete new base image can be built at regular intervals, as the maintainer feels necessary
Advantages
- We are preserving the old version of the image while creating the new security patched image, hence we can rollback to previous running image if necessary
- We are preserving the docker cache, hence less network transfer (only the changed layer gets on the wire)
- The upgrade process can be validated in staging before moving to prod
- This can be a controlled process, hence the security patches only when necessary/ deemed important can be pushed.