Keeping testing and production server environments clean, in sync, and consistent

Question

It seems that the company that I work for is always struggling with our customers’ server environments.

Specifically, we almost always encounter pro

Answer 1

You need to start keeping track of every change that you make to the testing environment and provide a way for propagating this to the production environment.

For code, this means a versioning systems such as CVS, Subversion or GIT.

For the database, it means a structure comparison tool or deploy scripts that update the production database.

For configuration, the two systems should be exactly the same and any 'tweaking' or changing needs to first be applied to the testing server then later applied to the production server during a deploy.

Until you have a process that works, you will continue to have problems.

Answer 2

You need to make sure that any changes to the environments are done in a consistent manner.

I'd consider either starting with fresh images and enforcing a strict modification log policy, or using something like Capistrano to execute remote commands on and deploy code to all machines simultaneously.

Ideally, all requirements should be checked into your version control system (along the lines of how Rails lets you store gems in the /vendor directory and preferentially loads those at runtime), along with a readme file that describes exactly how to set up the environment (required libraries, etc). The readme file needs to be rigorously updated by anyone who makes changes to the environment.

Answer 3

Your problems are quite normal. There are at least two strategies I know work fairly well:

If you're distributing on linux, you can build rpms/debs from your development process and use the package management function. I know lots of projects do this with great success for inhouse projects.

Another alternative is to package the whole environment as some kind of shell script. This shell script can/should configure the complete environment with all settings. Normally this script is maintained by develeopment, and this script overwrites any modifications that have been made manually. A script like this is usually maintained by development, kept under version control and sent to deployment as a full distribution. We use cygwin for this. Normally the script reads some kind of configuration that may be administered by operations. I've had scripts that'd actually set up the entire system from scratch as if installing on a totally blank, freshly installed machine.

Both of these strategies should preferably include automated production of these artifacts all the way from your build script/build system. The smoother this process runs, the better for all involved parties.

Answer 4

For keeping MySQL databases in sync, I just ran into this tool:

http://schemasync.org/

I haven't actually used it yet so I can't speak to whether it's any good, but we need some way to control schema drift between test and prod so we're going to give it a shot.

Answer 5

Of course we have a general idea of why this happens. Every cloned environment starts out the same and works the same the first couple of days, but sooner or later someone reconfigure something in only one of the server environments

I think you're being generous, or you've been very lucky. As often as not shops that need to contract out for development work don't really understand the development process. If they provide you with a test environment at all what you'll get is a their production system from before the last server refresh.