What are some good strategies to allow deployed applications to be hotfixable?

Question

In an ideal world, our development processes would be perfect, resulting in regular releases that were so thoroughly tested that it would never be necessary to \"hotfix\" a

Answer 1

One strategy is to heavily use declarative-style external configuration files for the different components.

Examples of this:

• Database access/object-relational mapping via a tool like IBatis/IBatis.NET
• Logging via a tool like JLog/NLog
• Dependency injection via a tool like Spring/Spring.NET

In this way, you can often keep key components separated into discrete parts, hotfix a running application without recompile, and seamlessly use source control (particularly in comparison to stored procedures, which usually require manual effort to source control).

Answer 2

[Even though we test a lot before we release, ] What we do is this:

Our SVN looks like this:

/repo/trunk/
/repo/tags/1.1
/repo/tags/1.2
/repo/tags/1.3

Now whenever we release, we create a tag which we eventually check out in production. Before we do production, we do staging which is [less servers but] pretty much the same as production.

Reasons to create a "tag" include that some of the settings of our app in production code are slightly different (e.g. no errors are emailed, but logged) from "trunk" anyway, so it makes sense to create the tag and commit those changes. And then checkout on the production cluster.

Now whenever we need to hotfix an issue, we fix it in tags/x first and then we svn update from the tag and are good. Sometimes we go through staging, with some issues (e.g. minor/trivial fixes like spelling) we by-pass staging.

The only thing to remember is to apply all patches from tags/x to trunk.

If you have more than one server, Capistrano is extremely helpful to run all those operations.

Answer 3

We divide our code in framework code and business customizations. Business customization classes are loaded using a separate classloader and we have tool to submit changes to a running instance of production. whenever we need a change in any class we change it and submit it to a running instance. the running instance will reject the old classloader and use a new classloader insance to load the classes again. This is similar to Jboss hot deploy of EJBs.