How to handle large number of configuration parameters across a program conceptually? [closed]

Question

Imagine a software system where you have like 100 parameters, each of which is sensible to change (in my case this is a reasearch system for machine learning). The obvious solution is to store all parameters in a config file, so that the system is easy to handle and see through.

Whis approach is better:

• a) load the config file at the entry point of the program and pass down a big collection of config variables through each method in the code
• b) load the config file at the entry point of the program and pass down the relevant collection of config variables through each method in the code
• c) load the config variables directly where they are needed
• d) load the config and make it global

I'm open for suggestions or examples of a particular implementation. At the moment I'm experimenting with nested config variables, each nested object storing config of different modules in the code.

Answer 1

I recommend you do the following steps.

First, use a configuration file syntax that supports the concept of multiple sections/scopes, and preferably nestable scopes. For example, an XML element is a nestable scope in the sense I am talking about, while a ".ini" file provides non-nestable sections. A Java properties file does not provide direct support for scopes, but you can emulate such support by using the syntax x.y.z to denote variable z, in scope y, which in turn is nested in scope x. Config4* (which I developed) provides direct support for nested scopes.

Second, write the constructor of a configurable Foo class as shown in the following pseudocode:

Foo(Configuration cfg, String scope) {
    _x = cfg.lookup(scope + ".x");
    _y = cfg.lookup(scope + ".y");
    _z = cfg.lookup(scope + ".z");
    _bar = new Bar(cfg, scope + ".bar");
}

In the above pseudo code, I use the _ prefix to denote an instance variable, and I assume the Configuration class has a lookup() operation that takes a scoped name, for example, cfg.lookup("foo.bar.abc") will return the value of the abc variable in the foo.bar scope.

Third, the main() function of your application can be written as shown in the following pseudocode:

main(...) {
    String configFileName = ...; // obtain from command-line argument
    String scope          = ...; // obtain from command-line argument
    Configuration cfg     = parseConfigurationFile(configFileName);
    Foo foo               = new Foo(cfg, scope + ".foo");
    ... // create other configured objects
    doRealWork(foo, ...);
}

Finally, command-line arguments to your application should specify: (1) the name of a configuration file, and (2) a top-level scope (within the configuration file) that holds configuration variables for running the application. For example, let's assume example.cfg is structured as follows:

instance1 {
    foo {
        x = "a value";
        y = "another value";
        z = "yet another value";
        bar {
           ...
        }
    }
    ... # configuration for other objects
}

instance2 {
    foo {
        x = "...";
        y = "...";
        z = "...";
        bar {
           ...
        }
    }
    ... # configuration for other objects
}

You might run your application as myApp.exe -cfg example.cfg -scope instance1.

The above advice provides the following benefits:

• Your application can create multiple Foo objects, each of which can be configured differently, simply by passing a different scope parameter to the constructor of each object.
• Users have the flexibility of being able to store multiple sets of configuration variables inside a single configuration file, if they want. For example, a user might have different sets of configuration variables for: (1) different unit tests; (2) development, UAT and production environments; (3) multiple instances of a replicated server application.
• Over time, you can write a library of configurable classes that follow the above design principle. That library of configurable classes can be reused across multiple applications.

I have used the above approach in several C++ and Java Config4*-based applications that I wrote, and it has worked well for me. If you are using a language that has built-in support for reflection (such as Java), then an alternative approach is to use a dependency injection framework. If you don't know what that is, then do an Internet search for "dependency injection", "inversion of control" or "Spring Framework".