What's the use of metaprogramming?

Question

I\'ve read:

• Wikipedia
• Code Generation vs. Metaprogramming
• The art of Metaprogramming
• Metaprogramming at c2.com

and I

Answer 1

A specific example of where it could be a useful approach.

You have a set of third-party classes, to which you want to add generic behaviour - for example some kind of security/access control, mapping out objects as JSON, etc.

You could write or generate sub-classes for everything, adding wrapper methods to add in access control and call the superclass. With meta-programming, you can do that at runtime, and also your changes will be automatically applied to any additional / changed third party classes.

With the JSON example, by using introspection of the class you should be able to generate the code to serialise an object, and then add this as a method to the class. The other extremes would be generating or writing the code upfront (before compilation) and impacting every time the class changes, or a completely generic approach that used introspection on each individual object, each time you wanted to map it.

Depending on the language and runtime in question, a metaprogamming approach is likely to be faster than the wholly generic/introspective one, but slower that upfront code, as you have reduced a lot of data lookups into code.

Where meta-programming doesn't exist directly in a language, it also seems to me that it is often re-invented through frameworks (i.e. IoC style containers like Spring).

Answer 2

Start your Visual Studio (Eclipse, Netbeans, whatever else). Create a new project. Surprise - you've just used some metaprogramming, by creating a project from a template. Isn't it practical?

Answer 3

You could look at Common Lisp's macros or C++'s templates and see how they're used. Both are metaprogramming in the sense you're using. You'll find that both are used heavily in a lot of code.

Lisp macros are often used to redefine the language. As an example, the last chapter of Paul Graham's On Lisp creates an working object-oriented extension for Common Lisp. Another example is the now-defunct Garnet.

The old Standard Template Library for C++ (mostly incorporated in the standard library) was a way of introducing a large number of containers and algorithms that worked as if they were built into the language, at least in terms of integration and efficiency (not syntactically).

Answer 4

I've gotten the most use out of metaprogramming for bridging between different APIs.

A working example would be FireBreaths JSAPIAuto¹ that eases writing C++ classes that are exposed to JavaScript. By providing a registering facility for the functions that are to be exposed, the argument types can be inspected and from that fitting code generated at compile-time that converts from the script-API-types to native C++ types and back, even directly supporting map, vector, etc.

As a simple example, consider an exposed add(a, b) function that uses some scripting API types:

ScriptVariant add(const std::vector<ScriptVariant>& values) {
    // have to check argument count
    if(values.size() != 2)
        throw script_error("wrong number of arguments");

    try {
        // have to convert from scripting-API types
        long a = values[0].convert_cast<long>();
        long b = values[0].convert_cast<long>();
        return a+b; // potentially need to convert back too
    } catch(ScriptVariant::bad_cast& e) {
        // need to handle conversion failure
        throw script_error("conversion failed :(");
    }
}

The actual logic buried in there is only one line, that checks and conversions are annoying and redundant. With the previously mentioned registration-facility (e.g. in the constructor):

registerMethod("add", make_method(this, &MyClass::add));

this can now simply be written as:

long add(long a, long b) {
    return a+b;
}

... and the framework takes care of generating the neccessary code for you.

^{1: Although i would do implementation a bit... cleaner... if i would have to start again}

Answer 5

I think of metaprogamming as "programs that write (or modify) other programs". (Another answer said "factories that make factories", nice analogy).

People find all sorts of uses for this: customizing applications, generating boilerplate code, optimizing a program for special circumstances, implementing DSLs, inserting code to handle orthogonal design issues ("aspects") ...

What's remarkable is how many different mechanisms have been invented to do this piecemeal: text-templates, macros, preprocessor conditionals, generics, C++-templates, aspects, reflection,... And usually some of these mechanisms are built into some languages, and other mechanisms into other languages, and most languages have no metaprogramming support at all. This scatter-shot distribution of capabilities means that you might be able to do some kinds of metaprogramming in one language, with limitations, and yet not be able to do those kinds in another. That's aggravating :-}

An observation that I have been following to the hilt is that one can build generic metaprogramming machinery that works with any language in the form of program transformations. A program transformation is a parameterized pattern: "if you see this syntax, replace it by that syntax".

One transformation by itself generally isn't impressive, but dozens or hundreds can make spectacular changes to code. Because (sophisticated) program transformations can in effect simulate a Turing machine, they can carry out arbitrary code changes, including all those point-wise techniques you find scatter-shotted about.

A tool that accepts language definitions. language-specific transformations and generates another to apply those transformations is a meta-metaprogramming tool: a program to write "programs that write programs".

The value is that you can apply such tool to carry out wide varieties of changes to arbitrary code. And, you don't need the language design committee to realize that you want a particular kind of metaprogramming support, and hurry up to provide it so you can get on with your job today.

An interesting lesson is that such machinery needs strong program analysis (symbol tables, control and data flow analysis, etc.) support to help it focus on where problems are in the code, so that metaprogramming machinery can do something at that point (a very weak kind of example of this are point-cut specifications in aspects, that say "make changes at places that look like this").

The OP asked for specific examples of where metaprogramming was applied. We've used our "meta"-metaprogramming tool (DMS Software Reengineering Toolkit) to carry out the following activities on large code bases automatically:

• Language Migration
• Implementing Test Coverage and Profilers
• Implementing Clone Detection
• Massive architecture reengineering
• Code generation for factory control
• SOAization of embedded network controllers
• Architecture extraction for mainframe software
• Generation of vector SIMD instructions from array computations
• Reverse engineering of code back to concepts

across many languages, including Java, C#, C++, PHP, ...

The OP also asked, "Why was this better than the alternative?" The answer has to do with scale, time, and accuracy.

For large applications, the sheer size of the code base means you don't have the resources or the time to make such analyses or changes by hand.

For code generation or optimization tasks, you might be able to do it by hand, but the tools can do it much faster and more accurately.

In essence, these tools do what human beings simply cannot.

It is worth noting that the tools have no creativity; you still need humans to determine what to have them do, e.g., to decide what the task is (see above list for examples) and determine how to define the analyses/transformations to achieve the effect. You still need meta-programmers. However, when a meta programmer arms such a tool with the right knowledge, the resulting code can appear to be built by an incredibly fast, creative, expert coder.

Answer 6

My recent (last 6 months) concrete example of code generation:

1. I have an SQL Plus script that generates and then executes other SQL Plus scripts. The generates script runs queries against some tables that have time-stamp fields, and when I designed the script, it was impossible to know what time window to select. So, the main script does its work, and figures out what time ranges need to be in the sub scripts. Then it generates the subscripts by writing their code to file (and substituting placeholders for the actual start and end times). Finally it executes the subscript(s). I've used this trick for a few situations now (though often more complicated than this one) where the structure of the substeps depends on results of earlier steps.

2. I once got a spreadsheet mapping elements from an XSD to table columns in a database. It was possible to generate XSL snippets and complete queries from the spreadsheet using macros and VBA. These snippets and queries were copied and pasted (mostly as-is with no neede changes) into the system that executed them and processed the results. Not a pretty solution but it certainly made a very tedious job a lot less tedious, and the code that resulted was probably a lot more consistent-looking than if I had spent a week or two writing it all by hand.

SO list of examples of metaprogramming: What are the coolest examples of metaprogramming that you've seen in C++?