Unreliable parallel builds in a makefile with .INTERMEDIATE?

Question

I have a tool which generates multiple output files, which is notoriously hard to model in make. I'm using the recipe at GNU Makefile rule generating a few targets from a single source file, which seems simple and reliable. And it almost works.

Unfortunately, I'm seeing some very odd behaviour with regard to parallel builds, where it seems to be dropping dependencies sometimes; and I don't understand why.

Here's my test case:

out3: out1 out2
    touch out3

.INTERMEDIATE: out.intermediate
out1 out2: out.intermediate
out.intermediate: in
    touch out1 out2

If I build it once, it works:

$ touch in
$ make -f test.mk out3 -j4
touch out1 out2
touch out3

out1 and out2 are built together, once, which is good; then out3 is built from the result.

Now I touch the input file, simulating an incremental build, and try again:

$ touch in
$ make -f test.mk out3 -j4
touch out1 out2

That's rebuild out1 and out2, correctly... but it hasn't rebuilt out3, which it should have. Then if I do another build:

$ make -f test.mk out3 -j4
touch out3

...and it catches up.

This only applies to parallel builds. -j1 builds work fine.

This is bad --- I need to be able to rely on correct builds. Does anyone have any idea as to what's going on?

This is GNU Make 4.1.

Answer 1

The SO answer you link to has an error; surprising that so many people have apparently used it with success.

What's happening (you can add the -rRd options to your make invocation to see in more detail) is a result of GNU make's directory caching. GNU make doesn't expect the filesystem state to change in any way other than what your makefile describes, and so it caches the contents of directories to provide a significant performance increase (for large makefiles/directories).

Basically, when make runs the rule:

out.intermediate: in
        touch out1 out2

it doesn't expect that this recipe will update any targets other than the one listed in the rule: out.intermediate. If the other files out1 and out2 are already internalized in the directory cache (which they will be since they exist and we've already checked them as prerequisites to out3), then make won't go back to the filesystem and see if they've been updated or not: make "knows" they can't have changed because no rule that it ran could have changed them, according to the makefile.

There's a simple, one-character fix to make all this work. Change this line:

out1 out2: out.intermediate

to this:

out1 out2: out.intermediate ;

If you want to be more explicit you could also use this:

out1 out2: out.intermediate
        @:

or even this, for debugging:

out1 out2: out.intermediate
        @echo Do nothing

What all these have in common is that you've now defined not just a dependency relationship between the targets and prerequisite, but you've also given a recipe that make should invoke for that rule. Even if the recipe is empty (as in the first example), so make doesn't actually run any commands, make will still infer that it's possible that the timestamps on out1 and/or out2 have changed, and it will invalidate the cached modified time for those targets and re-retrieve them from the filesystem.