Number of parallel build jobs in recursive make call

Question

I have a makefile which wraps the real build in a single recursive call, in order to grab and release a license around the build, regardless of whether the build succeeds. Example:

.PHONY main_target
main_target:
    @license_grab &
    @sleep 2
    -@$(MAKE) real_target
    @license_release

This works great, until I want to run the make in parallel. On some systems it works fine, but on other systems I run into the documented problem with the -j flag when passing options to a sub-make:

If your operating system doesn’t support the above communication, then ‘-j 1’ is always put into MAKEFLAGS instead of the value you specified. This is because if the ‘-j’ option were passed down to sub-makes, you would get many more jobs running in parallel than you asked for.

Since I only ever have one recursive call, I'd really like to pass the -j flag untouched to the sub-make. I don't want to hard-code the -j flag (with a number) into the makefile, because the makefile runs on multiple machines, with differing numbers of processors. I don't want to use -j with no limit, because that launches a bunch of processes right away rather than waiting for jobs to finish. I tried using the -l flag when I build, but I found that the limit doesn't apply right away, probably because limits don't apply until make can start sampling.

Is there a way to force a sub-make to use multiple jobs? Or, a way to make the -l flag actually accomplish something?

I think I could do it using a makefile modification like using -@$(MAKE) $(JOBS) real_target, and invoking make like make JOBS="-j4" main_target.

But, I'd prefer to just use standard make parameters, not adding extra dependencies on variables. I'd also prefer to modify the makefile as little as possible.

Answer 1

There is no way to change this behavior on systems which don't support jobserver capabilities. Something like your JOBS variable is the only way I can think of.

I'll point out a few things: first, I assume when you say other systems you mean Windows systems: that's the only commonly-used platform I'm aware of which doesn't support the jobserver. If so, note that as of GNU make version 4.0, jobserver support is implemented on Windows. So another option is to upgrade to a newer version of GNU make.

Second, the issues with the -l option were at least partly solved in GNU make version 3.81, where an extra algorithm was introduced to artificially adjust the load average based on the number of jobs make invokes. You shouldn't see this issue any longer with versions of make after that.

Answer 2

The problem is that for whatever reason my make does not support the job server, thus it does not pass the -j flag on in the $(MAKEFLAGS) variable. Since upgrading make to a version that does pass the flag is not an option, the solution is to pass the -j flag elsewhere. Just like the $(MAKE) variable can be abused to pass the -f flag, it can be used in the same way to pass the -j flag:

make MAKE="make -j4" main_target

This will start the main_target build with one job, but invoke make with 4 jobs on the sub-make process. Obviously, if you need a special make tool (the normal purpose of $(MAKE) then you'll need to specify it in the MAKE string as well as in the command.