How to use GNU make --max-load on a multicore Linux machine?

Question

From the documentation for GNU make: http://www.gnu.org/software/make/manual/make.html#Parallel

When the system is heavily loaded, you will probably w

Answer 1

Even for a build where the CPU is the bottleneck, -l is not ideal. I use -jN, where N is the number of cores that exist or that I want to spend on the build. Choosing a bigger number doesn't speed up the build in my situation. It doesn't slow it down either, as long as you don't go overboard (such as by specifying infinite through -j).

Using -lN is broadly equivalent to -jN, and can work better if the machine has other independent work to do, but there are two quirks (apart from the one you mentioned, the number of cores not accounted for):

• Initial spike: when the build starts, make launches a lot of jobs, many more than N. The system load number doesn't immediately increase when a process is forked. That's not a problem in my situation.
• Starvation: when some build jobs take a long time compared to others, at the moment the first M quick jobs have ended, the system load is still >N. Soon the system load drops to N - M, but as long as those few slow jobs are dragging on, no new jobs are launched, and cores are left hungry. Make only thinks about launching new jobs when an old job ends, and at the start. It doesn't notice the system load dropping in between.