GNU make: should the number of jobs equal the number of CPU cores in a system?

Question

There seems to be some controversy on whether the number of jobs in GNU make is supposed to be equal to the number of cores, or if you can optimize the build time by adding

Answer 1

Both are not wrong. To be at peace with yourself and with author of software you're compiling (different multi-thread/single-thread restrictions apply at software level itself), I suggest you use:

make -j`nproc`

Notes: nproc is linux command that will return number of cores/threads(modern CPU) available on system. Placing it under ticks ` like above will pass the number to the make command.

Additional info: As someone mentioned, using all cores/threads to compile software can literally choke your box to near death (being unresponsive) and might even take longer than using less cores. As I seen one Slackware user here posted he had dual core CPU but still provided testing up to j 8, which stopped being different at j 2 (only 2 hardware cores that CPU can utilize). So, to avoid unresponsive box i suggest you run it like this:

make -j`nproc --ignore=2`

This will pass the output of nproc to make and subtract 2 cores from its result.

Answer 2

YES! On my 3950x, I run -j32 and it saves hours of compile time! I can still watch youtube, browse the web, etc. during compile without any difference. The processor isn't always pegged even with a 1TB 970 PRO nvme or 1TB Auros Gen4 nvme and 64GB of 3200C14. Even when it is, I don't notice UI wise. I plan on testing with -j48 in the near future on some big upcoming projects. I expect, as you probably do, to see some impressive improvement. Those still with a quad-core might not get the same gains....

Linus himself just upgraded to a 3970x and you can bet your bottom dollar, he is at least running -j64.

Answer 3

I would say the best thing to do is benchmark it yourself on your particular environment and workload. Seems like there are too many variables (size/number of source files, available memory, disk caching, whether your source directory & system headers are located on different disks, etc.) for a one-size-fits-all answer.

My personal experience (on a 2-core MacBook Pro) is that -j2 is significantly faster than -j1, but beyond that (-j3, -j4 etc.) there's no measurable speedup. So for my environment "jobs == number of cores" seems to be a good answer. (YMMV)

Answer 4

Many years later, the majority of these answers are still correct. However, there has been a bit of a change: Using more jobs than you have physical cores now gives a genuinely significant speedup. As an addendum to Dascandy's table, here's my times for compiling a project on a AMD Ryzen 5 3600X on linux. (The Powder Toy, commit c6f653ac3cef03acfbc44e8f29f11e1b301f1ca2)

I recommend checking yourself, but I've found with input from others that using your logical core count for job count works well on Zen. Alongside that, the system does not seem to lose responsiveness. I imagine this applies to recent Intel CPUs as well. Do note I have an SSD, as well, so it may be worth it to test your CPU yourself.

scons -j1 --release --native  120.68s user 9.78s system 99% cpu 2:10.60 total
scons -j2 --release --native  122.96s user 9.59s system 197% cpu 1:07.15 total
scons -j3 --release --native  125.62s user 9.75s system 292% cpu 46.291 total
scons -j4 --release --native  128.26s user 10.41s system 385% cpu 35.971 total
scons -j5 --release --native  133.73s user 10.33s system 476% cpu 30.241 total
scons -j6 --release --native  144.10s user 11.24s system 564% cpu 27.510 total
scons -j7 --release --native  153.64s user 11.61s system 653% cpu 25.297 total
scons -j8 --release --native  161.91s user 12.04s system 742% cpu 23.440 total
scons -j9 --release --native  169.09s user 12.38s system 827% cpu 21.923 total
scons -j10 --release --native  176.63s user 12.70s system 910% cpu 20.788 total
scons -j11 --release --native  184.57s user 13.18s system 989% cpu 19.976 total
scons -j12 --release --native  192.13s user 14.33s system 1055% cpu 19.553 total
scons -j13 --release --native  193.27s user 14.01s system 1052% cpu 19.698 total
scons -j14 --release --native  193.62s user 13.85s system 1076% cpu 19.270 total
scons -j15 --release --native  195.20s user 13.53s system 1056% cpu 19.755 total
scons -j16 --release --native  195.11s user 13.81s system 1060% cpu 19.692 total
( -jinf test not included, as it is not supported by scons.)

Tests done on Ubuntu 19.10 w/ a Ryzen 5 3600X, Samsung 860 Evo SSD (SATA), and 32GB RAM

Final note: Other people with a 3600X may get better times than me. When doing this test, I had Eco mode enabled, reducing the CPU's speed a little.