Support parallel builds

[elitak]

I would like to be able to compile with make -j32 all. This works in betaflight. The regression here is caused by src/utils/build_stamp.rb being run simultaneously for each target in the Makefile, clobbering the same obj/build-stamp/object.cc (iirc) file, creating a race condition for each target's build.

I think this should be fairly trivial to fix: simply create a distinct stamp for each target, so they can be created and used in parallel without interference. I'm not familiar enough with the tooling and the purpose of the stamp to do it myself.

[digitalentity]

There's another issue to parallel builds. fc/settings.c is autogenerated by a script. This file also needs to be compiled per-target. @fiam do you think it would be possible to make parallel builds possible?

[elitak]

Thanks for addressing this, but the HEAD of master doesn't seem to parallel build anymore:

$ nix-build -A inav
these derivations will be built:
  /nix/store/cwpshhnh24y9l9yfrhcxkl2l06abrn9w-inav-1.9.1.drv
building '/nix/store/cwpshhnh24y9l9yfrhcxkl2l06abrn9w-inav-1.9.1.drv'...
unpacking sources
unpacking source archive /nix/store/q8vv6jysb058rdcbwh4x9mvq8k1zljkp-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
build flags: -j32 -l32 SHELL=/nix/store/m47apl3hq3i52fzy2cz24378p0xn4lyx-bash-4.4-p19/bin/bash all
echo "" && \
echo "Building AIRBOTF4" && \
make -j 8 TARGET=AIRBOTF4 && \
echo "Building AIRBOTF4 succeeded."

Building AIRBOTF4
make[1]: warning: -jN forced in submake: disabling jobserver mode.
make[1]: Entering directory '/tmp/nix-build-inav-1.9.1.drv-0/source'
settings.yaml -> settings_generated.h, settings_generated.c
%% startup_stm32f40xx.s
%% inverter.c
%% serial_softserial.c
%% serial_uart_stm32f4xx.c
%% system_stm32f4xx.c
%% timer_stm32f4xx.c
...

For some reason the -j option is no longer honored?

[elitak]

Sorry, nevermind. I wasn't in the development branch.

(may still have the problem stated above; investigating)

Even after building on the development branch with the -j8 args excised, the build targets are processed serially. Any advice?

[fiam]

@elitak Seems to be working fine here using GNU Make 3.81:

muk:inav fiam$ make -j32 all
echo "" && \
	echo "Building AIRBOTF4" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=AIRBOTF4 && \
	echo "Building AIRBOTF4 succeeded."
echo "" && \
	echo "Building AIRHEROF3" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=AIRHEROF3 && \
	echo "Building AIRHEROF3 succeeded."
echo "" && \
	echo "Building AIRHEROF3_QUAD" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=AIRHEROF3_QUAD && \
	echo "Building AIRHEROF3_QUAD succeeded."
echo "" && \
	echo "Building ALIENFLIGHTF3" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=ALIENFLIGHTF3 && \
	echo "Building ALIENFLIGHTF3 succeeded."
echo "" && \
	echo "Building ALIENFLIGHTF4" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=ALIENFLIGHTF4 && \
	echo "Building ALIENFLIGHTF4 succeeded."
echo "" && \
	echo "Building ALIENFLIGHTNGF7" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=ALIENFLIGHTNGF7 && \
	echo "Building ALIENFLIGHTNGF7 succeeded."
echo "" && \
	echo "Building ANYFC" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=ANYFC && \
	echo "Building ANYFC succeeded."
echo "" && \
	echo "Building ANYFCF7" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=ANYFCF7 && \
	echo "Building ANYFCF7 succeeded."
echo "" && \
	echo "Building ANYFCF7_EXTERNAL_BARO" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=ANYFCF7_EXTERNAL_BARO && \
	echo "Building ANYFCF7_EXTERNAL_BARO succeeded."

Building AIRBOTF4

Building AIRHEROF3

Building AIRHEROF3_QUAD
echo "" && \
	echo "Building ANYFCM7" && \
	/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 TARGET=ANYFCM7 && \
	echo "Building ANYFCM7 succeeded."

Building ALIENFLIGHTF3

Building ALIENFLIGHTNGF7
...

[elitak]

Oh, the -l flag was preventing more than one job from starting, my bad. Everything works as expected now. Thanks again for fixing the issue.

[fiam]

You’re welcome :-)