Parallel cmake and make #4783

publicarray · 2021-08-15T13:39:11Z

Motivation: Make compiling faster. I've only tested fish and umurmur so far. (I have an i7-10750H with 6 cores)

# fish before
real	2m29.187s
user	2m58.331s
sys	0m29.212s

# fish after
real	0m49.710s
user	4m31.117s
sys	0m35.185s

# fish ninja build (https://github.com/publicarray/spksrc/tree/cmake-ninja)
real	1m8.352s
user	6m17.500s
sys	0m47.527s

# fish https://github.com/SynoCommunity/spksrc/pull/4803
real    0m49.367s
user    4m33.264s
sys     0m35.001s

fish from 2m29s to 49 s

# python38 before 
real    10m44.831s
user    15m10.705s
sys 2m10.672s

# python38 after
real    6m6.519s
user    13m9.923s
sys 1m52.586s

publicarray · 2021-08-15T16:11:17Z

mk/spksrc.compile.mk

@@ -33,7 +33,7 @@ compile_msg:
 pre_compile_target: compile_msg

 compile_target:  $(PRE_COMPILE_TARGET)
-	@$(RUN) $(MAKE) $(COMPILE_MAKE_OPTIONS)
+	@$(RUN) $(MAKE) -j$(NCPUS) $(COMPILE_MAKE_OPTIONS)


i'm not sure if this one does anything

yes it is obsolete, as COMPILE_MAKE_OPTIONS += -j$(NCPUS) is already defined above in this file.
But all custom COMPILE_MAKE_OPTIONS must be defined with += too, to keep the -j option.

Thanks having it multiple times doesn't actually hurt AFAIK but makes things messy. Impressive improvements though when packages use the $(COMPILE_MAKE_OPTIONS) aka include the -j$(NCPUS)

hgy59 · 2021-08-15T20:58:28Z

cross/busybox/Makefile

 .PHONY: busybox_compile
 busybox_compile:
-	$(RUN) CROSS_COMPILE=$(TC_PATH)$(TC_PREFIX) $(MAKE) busybox
+	$(RUN) CROSS_COMPILE=$(TC_PATH)$(TC_PREFIX) $(MAKE) -j$(NCPUS) busybox

 .PHONY: busybox_install


most of the custom compile-targets adjusted here could be removed by definition of COMPILE_MAKE_OPTIONS
as the default compile_target executes $(RUN) $(MAKE) $(COMPILE_MAKE_OPTIONS)

busybox_compile can be removed by:
COMPILE_MAKE_OPTIONS += CROSS_COMPILE=$(TC_PATH)$(TC_PREFIX) busybox

this work only, when it does not matter that CROSS_COMPILE definition must not be left of $(MAKE) on the command line.

As spksrc.common.mk already defines COMPILE_MAKE_OPTIONS += -j$(NCPUS) this must not be added here again.

hgy59 · 2021-08-15T21:05:08Z

native/elixir/Makefile

@@ -22,7 +22,7 @@ include ../../mk/spksrc.native-cc.mk

 .PHONY: elixir_compile
 elixir_compile:
-	$(RUN) PATH=$$PATH:$(ERLANG_BIN_DIR) $(MAKE) $(MAKE_COMPILE_OPTIONS)
+	$(RUN) PATH=$$PATH:$(ERLANG_BIN_DIR) $(MAKE) -j$(NCPUS) $(MAKE_COMPILE_OPTIONS)


here the PATH definition must preceed the $(MAKE) call, but we can omit the -j$(NCPUS) , as it is already in MAKE_COMPILE_OPTIONS.

th0ma7 · 2021-08-22T14:38:30Z

Humm, rather interesting but there is a need to be really really careful with this... There are two things to be on the lookout:

Is this being called as a parallel build in the first place?
Will you overrun your total CPU core count?

Let's say you run make -jX all-supported (X being nproc). This will result in parallel-build up to npcroc times of arch-<arch>-<version>. Now if you add parallel builds within each cross/* you may end-up with a npcroc * npcroc resulting in overwhelming your system entirely.

Currently, as we call -@MAKEFLAGS= $(MAKE) ... to invoke parallel building of arch-<arch>-<version> so you won't know if a -j was passed over to make. So there would be a need to check that first, and only then when assuming not already building in parallel (or not exceeding the total amount of core), add parallel building of cross*.

publicarray · 2021-08-23T06:21:55Z

Hm that is a valid concern. I still think this is an overall improvement and adds consistency. AFAIK lots of packages are already running like you describe (npcroc * npcroc). I guess my focus is to make one arch builds faster to increase development productivity, everything else is secondary IMHO. As for overwhelming your system. It depends entirely how many cores you let docker use.

I guess my ultimate aim is to make one arch builds faster to increase development productivity and let the CI build all other architectures.

th0ma7 · 2021-08-23T12:29:29Z

I took a totally different approach (ref: b62495f), based on my current PR while fixing cross/* direct arch builds (PR #4803).

Running quick tests on my 12x core AMD show a significant reduction of build time for both regular make and cmake. Actually, zlib was sooo fast that I didn't thought it had built in the first place:

cross/SVT-AV1

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	2m51.662s
user	2m38.332s
sys	0m13.549s

$ time make arch-apollolake-7.0
real	0m32.660s
user	4m4.555s
sys	0m18.168s

cross/ZLIB

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	0m3.024s
user	0m2.190s
sys	0m0.888s

$ time make arch-apollolake-7.0
real	0m0.876s
user	0m2.553s
sys	0m1.146s

publicarray · 2021-08-25T02:13:02Z

spk/fish

Before (current master):

$ make -C spk/fish clean
$ time make -C spk/fish all-supported
real    22m45.112s
user    27m26.899s
sys     4m33.054s

After (this PR):

$ make -C spk/fish clean
$ time make -C spk/fish all-supported
real    7m33.658s
user    40m11.263s
sys     5m12.594s

$ make -C spk/fish clean
$ time make -j12 -C spk/fish all-supported
Out of memory (over 18GB)

With your PR: (I forgot to clean after 1 arch was build)

$ time make -j12 -C spk/fish all-supported
real    2m19.985s
user    19m54.434s
sys     2m30.063s

publicarray · 2021-08-31T01:16:54Z

Superseded by #4803

Parallel cmake

7a3e8ec

publicarray added the framework label Aug 15, 2021

speedup package compile time

f869973

publicarray commented Aug 15, 2021

View reviewed changes

publicarray changed the title ~~Parallel cmake~~ Parallel cmake and make Aug 15, 2021

hgy59 reviewed Aug 15, 2021

View reviewed changes

publicarray added 2 commits August 16, 2021 11:35

use COMPILE_MAKE_OPTIONS

46b89e1

remove some unnecessary compile targets

6499c84

publicarray requested a review from th0ma7 August 22, 2021 14:24

th0ma7 mentioned this pull request Aug 23, 2021

Fix per-arch builds & add multi-arch & single arch parallel builds #4803

Merged

3 tasks

publicarray closed this Aug 31, 2021

publicarray deleted the parallel-cmake branch August 31, 2021 01:17

publicarray restored the parallel-cmake branch August 31, 2021 01:17

publicarray mentioned this pull request Sep 12, 2021

parallel build: First set of fixes #4854

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallel cmake and make #4783

Parallel cmake and make #4783

publicarray commented Aug 15, 2021 •

edited

Loading

publicarray Aug 15, 2021 •

edited

Loading

hgy59 Aug 15, 2021 •

edited

Loading

publicarray Aug 16, 2021 •

edited

Loading

hgy59 Aug 15, 2021

hgy59 Aug 15, 2021

th0ma7 commented Aug 22, 2021

publicarray commented Aug 23, 2021 •

edited

Loading

th0ma7 commented Aug 23, 2021

publicarray commented Aug 25, 2021 •

edited

Loading

publicarray commented Aug 31, 2021

Parallel cmake and make #4783

Parallel cmake and make #4783

Conversation

publicarray commented Aug 15, 2021 • edited Loading

publicarray Aug 15, 2021 • edited Loading

Choose a reason for hiding this comment

hgy59 Aug 15, 2021 • edited Loading

Choose a reason for hiding this comment

publicarray Aug 16, 2021 • edited Loading

Choose a reason for hiding this comment

hgy59 Aug 15, 2021

Choose a reason for hiding this comment

hgy59 Aug 15, 2021

Choose a reason for hiding this comment

th0ma7 commented Aug 22, 2021

publicarray commented Aug 23, 2021 • edited Loading

th0ma7 commented Aug 23, 2021

cross/SVT-AV1

cross/ZLIB

publicarray commented Aug 25, 2021 • edited Loading

publicarray commented Aug 31, 2021

publicarray commented Aug 15, 2021 •

edited

Loading

publicarray Aug 15, 2021 •

edited

Loading

hgy59 Aug 15, 2021 •

edited

Loading

publicarray Aug 16, 2021 •

edited

Loading

publicarray commented Aug 23, 2021 •

edited

Loading

publicarray commented Aug 25, 2021 •

edited

Loading