Fix per-arch builds & add multi-arch & single arch parallel builds #4803

th0ma7 · 2021-08-22T13:09:41Z

Motivation: Since code was changed to all-supported in order to allow full parallel builds and align github action workflow, it wasn't possible to build package simply using make arch-apollolake (required full version such as make arch-apollolake-7.0). The following code change does:

Builds all $DEFAULT_TC packages for same arch when no version passed
Allows per arch parallel builds when calling -jX whith multiple archs passed in argument such as make -j12 arch-evansport arch-apollolake-7.0. Can be disabled with PARALLEL_MAKE=nop
Enable full parallel builds for single arch compilation at per-dependency level (e.g. make arch-apollolake-7.0)
Adds the build-<arch>-<version>.log for arch builds (e.g. similar to group builds all-suported) for spk/* and cross/*

Linked issues: #4783 (in theory it supersedes its parallel build functionality)

Checklist

Build rule all-supported completed successfully
Package upgrade completed successfully
New installation of package completed successfully

Test restults on AMD Ryzen 5 1600xX - 6x cores / 12 threads

spk/ffmpeg

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	100m14.571s
user	90m13.618s
sys	11m16.448s

$ time make arch-apollolake-7.0
real	27m35.783s
user	131m24.426s
sys	17m6.737s

spk/fish

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	3m46.047s
user	3m9.937s
sys	0m37.642s

$ time make arch-apollolake-7.0
real	0m57.700s
user	4m50.890s
sys	0m59.453s

spk/python38

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	13m39.728s
user	11m44.916s
sys	2m29.929s

$ time make arch-apollolake-7.0
real	7m16.851s
user	14m36.478s
sys	3m17.609s

spk/rutorrent

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	20m39.915s
user	18m1.128s
sys	3m46.232s

$ time make arch-apollolake-7.0
real	6m17.894s
user	26m21.084s
sys	5m39.970s

cross/bzip2

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	0m3.837s
user	0m3.586s
sys	0m0.290s

$ time make arch-apollolake-7.0
real	0m1.960s
user	0m4.219s
sys	0m0.436s

cross/lua

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	1m6.893s
user	0m47.150s
sys	0m20.635s

$ time make arch-apollolake-7.0
real	0m24.443s
user	1m3.870s
sys	0m34.759s

cross/svt-av1

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	2m51.662s
user	2m38.332s
sys	0m13.549s

$ time make arch-apollolake-7.0
real	0m32.660s
user	4m4.555s
sys	0m18.168s

spk/umurmur

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	2m9.403s
user	1m36.813s
sys	0m30.935s

$ time make arch-apollolake-7.0
real	0m35.695s
user	2m29.563s
sys	0m47.700s

cross/x265

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	8m59.325s
user	8m40.300s
sys	0m18.258s

$ time make arch-apollolake-7.0
real	3m18.044s
user	11m20.038s
sys	0m27.208s

cross/zlib

$ time PARALLEL_MAKE=nop make arch-apollolake-7.0
real	0m3.024s
user	0m2.190s
sys	0m0.888s

$ time make arch-apollolake-7.0
real	0m0.876s
user	0m2.553s
sys	0m1.146s

th0ma7 · 2021-08-22T13:12:58Z

@hgy59 This solves the issue (can't find from which #issue) where make arch-<arch> would not work anymore. This relatively simple code change solves this.

Was there anything else to look into as well?

PS: slowly catching up...

th0ma7 · 2021-08-23T17:09:23Z

@hgy59 and @publicarray this is good for your review.
Honestly I think I finally created the "nirvana" for me to build ffmpeg & all packages.
Really much happy with the proposed changes.
In theory supersedes #4783

publicarray · 2021-08-24T12:15:00Z

Would you mind doing one benchmark with all-suported? I'm hoping to be little regression, but still like to see if any and how much. I'm using fish as a good baseline. only 2 dependencies, but builds long enough, but not too long on a single core. (With zlib you are effectively benchmarking your storage medium and docker overhead 🤣)

th0ma7 · 2021-08-24T14:48:25Z

@publicarray I noticed one yesterday... I am currently figuring out how to best handle the all-supported part as I want it to be able to do both:

accept the -jX passed argument in order to build multiple archs in parallel, each single threaded (PARALLEL_MAKE=nop)
build all arch in sequence BUT each one built with PARALLEL_MAKE=max (reverse behavior of previous)
and/or a mixture of both using -l (--max-load) parameter option

Preliminary code for this is in the work...

publicarray

From my testing if max is used this works as intended and improves compile time. But I'm not too qualified to talk about the .PARALLEL and .NOTPARALLEL commands.

Are there any packages that don't want to have the -j option because it breaks something? This PR adds it by default, just something to consider. But I'm all for it 👍

publicarray · 2021-08-25T04:51:22Z

mk/spksrc.cross-env.mk

-#endif
+# Set parallel options in caller
+ifeq ($(PARALLEL_MAKE),max)
+MAKEFLAGS += -j$(shell nproc)


What happens when PARALLEL_MAKE is a number? IIRC PARALLEL_MAKE was used to limit how many cores to use. I guess it doesn't actually matter, since we can just limit docker to the number of cores we want to give it. Any thoughts @hgy59? If the new behaviour stays, rather than max, true/false would make more sense then.

$ time PARALLEL_MAKE=3 make -C spk/fish arch-x64-7.0 real 3m8.738s user 2m44.456s sys 0m24.859s

@publicarray some early code and excellent results in hope to address this :

spksrc@spksrc-debian10:~/all-supported/spksrc/spk/lua$ time PARALLEL_MAKE=nop make all-supported real 12m1.839s user 8m35.287s sys 3m39.685s spksrc@spksrc-debian10:~/all-supported/spksrc/spk/lua$ time make all-supported real 4m10.488s user 10m42.093s sys 5m19.238s spksrc@spksrc-debian10:~/all-supported/spksrc/spk/lua$ time make -j4 all-supported real 2m23.854s user 10m3.423s sys 4m35.352s spksrc@spksrc-debian10:~/all-supported/spksrc/spk/lua$ time PARALLEL_MAKE=max make -j6 all-supported real 2m5.130s user 10m7.812s sys 4m37.762s spksrc@spksrc-debian10:~/all-supported/spksrc/spk/lua$ time PARALLEL_MAKE=4 make -j4 all-supported real 1m54.972s user 10m40.255s sys 5m6.501s spksrc@spksrc-debian10:~/all-supported/spksrc/spk/lua$ time PARALLEL_MAKE=6 make -j6 all-supported real 1m47.070s user 11m42.589s sys 5m29.545s

This commit also add the ability to create stats from parallel make being invoked

th0ma7 · 2021-08-27T13:00:03Z

@publicarray can you give this a shot? My initial take looks promising.

Basically it is capable of following the -j flag to build multiple archs in parallel and uses the PMAKE variable (default to max) to define how each dependency is being parallelized during their specific build. A really bad representation of things could look like:

$ time PMAKE=4 make -j2 all-supported
-j2 (2x arch parallel)
1. work-88f6281-6.1 -> gcc gcc gcc gcc (PMAKE=4)
1. work-aarch64-6.1 -> gcc gcc gcc gcc (PMAKE=4) -> done -> next

-j2 (2x arch parallel)
2. work-aarch64-7.0 -> gcc gcc gcc gcc (PMAKE=4)
2. work-armv7-6.1 -> gcc gcc gcc gcc (PMAKE=4) -> done -> next
...

By default it defaults to max with no -j flag such as, on a 12x core CPU:

no -j flag, 1x arch sequence
1. work-88f6281-6.1 -> gcc gcc gcc gcc gcc gcc gcc gcc gcc gcc gcc gcc (PMAKE=max) -> done -> next
2. work-aarch64-6.1 -> gcc gcc gcc gcc gcc gcc gcc gcc gcc gcc gcc gcc
3. work-aarch64-7.0 -> ...
4. work-armv7-6.1 ->
...

Add the following to you local.mk (variable names could be reviewed as needed):

PSTAT = ON
PMAKE = max

You can then try it out using variations on the theme such as (with combinations of PMAKE and -jX):

$ time PMAKE=nop make all-supported
$ time PMAKE=max make arch-apollolake-6.1 arch-apollolake-7.0
$ time PMAKE=4 make -j4 all-supported

Having PSTAT=ON will generate a build.stat file providing info on how decent was parallelism during your build.

There currently is one corner-case that I'll have to dig a little more into, it's when building under the cross/<depend> it does not follow the -jX flag passed to make like it does otherwise under spk/<package> (not even sure it ever did it). Something to look into...

publicarray · 2021-08-27T13:19:10Z

Awesome work. Pretty please rename PMAKE back to PARALLEL_MAKE. It's also used in GitHub actions and I also think the longer name is more descriptive to what it does.

mk/spksrc.common.mk

th0ma7 · 2021-08-29T01:34:03Z

@publicarray looks pretty good now. I think all issues you found so far are now solved.

Item still remaining (for a later time) is to allow full parallel build within cross/*... For now it does handle PARALLEL_MAKE option but doesn't build multiple archs in parallel following user passed option -jX.

EDIT: I found where the issue is with the parallel build within cross/*. It's caused by the .NOTPARALLEL: in spksrc.download.mk. It is mandatory to make spk/* to fully work, but somehow is disables the user parameter -jX flag from cross/*.

publicarray

I'm happy with this.
Before merging, I do want to give @hgy59 a little time to have a look if he wants to.

th0ma7 · 2021-08-29T03:12:41Z

@publicarray I was finally able to circumvent the issue with building from cross/* ...

Long story short, to use PARALLEL_MAKE flag it is mandatory to set .NOTPARALLEL from within spksrc.cross-cc.mk. This way using both -j with PARALLEL_MAKE works great from spk/* directory. Side effect is, it doesn't honor the -jX flag when ran from the cross/* directory.

As such the .NOTPARALLEL is only set if PARALLEL_MAKE is not nop. This allows using the -jX flag like previously but it won't honor that -jX flag when ran in conjunction with PARALLEL_MAKE. Its one or the other, and only cross/* is affected.

And sure, such a change requires a few pair of eyes (@hgy59) to confirm all looks good. I really wanted to complete this before disconnecting for the comming week. If you find it relevant for a merge please proceed as you see fit in my absence, cheers!

th0ma7 · 2021-09-09T00:26:37Z

@publicarray while playing a bit with #4797 I noticed a bizarre failure that may be due to this PR. I'm unable to reproduce locally, perhaps can you?

I'm thinking perhaps we should revert this PR and look at it a little further? I could then add a "false" change to ffmpeg which is a perfect testing candidate to confirm auto-builds are still ok?

publicarray · 2021-09-10T11:34:59Z

Hm what failures? Would it help to remove the -j flag if this is a package specific issue? To remove the -j flag you can use $(filter-out -j%, $(TMP_MAKEFLAGS)) a4fc85a

publicarray · 2021-09-10T11:40:28Z

Do you mean

[ 81%] Built target aom_static
make[4]: Leaving directory '/github/workspace/spk/ffmpeg/work-evansport-7.0/libaom-git33aff8586e35ffe385637f6aa2c991858c0fb58e/build'
make[3]: *** [Makefile:130: all] Error 2

?

publicarray · 2021-09-10T12:00:53Z

publicarray@41beb00 should fix it

As a patch: https://github.com/publicarray/spksrc/commit/41beb00b1142048e85f151a7b3a43a2f91eed2a2.patch

th0ma7 · 2021-09-10T12:39:08Z

@publicarray actually, #4849 solves it entirely.
And is ready for a quick review & merge.

…ynoCommunity#4803) * spk.mk: Handle properly arch-<arch> vs arch-<arch>-X.Y use cases * spk.mk: Rework code and optimize for parallel builds * spk.mk: Allow full parallel builds * spk.mk: Optimize parallel build functionality This commit also add the ability to create stats from parallel make being invoked * spk.mk: rename PMAKE back to PARALLEL_MAKE * spk.mk: Revert NCPUS, force download to .NOTPARALLEL, fix cmake * cmake.mk: Use NCPUS instead of call to shell * cmake.mk: No need to adjust MAKEFLAGS for cmake * compile.mk: Move MKFLAGS to compile.mk and set if MAKELEVEL>=2 * cross-cc.mk: Allow either -jX or PARALLEL_MAKE from cross/*

th0ma7 added 2 commits August 22, 2021 11:46

spk.mk: Handle properly arch-<arch> vs arch-<arch>-X.Y use cases

8d0dae6

spk.mk: Rework code and optimize for parallel builds

3992b70

th0ma7 self-assigned this Aug 22, 2021

th0ma7 requested a review from hgy59 August 22, 2021 13:11

spk.mk: Allow full parallel builds

b62495f

th0ma7 mentioned this pull request Aug 23, 2021

Parallel cmake and make #4783

Closed

th0ma7 requested a review from publicarray August 23, 2021 12:30

th0ma7 added framework status/needs-review labels Aug 23, 2021

th0ma7 changed the title ~~[WIP] Fix per-arch builds~~ Fix per-arch builds & add multi-arch & single arch parallel builds Aug 23, 2021

publicarray approved these changes Aug 25, 2021

View reviewed changes

publicarray linked an issue Aug 26, 2021 that may be closed by this pull request

Strange error when building package using Wiki procedure #4750

Closed

spk.mk: Optimize parallel build functionality

a2b0696

This commit also add the ability to create stats from parallel make being invoked

spk.mk: rename PMAKE back to PARALLEL_MAKE

9ff47fa

publicarray approved these changes Aug 28, 2021

View reviewed changes

mk/spksrc.common.mk Outdated Show resolved Hide resolved

th0ma7 added 2 commits August 29, 2021 01:23

spk.mk: Revert NCPUS, force download to .NOTPARALLEL, fix cmake

88876b8

cmake.mk: Use NCPUS instead of call to shell

a7a0c11

th0ma7 added 2 commits August 29, 2021 02:00

cmake.mk: No need to adjust MAKEFLAGS for cmake

85b7ba2

compile.mk: Move MKFLAGS to compile.mk and set if MAKELEVEL>=2

767052a

publicarray approved these changes Aug 29, 2021

View reviewed changes

cross-cc.mk: Allow either -jX or PARALLEL_MAKE from cross/*

c86cc91

publicarray removed the status/needs-review label Aug 31, 2021

publicarray added the status/ready-to-merge label Aug 31, 2021

publicarray mentioned this pull request Sep 5, 2021

java11: update no parallel make flags #4839

Merged

publicarray merged commit a442e16 into SynoCommunity:master Sep 5, 2021

th0ma7 deleted the arch-build branch December 17, 2021 10:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix per-arch builds & add multi-arch & single arch parallel builds #4803

Fix per-arch builds & add multi-arch & single arch parallel builds #4803

th0ma7 commented Aug 22, 2021 •

edited

Loading

th0ma7 commented Aug 22, 2021 •

edited

Loading

th0ma7 commented Aug 23, 2021

publicarray commented Aug 24, 2021 •

edited

Loading

th0ma7 commented Aug 24, 2021 •

edited

Loading

publicarray left a comment

publicarray Aug 25, 2021

publicarray Aug 25, 2021

th0ma7 Aug 26, 2021

th0ma7 commented Aug 27, 2021

publicarray commented Aug 27, 2021

th0ma7 commented Aug 29, 2021 •

edited

Loading

publicarray left a comment

th0ma7 commented Aug 29, 2021

th0ma7 commented Sep 9, 2021

publicarray commented Sep 10, 2021 •

edited

Loading

publicarray commented Sep 10, 2021 •

edited

Loading

publicarray commented Sep 10, 2021

th0ma7 commented Sep 10, 2021

Fix per-arch builds & add multi-arch & single arch parallel builds #4803

Fix per-arch builds & add multi-arch & single arch parallel builds #4803

Conversation

th0ma7 commented Aug 22, 2021 • edited Loading

Checklist

Test restults on AMD Ryzen 5 1600xX - 6x cores / 12 threads

spk/ffmpeg

spk/fish

spk/python38

spk/rutorrent

cross/bzip2

cross/lua

cross/svt-av1

spk/umurmur

cross/x265

cross/zlib

th0ma7 commented Aug 22, 2021 • edited Loading

th0ma7 commented Aug 23, 2021

publicarray commented Aug 24, 2021 • edited Loading

th0ma7 commented Aug 24, 2021 • edited Loading

publicarray left a comment

Choose a reason for hiding this comment

publicarray Aug 25, 2021

Choose a reason for hiding this comment

publicarray Aug 25, 2021

Choose a reason for hiding this comment

th0ma7 Aug 26, 2021

Choose a reason for hiding this comment

th0ma7 commented Aug 27, 2021

publicarray commented Aug 27, 2021

th0ma7 commented Aug 29, 2021 • edited Loading

publicarray left a comment

Choose a reason for hiding this comment

th0ma7 commented Aug 29, 2021

th0ma7 commented Sep 9, 2021

publicarray commented Sep 10, 2021 • edited Loading

publicarray commented Sep 10, 2021 • edited Loading

publicarray commented Sep 10, 2021

th0ma7 commented Sep 10, 2021

th0ma7 commented Aug 22, 2021 •

edited

Loading

th0ma7 commented Aug 22, 2021 •

edited

Loading

publicarray commented Aug 24, 2021 •

edited

Loading

th0ma7 commented Aug 24, 2021 •

edited

Loading

th0ma7 commented Aug 29, 2021 •

edited

Loading

publicarray commented Sep 10, 2021 •

edited

Loading

publicarray commented Sep 10, 2021 •

edited

Loading