list_hardware.py: sort rglob(SOC_YML) HWMv2 results #70132
Conversation
This makes .config, autoconf.h, and configs.c deterministic again. Directory listing is not deterministic, it must always be sorted. https://reproducible-builds.org/docs/stable-inputs/ Fixes commit 61bbfb5 ("scripts: introduce list_hardware.py for listing of architectures and SoCs") in collab-hwm branch which was squashed in mega HWMv2 commit 8dc3f85 ("hwmv2: Introduce Hardware model version 2 and convert devices") SOF CI builds with both Windows and Linux and compares the outputs. This catches practically 100% of build reproducibility issues and caught this one too: https://github.com/thesofproject/sof/actions/runs/8241692987/job/22539664560 HWMv2 was a "big bang" integration on both the Zephyr and SOF sides. So this `rglob()` was a needle in a haystack but with hindsight, this issue is really trivial to reproduce and verify: ``` apt-get install disorderfs mkdir disorderedWorkspace/ disorderfs --shuffle-dirents=yes workspace/ disorderedWorkspace/ ``` ... then just build `samples/hello_world/` twice in disorderedWorkspace/ with any --board and compare the build directories. Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Testing zephyrproject-rtos/zephyr#70132 Signed-off-by: Marc Herbert <marc.herbert@intel.com>
zephyrbot
added
the
Trivial
marc-hb
added
bug
|
@tejlmand : based on the git blame you probably want to add yourself (and others?) to the MAINTAINERS.yml section for this file; and maybe for some other HWMv2 files too. The "Pull Request" assigner found no reviewer and no assignee. |
There was a problem hiding this comment.
lgtm, and I agree build reproducibility is very important and in this case trivial to fix.
But i'm a little worried if we suddenly start to require that every time we search for files (or create any other kind of list) that we start to require sorting of such lists.
I'm afraid this is a price to pay. Directory listing has never been deterministic and I don't see this changing any time soon. The good news is: it's a relatively small price to pay. The hardest part is to automate the testing in CI but SOF already has that thanks to Windows builds. Zephyr came close but you pushed back here: |
This makes .config, autoconf.h, and configs.c deterministic again.
Directory listing is not deterministic, it must always be sorted. https://reproducible-builds.org/docs/stable-inputs/
Fixes commit 61bbfb5 ("scripts: introduce list_hardware.py for listing of architectures and SoCs") in collab-hwm branch which was squashed in mega HWMv2 commit 8dc3f85 ("hwmv2: Introduce Hardware model version 2 and convert devices")
SOF CI builds with both Windows and Linux and compares the outputs. This catches practically 100% of build reproducibility issues and caught this one too:
https://github.com/thesofproject/sof/actions/runs/8241692987/job/22539664560
HWMv2 was a "big bang" integration on both the Zephyr and SOF sides. So this
rglob()was a needle in a haystack but with hindsight, this issue is really trivial to reproduce and verify:... then just build
samples/hello_world/twice in disorderedWorkspace/ with any --board and compare the build directories.cc: