phase1: IGNORE_ERRORS hides package build issues

[ynezz]

Problem

Following IGNORE_ERRORS constructs in https://github.com/openwrt/buildbot/blob/master/phase1/master.cfg#L1172

make package/compile IGNORE_ERRORS=n m
make target/compile IGNORE_ERRORS=n m

are effectively hiding the build issues, leading to hardly spottable issues such as recent kmod-ltql-vmmc build breakage caused by default GCC version bump, where the buildbot step succeeds, but some target's optional kernel module (or package) might fail to build. Contrary to phase2, there are no faillogs generated on phase1 for failing packages, thus we're completely unaware about the breakage.

For a complete overview see the discussion here openwrt/openwrt#15064

Possible solutions

1. Get rid of IGNORE_ERRORS=n m and make the build fails, thus making sure, that all packages are building
2. Provide faillogs/images (or faillogs-images) for phase1 package build issues as we've for phase2

I'm more inclined towards 1. as there is quite good CI test coverage on GitHub so I believe, that when something fails to build, like in the recent kmod-ltql-vmmc case, it should be spotted and fixed like in openwrt/openwrt#15065 and openwrt/openwrt#15098.

[Ansuel]

Errors might also be triggered for some images that are too big. I'm more inclined to 2? Also can the failogs handling be improved to also delete the directory if it's empty? Il Dom 7 Apr 2024, 22:17 Petr Štetiar ***@***.***> ha scritto:

…

Problem Following IGNORE_ERRORS constructs in https://github.com/openwrt/buildbot/blob/master/phase1/master.cfg#L1172 make package/compile IGNORE_ERRORS=n m make target/compile IGNORE_ERRORS=n m are effectively hiding the build issues, leading to hardly spottable issues such as recent kmod-ltql-vmmc <openwrt/openwrt#15064 (comment)> build breakage caused by default GCC version bump, where the buildbot step succeeds, but some target's optional kernel module (or package) might fail to build. Contrary to phase2, there are no faillogs generated <https://downloads.openwrt.org/snapshots/faillogs/> on phase1 for failing packages, thus we're completely unaware about the breakage. For a complete overview see the discussion here openwrt/openwrt#15064 <openwrt/openwrt#15064> Possible solutions 1. Get rid of IGNORE_ERRORS=n m and make the build fails, thus making sure, that all packages are building 2. Provide faillogs/images (or faillogs-images) for phase1 package build issues as we've for phase2 I'm more inclined towards 1. as there is quite good CI test coverage on GitHub so I believe, that when something fails to build, like in the recent kmod-ltql-vmmc case, it should be spotted and fixed like in #15065 and #15098. — Reply to this email directly, view it on GitHub <#27>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE2ZMQUXM3XK7U4FQORAV23Y4GSUPAVCNFSM6AAAAABF3SZWROVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIZDSOJUGA3TSMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ynezz]

Errors might also be triggered for some images that are too big.

Yep, but this is make target/install step (no IGNORE_ERRORS=n m there) so if that fails, it would result in the build failure of the target and we would notice it on IRC as a failed build, investigate and fix. We don't get that for failing package currently as you can see in openwrt/openwrt#15065

Also can the failogs handling be improved to also delete the directory if it's empty?

Do you've such example? The code seems to delete it already https://github.com/openwrt/buildbot/blob/master/phase2/master.cfg#L549C13-L549C65 so I'm wondering whats the problem.

[Ansuel]

@ynezz oh ok wasn't aware the target were split. Then yes it's ok for me to make them fail. (but I would still upload the faillogs? instead of checking tons of line in buildbot searching for the error)

For example... we don't have one cause packages got broken again lately. We had case where https://downloads.openwrt.org/snapshots/faillogs/aarch64_cortex-a53/ was still there in https://downloads.openwrt.org/snapshots/faillogs/ but the directory was empty. (making it difficult to check targets with failed packages)