-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Race condition on nightlies generation #644
Comments
Another incident today on Bionic:
The start order is clearly wrong |
Nightly generation starts at 9:00am UTC. The last night the last of the packages was uploaded at 10:30 UTC. |
I think the builds are triggered in a reasonable order, but they are all immediately put into the queue and there are perhaps too many nightly executors acting at once to enforce the ordering. In the extreme case, I think it would work fine if we only had 1 machine for executing nightly builds, or 1 machine for each distro ( |
After implementing #650, while debugging other issues I've triggered testing runs (idea was to get only Focal but failed, so we also get a nice show of what Bionic cancel builds useful to see the order): The order matches the packages in https://build.osrfoundation.org/job/ignition-garden-nightly-scheduler/122/ parameter, which is good. The order is currently: I can confirm that latest build seems just fine. I'm going to leave the issue open until I implement in the deployment code the necessary tags in the nodes. |
ok, I think the order is ok except for |
|
I think this is fixed now that we only build one package at a time |
Some nightly builds are broken in the dependency resolution. The effect seems to happen in the top level packages of the ignition family, mainly ign-gazebo and ign-launch.
Example from today: https://build.osrfoundation.org/job/ign-launch6-debbuilder/205/
The build is broken to the different versions available at the building time of ignition-sensors7 (note the version, one is 20220214 and the other is 20220215:
The times of the uploader and ign-launch-debbuilder also confirm this possibility:
Sensors uploading: https://build.osrfoundation.org/job/repository_uploader_packages/22310/ Feb 15, 2022 10:09 AM
Launch generation: https://build.osrfoundation.org/job/ign-launch6-debbuilder/205/ Feb 15, 2022 10:07:24 AM
The uploading of sensors needs almost 3min, #643 was not enough to give the necessary time.
Real fix: to properly fix the problem we would need a bit of orchestration in the packages generation to be able to launch some of them at the same time. First step could be to covert the repository_uploader call to a 'Build Step' (instead of a Post Build Step) so we can define it as a blocker to the
-debbuilder-
job. That would open the door to implement ignitions interdependencies blockers among their respective-debbuilders-
.The text was updated successfully, but these errors were encountered: