Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition on nightlies generation #644

Closed
j-rivero opened this issue Feb 15, 2022 · 8 comments
Closed

Race condition on nightlies generation #644

j-rivero opened this issue Feb 15, 2022 · 8 comments
Labels
bug Something isn't working buildfarm infra-team

Comments

@j-rivero
Copy link
Contributor

j-rivero commented Feb 15, 2022

Some nightly builds are broken in the dependency resolution. The effect seems to happen in the top level packages of the ignition family, mainly ign-gazebo and ign-launch.

Example from today: https://build.osrfoundation.org/job/ign-launch6-debbuilder/205/

The build is broken to the different versions available at the building time of ignition-sensors7 (note the version, one is 20220214 and the other is 20220215:

Investigating (0) libignition-sensors7-segmentation-camera:amd64 < none -> 6.999.999+nightly+git20220214+1rb1f78a5b75a62680a54e7e880d9fc301c2814a0b-1~focal @un uN Ib >^M
Broken libignition-sensors7-segmentation-camera:amd64 Depends on libignition-sensors7-camera:amd64 < none | 6.999.999+nightly+git20220215+1rb1f78a5b75a62680a54e7e880d9fc301c2814a0b-1~focal @
  Considering libignition-sensors7-camera:amd64 0 as a solution to libignition-sensors7-segmentation-camera:amd64 2^M

The times of the uploader and ign-launch-debbuilder also confirm this possibility:
Sensors uploading: https://build.osrfoundation.org/job/repository_uploader_packages/22310/ Feb 15, 2022 10:09 AM
Launch generation: https://build.osrfoundation.org/job/ign-launch6-debbuilder/205/ Feb 15, 2022 10:07:24 AM

The uploading of sensors needs almost 3min, #643 was not enough to give the necessary time.

Real fix: to properly fix the problem we would need a bit of orchestration in the packages generation to be able to launch some of them at the same time. First step could be to covert the repository_uploader call to a 'Build Step' (instead of a Post Build Step) so we can define it as a blocker to the -debbuilder- job. That would open the door to implement ignitions interdependencies blockers among their respective -debbuilders-.

@j-rivero
Copy link
Contributor Author

The uploading of sensors needs almost 3min, #643 was not enough to give the necessary time.

#645 tries to fix this but this is no more than a workaround.

@j-rivero
Copy link
Contributor Author

j-rivero commented Feb 17, 2022

Another incident today on Bionic:

  • started repository uploader for sensors7 upload Feb 17, 2022 9:26:03 AM, took ~3min
  • started gazebo7 Feb 17, 2022 9:23:11 AM, took ~9 min (5 waiting)

The start order is clearly wrong so the waiting workarounds can not work in anyway Update: the workaround could work if an undetermined amount of time it would be waiting until the upload is done.

@j-rivero
Copy link
Contributor Author

Nightly generation starts at 9:00am UTC. The last night the last of the packages was uploaded at 10:30 UTC.

@scpeters
Copy link
Contributor

I think the builds are triggered in a reasonable order, but they are all immediately put into the queue and there are perhaps too many nightly executors acting at once to enforce the ordering. In the extreme case, I think it would work fine if we only had 1 machine for executing nightly builds, or 1 machine for each distro (nightly-focal and nightly-bionic).

@j-rivero
Copy link
Contributor Author

j-rivero commented Mar 1, 2022

After implementing #650, while debugging other issues I've triggered testing runs (idea was to get only Focal but failed, so we also get a nice show of what Bionic cancel builds useful to see the order):

image

The order matches the packages in https://build.osrfoundation.org/job/ignition-garden-nightly-scheduler/122/ parameter, which is good. The order is currently: ign-tools ign-cmake2 ign-math7 ign-plugin ign-utils1 ign-common5 ign-msgs9 ign-rendering7 sdformat13 ign-fuel-tools8 ign-transport12 ign-gui7 ign-sensors7 ign-physics6 ign-gazebo7 ign-launch6 ign-garden

I can confirm that latest build seems just fine. I'm going to leave the issue open until I implement in the deployment code the necessary tags in the nodes.

@scpeters
Copy link
Contributor

scpeters commented Mar 1, 2022

ok, I think the order is ok except for ignition-utils1 should come before ignition-math7. I'll make a PR to fix that

@scpeters
Copy link
Contributor

scpeters commented Mar 1, 2022

ok, I think the order is ok except for ignition-utils1 should come before ignition-math7. I'll make a PR to fix that

#651

@scpeters
Copy link
Contributor

I think this is fixed now that we only build one package at a time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working buildfarm infra-team
Projects
None yet
Development

No branches or pull requests

2 participants