Transitioning Rolling builds to Ubuntu 20.04 (Jammy)

[nuclearsandwich]

This is a subset of the greater Rolling transition to 22.04 ros2/ros2#1221

• Disable management jobs during target transitions:
  • rolling_rosdistro-cache
  • rolling_failing-jobs
  • Rci_reconfigure*
  • Rdev_reconfigure*
  • Rdev_trigger*
  • Rdoc_reconfigure*
  • Rdoc_trigger*
  • Rrel_*
• Disable jobs running on Ubuntu Focal
  • default and ufv8 release jobs
    • Rsrc_uF__*
    • Rbin_uF64__*
    • Rbin_ufv8_uFv8__*
  • Devel and PR jobs
    • Rdev__*
    • Rpr__*
  • Doc jobs
    • Rdoc__*
  • Not disabling CI jobs for now.
• Release builds
  • Update release build target platforms (Rsrc and Rbin jobs) Update build targets for Rolling release configurations. #209
  • Merge Prepare Rolling for platform transition. ros/rosdistro#32036 updating rolling/distribution.yaml
  • Re-deploy rolling release management jobs
  • Re-deploy release trigger upload jobs
  • Wait for ros-rolling-ros-workspace to become available in testing on Jammy
• Source builds
  • Update source build target platforms (Rdev and Rpr jobs) Update source (dev and PR jobs) build target to Jammy. #212
  • Re-deploy rolling source management jobs
• Doc builds
  • Update doc build target platform (Rdoc jobs) Update rolling doc build target to Jammy. #213
  • Re-deploy rolling doc management jobs
• Ci builds
  • Update ci build target platforms (Rci jobs) Update CI jobs to run on Ubuntu Jammy #217
  • Re-deploy rolling CI management jobs

Since this will be the first time I've updated the platform of an existing rosdistro there are a few things I'm trying to work out to smooth this out going forward.

The target platform and architecture is embedded into all job names, but is only part of the Jenkins view name for release jobs.
Non-default build files have the build file name embedded in the view but default jobs do not. This means that when we bring up the Jammy source (Rdev) jobs they'll create doubled jobs for everything on the build farm or de-list the old focal jobs. I haven't yet checked how the views are updated to check if the old jobs will just remain or be removed. Since we aren't supporting the focal jobs any more the natural thing to do would be to delete them clearing space for the new jammy jobs. However that also means that we lose the build history of those jobs immediately and there is probably value in having a sunset period where the jobs are disabled but not deleted just like the release jobs will be. The same issue true of Rpr, Rci, and Rdoc jobs.
I see a couple of methods to resolve this:

1. We could update ros_buildfarm to include a target identifier in all views regardless of job type. I've not yet looked to see how feasible this would be to change. Sometimes ros_buildfarm changes are small thanks to templating and code re-use and sometimes the same function is implemented across several different layers or in python and bash or groovy. This change would also affect all distributions (and ROS 1 distros) not just Rolling which may or may not be desirable.

2. We could add the Jammy jobs by introducing new non-default targets which would put them in new views based on the build file identifier used in the config index. If we did this I'd suggest that for Rolling we abandon the use of default build files for all build types except release build files where the target is encoded in the view already. This has the benefit of requiring no changes in ros_buildfarm but the downside of creating separate conventions for release build files and every other build file as well as for Rolling compared to every other distribution where there is less of an issue using default build files because the platform list is less likely to change during the life of the distribution.

3. After updating build files and deploying the new jobs we manually move the focal jobs into manually created views which we can clean up at any later date. This is pretty straightforward to do with a Groovy script or Jenkins configuration UI but it's a chore and will have to be maintained separately.

4. We could do nothing and let the jobs fall where they will, and see how much, if any trouble the extra jobs give us.

5. We could delete outright the focal jobs with no sunset period in order to clear the way for the Jammy jobs and not change anything else.

[tfoote]

It probably makes sense to disable them and then rename them with a prefix to push into another view. Something like LegacyFocal_ as a prefix before R...

[nuclearsandwich]

It probably makes sense to disable them

Yes. I should have clarified that no matter what we'll be disabling the focal jobs in advance of the Jammy rollout and leaving them disabled until we decide it's time to remove them (most likely in tandem with removing the last rolling binaries from the focal repositories).

It probably makes sense to ... rename them with a prefix to push into another view. Something like LegacyFocal_ as a prefix before R...

We shouldn't rename the jobs themselves otherwise links to the jobs will 404 as Jenkins has no internal redirect handling for renamed jobs. As an extended bummer, many of the links we publish for the buildfarm embed the view in the url rather than linking to the job directly (i.e. /view/Rci/job/Rci__... rather than /job/Rci__...) although thankfully the build farm emails do not include the view in the url.

[nuclearsandwich]

I just double checked the views and they're implemented as regex filters on the job list. So Rdev is implemented as

Rdev__.+

This means that when we deploy the Jammy jobs the Rdev view will include both the Jammy and focal jobs unless we modify the view regex.

[cottsay]

1. We could update ros_buildfarm to include a target identifier in all views regardless of job type.

I don't think we're doing this anywhere on build.ros*.org, but I'm not sure this could be implemented cleanly to handle multiple OS targets in a single build file, which is currently a supported scenario. -1 from me.

2. We could add the Jammy jobs by introducing new non-default targets...

I like this in principle, but I really like having a default target. +0 from me.

3. After updating build files and deploying the new jobs we manually move the focal jobs into manually created views...

Not possible, AFAIK.

4. We could do nothing and let the jobs fall where they will, and see how much, if any trouble the extra jobs give us.

This is my vote, and if Jenkins gives out, fall back to (5).

5. We could delete outright the focal jobs with no sunset period...

I'm OK with this for most of the job types, but the performance/benchmark jobs hold history that we've found useful in the past. If we're copying the history over to the Jammy jobs, then I have no qualms here. This is Rolling, not an LTS release. It comes with the territory.

[nuclearsandwich]

Thanks for the feedback Scott. Your comments on 1, 2, and 3 all bear out for me so I don't think any of them are ready options.

I think we're going with 4. AKA Fool around and find out.

[nuclearsandwich]

When running https://build.ros2.org/job/Rdoc_reconfigure-jobs/11 "obsolete" doc jobs from Ubuntu Focal were automatically deleted. I think we can probably expect the same for dev and pr jobs. A dry run of the Rdev reconfiguration shows that it will do the same https://build.ros2.org/job/Rdev_reconfigure-jobs/788

[clalancette]

I think this is all done now, so closing out. Feel free to reopen if this is still needed for some reason.