Workspace controller nightly builds will break whenever CRDs are modified

[amisevsk]

Describe the bug

We currently build images of the che workspace controller nightly: https://ci.centos.org/view/Devtools/job/devtools-che-workspace-operator-nightly/

However, a release of the controller is not just the container image, but also the CRDs and other yaml files required for deploying the controller to a cluster. Since these are distributed via the repository, if they are updated at any point, the nightly build will stop working until the next nightly is triggered.

Solutions

Option 1: nightly = every commit

Instead of building nightlies on a timer (midnight UTC), push to the nightly tag after every commit. This would fix the problem, but mean more builds. It also violates the notion of "nightly".

Option 2: configure master CI to push a latest image in addition to tagged.

Currently, the master CI builds every commit in master and could be used, but it does not push a latest image, so updating would be manual. The master CI could push latest on every build. However, it would increase the chances that the image being used changes suddenly.

Option 3: Figure out way to include yamls with a build, as though it was a regular release.

Probably not a great option, since it is a lot of compliexity and CRDs do not change that often.

Additional context

This issue was triggered by merging PR devfile/devworkspace-operator#50, which set routingSuffix as a necessary field in the workspaceRouting spec. As the nightly at the time was built before this PR was merged, it would never set routingSuffix, cauing an reconciles to loop in error and workspaces to be stuck in the Starting phase.

[sleshchenko]

I this it's going to be fixed by switching to next tag built from master branch #17175