Move Che release workflow from Centos CI

[mkuznyetsov]

Is your task related to a problem? Please describe.

Centos CI was announced to be deprecated around the end of this autumn. Before that happens, we would want to move to the alternative options, that would allow us to perform release processes.

This issue is only about Che release related projects, for example, QE test jobs are not covered here.

List of projects, that require CI for release:

• machine-exec (docker image, Golang project)
  • remove cico action che-machine-exec-release
• devfile-registry + plugin-registry (docker image)
  • remove cico actions for release branch
• che-theia (docker images)
  • remove cico action for release branch, che-theia-che-release
• che-dashboard (docker image, Typescript project)
• che-workspace-loader (docker image, Golang project)
• che-server (docker images, Java Maven project + publishing on Nexus, publishing Typescipt DTO on npmjs.org)
• che-docs (simple tagging GH action)
  • remove cico action for release branch, che-docs-che-release
• che-operator (docker image, generating & pushing OLM files)
• community PRs (generated PRs only)
• chectl (node JS build)
• overall che-release orchestrator
  • add PHASES
  • verify work for full set of phases 1-7
• remove cico che-* credentials -- won't do, too many remaning jobs to migrate first - see Remove obsolete nightly/RC jobs from ci centos #19287
• overall che-release readme/docs
  • update che-release issue template: add che-docs step; refactor table to drop... #19063
  • update che-release docs: remove stuff that no longer applies, document new process triggers (GH actions) https://github.com/eclipse/che-release/blob/master/README.md#phase-2---manual-steps

We should also ensure that CI builds (nightly, latest, next), tests, and Hosted Che jobs are also migrated away from Centos CI.

See https://gitlab.cee.redhat.com/service/app-interface/-/blob/master/resources/jenkins/ci-centos/defaults.yaml

Ideally, we would want to preserve a workflow, of having 1 primary workflow, that would run all the subsequent tasks in the correct order, considering that some builds must be triggered after others, some can be run in parallel etc..

One of the alternatives to be considered is Github Actions. Some of the projects, e.g. Che Dashboard already run daily builds, so we should try creating a similar job, that would cover full release workflow for it.

[nickboldt]

chectl moved to GH action in https://github.com/che-incubator/chectl/runs/1494300249?check_suite_focus=true

[nickboldt]

Initial attempt to (easily) move to a GH action to invoke che-release process has failed, see #18576

A day later, it's better... but we need to load some secrets to try it out properly.

[nickboldt]

plugin and devfile registry RELEASE moved to GH actions:

PR to remove plugin and devfile release actions from cico: https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/13061

NOTE: this does not affect the release-preview job that Hosted Che uses https://ci.centos.org/job/devtools-che-plugin-registry-release-preview/ which should ALSO be migrated away from CICO eventually.

[nickboldt]

Looks like the che-bot (or the GH action bot?) doesn't have permission to push to che-incubator/community-operators.

 create mode 100644 upstream-community-operators/eclipse-che/7.23.0/checlusters.org.eclipse.che.crd.yaml
 create mode 100644 upstream-community-operators/eclipse-che/7.23.0/eclipse-che.v7.23.0.clusterserviceversion.yaml

   - Push branch update-eclipse-che-upstream-operator-7.23.0 to the 'che-incubator/community-operators' GitHub repository
Warning: Permanently added the RSA host key for IP address '140.82.114.3' to the list of known hosts.
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Error: Process completed with exit code 128.

-- https://github.com/eclipse/che-operator/runs/1557932455?check_suite_focus=true

[benoitf]

I've checked that che-bot has write permissions to che-incubator/community-operators github repository

[benoitf]

Token needs to be given to git tool (as it's an external org/repository)

[nickboldt]

@benoitf has created a new bot: https://github.com/che-incubator-bot and shared its credentials with the che-incubator org.

Latest news on the ci centos shutdown can be seen in https://issues.redhat.com/browse/APPSRE-2140

[nickboldt]

Only a few weeks left (one sprint) before cico is decommissioned!

Do we have all the nightlies, tests, and releases moved over to GH action? cc: @rhopp @benoitf

Bumping from P1 to P0 so we make sure this is the highest priority in backlog.

[nickboldt]

@mkuznyetsov I've added a couple TODOs to this issue:

• update che-release issue template: add che-docs step; refactor table to drop... #19063
• update che-release docs: remove stuff that no longer applies, document new process triggers (GH actions) https://github.com/eclipse/che-release/blob/master/README.md#phase-2---manual-steps

Please review my PR for the first, and submit a PR for the second.

[mkuznyetsov]

and looks like all essential release jobs for upstream Che release are now finally done on GitHub Actions workflows. So it's time for old jobs & docs cleanup.

[nickboldt]

Still waiting on a review of #19063 so I can merge it... :( 7 days = sadness

Also waiting on a rebase / fix in https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/15206/

[benoitf]

you can ping me before if I forgot some reviews

[nickboldt]

getting closer to being done here - both PR and MR from my previous comment are merged.

[nickboldt]

We could consider removing these chunks of code:

• https://gitlab.cee.redhat.com/service/app-interface/-/blob/master/resources/jenkins/ci-centos/defaults.yaml#L2133 - 2152
• https://gitlab.cee.redhat.com/service/app-interface/-/blob/master/resources/jenkins/ci-centos/defaults.yaml#L2573 - 2592
• https://gitlab.cee.redhat.com/service/app-interface/-/blob/master/resources/jenkins/ci-centos/defaults.yaml#L325 - 406

but not sure if that will break anything that @ibuziuk (rh-che, rhche, osio) or @rhopp (pr checks, happy paths) still need to run there.

[mkuznyetsov]

I've removed credentials for che-operator, that I have myself added for operator release on CentOS CI, so I know that only I have used them

But as for remove cico che-* credentials? section, I'm not sure if I can do it, since those still may be used by some nigthly/Hosted Che related jobs

[nickboldt]

Seems we still have some actively running che jobs on CICO:

Also seeing lots of recently-run builds here:

https://ci.centos.org/view/Devtools/

[ericwill]

The release preview jobs are for Hosted Che, which will EOL in 4 days. After that we can remove them.

[nickboldt]

oh, good! what about ...

and

[nickboldt]

For example
https://ci.centos.org/view/Devtools/job/devtools-che-devfile-registry-build-master/261/console ->
https://quay.io/repository/eclipse/che-devfile-registry?tag=ecabf9b&tab=tags seems to be the current process for publishing nightlies.

but there's also https://github.com/eclipse/che-devfile-registry/blob/master/.github/workflows/nightly-build-publish.yml -> https://github.com/eclipse/che-devfile-registry/runs/1928193391?check_suite_focus=true which seems to have attempted to push the same tags (which already exist in quay due to the cico build?)

Feb 18 10:33 GMT-5 or 3:33pm (according to CICO) trigger:

• GH action: finished in 20m59s -> 10:54 gmt-5 (pushed two tags nightly + ecabf9b)
• CICO: finished in 2m43s (pushed only the ecabf9b tag)

[ericwill]

For example
https://ci.centos.org/view/Devtools/job/devtools-che-devfile-registry-build-master/261/console ->
https://quay.io/repository/eclipse/che-devfile-registry?tag=ecabf9b&tab=tags seems to be the current process for publishing nightlies.

but there's also https://github.com/eclipse/che-devfile-registry/blob/master/.github/workflows/nightly-build-publish.yml

It's a duplicate job, we've had successful nightly build jobs via GitHub action for ages now:
https://github.com/eclipse/che-devfile-registry/runs/1928193391?check_suite_focus=true

I'll remove it from the cico job list

[ericwill]

For example
https://ci.centos.org/view/Devtools/job/devtools-che-devfile-registry-build-master/261/console ->
https://quay.io/repository/eclipse/che-devfile-registry?tag=ecabf9b&tab=tags seems to be the current process for publishing nightlies.
but there's also https://github.com/eclipse/che-devfile-registry/blob/master/.github/workflows/nightly-build-publish.yml

It's a duplicate job, we've had successful nightly build jobs via GitHub action for ages now:
https://github.com/eclipse/che-devfile-registry/runs/1928193391?check_suite_focus=true

I'll remove it from the cico job list

It's done: https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/15557

[nickboldt]

any updates here re: doc or removal of unneeded credentials?

[nickboldt]

Re: remove cico che-* credentials
We won't do this - there are too many remaning jobs to migrate first - see #19287 for followup (which probably needs to be split into prod and qe sub tasks).

[nickboldt]

Doc update:

• Update release docs che-release#32 (merged)