How to update che which was installed using chectl server:deploy --installer=helm as described in the official documentation for "Installing Che on AWS"

[reimer-atb]

Summary

How does one update che, which has been deployed on AWS following the official documentation for Installing Che on AWS?

Relevant information

Che was installed following the official documentation for Installing Che on AWS, i.e.:

chectl server:deploy --installer=helm --platform=k8s --domain=...

When trying to follow the instructions for Upgrading Che using the CLI management tool, I get the following error:

  ❯ 🏃‍  Running the Eclipse Che operator Update
    ✖ Checking existing operator deployment before update
      → Probably Eclipse Che was initially deployed with another installer
      Detecting existing version...
    Error: Command server:update failed. Error log:

The error log says: Cause: Error: che-operator deployment is not found in namespace ...

But server:update does not allow to specify --installer=helm like server:deploy does.

So how to tell chectl to look for a helm-based deployment?

[svor]

cc @mmorhun

[mmorhun]

Hello @philipreimer. Helm installer that deploys Che is deprecated, there is issue #20552 about that.
I am afraid, you'll have to redeploy Che from scratch using operator installer.
chectl server:update expects that Che is deployed with operator or olm installer. In case of helm the installation had to be updated via the Helm itself, but since we do not support it anymore, there is no way to do it, sorry. We deprecate current helm installer because it cannot provide flexibility that operator does. In case of operator one may change Che configuration and the operator will update the installation in a moment. Helm cannot do that...
I'd recommend to commit all your changes, save your workspaces' configs and redeploy Che with operator installer.
If you have any questions, feel free to ask.

[reimer-atb]

save your workspaces' configs

What is the recommended way to do that?

redeploy Che with operator installer

How do I do that? Is there some kind of migration guide available?

We are using EKS ("plain" kubernetes, no openshift). The che docs only mention the "helm-installer" for AWS. In fact table 3 on the Supported Platforms page seems to indicate that the operator only works with openshift?

How do I migrate values from helm-patch-yaml to operator?

How do I make sure that all configurations (keycloak users, etc.) are migrated to my new deployment?

[mmorhun]

Hello @philipreimer. Sorry for delay.

What is the recommended way to do that?

Ideally, devfile should be present in the repository, so Che can just create workspace from the repository (just paste repository URL in the dashboard). Of course, before moving commit and push all your changes.
If it is not so, go to dashboard and save devfiles from each of your workspace manually.
The devfiles are stored in database, so if database is migrated, the workspaces will be in place, however volumes will not, so commit and push changes or see how to backup volumes, but much easier is to commit and push).

How do I do that? Is there some kind of migration guide available?

Unfortunately, there is no migration guide I know about. But it is very easy to deploy Che with Operator installer even on vanilla Kubernetes (ingress controller should be set up, though). Just use chectl (your should be logged into the cluster as kubeadmin):

chectl server:deploy --platform=kubernetes --installer=operator --domain=your.domain.com

In fact table 3 on the Supported Platforms page seems to indicate that the operator only works with openshift?

The doc is wrong. Operator installer works on vanilla Kubernetes for a long time already (two years for sure). I use this installation method for testing very often.

How do I migrate values from helm-patch-yaml to operator?

You have to provide --che-operator-cr-patch-yaml flag that points to a yaml with your customizations. Please note, structure could be somewhat different from Helm patch. See doc here for available options, but they should be in appropriate sections, for example:
patch.yaml

spec:
  server:
    disableInternalClusterSVCNames: true
    devfileRegistryImage: docker.io/user/image:tag
  k8s:
    singleHostExposureType: native

And sources might be helpful as well.

How do I make sure that all configurations (keycloak users, etc.) are migrated to my new deployment?

That's the hardest part and I don't have a good answer, really sorry...
I'd do the following: install Che on, say, minikube using helm installer (with the custom settings). Create a few users. Then play with the migration.

Attempt 1
Install Operator in the namespace where Che is already installed via Helm. It will try to pickup existing installation, but this is more a trick than a method. Might fail.

Attempt 2

1. Backup databases (keycloak and che). It could be done easily:
   1.1. Get name of Postgres pod: kubectl get pods -n eclipse-che
   1.2. Run terminal inside the pod: kubectl exec -n eclipse-che -it <pod-name> -- sh
   1.3. Dump databases into tmp: pg_dump che > /tmp/che.sql && pg_dump keycloak > /tmp/keycloak.sql .
   1.4 Get the dumps from the pod: kubectl cp -n eclipse-che <pod-name>:/tmp/che.sql ./che.sql. The same for keycloak.
2. Deploy a new Che installation using operator installer (into a separate namespace or after cleaning existing one).
3. Scale down operator deployment, so it will not touch anything while the migration is in progress.
4. Restore databases
   4.1. Get Postgres pod name
   4.2. Copy the dumps inside.
   4.3. Apply them. Here is the script.
5. Scale up the operator.
   You might need to restart keycloak and che server pods if they are stuck.

Actually Operator installer is much better because it can dynamically change Che configuration. Say, one has deployed Che and then decide to change some settings. No need to redeploy. Just run kubectl edit checluster eclipse-che -n eclipse-che and put needed values into the yaml and save. Wait a minute and Che is ready!

If something is not clear, feel free to ask more questions. Sorry for inconvenience...

[tolusha]

1. Save and push changes back to the Git repositories for all running workspaces of the Che instance.
2. Stop all workspaces in the Che instance.
3. Scale down the Che and Keycloak deployments:

kubectl scale deployment che --replicas=0 -n eclipse-che
kubectl scale deployment keycloak --replicas=0 -n eclipse-che

4. Backup available databases:

POSTGRES_POD=$(kubectl get pods -n eclipse-che | grep postgres | awk '{print $1}')
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "pg_dump dbche > /tmp/che.sql"
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "pg_dump keycloak > /tmp/keycloak.sql"

5. Copy the obtained backups to a local file system:

kubectl cp eclipse-che/$POSTGRES_POD:/tmp/che.sql che.sql
kubectl cp eclipse-che/$POSTGRES_POD:/tmp/keycloak.sql keycloak.sql

6. Remove Eclipse Che deployed with helm

chectl server:delete 

7. Deploy Eclipse Che using operator

chectl server:deploy --installer operator ...

8. Scale down the Che and Keycloak deployments:

kubectl scale deployment che --replicas=0 -n eclipse-che
kubectl scale deployment keycloak --replicas=0 -n eclipse-che

9. Scale down the PostgreSQL deployment:

kubectl scale deployment postgres --replicas=0 -n eclipse-che

10. Delete the corresponding PVC unit to clean up old data:

kubectl delete pvc postgres-data -n eclipse-che

11. Scale up the PostgreSQL deployments:

kubectl scale deployment postgres --replicas=1 -n eclipse-che
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=postgres -n eclipse-che --timeout=120s

If PostgreSQL pod failed to start, then repeat steps 9-10 again.

12. Provision a database:

POSTGRES_POD=$(kubectl get pods -n eclipse-che | grep postgres | awk '{print $1}')
OPERATOR_POD=$(kubectl get pods -n eclipse-che | grep che-operator | awk '{print $1}')

IDENTITY_POSTGRES_SECRET=$(kubectl get checluster/eclipse-che -n eclipse-che -o json | jq -r '.spec.auth.identityProviderPostgresSecret')
IDENTITY_POSTGRES_PASSWORD=$(if [ -z "$IDENTITY_POSTGRES_SECRET" ] || [ $IDENTITY_POSTGRES_SECRET = "null" ]; then kubectl get checluster/eclipse-che  -n eclipse-che -o json | jq -r '.spec.auth.identityProviderPostgresPassword'; else kubectl get secret $IDENTITY_POSTGRES_SECRET -n eclipse-che -o json | jq -r '.data.password' | base64 -d; fi)

kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"CREATE USER keycloak WITH PASSWORD '$IDENTITY_POSTGRES_PASSWORD'\""
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"CREATE DATABASE keycloak\""
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak\""

POSTGRES_SECRET=$(kubectl get checluster/eclipse-che -n eclipse-che -o json | jq -r '.spec.database.chePostgresSecret')
CHE_USER=$(if [ -z "$POSTGRES_SECRET" ] || [ $POSTGRES_SECRET = "null" ]; then kubectl get checluster/eclipse-che  -n eclipse-che -o json | jq -r '.spec.database.chePostgresUser'; else kubectl get secret $POSTGRES_SECRET -n eclipse-che -o json | jq -r '.data.user' | base64 -d; fi)

kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"ALTER USER $CHE_USER WITH SUPERUSER\""

13. Copy the backups to the PostgreSQL Pod:

kubectl cp che.sql eclipse-che/$POSTGRES_POD:/tmp/che.sql
kubectl cp keycloak.sql eclipse-che/$POSTGRES_POD:/tmp/keycloak.sql

14. Restore the database:

CHE_POSTGRES_DB=$(kubectl get checluster/eclipse-che -n eclipse-che -o json  | jq '.spec.database.chePostgresDb')
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql keycloak < /tmp/keycloak.sql"
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql $CHE_POSTGRES_DB < /tmp/che.sql"

15. Scale up the Keycloak and Che deployments:

kubectl scale deployment keycloak --replicas=1 -n eclipse-che
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=keycloak -n eclipse-che --timeout=120s
kubectl scale deployment che --replicas=1 -n eclipse-che
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=che -n eclipse-che --timeout=120s

16. Open dashboard

chectl dashboard:open

[tolusha]

Finally guide above #20712 (comment) works

[benoitf]

@tolusha @mmorhun 7.41.0 release will be cut tomorrow. This issue is not yet closed, so please close it if it's done or move it to 7.42

[reimer-atb]

@tolusha

Stop all workspaces in the Che instance.

How would you do that? It seems that not even an admin account can list - let alone manage - all users' workspaces. Neither with chectl nor with the REST API, nor through the web interface

[tolusha]

@philipreimer
Unfortunately, there is no API to do this.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[reimer-atb]

/remove-lifecycle stale

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[reimer-atb]

/remove-lifecycle stale

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[reimer-atb]

/remove-lifecycle stale

[reimer-atb]

Please re-open. The issue is not resolved.