Error is not meanful when workspace startup fails because route quota is exceeded

[l0rd]

Describe the bug

The default workspace has 3 routes. If routes count quota is configured in the namespace of the workspace pod and if the creation of 3 new routes will exceed the quota, then workspaces creation will fail.

The problem workspace creation will fails for timeout (so after waiting for 5 minutes) and the developer have no clue how to address the problem and may spend hours trying to start a workspace without success (deleting an existing workspace would solve the problem):

This is even more problematic as there is no Kubernetes event during the startup:

Che version

7.61@latest

Steps to reproduce

Start 4 workspaces on developer sandbox. The 4th will fail (quota is going to be updated so you may need to start more workspaces)

Expected behavior

We should catch the error and present the following message "The workspace cannot be started because it requires the creation a number of routes () that exceed the quota () for the current namespace (). Deleting an existing workspace may solve the problem."

Runtime

OpenShift

Screenshots

No response

Installation method

OperatorHub, other (please specify in additional context)

Environment

Dev Sandbox (workspaces.openshift.com)

Eclipse Che Logs

$ kubectl get dw
NAME                      DEVWORKSPACE ID             PHASE      INFO
python-hello-world        workspace85995a207cb641a2   Stopped    Stopped
python-hello-world-4iqo   workspacee0385883730b4ef8   Stopped    Stopped
python-hello-world-kqii   workspacec9211b3e57f1461f   Stopped    Stopped
python-hello-world-qvbu   workspacedbf3bb93a7294cf4   Starting   Preparing routes

$ kubectl get dwr
NAME                                DEVWORKSPACE ID             PHASE       INFO
routing-workspace85995a207cb641a2   workspace85995a207cb641a2   Ready       DevWorkspaceRouting prepared
routing-workspacec9211b3e57f1461f   workspacec9211b3e57f1461f   Ready       DevWorkspaceRouting prepared
routing-workspacedbf3bb93a7294cf4   workspacedbf3bb93a7294cf4   Preparing   Preparing routes
routing-workspacee0385883730b4ef8   workspacee0385883730b4ef8   Ready       DevWorkspaceRouting prepared

Additional context

There is no Kubernetes event during the startup (zero).

[ibuziuk]

Regarding the Sandbox case, there are PRs for increasing the number of routes to 30:

• chore: increase quota for ingresses / routes to 30 codeready-toolchain/host-operator#768
• test: increase quota for ingresses / routes to 30 codeready-toolchain/toolchain-e2e#683
  Downstream issue - https://issues.redhat.com/browse/CRW-4124

[l0rd]

@ibuziuk this is good, users will be able to have 4 workspace.

But workspaces creation will fail (with no clue why) if the user have 9 stopped workspaces. And it's less then 9 if workspaces have endpoints (and usually they do). I think we have the opportunity to limit the workspace count now (because of this issue users cannot have created a lot of workspaces lately) and I have created #22069.

[amisevsk]

There is also the DevWorkspaceOperatorConfig .config.workspace.cleanupOnStop setting, which configures DWO to delete owned resources of the DevWorkspace when it is stopped. This would delete the DevWorkspace's DevWorkspaceRouting CR, which should clean up any routes created for the workspace.

[amisevsk]

To resolve this bug, the Che Operator should detect this condition (cannot create routes due to quota) and set the DWR to a failed state. This would trigger workspace failure.

[l0rd]

There is also the DevWorkspaceOperatorConfig .config.workspace.cleanupOnStop setting, which configures DWO to delete owned resources of the DevWorkspace when it is stopped. This would delete the DevWorkspace's DevWorkspaceRouting CR, which should clean up any routes created for the workspace.

@amisevsk this is good to know but why isn't it the default then? does it have an impact on the restart time of the workspace?

[amisevsk]

When the setting was added, the default was set to the earlier behaviour as we didn't have bandwidth to test all edge cases -- if all resources are removed on stop, they need to be re-created when the workspace is restarted and this could have implications on scalability. For instance, this would require the Che gateway to un-expose and re-expose all workspace routes on every stop/start.

It also would have some impact on restart time, though the size of the effect is not clear. Enabling the setting means workspace starts require as much work as new workspaces on the cluster (everything needs to be re-created).

We can test enabling it in our development environments to see if it impacts availability, and make it a default in a future release if there are no issues.

[ibuziuk]

enabled on the dogfooding instance

config:
  workspace:
    cleanupOnStop: true

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[l0rd]

/remove-lifecycle stale

[amisevsk]

I've opened a PR in the DWO repository that should fix this issue: devfile/devworkspace-operator#1199

However, since the Che Operator runs its own DevWorkspaceRouting reconciler, fixing this problem in Che will require the Che Operator to pull in a newer DevWorkspaceOperator dependency, so I'm leaving this issue open until that is completed.

[ibuziuk]

@amisevsk do you think we can close the issue?

[amisevsk]

@ibuziuk thanks for the reminder -- I've opened eclipse-che/che-operator#1805 to update the Che Operator to DevWorkspace v0.25.0, which will close this issue.