Stop workspace doesn't work properly

[artaleks9]

Describe the bug

• I faced out with a problem to stop a workspace. When there is running the workspace and I try to stop it on the dashbord - this doesn't work, the workspace is still running. I do several tries without expected result, after some time I could stop the workspace. It seems this was broken from 11.03.2021.
  Also one of the our E2E test tries to stop test workspace via API, but it is fail on this step (see logs) :

19:38:36  Stopping workspace failed. URL used: api/workspace/workspacegfkgaa8u4qeai7bo
19:38:36      1) Change commit and push
19:38:36    [ERROR] CheReporter runner.on.fail: Git with ssh workflow Change commit and push failed after 43828ms
19:38:36  
19:38:36  
19:38:36    3 passing (3m)
19:38:36    1 failing
19:38:36  
19:38:36    1) Git with ssh workflow
19:38:36         Change commit and push:
19:38:36       TimeoutError: The workspace was not stopped in 20000 ms. Currnet status is: RUNNING

Che version

• nightly

Steps to reproduce

1. Create and open some workspace
2. When the workspace is running go to dashboard
3. Try to stop the workspace

Expected behavior

• The workspace should be stopped immidiately with one click

Runtime

• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)

Screenshots

Installation method

• chectl

Eclipse Che Logs

2021-03-17 09:36:29,098[nio-8080-exec-6]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 694]    - Workspace 'admin/checkGeneratingSsh' with id 'workspaceu7llzgvlfcvc245w' created by user 'admin'
2021-03-17 09:36:29,760[nio-8080-exec-9]  [WARN ] [.c.w.i.k.n.KubernetesNamespace 193]  - Can't label the namespace due to lack of permissions. Grant cluster-wide permissions to `get` and `update` the `namespaces` to the `che` service account (Che operator might have already prepared a cluster role called `che-namespace-editor` for this, depending on its configuration). Alternatively, consider disabling the feature by setting `che.infra.kubernetes.namepsace.label` to `false`.
2021-03-17 09:36:29,882[nio-8080-exec-9]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 480]   - Starting workspace 'admin/checkGeneratingSsh' with id 'workspaceu7llzgvlfcvc245w' by user 'admin'
2021-03-17 09:36:30,105[aceSharedPool-4]  [WARN ] [.c.w.i.k.n.KubernetesNamespace 193]  - Can't label the namespace due to lack of permissions. Grant cluster-wide permissions to `get` and `update` the `namespaces` to the `che` service account (Che operator might have already prepared a cluster role called `che-namespace-editor` for this, depending on its configuration). Alternatively, consider disabling the feature by setting `che.infra.kubernetes.namepsace.label` to `false`.
2021-03-17 09:36:30,330[aceSharedPool-4]  [WARN ] [.c.w.i.k.n.KubernetesNamespace 193]  - Can't label the namespace due to lack of permissions. Grant cluster-wide permissions to `get` and `update` the `namespaces` to the `che` service account (Che operator might have already prepared a cluster role called `che-namespace-editor` for this, depending on its configuration). Alternatively, consider disabling the feature by setting `che.infra.kubernetes.namepsace.label` to `false`.
2021-03-17 09:37:11,240[aceSharedPool-4]  [WARN ] [.c.w.i.k.n.KubernetesNamespace 193]  - Can't label the namespace due to lack of permissions. Grant cluster-wide permissions to `get` and `update` the `namespaces` to the `che` service account (Che operator might have already prepared a cluster role called `che-namespace-editor` for this, depending on its configuration). Alternatively, consider disabling the feature by setting `che.infra.kubernetes.namepsace.label` to `false`.
2021-03-17 09:37:57,007[aceSharedPool-4]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 974]   - Workspace 'admin:checkGeneratingSsh' with id 'workspaceu7llzgvlfcvc245w' started by user 'admin'
2021-03-17 09:38:52,971[nio-8080-exec-5]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 541]   - Workspace 'admin/checkGeneratingSsh' with id 'workspaceu7llzgvlfcvc245w' is stopping by user 'admin'
2021-03-17 09:38:54,043[aceSharedPool-5]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 1042]  - Workspace 'admin/checkGeneratingSsh' with id 'workspaceu7llzgvlfcvc245w' is stopped by user 'admin'
2021-03-17 09:39:16,200[nio-8080-exec-7]  [WARN ] [.c.w.i.k.n.KubernetesNamespace 193]  - Can't label the namespace due to lack of permissions. Grant cluster-wide permissions to `get` and `update` the `namespaces` to the `che` service account (Che operator might have already prepared a cluster role called `che-namespace-editor` for this, depending on its configuration). Alternatively, consider disabling the feature by setting `che.infra.kubernetes.namepsace.label` to `false`.
2021-03-17 09:39:16,312[nio-8080-exec-7]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 480]   - Starting workspace 'admin/checkGeneratingSsh' with id 'workspaceu7llzgvlfcvc245w' by user 'admin'
2021-03-17 09:39:16,644[aceSharedPool-6]  [WARN ] [.c.w.i.k.n.KubernetesNamespace 193]  - Can't label the namespace due to lack of permissions. Grant cluster-wide permissions to `get` and `update` the `namespaces` to the `che` service account (Che operator might have already prepared a cluster role called `che-namespace-editor` for this, depending on its configuration). Alternatively, consider disabling the feature by setting `che.infra.kubernetes.namepsace.label` to `false`.
2021-03-17 09:39:16,879[aceSharedPool-6]  [WARN ] [.c.w.i.k.n.KubernetesNamespace 193]  - Can't label the namespace due to lack of permissions. Grant cluster-wide permissions to `get` and `update` the `namespaces` to the `che` service account (Che operator might have already prepared a cluster role called `che-namespace-editor` for this, depending on its configuration). Alternatively, consider disabling the feature by setting `che.infra.kubernetes.namepsace.label` to `false`.
2021-03-17 09:39:38,769[aceSharedPool-6]  [WARN ] [.c.w.i.k.n.KubernetesNamespace 193]  - Can't label the namespace due to lack of permissions. Grant cluster-wide permissions to `get` and `update` the `namespaces` to the `che` service account (Che operator might have already prepared a cluster role called `che-namespace-editor` for this, depending on its configuration). Alternatively, consider disabling the feature by setting `che.infra.kubernetes.namepsace.label` to `false`.
2021-03-17 09:40:05,398[aceSharedPool-6]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 974]   - Workspace 'admin:checkGeneratingSsh' with id 'workspaceu7llzgvlfcvc245w' started by user 'admin'
2021-03-17 09:46:30,479[nio-8080-exec-8]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 541]   - Workspace 'admin/checkGeneratingSsh' with id 'workspaceu7llzgvlfcvc245w' is stopping by user 'admin'
2021-03-17 09:46:31,393[aceSharedPool-7]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 1042]  - Workspace 'admin/checkGeneratingSsh' with id 'workspaceu7llzgvlfcvc245w' is stopped by user 'admin'

Additional context

[sleshchenko]

Bug is investigated and confirmed.
Che Workspace Client used the token that it got initially, instead of updated one, provided by Dashboard. I can't reproduce it on my RHPDS cluster, I assume mainly because there are small connection timeout and Websockets are reconnected each 1-2 minutes, which probably lead to token refresh on Workspace Client side.

Adding it into current sprint, since it may be annoying bug that will be pulled into CRW with a new Dashboard.

[sleshchenko]

It needs to be backported to 7.28.x to be in CRW 2.0

[sleshchenko]

Moving back to 7.28 since it seems the merged PR is going to be part of that release.