Cannot start workspace

[SkorikSergey]

Describe the bug

Workspace starting fails with Internal Server Error occurred error message.

Failed Jenkins build logs - https://ci.centos.org/view/Devtools/job/devtools-che-nightly-multiuser-stable-test/75/console
Part of che-server log:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1/apis/authorization.openshift.io/v1/namespaces/eclipse-che/roles/workspace-stop. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. roles.authorization.openshift.io "workspace-stop" is forbidden: User "system:serviceaccount:eclipse-che:che" cannot get roles.authorization.openshift.io in the namespace "eclipse-che": no RBAC policy matched.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:505)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:395)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:376)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:845)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:214)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:168)
	at org.eclipse.che.workspace.infrastructure.openshift.provision.OpenShiftStopWorkspaceRoleProvisioner.provision(OpenShiftStopWorkspaceRoleProvisioner.java:53)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProjectFactory.getOrCreate(OpenShiftProjectFactory.java:100)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:69)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:31)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startAsync(WorkspaceRuntimes.java:466)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.startAsync(WorkspaceManager.java:488)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.startWorkspace(WorkspaceManager.java:373)
	at org.eclipse.che.multiuser.resource.api.workspace.LimitsCheckingWorkspaceManager.startWorkspace(LimitsCheckingWorkspaceManager.java:132)
	at org.eclipse.che.api.workspace.server.WorkspaceService.startById(WorkspaceService.java:469)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
	at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
	at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:297)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:233)

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

1. Login to Openshift by oc login command.
2. Download custom resource yaml file from https://raw.githubusercontent.com/eclipse/che-operator/master/deploy/crds/org_v1_che_cr.yaml and set tlsSupport property to false.
3. Deploy Che by chectl server:start -p=openshift --chenamespace=che --che-operator-cr-yaml=/tmp/custom-resource.yaml
4. Create and start workspace from any devfile

Runtime

• kubernetes (include output of kubectl version)
• Openshift 3.11 (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl (chectl/7.11.0 linux-x64 node-v10.19.0)
• che-operator
• minishift-addon
• I don't know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

[ibuziuk]

@SkorikSergey could you clarify what is the precise installation method?

[dmytro-ndp]

t seems OpenShift specific problem, because it works on minikube it.

[ibuziuk]

@dmytro-ndp @SkorikSergey we need to understand why the CRW tests has passed during the PR check. Any ideas - #16532 ?

[SkorikSergey]

It was started on minikube.

[ibuziuk]

crw-ci-test tests are started against minikube? no coverage of openshift during PR checks?

[SkorikSergey]

[ibuziuk]

ok, so no openshift specific tests are running at all during the PR check, right?
@rhopp is it expected?

[SkorikSergey]

You can start java selenium tests by [ci-test] comment. It doesn't start on PR automatically.

[ibuziuk]

@SkorikSergey so [ci-test] will do the verification against OpenShift?
also, please update the description with the precise installation command

[SkorikSergey]

@SkorikSergey so [ci-test] will do the verification against OpenShift?

Yes. It uses cico_pr_test script.

[ibuziuk]

hmmm.. a couple of things here:

chectl server:start -p=openshift --chenamespace=che --che-operator-cr-yaml=/tmp/custom-resource.yaml

• why the custom-resource yaml isused ?
• it looks like namespace detection does not work correctly

che was installed in the che namespace, but according to the logs the detected namespace was eclipse-che
@SkorikSergey could you please check the KUBERNETES_NAMESPACE env var in the che-server deployment?

[SkorikSergey]

custom-resource.yaml uses to disable tlsSupport property.

[ibuziuk]

@SkorikSergey could you please post tmp/custom-resource.yaml ?
My idea is that KUBERNETES_NAMESPACE / POD_NAMESPACE env vars are not set correctly and do not point to che namespace

[ibuziuk]

PR with hot-fix #16725

[ibuziuk]

@SkorikSergey PR merged. I believe we can close this now