Unable to create workspace on Openshift 4.8

[flacatus]

Describe the bug

During 7.32.0 release, the tests fails in openshift 4.8 nightly when create a workspace throwing the next error:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1/apis/authorization.openshift.io/v1/namespaces/admin-che/rolebindings/admin. Message: Get "https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/namespaces/admin-che/rolebindings/admin": net/http: invalid header field name "Impersonate-Extra-authentication.kubernetes.io/pod-uid". Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=Get "https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/namespaces/admin-che/rolebindings/admin": net/http: invalid header field name "Impersonate-Extra-authentication.kubernetes.io/pod-uid", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).

Openshift CI JOB: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/eclipse-che_che-operator/871/pull-ci-eclipse-che-che-operator-7-v8-che-operator-update/1405134709763084288/

Che Full Log: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/eclipse-che_che-operator/863/pull-ci-eclipse-che-che-operator-main-v8-single-host-nightly-deployment/1405134773873020928/artifacts/single-host-nightly-deployment/gather-extra/artifacts/pods/eclipse-che_che-7cc7f66469-hmtkf_che.log

21-06-16 15:25:15,835[nio-8080-exec-2]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 693]    - Workspace 'admin/golang-htlm8' with id 'workspace4grkhcj76pzaspbk' created by user 'admin'
2021-06-16 15:25:19,254[nio-8080-exec-1]  [ERROR] [c.a.c.r.RuntimeExceptionMapper 47]   - Internal Server Error occurred, error time: 2021-06-16 15:25:19
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1/apis/authorization.openshift.io/v1/namespaces/admin-che/rolebindings/admin. Message: Get "https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/namespaces/admin-che/rolebindings/admin": net/http: invalid header field name "Impersonate-Extra-authentication.kubernetes.io/pod-uid". Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=Get "https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/namespaces/admin-che/rolebindings/admin": net/http: invalid header field name "Impersonate-Extra-authentication.kubernetes.io/pod-uid", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:639)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:578)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:543)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:504)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:471)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:453)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:947)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:221)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:187)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:86)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.lambda$createOrReplace$1(BaseOperation.java:416)
	at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:59)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:419)

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

1. Create a ocp4.8 nightly cluster using clusterbot
2. . Download operator code and checkout to 7.32.0 branch
3. Execute /bin/bash .ci/cico_updates_openshift.sh

Expected behavior

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)-> OCP 4.8-nightly
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl
  • provide a full command that was used to deploy Eclipse Che (including the output)
  • provide an output of chectl version command
• OperatorHub
• I don't know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• Dev Sandbox (workspaces.openshift.com)
• other: please specify

Eclipse Che Logs

Additional context

[flacatus]

I was able to reproduce from UI dashboard the problem:

1. I've installed CRW in fresh cluster
2. Start Java Maven namespace and crash
3. Start again Java maven workspace and start to work

[xbaran4]

This is a verified bug on openshift 4.8. https://bugzilla.redhat.com/show_bug.cgi?id=1972687
Should be fixed by openshift/openshift-apiserver#219 but is not yet known when it will show up in new 4.8 RC (openshift/openshift-apiserver#219 (comment))

[flacatus]

@xbaran4 @skabashnyuk Thank you very much for your verifications...
I saw the fix is already present in OCP new nightly build and was tested against Eclipse 7.32.1 successfully.

New Openshift 4.8 install in AWS: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/eclipse-che_che-operator/873/pull-ci-eclipse-che-che-operator-7-v8-che-operator-update/1407664883419320320/artifacts/che-operator-update/ipi-install-install/artifacts/.openshift_install.log

Che Log in 4.8: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/eclipse-che_che-operator/873/pull-ci-eclipse-che-che-operator-7-v8-che-operator-update/1407664883419320320/artifacts/che-operator-update/ipi-install-install/artifacts/.openshift_install.log

Thank you!