Cannot start any workspaces anymore: Found '2' matching labeled namespaces

[sunix]

Describe the bug

when switching to per user namespaces some times ago, all my new workspaces were created in a new dedicated namespace. It was working well for a while.
However, after an upgrade, I cannot start any workspaces

[WARN ] [k.n.KubernetesNamespaceFactory 496] - found '2' matching labeled namespaces [KubernetesNamespaceMetaImpl{name='che', attributes={phase=Active}}, KubernetesNamespaceMetaImpl{name='che-ws-sunix', attributes={phase=Active}}]. Using 'che'.

I guess it is starting using the wrong one ... che instead of che-ws-sunix
and obviously I have some permission issue and cannot start any new workspaces:

Failure executing: PUT at: https://x.x.x.x/apis/rbac.authorization.k8s.io/v1/namespaces/che/rolebindings/che-workspace-view. Message: RoleBinding.rbac.authorization.k8s.io "che-workspace-view" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"Role", Name:"workspace-view"}: cannot change roleRef. 

Che version

7.39@latest

Steps to reproduce

N/A

Expected behavior

Should be able to start a workspace

Runtime

Kubernetes (vanilla)

Screenshots

No response

Installation method

chectl/latest

Environment

GCE

Eclipse Che Logs

[WARN ] [k.n.KubernetesNamespaceFactory 496] - found '2' matching labeled namespaces [KubernetesNamespaceMetaImpl{name='che', attributes={phase=Active}}, KubernetesNamespaceMetaImpl{name='che-ws-sunix', attributes={phase=Active}}]. Using 'che'.
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://x.x.x.x/apis/rbac.authorization.k8s.io/v1/namespaces/che/rolebindings/che-workspace-view. Message: RoleBinding.rbac.authorization.k8s.io "che-workspace-view" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"Role", Name:"workspace-view"}: cannot change roleRef. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=roleRef, message=Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"Role", Name:"workspace-view"}: cannot change roleRef, reason=FieldValueInvalid, additionalProperties={})], group=rbac.authorization.k8s.io, kind=RoleBinding, name=che-workspace-view, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=RoleBinding.rbac.authorization.k8s.io "che-workspace-view" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"Role", Name:"workspace-view"}: cannot change roleRef, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:686) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:625) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:565) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:526) at

Additional context

No response

[sunix]

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    che.eclipse.org/username: sunix
[...]
  labels:
    app.kubernetes.io/component: workspaces-namespace
    app.kubernetes.io/part-of: che.eclipse.org
  name: che
[...]

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    che.eclipse.org/username: sunix
[...]
  labels:
    app.kubernetes.io/component: workspaces-namespace
    app.kubernetes.io/part-of: che.eclipse.org
  name: che-ws-sunix
[...]

[sunix]

trying

$ kubectl label namespace che app.kubernetes.io/component-
namespace/che labeled
$ kubectl label namespace che app.kubernetes.io/part-of-
namespace/che labeled

[skabashnyuk]

@sunix can you remove labels and annotations from che namespace?

[sunix]

hi @skabashnyuk it works, I am closing this one