Route ConfigMaps are not removed after workspace removal

[ibuziuk]

Describe the bug

Che Operator creates workspace<>-route configmap for every workspace in order to support the single-host strategy. However, those cms are not removed after the workspace removal.

Che version

7.53@latest

Steps to reproduce

1. Create and start a workspace
2. In the namespace where CheCluster is created see that new route cm is created
   
3. Remove the workspace
4. ERROR: cm is still there after workspace removal

Expected behavior

route cm is deleted together with workspace

Runtime

OpenShift

Screenshots

No response

Installation method

OperatorHub

Environment

Linux

Eclipse Che Logs

No response

Additional context

This isssue could be critical for long-running instancies where thousands of workspaces are created

[ibuziuk]

looks like the issue is not reproducible anymore on dogfooding, after workspace removal route cm is deleted correctly. Production clusters have been analyzed and we only found two orphaned / abandoned cms at this point

[tolusha]

Since it is not possible to reproduce the issue, I suggest running the script once to clean up abandoned ConfigMaps.

#!/bin/bash

set -e

usage () {
    echo "Deletes abandoned workspaces ConfigMaps."
    echo
    echo "Usage:   $0 [--namespace NAMESPACE] [--delete] [--dump]"
    echo
    echo "OPTIONS:"
    echo -e "\t-n,--namespace           Kubernetes namespace where Eclipse is deployed into"
    echo -e "\t-n,--delete              [default: no] Indicates to delete abandoned ConfigMap"
    echo -e "\t-n,--dump                [default: no] Indicates to make a dump of the ConfigMap before deletion"
    echo    
    echo "Example: $0 --namespace eclise-che"
    echo "Example: $0 --namespace eclise-che --delete --dump"
}

unset NAMESPACE
unset DELETE
unset DUMP

while [[ "$#" -gt 0 ]]; do
  case $1 in
    '--namespace'|'-n') NAMESPACE="$2"; shift 1;;
    '--delete') DELETE="true";;
    '--dump') DUMP="true";;
    '--help'|'-h') usage; exit;;
  esac
  shift 1
done

if [[ ! ${NAMESPACE} ]]; then usage; exit 1; fi

dumpAndDeleteConfigMap() {
    if [[ ${DUMP} == "true" ]]; then
        oc get configmap "${CM_NAME}" -n "${NAMESPACE}" -o yaml > "${CM_NAME}.yaml"
    fi
    oc delete configmap "${CM_NAME}" -n "${NAMESPACE}"
}

run() {
    DEVWORKSPACES_IDS=(
        $(oc get devworkspace -A --no-headers=true -o custom-columns=DW_ID:status.devworkspaceId)
    )
    WORKSPACE_CONFIG_MAPS=(
         $(oc get configmap -n "${NAMESPACE}" --no-headers \
        -o custom-columns=NAME:metadata.name \
        -l app.kubernetes.io/part-of=che.eclipse.org,app.kubernetes.io/component=gateway-config)
    )

    for CM_NAME in "${WORKSPACE_CONFIG_MAPS[@]}"; do
        CM_ITEM=($(oc get configmap "${CM_NAME}" -n "${NAMESPACE}" --no-headers -o custom-columns=CREATION_TIMESTAMP:metadata.creationTimestamp,DW_ID:metadata.labels."controller\.devfile\.io/devworkspace_id"))
        CM_CREATION_TIME=${CM_ITEM[0]}
        CM_DW_ID=${CM_ITEM[1]}

        if [[ -z ${CM_DW_ID} ]]; then
            echo "Abandoned Gateway workspace ConfigMap without devWorkspaceId: ${CM_NAME}, created: ${CM_CREATION_TIME}"
        elif [[ ! ${DEVWORKSPACES_IDS[*]}  =~ ${CM_DW_ID} ]]; then
            if [[ ${DELETE} == "true" ]]; then
                dumpAndDeleteConfigMap
            else
                echo "Abandoned Gateway workspace ConfigMap: ${CM_NAME}, devWorkspaceId: ${CM_DW_ID}, created: ${CM_CREATION_TIME}"
            fi
        fi
    done
    
    WORKSPACE_CONFIG_MAPS=(
        $(oc get configmap -n "${NAMESPACE}" --no-headers -o custom-columns=NAME:metadata.name -l app.kubernetes.io/part-of=che.eclipse.org)
    )

    for CM_NAME in "${WORKSPACE_CONFIG_MAPS[@]}"; do
        WS_ID=$(oc get configmap "${CM_NAME}" -n "${NAMESPACE}" -o=jsonpath='{.metadata.labels.che\.workspace_id}')

        if [[ ${WS_ID} ]]; then
            if [[ ${DELETE} == "true" ]]; then
                dumpAndDeleteConfigMap
            else
                echo "Abandoned Che workspace ConfigMap: ${CM_NAME}, created: ${CM_CREATION_TIME}"
            fi        
        fi
    done
}

run

[tolusha]

Closed as can't reproduce.
Abandoned ConfigMap can be deleted by running #21676 (comment)