Scale down of aws-custom-route-controller is not working when it is disabled

[ialidzhikov]

How to categorize this issue?

/area control-plane
/kind bug
/platform aws

What happened:
Shoots created with provider-aws@v1.38.1 (hence affected by #619) have their aws-route-controller in CrashLoopBackOff:

$ k get po -A -l role=aws-custom-route-controller
NAMESPACE                        NAME                                           READY   STATUS             RESTARTS          AGE
<omitted>                        aws-custom-route-controller-56bd5c96dd-lt9b8   0/1     CrashLoopBackOff   112 (2m28s ago)   9h
<omitted>                        aws-custom-route-controller-69b6484dd-6xxjx    0/1     CrashLoopBackOff   253 (58s ago)     21h
<omitted>                        aws-custom-route-controller-5b7fd48466-hsk9w   0/1     CrashLoopBackOff   168 (115s ago)    14h
<omitted>                        aws-custom-route-controller-7d45d99468-xnhgw   0/1     Running            161 (8s ago)      9h
<omitted>                        aws-custom-route-controller-7dbf58fd6-52ltx    0/1     CrashLoopBackOff   33 (2m23s ago)    102m
<omitted>                        aws-custom-route-controller-6cccdd7cdb-87kml   0/1     Running            14 (5m23s ago)    37m
<omitted>                        aws-custom-route-controller-59668966cb-999jp   0/1     CrashLoopBackOff   210 (113s ago)    18h
<omitted>                        aws-custom-route-controller-545f9b976-66vcw    0/1     CrashLoopBackOff   209 (3m1s ago)    18h
<omitted>                        aws-custom-route-controller-57b4fbddf6-8f9jj   0/1     CrashLoopBackOff   211 (2m57s ago)   18h
<omitted>                        aws-custom-route-controller-5bff8bc576-gjtzc   0/1     Running            183 (5m18s ago)   15h
<omitted>                        aws-custom-route-controller-7f5fd4744-r2866    0/1     Running            301 (22s ago)     17h
<omitted>                        aws-custom-route-controller-65866fdccb-c57vg   0/1     CrashLoopBackOff   300 (4m52s ago)   16h
<omitted>                        aws-custom-route-controller-857567754-pspxf    0/1     CrashLoopBackOff   225 (4m42s ago)   12h
<omitted>                        aws-custom-route-controller-68dbfc7cf4-qnw8r   0/1     CrashLoopBackOff   201 (4m1s ago)    12h
<omitted>                        aws-custom-route-controller-6c9bd4b866-dbhv5   0/1     CrashLoopBackOff   163 (118s ago)    9h
<omitted>                        aws-custom-route-controller-945cd44-s4wrv      0/1     CrashLoopBackOff   181 (3m15s ago)   10h
<omitted>                        aws-custom-route-controller-f748868f6-xqgqn    0/1     CrashLoopBackOff   147 (4m38s ago)   12h
<omitted>                        aws-custom-route-controller-5c8c78c4c6-h5f4b   0/1     CrashLoopBackOff   65 (2m26s ago)    3h31m
<omitted>                        aws-custom-route-controller-5bfcb8d7fb-9qjtl   0/1     CrashLoopBackOff   272 (4m46s ago)   15h
<omitted>                        aws-custom-route-controller-696775794b-8b748   0/1     CrashLoopBackOff   188 (4m45s ago)   16h
<omitted>                        aws-custom-route-controller-6dcdb5cddd-fzz2l   0/1     CrashLoopBackOff   140 (18s ago)     9h
<omitted>                        aws-custom-route-controller-85b865f6c6-2g6jl   0/1     CrashLoopBackOff   157 (4m49s ago)   9h
<omitted>                        aws-custom-route-controller-56659d5868-6wgx7   0/1     CrashLoopBackOff   164 (80s ago)     9h
<omitted>                        aws-custom-route-controller-6974fbc456-842ts   0/1     CrashLoopBackOff   184 (11s ago)     16h
<omitted>                        aws-custom-route-controller-ccf955574-fh4tk    0/1     Running            206 (5m10s ago)   17h
<omitted>                        aws-custom-route-controller-66b8b678f8-h55nd   0/1     CrashLoopBackOff   218 (22s ago)     13h
<omitted>                        aws-custom-route-controller-c8dc7dfbb-69wjj    0/1     CrashLoopBackOff   197 (3m18s ago)   17h
<omitted>                        aws-custom-route-controller-7b56c5f75b-nkgcm   0/1     CrashLoopBackOff   152 (2m23s ago)   12h
<omitted>                        aws-custom-route-controller-c8dfbbd48-pzwmz    0/1     CrashLoopBackOff   260 (3m40s ago)   22h

The reason for it is that the controller is not managed by MR while its RBAC deployed in the Shoot is managed by MR. When aws-custom-route-controller gets disabled, its RBAC is deleted and the controller is not deleted -> it gets in CrashLoopBackOff because of the missing RBAC.

#617 was attempt to mitigate this issue by scaling down the aws-custom-route-controller when it is disabled. But actually this PR does not mitigate/fix anything.

The corresponding scaledDown parameter in GetControlPlaneReplicas is only evaluated when when the Shoot hibernation is enabled:

gardener-extension-provider-aws/vendor/github.com/gardener/gardener/extensions/pkg/controller/shoot.go

Lines 105 to 110 in c9f432d

	func GetControlPlaneReplicas(cluster *Cluster, scaledDown bool, wokenUp int) int {
	if cluster.Shoot != nil && cluster.Shoot.DeletionTimestamp == nil && IsHibernationEnabled(cluster) && scaledDown {
	return 0
	}
	return wokenUp
	}

What you expected to happen:
aws-custom-route-controller to be deleted/scaled-down when it is disabled.

How to reproduce it (as minimally and precisely as possible):

1. Create a Shoot that enables aws-custom-route-controller.

2. Disable the aws-custom-route-controller

3. Make sure that after the disablement the aws-custom-route-controller is not deleted/scaled down.

Anything else we need to know?:

Environment:

• Gardener version (if relevant):
• Extension version: 1e9b376
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others: