Upgrade cluster autoscaler for k8s version 1.28

[poornima-krishnasamy]

The Cloud Platform Cluster is in k8s version . Hence upgrade the cluster-autoscaler to match the k8s version.

https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler
https://github.com/ministryofjustice/cloud-platform-terraform-cluster-autoscaler

Related to: #5569

[sj-williams]

Target release for 1.28
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/README.md#releases

Current CA image v1.27.5

[sj-williams]

Checks

• release notes
• bump release in test cluster
• compare before after deploy / pod configs
• int tests
• check autoscaler logs
• scale up/down pods and nodes

Scale logs

cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 11:39:50.452422       1 orchestrator.go:157] No pod can fit to eks-cp-2506-0926-mon-ng-20240625093637764100000019-66c82760-8fdd-0926-3740-e109e77b8039
cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 11:39:50.452443       1 orchestrator.go:185] Best option to resize: eks-cp-2506-0926-def-ng-2024062509363778210000001b-2ac82760-8fe0-98c3-b55a-de266e4c42b8
cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 11:39:50.452452       1 orchestrator.go:189] Estimated 1 nodes needed in eks-cp-2506-0926-def-ng-2024062509363778210000001b-2ac82760-8fe0-98c3-b55a-de266e4c42b8
cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 11:39:50.452472       1 orchestrator.go:295] Final scale-up plan: [{eks-cp-2506-0926-def-ng-2024062509363778210000001b-2ac82760-8fe0-98c3-b55a-de266e4c42b8 3->4 (max: 85)}]
cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 11:39:50.452504       1 executor.go:147] 

cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 12:14:04.779295       1 nodes.go:134] ip-172-20-112-201.eu-west-2.compute.internal was unneeded for 2m40.731078688s
cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 12:14:04.779312       1 nodes.go:134] ip-172-20-78-241.eu-west-2.compute.internal was unneeded for 3m1.017977204s

cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 12:16:35.969759       1 cluster.go:178] node ip-172-20-78-241.eu-west-2.compute.internal may be removed
cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 12:16:35.969770       1 cluster.go:155] ip-172-20-112-201.eu-west-2.compute.internal for removal

cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 12:21:08.233943       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"d1b113ff-7e74-45df-b853-9bfc3f8602d3", APIVersion:"v1", ResourceVersion:"114136", FieldPath:""}): type: 'Normal' reason: 'ScaleDownEmpty' Scale-down: removing empty node "ip-172-20-78-241.eu-west-2.compute.internal"
cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 12:21:13.486431       1 auto_scaling_groups.go:345] Terminating EC2 instance: i-0f5bf90a389d71b82

cluster-autoscaler-aws-cluster-autoscaler-74f7bcfb9f-p9fc6 aws-cluster-autoscaler I0625 12:21:13.486522       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"d1b113ff-7e74-45df-b853-9bfc3f8602d3", APIVersion:"v1", ResourceVersion:"114198", FieldPath:""}): type: 'Normal' reason: 'ScaleDownEmpty' Scale-down: empty node ip-172-20-78-241.eu-west-2.compute.internal removed

Release notes

1.28.5
1.28.4
1.28.3
1.28.2
1.28.1
1.28