Upstream core Kubernetes HPA loop is single-threaded

[bpinske]

Report

I am leveraging Keda to query Prometheus for metrics to use in scaling decisions. Core Kubernetes appears to have some poor performance when it comes to operation with many external metrics based queries. The core HPA loop is single-threaded and acts on a single HPA object at a time in a blocking fashion on every loop iteration.

This core HPA loop should be concurrent and process every HPA object of the cluster at once as there is no mutability of the HPA objects themselves caused by other HPA objects.

This has an associated upstream issue opened describing the issuehere

The single threaded nature can be found here

Expected Behavior

Kubernetes' core HPA loop to be performant

Actual Behavior

It's possible for scaling decisions to be greatly delayed due to the sequential blocking nature of the core HPA loop. HPA simply cannot process all HPA objects sequentially in a timely manner.

Steps to Reproduce the Problem

1. Create and install the https://github.com/kubernetes-sigs/custom-metrics-apiserver and alter its code to add a delay in the GetExternalMetric method of the Testing Provider

2. Also add logging to the method with timestamps logrus supports timestamps, e.g.

3. Create 1,000 HPA objects referring to that test metric - possibly 1,000 deployments as well, but one will work fine too even if not a real use case.

4. Pick one of your HPA's and watch for the time between its call to GetExternalMetric.

Logs from KEDA operator

.

KEDA Version

2.5.0

Kubernetes Version

1.20

Platform

Any

Scaler Details

All of them!

Anything else?

No response

[zroubalik]

Thanks for documenting this, we should try to work with SIG Autoscaling to tackle this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[zroubalik]

This has been fixed upstream.