Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ScaledObject minReplicaCount is not work when it set to 0 #6414

Open
Liufangyu opened this issue Dec 10, 2024 · 3 comments
Open

ScaledObject minReplicaCount is not work when it set to 0 #6414

Liufangyu opened this issue Dec 10, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@Liufangyu
Copy link

Report

you can see, min could set to 0, and when i see Scaledobjects,it's work
jietu-1733824444165
but when i see hpa, it's MINPODS had been set to 1, and the actual number of replicas remains at 1 and has not changed.
image

`apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
creationTimestamp: "2024-12-10T06:56:36Z"
finalizers:

  • finalizer.keda.sh
    generation: 9
    labels:
    app: ap-77ae9b7d
    component: aihc-pom
    scaledobject.keda.sh/name: aihcpom-ap-77ae9b7d
    name: aihcpom-ap-77ae9b7d
    namespace: aihc-pom
    resourceVersion: "10652273"
    uid: 092a2021-d036-429f-a42d-e7a6cfc2a051
    spec:
    advanced:
    horizontalPodAutoscalerConfig:
    behavior:
    scaleDown:
    stabilizationWindowSeconds: 0
    scaleUp:
    stabilizationWindowSeconds: 0
    maxReplicaCount: 1
    minReplicaCount: 0
    scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: ap-77ae9b7d
    triggers:
  • metadata:
    value: "50"
    metricType: Utilization
    type: cpu
    status:
    conditions:
  • message: ScaledObject is defined correctly and is ready for scaling
    reason: ScaledObjectReady
    status: "True"
    type: Ready
  • message: Scaling is performed because triggers are active
    reason: ScalerActive
    status: "True"
    type: Active
  • status: Unknown
    type: Fallback
    hpaName: keda-hpa-aihcpom-ap-77ae9b7d
    lastActiveTime: "2024-12-10T10:01:01Z"
    originalReplicaCount: 1
    resourceMetricNames:
  • cpu
    scaleTargetGVKR:
    group: argoproj.io
    kind: Rollout
    resource: rollouts
    version: v1alpha1
    scaleTargetKind: argoproj.io/v1alpha1.Rollout`

Expected Behavior

when metrics is low and minReplicaCount set to 0, the actual number of replicas should be set to 0.

Actual Behavior

the actual number of replicas remains at 1 .

Steps to Reproduce the Problem

  1. minReplicaCount set to 0 is not work

Logs from KEDA operator

example

KEDA Version

< 2.12.0

Kubernetes Version

< 1.28

Platform

Other

Scaler Details

No response

Anything else?

No response

@Liufangyu Liufangyu added the bug Something isn't working label Dec 10, 2024
@josegonzalez
Copy link

I'm also seeing this issue. I'm running Keda 2.13.1 and using the Keda HTTP Add on at version 0.8.0.

$ kubectl get httpscaledobject screencasts-web
NAME              TARGETWORKLOAD                       TARGETSERVICE        MINREPLICAS   MAXREPLICAS   AGE   ACTIVE
screencasts-web   apps/v1/Deployment/screencasts-web   screencasts-web:80   0             5             26d
$ kubectl get scaledobject screencasts-web
NAME              SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   TRIGGERS   AUTHENTICATION   READY   ACTIVE   FALLBACK   PAUSED    AGE
screencasts-web   apps/v1.Deployment   screencasts-web   0     5                                 True    True     Unknown    Unknown   62d
$ kubectl get hpa keda-hpa-screencasts-web
NAME                       REFERENCE                    TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-screencasts-web   Deployment/screencasts-web   0%/80%    1         5         1          60d

Looking at the code that generates the HorizontalPodAutoscaler, I suppose we're setting the min pods to 0 in the HPA. Since thats a limit of the HPA, it's not super clear to me how we're supposed to scale to zero in any case.

I've been diving into the code to see if we maybe delete the HPA and manually set the deployment to 0 scale, but that doesn't seem like it's happening anywhere. Is there something I'm missing here?


httpscaledobject
apiVersion: http.keda.sh/v1alpha1
kind: HTTPScaledObject
metadata:
  annotations:
    app.kubernetes.io/version: "1733278908"
    dokku.com/managed: "true"
    httpscaledobject.keda.sh/skip-scaledobject-creation: "true"
    meta.helm.sh/release-name: screencasts
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2024-11-14T23:27:45Z"
  finalizers:
  - httpscaledobject.http.keda.sh
  generation: 4
  labels:
    app.kubernetes.io/instance: screencasts-web
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: web
    app.kubernetes.io/part-of: screencasts
  name: screencasts-web
  namespace: default
  resourceVersion: "132041184"
  uid: 95cedefd-26dc-42cb-b82b-248be8ed847d
spec:
  hosts:
  - screencasts.dokku.com
  replicas:
    max: 5
    min: 0
  scaleTargetRef:
    apiVersion: apps/v1
    deployment: ""
    kind: Deployment
    name: screencasts-web
    port: 80
    service: screencasts-web
  scaledownPeriod: 120
  scalingMetric:
    concurrency:
      targetValue: 2
status:
  conditions:
  - message: Identified HTTPScaledObject creation signal
    reason: PendingCreation
    status: Unknown
    timestamp: "2024-12-11T00:59:45Z"
    type: Ready
  - message: Finished object creation
    reason: HTTPScaledObjectIsReady
    status: "True"
    timestamp: "2024-12-11T00:59:45Z"
    type: Ready
  targetService: screencasts-web:80
  targetWorkload: apps/v1/Deployment/screencasts-web
scaledobject
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  annotations:
    app.kubernetes.io/version: "1733278908"
    dokku.com/managed: "true"
    meta.helm.sh/release-name: screencasts
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2024-10-09T04:20:19Z"
  finalizers:
  - finalizer.keda.sh
  generation: 17
  labels:
    app.kubernetes.io/instance: screencasts-web
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: web
    app.kubernetes.io/part-of: screencasts
    scaledobject.keda.sh/name: screencasts-web
  name: screencasts-web
  namespace: default
  resourceVersion: "132046385"
  uid: b5a3dac1-da92-417c-80f4-10b234321e30
spec:
  cooldownPeriod: 300
  fallback:
    failureThreshold: 3
    replicas: 0
  maxReplicaCount: 5
  minReplicaCount: 0
  pollingInterval: 30
  scaleTargetRef:
    apiVersion: apps/v1
    envSourceContainerName: screencasts-web
    kind: Deployment
    name: screencasts-web
  triggers: []
status:
  conditions:
  - message: ScaledObject is defined correctly and is ready for scaling
    reason: ScaledObjectReady
    status: "True"
    type: Ready
  - message: Scaling is performed because triggers are active
    reason: ScalerActive
    status: "True"
    type: Active
  - status: Unknown
    type: Fallback
  - status: Unknown
    type: Paused
  hpaName: keda-hpa-screencasts-web
  lastActiveTime: "2024-12-11T01:14:33Z"
  originalReplicaCount: 1
  scaleTargetGVKR:
    group: apps
    kind: Deployment
    resource: deployments
    version: v1
  scaleTargetKind: apps/v1.Deployment

Important

I am not managing the HPA at all, thats all generated by Keda. It's not clear to me why there isn't an external metric there and it's referencing the CPU, but changing the max value of the ScaleObject/HTTPScaleObject appears to reflect on it, so maybe thats as expected?

httpscaledobject
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    app.kubernetes.io/version: "1733278908"
    dokku.com/managed: "true"
    meta.helm.sh/release-name: screencasts
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2024-10-11T22:41:59Z"
  labels:
    app.kubernetes.io/instance: screencasts-web
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: web
    app.kubernetes.io/part-of: screencasts
    app.kubernetes.io/version: 2.13.1
    scaledobject.keda.sh/name: screencasts-web
  name: keda-hpa-screencasts-web
  namespace: default
  ownerReferences:
  - apiVersion: keda.sh/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ScaledObject
    name: screencasts-web
    uid: b5a3dac1-da92-417c-80f4-10b234321e30
  resourceVersion: "132048058"
  uid: d93e4aba-754c-4d88-91cb-c8bb72bc81b5
spec:
  maxReplicas: 5
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 80
        type: Utilization
    type: Resource
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: screencasts-web
status:
  conditions:
  - lastTransitionTime: "2024-10-11T22:42:14Z"
    message: recent recommendations were higher than current one, applying the highest
      recent recommendation
    reason: ScaleDownStabilized
    status: "True"
    type: AbleToScale
  - lastTransitionTime: "2024-12-10T22:01:01Z"
    message: the HPA was able to successfully calculate a replica count from cpu resource
      utilization (percentage of request)
    reason: ValidMetricFound
    status: "True"
    type: ScalingActive
  - lastTransitionTime: "2024-12-11T01:16:40Z"
    message: the desired count is within the acceptable range
    reason: DesiredWithinRange
    status: "False"
    type: ScalingLimited
  currentMetrics:
  - resource:
      current:
        averageUtilization: 0
        averageValue: 2m
      name: cpu
    type: Resource
  currentReplicas: 3
  desiredReplicas: 3
  lastScaleTime: "2024-12-11T01:16:55Z"

@josegonzalez
Copy link

This might actually be related to having an empty triggers list? In my case, I'm manually creating the ScaledObject separate from the HTTPScaledObject, so it's not super clear what should be in the triggers list.

Going to see if disabling the manual creation causes this to work as expected.

@josegonzalez
Copy link

Yeah that was the case for me. Either letting the HTTPScaledObject create the entry or adding in the entry myself allows the replica count to climb back down to zero (after hitting the cooldown).

See this PR for more context. I also upgraded the chart to 2.16.0 at the same time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: To Triage
Development

No branches or pull requests

2 participants