Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Horizontal Pod Autoscaler (HPA) is not working as expected, and the specific error is related to the missing selector #7655

Closed
converge opened this issue Feb 2, 2024 · 1 comment
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@converge
Copy link
Member

converge commented Feb 2, 2024

Describe the bug
The Horizontal Pod Autoscaler (HPA) is not working as expected, and the specific error is related to the missing selector.

Expected behavior
The HPA should successfully scale based on the provided metrics and selector.

To Reproduce

  1. create the kafka source
  2. add a topic to the kafka source
  3. update config-kafka-features configmap value controller.autoscaler: enabled to use Keda to scale
  4. everything seems to work, but the HPA doesn't seems to work properly
k get hpa -n <namespace>
NAME                                               REFERENCE                                            TARGETS               MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-so-0c156161-02f1-4eb0-8bcc-9f8ac37ed66c   ConsumerGroup/878bbfc0-fd37-4e95-84fe-acd2e2cb2a5a   <**unknown**>/100 (avg)   1         50        0          28m

Knative release version
Knative serving 1.13
Knative eventing 1.13

Additional context
HPA description in two different moments:

Conditions:
  Type           Status  Reason             Message
  ----           ------  ------             -------
  AbleToScale    True    SucceededGetScale  the HPA controller was able to get the target's current scale
  ScalingActive  False   InvalidSelector    the HPA target's scale is missing a selector
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedComputeMetricsReplicas  65m (x12 over 68m)   horizontal-pod-autoscaler  selector is required
  Warning  SelectorRequired              59s (x140 over 68m)  horizontal-pod-autoscaler  selector is required

after a long waiting, the status become:

Conditions:
  Type           Status  Reason             Message
  ----           ------  ------             -------
  AbleToScale    True    SucceededGetScale  the HPA controller was able to get the target's current scale
  ScalingActive  False   ScalingDisabled    scaling is disabled since the replica count of the target is zero
Events:
  Type     Reason            Age                  From                       Message
  ----     ------            ----                 ----                       -------
  Warning  SelectorRequired  13h (x259 over 16h)  horizontal-pod-autoscaler  selector is required
  Warning  FailedGetScale    10h (x5 over 15h)    horizontal-pod-autoscaler  Unauthorized

the consumergroup has the selector:

k describe consumergroup 878bbfc0-fd37-4e95-84fe-acd2e2cb2a5a -n <namespace>
Name:         878bbfc0-fd37-4e95-84fe-acd2e2cb2a5a
Namespace:    10ec4d85e5e44990b1716d3f26b87be5
Labels:       kafka.eventing.knative.dev/metadata.kind=kafkasource
Annotations:  internal.kafka.eventing.knative.dev/lastModifier: system:serviceaccount:knative-eventing:kafka-controller
API Version:  internal.kafka.eventing.knative.dev/v1alpha1
Kind:         ConsumerGroup
Metadata:
  Creation Timestamp:  2024-01-31T23:57:53Z
  Finalizers:
    consumergroups.internal.kafka.eventing.knative.dev
  Generation:  53
  Owner References:
    API Version:           sources.knative.dev/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  KafkaSource
    Name:                  kafka-source-eventing
    UID:                   878bbfc0-fd37-4e95-84fe-acd2e2cb2a5a
  Resource Version:        490352
  UID:                     0c156161-02f1-4eb0-8bcc-9f8ac37ed66c
Spec:
  Replicas:  0
  **Selector:**
    kafka.eventing.knative.dev/metadata.uid:  878bbfc0-fd37-4e95-84fe-acd2e2cb2a5a

the selector that HPA is looking:

spec:
  maxReplicas: 50
  metrics:
  - external:
      metric:
        name: s0-kafka-topic-name-notify-friends-1
        **selector:**
          matchLabels:
            scaledobject.keda.sh/name: so-0c156161-02f1-4eb0-8bcc-9f8ac37ed66c
@Cali0707
Copy link
Member

Closing as fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

2 participants