HPA does not scale out my Deployment

[clementrey]

Dear Keda users/developers,

I tried to create an SQS and a CloudWatch ScaledObject but without success.
Once created I the scaled object the HPA has been created. The HPA can scale-in but never scale out

Kubernetes v: 1.14
cloud provider: AWS
Keda version: 1.3 (deployed with Helm 3)

ScaledObject:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-scaler
  namespace: default
  labels:
    app: agent
spec:
  scaleTargetRef:
    deploymentName: agent
  pollingInterval: 10  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 1   # Optional. Default: 0
  maxReplicaCount: 100 # Optional. Default: 100
  triggers:
  - type: aws-cloudwatch
    authenticationRef:
      name: trigger-auth-aws-role
    metadata:
      namespace: AWS/SQS
      dimensionName: job-queue
      dimensionValue: keda
      metricName: ApproximateNumberOfMessagesVisible
      targetMetricValue: "500"
      minMetricValue: "0"
      awsAccessKeyID: AWS_ACCESS_KEY_ID # default AWS_ACCESS_KEY_ID
      # Optional: AWS Secret Access Key, can use TriggerAuthentication as well
      awsSecretAccessKey: AWS_SECRET_ACCESS_KEY # default AWS_SECRET_ACCESS_KEY
      awsRegion: "eu-west-1"

I watch the keda controller and metrics server but I could not detect any error message.

Running the describe command on the created HPA:
ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric default/AWS-SQS-job-queue-keda/&LabelSelector{MatchLabels:map[string]string{deploymentName: agent,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: Exactly one scaled object should match label deploymentName=agent

Does someone knows why the external metrics is not found ? and why my deployment does not scale out ?

Best Regards
Clem

[tomkerkhove]

Hi Clem,

I presume AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY are configured on your container inside the agent deployment?

I see you also use trigger authentication, can you paste that one please?

[clementrey]

Hi,

I presume AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY are configured on your container inside the agent deployment?

Yes

Please find below the trigger description:

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-aws-role
spec:
  secretTargetRef:
  #- key: AWS_ROLE_ARN
  #  name: keda-aws-secrets
  #  parameter: awsRoleArn
  - parameter: awsAccessKeyID     # Required.
    name: keda-aws-secrets        # Required.
    key: AWS_ACCESS_KEY_ID        # Required.
  - parameter: awsSecretAccessKey # Required.
    name: keda-aws-secrets        # Required.
    key: AWS_SECRET_ACCESS_KEY    # Required.

[tomkerkhove]

Oh so you have it configured twice then? In the trigger authentication and on the trigger specification itself; I'd recommend to only use one approach to avoid confusion.

Interesting, does your deployment contain 1 or more containers?

[clementrey]

My deployment contains only 1 container

[zroubalik]

@clementrey please try to add deploymentName: agent label to your ScaledObject. The problem you are facing might be caused by #708
This fix will be present in the next release (v1.4), in the meantime you need to provide the label for each ScaledObject

[clementrey]

I did the change but unfortunately it did not chnage the error itself
Now I am even getting something like:
Warning FailedGetScale 12m (x4566 over 20h) horizontal-pod-autoscaler deployments/scale.apps "agent" not found

But I double check and my agent is well running.

[zroubalik]

@clementrey your ScaledObject looks like this?

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-scaler
  namespace: default
  labels:
    app: agent
    deploymentName: agent
spec:
  scaleTargetRef:
    deploymentName: agent
  pollingInterval: 10  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 1   # Optional. Default: 0
  maxReplicaCount: 100 # Optional. Default: 100
  triggers:

Could you please try to delete scaledObject and any leftover HPA and redeploy? Is your deployment agent running in the same namespace as the ScaledObject ?

[clementrey]

Hi,
Sorry I managed to solve my issue. This was related to credentials finally.

Sorry
Clement

[zroubalik]

@clementrey no worries, I am glad it is working 😄