TriggerAuthentication.keda.sh "xxxx" not found

[galvesribeiro]

Hello!

I've deployed KEDA on AWS EKS using the pod identity mode. The KEDA system pods doesn't show any errors starting and anything seems to be fine.

However, whenever a deploy an ScaledObject it doesn't seems that the trigger is able to find the authentication object to talk to the SQS queue.

Here are the two objects:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: worker-node-credentials
  namespace: {{ .Release.Namespace }}
spec:
  podIdentity:
    provider: aws
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-node-aws-sqs-queue-scaledobject
  namespace: {{ .Release.Namespace }}
spec:
  scaleTargetRef:
    name: worker-node
  minReplicaCount: 4
  maxReplicaCount: 200
  triggers:
  - type: aws-sqs-queue
    authenticationRef:
      name: worker-node-credentials
      namespace: {{ .Release.Namespace }}
    metadata:
      queueURL: {{ .Values.posta.workerQueueName }}
      queueLength: "5"
      awsRegion: "us-east-1"

Right after applying those, I see the following errors in the operator pod:

2024-11-13T11:38:20Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "fa06c1bf-1c6e-4a97-a44c-ca859c06443b"}
2024-11-13T11:38:20Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "fa06c1bf-1c6e-4a97-a44c-ca859c06443b", "resource": "apps/v1.Deployment", "name": "worker-node"}
2024-11-13T11:38:20Z	INFO	Creating a new HPA	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "fa06c1bf-1c6e-4a97-a44c-ca859c06443b", "HPA.Namespace": "workers", "HPA.Name": "keda-hpa-worker-node-aws-sqs-queue-scaledobject"}
2024-11-13T11:38:20Z	ERROR	scale_handler	error getting triggerAuth	{"type": "ScaledObject", "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "triggerAuthRef.Name": "worker-node-credentials", "error": "TriggerAuthentication.keda.sh \"worker-node-credentials\" not found"}
github.com/kedacore/keda/v2/pkg/scaling/resolver.resolveAuthRef
	/workspace/pkg/scaling/resolver/scale_resolvers.go:238
github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
	/workspace/pkg/scaling/resolver/scale_resolvers.go:183
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
	/workspace/pkg/scaling/scalers_builder.go:72
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scalers_builder.go:90
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:354
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:217
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:72
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/workspace/controllers/keda/hpa.go:45
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:452
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:291
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:193
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2024-11-13T11:38:20Z	ERROR	scale_handler	error resolving auth params	{"type": "ScaledObject", "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "triggerIndex": 0, "error": "error parsing SQS queue metadata: awsAccessKeyID not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scalers_builder.go:93
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:354
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:217
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:72
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/workspace/controllers/keda/hpa.go:45
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:452
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:291
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:193
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2024-11-13T11:38:20Z	ERROR	Error getting scalers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "fa06c1bf-1c6e-4a97-a44c-ca859c06443b", "error": "error parsing SQS queue metadata: awsAccessKeyID not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:219
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:72
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/workspace/controllers/keda/hpa.go:45
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:452
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:291
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:193
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2024-11-13T11:38:20Z	ERROR	Failed to create new HPA resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "fa06c1bf-1c6e-4a97-a44c-ca859c06443b", "HPA.Namespace": "workers", "HPA.Name": "keda-hpa-worker-node-aws-sqs-queue-scaledobject", "error": "error parsing SQS queue metadata: awsAccessKeyID not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/workspace/controllers/keda/hpa.go:47
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:452
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:291
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:193
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2024-11-13T11:38:20Z	ERROR	failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "fa06c1bf-1c6e-4a97-a44c-ca859c06443b", "error": "error parsing SQS queue metadata: awsAccessKeyID not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:195
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2024-11-13T11:38:20Z	INFO	TriggerAuthentication Not Found	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "fa06c1bf-1c6e-4a97-a44c-ca859c06443b"}
2024-11-13T11:38:20Z	ERROR	Failed to get TriggerAuthentication	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "fa06c1bf-1c6e-4a97-a44c-ca859c06443b", "error": "TriggerAuthentication.keda.sh \"worker-node-credentials\" not found"}
github.com/kedacore/keda/v2/pkg/status.updateTriggerAuthenticationStatus
	/workspace/pkg/status/status.go:121
github.com/kedacore/keda/v2/pkg/status.UpdateTriggerAuthenticationStatusFromTriggers
	/workspace/pkg/status/status.go:157
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateTriggerAuthenticationStatus
	/workspace/controllers/keda/scaledobject_controller.go:615
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:217
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2024-11-13T11:38:20Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "fa06c1bf-1c6e-4a97-a44c-ca859c06443b", "error": "error parsing SQS queue metadata: awsAccessKeyID not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2024-11-13T11:38:20Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "31ee2284-6b2d-4674-ad9b-3c9fe4bd24ac"}
2024-11-13T11:38:20Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "31ee2284-6b2d-4674-ad9b-3c9fe4bd24ac", "resource": "apps/v1.Deployment", "name": "worker-node"}
2024-11-13T11:38:20Z	INFO	Adding Finalizer for the TriggerAuthentication	{"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "TriggerAuthentication": {"name":"worker-node-credentials","namespace":"workers"}, "namespace": "workers", "name": "worker-node-credentials", "reconcileID": "90f217ce-4ada-40b3-95b0-5a48c4f52861"}
2024-11-13T11:38:20Z	INFO	Creating a new HPA	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "31ee2284-6b2d-4674-ad9b-3c9fe4bd24ac", "HPA.Namespace": "workers", "HPA.Name": "keda-hpa-worker-node-aws-sqs-queue-scaledobject"}
2024-11-13T11:38:20Z	INFO	Initializing Scaling logic according to ScaledObject Specification	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "31ee2284-6b2d-4674-ad9b-3c9fe4bd24ac"}
2024-11-13T11:38:20Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "347df847-528b-49e7-bb97-b693ed3d6579"}
2024-11-13T11:38:20Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"worker-node-aws-sqs-queue-scaledobject","namespace":"workers"}, "namespace": "workers", "name": "worker-node-aws-sqs-queue-scaledobject", "reconcileID": "347df847-528b-49e7-bb97-b693ed3d6579", "resource": "apps/v1.Deployment", "name": "worker-node"}
2024-11-13T11:38:20Z	INFO	scaleexecutor	Successfully set ScaleTarget replicas count to ScaledObject minReplicaCount	{"scaledobject.Name": "worker-node-aws-sqs-queue-scaledobject", "scaledObject.Namespace": "workers", "scaleTarget.Name": "worker-node", "Original Replicas Count": 1, "New Replicas Count": 4}

It is able to scale the pods from the default 1 to the minimum 4, but it can't authenticate with AWS to access the queue.

Both TriggerAuthentication and ScaledObject are on the same namespace, so I would expect them to just work without having to set the namespace explicitly but even then, it still fail.

Am I missing something?

Any light would be appreciated.

Thanks!