Metrics server suddenly stoped getting metrics from rabbitmq

[alexbanditos]

Report

Without known reason keda metrics server started getting errors for all auth triggers -

E0111 20:51:14.833101 1 scale_resolvers.go:164] keda_metrics_adapter/scalehandler "msg"="Error getting triggerAuth" "error"="context canceled" "name"="service-hpa-rmq" "namespace"="default" "type"="ScaledObject" "triggerAuthRef.Name"="prod-something-rmq"
E0111 20:51:14.845428 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error parsing rabbitmq metadata: no host setting given" "scaledObject.Name"="service-rmq" "scaledObject.Namespace"="default" "scaler"={}

we are using TriggerAuthentication with vault , no issues with vault cluster
the trigger and the scaledobject are in the same ns

trigger:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: prod-something-rmq
spec:
hashiCorpVault:
address: https://vault.cluster.com
authentication: kubernetes
role: keda-operator
mount: 'kubernetes/prod-cluster/'
credential:
serviceAccount: '/var/run/secrets/kubernetes.io/serviceaccount/token'
secrets:
- parameter: host
key: host
path: "secrets/data/infra/keda/prod-something-rmq"

Only after I delete the metrics server pod and it recreated scaling started working.

Expected Behavior

Metrics server able to monitor rabbitmq queues

Actual Behavior

keda in not scaling hpa based on rmq

Steps to Reproduce the Problem

1. create TriggerAuthentication with vault auth (external vault)
2. create scaledobject based on rabbit queue
3. wait for metrics server send errors.

Logs from KEDA operator

error	Exception (504) Reason: "channel/connection is not open"
level	error
logger	rabbitmq_scaler
stacktrace	github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive /workspace/pkg/scaling/cache/scalers_cache.go:89 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers /workspace/pkg/scaling/scale_handler.go:261 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop /workspace/pkg/scaling/scale_handler.go:146
ts	1641934785.0178342
}

KEDA Version

2.5.0

Kubernetes Version

1.18

Platform

Amazon Web Services

Scaler Details

rabbitmq

Anything else?

No response

[JorTurFer]

Hi @alexbanditos
Could you share more logs please?

[alexbanditos]

@JorTurFer
from metrics server:

• HH:mm:ss.SSSSSS 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="error parsing rabbitmq metadata: no host setting given" "scaledObject.Name"="*" "scaledObject.Namespace"="default" "scaler"={}
  1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"No matching metrics found for *"}: No matching metrics found for *
• HH:mm:ss.SSSSSS 1 scale_resolvers.go:164] keda_metrics_adapter/scalehandler "msg"="Error getting triggerAuth" "error"="context canceled" "name"="*" "namespace"="default" "type"="ScaledObject" "triggerAuthRef.Name"="prod-something-rmq"

from keda operator:

{

error | Exception (504) Reason: "channel/connection is not open"
level | error
logger | rabbitmq_scaler
stacktrace | github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive /workspace/pkg/scaling/cache/scalers_cache.go:89 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers /workspace/pkg/scaling/scale_handler.go:261 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop /workspace/pkg/scaling/scale_handler.go:146
ts | 1641934785.0183375
}

{

This is the only errors I got

[JorTurFer]

I think that there are missing logs :(
Could you collect more logs during the next time that this happens, please? Another thing, please paste them as a code because now they are super difficult for reading

[zroubalik]

Might be related to this: #2415

[JorTurFer]

The metrics server logs show one error, I think that not because #2415 is related with the operator

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.