RabbitMQ scaler is watching the number of Ready instead of Total number of messages

[krisowski]

Report

When the RabbitMQ scaler is configured with the amqp protocol it is watching the number of Ready and not the Total number of messages as the code and docs suggest.

The number of messages is taken from the QueueDeclarePassive API here: https://github.com/kedacore/keda/blob/main/pkg/scalers/rabbitmq_scaler.go#L491-L496. Unfortunately, this API returns the number of Ready messages, excluding the unacked ones, and not the total number of them according to the docs here: https://www.rabbitmq.com/dotnet-api-guide.html#passive-declaration

The items.Messages variable name suggests that this is the total number of messages but should be called items.MessagesReady. However, changing that doesn't solve the issue. There should be another way of getting the number of total messages by using the amqp protocol. I'm not sure if there is such method

Expected Behavior

The RabbitMQ scaler with the amqp configuration should return the number of Total messages in the queue and not the number of Ready messages.

I also believe, that if there was a way to get both, Ready and Total, by using the amqp protocol then the excludeUnacknowledged option could albo be supported with the amqp protocol and not with the http only

Actual Behavior

KEDA autoscaling doesn't work as expected.

Let's say it is configured with the QueueLength and value set to 1.
The number of messages in the queue "now":
Number of Total messages (ready+unacked): 50
Number of Unacked messages: 40 (the messages are being processed now)
Number of Ready messages: 10 (waiting for processing)
Number of running pods/consumers: 40

Result: no more pods are spawned even though there are 10 messages waiting and only 40 pods running. KEDA doesn't add more pods before it thinks there are more pods than pending messages.

Steps to Reproduce the Problem

1. Configure KEDA with RabbitMQ: mode: QueueLength and value: 1
2. Publish 10 messages
3. Start 8 customers and let them process the messages without acknowledging them
4. Observe that only 8 consumers are running and not 10 as expected

Logs from KEDA operator

example

KEDA Version

2.11.0

Kubernetes Version

1.24

Platform

Google Cloud

Scaler Details

RabbitMQ

Anything else?

This issue exists also in the latest version.

[JorTurFer]

Hi,
Thanks for your comment, but sadly amqp protocol doesn't support it. There is nothing that we can do if the underlying SDK doesn't support it, sorry.
If you need the total messages, I'd suggest using the http protocol.

As this can't be fixed, I close this issue as not planned. If you think that the SDK supports somehow this feature (it could be, the SDKs grow and we can't track all of them), just comment the issue and I'll reopen it

[krisowski]

I see, but I suggest two changes though:

1. Update the docs to make it clear that the QueueLength value for amqp works for Ready messages and not for all messages in the queue. Today this is misleading and that's why we have this ticket
2. Add a comment here

   keda/pkg/scalers/rabbitmq_scaler.go

   Line 496 in dbb906f

   return int64(items.Messages), 0, nil

   explaining that the Messages from RabbitMQ response is actually the number of messages in Ready state

[zroubalik]

@krisowski sounds reasonable, are you willing to contribute that suggestion? thanks

[krisowski]

@zroubalik I can contribute but where do I find the docs? It looks like it's not part of this repo

[zroubalik]

@krisowski see this repo: https://github.com/kedacore/keda-docs/tree/main/content/docs