Add support for scaling by Google Pub/Sub subscription publish rate

[mbrancato]

Proposal

Scaling by unacknowledged messages and oldest message are good, reactive scaling metrics. If a queue is behind, both of these are good for responding to how much is currently in the queue. They are, however, reactive and likely can lead to constant scaling up and down to keep up with consistent message volumes.

It would be nice if there was a trigger based on the rate of incoming messages (pubsub.googleapis.com/topic/message_sizes) to keep up with current rate, and use a reactive metric like unacknowledged messages to scale up when things get behind.

Scaler Source

Google Pub/Sub incoming messages

Scaling Mechanics

Scale based on metric

Authentication Source

GCP

Anything else?

No response

[JorTurFer]

Hello!
It's a nice idea, but I think we can just add it as another flavor of the current scaler. Does GCP expose the publishing rate metric?
BTW, are you willing to contribute with the feature?

[kevinmingtarja]

Hey, I'm interested in taking on this feature!

[kevinmingtarja]

I see that there's already pkg/scalers/gcp_pubsub_scaler.go for Google Pub/Sub, so I'm guessing this should be a good place to start, along with other scalers in this package.

But please let me know if you have any additional pointer, thank you!

[zroubalik]

Thanks @kevinmingtarja, yeah, that is a good starting point. Here are docs for contributing a new scaler, you might find something valuable there.

[kevinmingtarja]

Got it, thanks for the pointers! I'll get started soon.

[kevinmingtarja]

Hi @zroubalik @JorTurFer, I did a bit of reading and have a question regarding how we should approach this.

Right now for the GCP Pub/Sub scaler, the level of granularity is subscriptions, so as part of the metadata users supply subscriptionName and mode (which can be any metrics that starts with subscription/) as inputs.

But the proposal here is to use the metric topic/message_sizes, which is a different level of granularity than subscription/, and thus not supported right now. The relationship between the two is that a topic can have multiple subscriptions, but a given subscription belongs to a single topic.

With this in mind, I think there are two options at least:

1. Extend the existing scaler. Perhaps we can have a new input topicName, and make one of subscriptionName | topicName required. Based on which of the two has a non nil value, we're able to figure out whether the mode will be a subscription or a topic metric, so we can prefix the mode accordingly when looking up the metric via the GCP API.
2. Create a new scaler, so we have one scaler working at the level of subscriptions (the existing one), and a new one for topics.

I'm kind of leaning towards option (1), since the API change is very limited and non-breaking, so by default it's backwards compatible as well. But I wanted to get a second opinion before starting to code. Thank you!

[JorTurFer]

I'm kind of leaning towards option (1), since the API change is very limited and non-breaking, so by default it's backwards compatible as well. But I wanted to get a second opinion before starting to code. Thank you!

I also think that the option 1 is the best if it's not a breaking change

[kevinmingtarja]

Got it, thanks for the input Jorge

[zroubalik]

I agree with that, thanks for the investigation.

[kevinmingtarja]

After starting to code, I think I realized the current code assumes that the metric/mode is a single value, and doesn't handle the case if it's a distribution [1], like for topic/message_sizes.

The main difference I can see is that for distribution, we need to specify an aggregation like sum, mean, p99, etc (example below) in order to get a single value. So I'll address this as well in my changes.

---

References:
[1] GCP value types: https://cloud.google.com/monitoring/api/v3/kinds-and-types

[kevinmingtarja]

Hi @mbrancato, just a heads up, once the PR for this gets in on the next release, you will be able to do what you proposed with the following params with the GCP PubSub scaler!

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
...
spec:
  ...
  triggers:
  - type: gcp-pubsub
    metadata:
      mode: "MessageSizes"
      aggregation: "count"
      topicName: "mytopic"
      ...