Provide e2e test for all scalers and secret providers (Help Wanted)

[tomkerkhove]

We must provide end-to-end tests for all our scalers and secret providers.

However, the following are missing:

Scalers

• Add e2e test for CPU Scaler #2219
• Add e2e test for Memory Scaler #2220
• Add e2e test for Liiklus Topic Scaler #1529
• Add e2e test for AWS Kinesis Stream Scaler #1526
• Add e2e test for AWS SQS Queue Scaler #1527
• Add e2e test for AWS CloudWatch scaler #1525
• Add e2e test for Huawei Cloudeye Scaler  #1524
• Add e2e test for Google Cloud Pub/Sub Scaler #1528
• Add e2e test for external scaler #2697
• Add e2e test for external push scaler #2698
• Add e2e test for Azure Event Hub scaler #2792
• Add e2e test for Azure Service Bus scaler (Topic) #2731
• Add e2e test for Azure Service Bus scaler (Queue) #2732
• Add e2e test for Metric API Scaler #2791
• Add e2e test for Azure Monitor #2928
• Add e2e test for IBM MQ Scaler #1287
• Add e2e test for Openstack Metrics Scaler #4155
• Implement e2e test for Fallback #4129
• Create e2e tests for all exposed Prometheus metrics #4127
• Add e2e test for Azure Blob Storage using Workload Identity and Pod Identity #3895
• Add e2e test for Azure Service Bus Topic using Workload Identity and Pod Identity #3894
• Add e2e test for Azure Service Bus Queue using Workload Identity and Pod Identity #3893
• Add e2e test for Azure Log Analytics using Workload Identity and Pod Identity #3892
• Add e2e test for Azure Application Insights using Workload Identity and Pod Identity #3891
• Add e2e test for Azure Data Explorer using Workload Identity and Pod Identity #3890
• Add e2e test for Azure Storage Queue using Workload Identity and Pod Identity #3889
• Add e2e test for Azure Event Hubs using Workload Identity and Pod Identity #3888
• Add e2e test for Azure Monitor using Workload Identity and Pod Identity #3887
• Implement e2e tests where ScaledObject is updated #3870
• Implement e2e test where mutliple triggers in ScaledObject are being used #3868
• Extend e2e and unit test coverage for ScaledJob #3661
• Add e2e test for SO updates of pollingInterval #4266
• Add e2e test for SO updates of cooldownPeriod #4267

Secret providers

• Add e2e test for Hashicorp Vault #2842
• Add e2e test for Azure Key Vault #2843

[RamCohen]

Hi @tomkerkhove , will be happy to work on this, but for it to be submitted, the issue of having a secret with the credentials for a GCP service account to use should be resolved (#2648)

[v-shenoy]

@tomkerkhove The Azure Monitor scaler is also missing an e2e test.

[tomkerkhove]

Nice find, opened #2928 - Thanks!

[nilayasiktoprak]

Would it be good to mention here that now e2e tests are migrating to Go in case someone wants to take any remaining tests above?

[v-shenoy]

@tomkerkhove @JorTurFer @zroubalik We need to create new issues for scalers that have e2e tests in ts in need of migration to go, and add them here.

[tomkerkhove]

Feel free to open them, thanks!

[v-shenoy]

Okay, I created a few. You can edit and add them to the issue. @tomkerkhove

But I also realize that there's too many of them. What's the smoothest way here to transition?

[zroubalik]

I'd create one umbrella issue, with tasks

• scaler X
• scaler Y

The task could be transferred into an issue if there's a person who is willing to contribute that.

[v-shenoy]

I'd create one umbrella issue, with tasks

• scaler X
• scaler Y

The task could be transferred into an issue if there's a person who is willing to contribute that.

Here's a list. Maybe Tom can edit and add them as a section in this issue itself. Or maybe here #2737

• ActiveMQ
• Argo Rollouts
• Artemis
• AWS CloudWatch Expression
• AWS CloudWatch (Migrate e2e test for AWS Cloudwatch scaler to Go #3173)
• AWS DynamoDB
• AWS Kinesis Stream (Migrate e2e test for AWS Kinesis Stream scaler to Go. #3171)
• AWS SQS Queue (Migrate e2e test for AWS SQS Queue scaler to Go. #3172)
• Azure App Insights
• Azure Blob
• Azure Data Explorer
• Azure Log Analytics
• Azure Pipelines
• Azure Queue (Idle Replicas)
• Azure Queue (Pause At N)
• Azure Queue (Pause)
• Azure Queue (Restore Original Replicas)
• Azure Service Bus (Queue Workload Identity)
• Cassandra
• CPU (Migrate e2e test for CPU scaler to Go. #3169)
• Datadog
• Elastic Search
• External Scaler (ScaledJob + ScaledObject) (Migrate e2e test for external scaler to Go. #3175)
• GCP PubSub (Migrate e2e test for Google Cloud Pub/Sub scaler to Go. #3174)
• GCP StackDriver
• GCP Storage
• Graphite
• InfluxDB
• Kafka
• Kubernetes Workload
• Memory (Migrate e2e test for memory scaler to Go. #3170)
• Metrics API (Trigger Auth)
• Metrics API
• MonboDB
• MSSQL
• MySQL
• NewRelic
• OpenStack Swift
• PostgreSQL (Hashicorp) (Migrate e2e test for Hashicorp Vault to Go. #3176)
• PostgreSQL
• PredictKube
• Prometheus (Value Metric Type)
• Prometheus
• RabbitMQ (AMQP)
• RabbitMQ (HTTP Regex)
• RabbitMQ (HTTP Regex Vhost)
• RabbitMQ (HTTP)
• RabbitMQ (TriggerAuth)
• Redis Cluster (Lists + Streams)
• Redis (Lists + Streams)
• Redis Sentinel (Lists + Streams)
• Selenium Grid
• Solace
• Stan

[zroubalik]

Thanks!

[zroubalik]

Updated here: #2737

[tomkerkhove]

I'd create one umbrella issue, with tasks

• scaler X
• scaler Y

The task could be transferred into an issue if there's a person who is willing to contribute that.

Here's a list. Maybe Tom can edit and add them as a section in this issue itself. Or maybe here #2737

• ActiveMQ
• Argo Rollouts
• Artemis
• AWS CloudWatch Expression
• AWS CloudWatch (Migrate e2e test for AWS Cloudwatch scaler to Go. #3173)
• AWS DynamoDB
• AWS Kinesis Stream (Migrate e2e test for AWS Kinesis Stream scaler to Go. #3171)
• AWS SQS Queue (Migrate e2e test for AWS SQS Queue scaler to Go. #3172)
• Azure App Insights
• Azure Blob
• Azure Data Explorer
• Azure Log Analytics
• Azure Pipelines
• Azure Queue (Idle Replicas)
• Azure Queue (Pause At N)
• Azure Queue (Pause)
• Azure Queue (Restore Original Replicas)
• Azure Service Bus (Queue Workload Identity)
• Cassandra
• CPU (Migrate e2e test for CPU scaler to Go. #3169)
• Datadog
• Elastic Search
• External Scaler (ScaledJob + ScaledObject) (Migrate e2e test for external scaler to Go. #3175)
• GCP PubSub (Migrate e2e test for Google Cloud Pub/Sub scaler to Go. #3174)
• GCP StackDriver
• GCP Storage
• Graphite
• InfluxDB
• Kafka
• Kubernetes Workload
• Memory (Migrate e2e test for memory scaler to Go. #3170)
• Metrics API (Trigger Auth)
• Metrics API
• MonboDB
• MSSQL
• MySQL
• NewRelic
• OpenStack Swift
• PostgreSQL (Hashicorp) (Migrate e2e test for Hashicorp Vault to Go. #3176)
• PostgreSQL
• PredictKube
• Prometheus (Value Metric Type)
• Prometheus
• RabbitMQ (AMQP)
• RabbitMQ (HTTP Regex)
• RabbitMQ (HTTP Regex Vhost)
• RabbitMQ (HTTP)
• RabbitMQ (TriggerAuth)
• Redis Cluster (Lists + Streams)
• Redis (Lists + Streams)
• Redis Sentinel (Lists + Streams)
• Selenium Grid
• Solace
• Stan

Why do we have Argo Rollouts in the list? This is not a scaler we have today.

[v-shenoy]

I'd create one umbrella issue, with tasks

• scaler X
• scaler Y

The task could be transferred into an issue if there's a person who is willing to contribute that.

Here's a list. Maybe Tom can edit and add them as a section in this issue itself. Or maybe here #2737

• ActiveMQ
• Argo Rollouts
• Artemis
• AWS CloudWatch Expression
• AWS CloudWatch (Migrate e2e test for AWS Cloudwatch scaler to Go. #3173)
• AWS DynamoDB
• AWS Kinesis Stream (Migrate e2e test for AWS Kinesis Stream scaler to Go. #3171)
• AWS SQS Queue (Migrate e2e test for AWS SQS Queue scaler to Go. #3172)
• Azure App Insights
• Azure Blob
• Azure Data Explorer
• Azure Log Analytics
• Azure Pipelines
• Azure Queue (Idle Replicas)
• Azure Queue (Pause At N)
• Azure Queue (Pause)
• Azure Queue (Restore Original Replicas)
• Azure Service Bus (Queue Workload Identity)
• Cassandra
• CPU (Migrate e2e test for CPU scaler to Go. #3169)
• Datadog
• Elastic Search
• External Scaler (ScaledJob + ScaledObject) (Migrate e2e test for external scaler to Go. #3175)
• GCP PubSub (Migrate e2e test for Google Cloud Pub/Sub scaler to Go. #3174)
• GCP StackDriver
• GCP Storage
• Graphite
• InfluxDB
• Kafka
• Kubernetes Workload
• Memory (Migrate e2e test for memory scaler to Go. #3170)
• Metrics API (Trigger Auth)
• Metrics API
• MonboDB
• MSSQL
• MySQL
• NewRelic
• OpenStack Swift
• PostgreSQL (Hashicorp) (Migrate e2e test for Hashicorp Vault to Go. #3176)
• PostgreSQL
• PredictKube
• Prometheus (Value Metric Type)
• Prometheus
• RabbitMQ (AMQP)
• RabbitMQ (HTTP Regex)
• RabbitMQ (HTTP Regex Vhost)
• RabbitMQ (HTTP)
• RabbitMQ (TriggerAuth)
• Redis Cluster (Lists + Streams)
• Redis (Lists + Streams)
• Redis Sentinel (Lists + Streams)
• Selenium Grid
• Solace
• Stan

Why do we have Argo Rollouts in the list? This is not a scaler we have today.

I think it's to test that KEDA is able to scale custom resources besides scaling deployments, and spawning jobs.

[JorTurFer]

yes, we use argo-rollouts to test KEDA with custom resources with /scale but we don't support scaling base on them

[tomkerkhove]

Is the above list still accurate @JorTurFer ?

[JorTurFer]

Is the above list still accurate @JorTurFer ?

I'll check it during the day and I'll update the issue if needed

[JorTurFer]

I have just updated the description with the new issues. All the Azure Workload Identity apply also for AAD-Pod-Idendity, I have to update the issues to include both (as the tests are quite similar, we can do both at once)

[tomkerkhove]

Awesome, thanks! Do you think it makes sense splitting scalers from auth support?

I'm asking because the list gets long, while majority is just for Azure auth e2e.

[JorTurFer]

Yeah,
As the authentication code is partially part of the scaler code, I think we should have those test cases covered. I mean, for example, in azure scalers (where we use multiple SDKs) having one covered with Workload Identity doesn't guarantee that the other scalers will work.
The only case where it can be duplicated is in scalers like servicebus where we have topics and queues, but the majority of the code is the same.
We can remove them if you think isn't worth, but I prefer to cover them, because otherwise we don't have guarantees of them (a typo could be introduced in scaler code for example, removing the pod identity support). From the e2e test duration pov, we could go more aggresive running more test together if we want to reduce the e2e test duration.

I'm asking because the list gets long, while majority is just for Azure auth e2e.

Yes, because I already added them for AWS and GCP scalers xD By AWS and GCP scalers are also covered alone and using pod Identity, the problem here is that azure has 2 supported pod identities and has more scalers than others, which means more effort covering them.

My only doubt is if we should cover aad-pod-identity as has EOL already defined, but... a year means 3-4 releases where we can potentially break it during the support period