Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve e2e test reliability #2580

Merged
merged 48 commits into from
Feb 11, 2022
Merged

Improve e2e test reliability #2580

merged 48 commits into from
Feb 11, 2022

Conversation

JorTurFer
Copy link
Member

@JorTurFer JorTurFer commented Jan 29, 2022 •
edited
Loading

Signed-off-by: jorturfer jorge_turrado@hotmail.es

This PR increases some timeouts across different e2e test and does these relevant changes:

  • Azure Pipelines: e2e tests now check scaling from/to 0 instead of between 1 and 3 because otherwise we have random fails if the removed pods during the scale in are still executing an AzDO job.
  • New Relic: e2e test has changed a bit to maintain the load till the scaling out is done.
  • Selenium: e2e test has changed, now nodes starts from 0 and check scaling to 0 instead of job status because sometimes the job started before the hub is ready and has more than 1 result (and the test fails)
  • Global: The timeout for each scaler e2e tests has been increased from 10m to 30m.
  • Global: Removed all the resources except for cpu e2e tests
  • Global: Failing e2e tests are retried one more time

Checklist

  • Commits are signed with Developer Certificate of Origin (DCO - learn more)
  • Tests have been added
  • A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
  • A PR is opened to update the documentation on (repo) (if applicable)
  • Changelog has been updated

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 29, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 29, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 29, 2022 •
edited by github-actions bot
Loading

/run-e2e

Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 29, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 30, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 30, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 30, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 30, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 30, 2022 •
edited by github-actions bot
Loading

/run-e2e

Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 30, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 30, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 31, 2022 •
edited by github-actions bot
Loading

/run-e2e

Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 31, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 31, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@tomkerkhove
Copy link
Member

Sorry but I cancelled the test run so that #2583 can be triggered, verified and merged for our release first.

/run-e2e

@JorTurFer
Copy link
Member Author

no problem

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 31, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Jan 31, 2022 •
edited by github-actions bot
Loading

/run-e2e

Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 1, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 1, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 1, 2022 •
edited by github-actions bot
Loading

/run-e2e

Update: You can check the progres here

@JorTurFer
Copy link
Member Author

/run-e2e

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 2, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 2, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 2, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 2, 2022 •
edited by github-actions bot
Loading

/run-e2e new-relic*
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 2, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 2, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 2, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

Jorge Turrado and others added 11 commits February 10, 2022 21:30
@JorTurFer
Update new relic test
ebd1447 
Signed-off-by: Jorge Turrado <jorge.turrado@docplanner.com>
@JorTurFer
set max instances for cassandra test to 2
86db63c 
Signed-off-by: jorturfer <jorge_turrado@hotmail.es>
@JorTurFer
Increase argo timeouts
7333e2a 
Signed-off-by: jorturfer <jorge_turrado@hotmail.es>
@JorTurFer
Add some configs to selenium
ddcbb9e 
Signed-off-by: jorturfer <jorge_turrado@hotmail.es>
@JorTurFer
undo change in integration test
899d0b6 
Signed-off-by: jorturfer <jorge_turrado@hotmail.es>
@JorTurFer
Update selenium test
4e3a488 
Signed-off-by: jorturfer <jorge_turrado@hotmail.es>
@JorTurFer
Add retry for failing e2e tests
7a18cdb 
Signed-off-by: Jorge Turrado <jorge.turrado@docplanner.com>
@JorTurFer
Update retry system
95338db 
Signed-off-by: Jorge Turrado <jorge.turrado@docplanner.com>
@JorTurFer
fix and error waiting
3ce106a 
Signed-off-by: Jorge Turrado <jorge.turrado@docplanner.com>
@JorTurFer
FORCE TEST FAILURE
1455c79 
Signed-off-by: jorturfer <jorge_turrado@hotmail.es>
@JorTurFer
Undo test failure probe
cf24adf 
Signed-off-by: jorturfer <jorge_turrado@hotmail.es>
@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 10, 2022 •
edited by github-actions bot
Loading

/run-e2e

Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 10, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Fix azure-queue-restore-original-replicas.test.ts
162e09a 
Signed-off-by: jorturfer <jorge_turrado@hotmail.es>
@JorTurFer
Update azure queue tests to avoid condition races
1621292 
Signed-off-by: jorturfer <jorge_turrado@hotmail.es>
@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 10, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 11, 2022 •
edited by github-actions bot
Loading

/run-e2e

Update: You can check the progres here

Fix some styles
16ba72d 
Signed-off-by: Jorge Turrado <jorge.turrado@docplanner.com>
@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 11, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 11, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

Improve azure queues
149cada 
Signed-off-by: Jorge Turrado <jorge.turrado@docplanner.com>
@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 11, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

@JorTurFer
Copy link
Member Author

JorTurFer commented Feb 11, 2022 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progres here

zroubalik
zroubalik approved these changes Feb 11, 2022
View reviewed changes
Copy link
Member

@zroubalik zroubalik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! Tremendous job @JorTurFer !

@zroubalik zroubalik merged commit 130bc93 into kedacore:main Feb 11, 2022
@tomkerkhove
Copy link
Member

Thanks a ton @JorTurFer!

@JorTurFer JorTurFer deleted the improve_e2e branch February 14, 2022 07:52
@JorTurFer
Copy link
Member Author

you're welcome ❤️

@JorTurFer JorTurFer mentioned this pull request Apr 27, 2022
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tomkerkhove tomkerkhove tomkerkhove approved these changes

@zroubalik zroubalik zroubalik approved these changes

Assignees
No one assigned
Labels
None yet
Projects
Roadmap - KEDA Core
Archived in project
Milestone
v2.7.0
Development

Successfully merging this pull request may close these issues.
3 participants
@JorTurFer @tomkerkhove @zroubalik