Skip to content

Add test retry logic for flaky tests #8654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
mtrezza opened this issue Jun 22, 2023 · 6 comments · Fixed by #9218
Closed
3 tasks done

Add test retry logic for flaky tests #8654

mtrezza opened this issue Jun 22, 2023 · 6 comments · Fixed by #9218
Labels
bounty:$20 Bounty applies for fixing this issue (Parse Bounty Program) state:released Released as stable version state:released-alpha Released as alpha version state:released-beta Released as beta version type:ci CI related issue

Comments

@mtrezza
Copy link
Member

mtrezza commented Jun 22, 2023

New Feature / Enhancement Checklist

Current Limitation

There are still some flaky tests;

  • requires more effort to merge a PR
  • shows ci: failed labels on README

Feature / Enhancement Description

Add a test-retry logic on jasmine-level (not GitHub workflow level) specifically for flaky tests.

  1. Implement logic to retry specific tests if they fail, in the style of the existing custom test specifiers:
    global.it_exclude_dbs = excluded => {
  2. Identify flaky tests
  3. Use retry-test-specifier for flaky tests

Alternatives / Workarounds

Manually re-run tests (status quo)

References

@parse-github-assistant
Copy link

parse-github-assistant bot commented Jun 22, 2023

Thanks for opening this issue!

  • 🎉 We are excited about your ideas for improvement!

@mtrezza mtrezza added type:ci CI related issue bounty:$20 Bounty applies for fixing this issue (Parse Bounty Program) labels Jun 22, 2023
@dplewis
Copy link
Member

dplewis commented Jul 17, 2024

@mtrezza I was just looking into doing something like this. Using the test name as an identifier would be the easiest way to go.

@mtrezza
Copy link
Member Author

mtrezza commented Jul 17, 2024

There may be ways; I've looked into this quite extensively with jasmine but I couldn't find a way given its architecture. Hence I published #8655 to document my findings, closed it and kept this issue open, in case someone finds a way in the future. I think I've spent more than a day on it. The challenge is to build something on top of, or integrated into jasmine, so we have the test stats, etc. It may not be difficult to re-run a test, but I couldn't find a way to do that so that jasmine understands what's happening.

Another approach that came to my mind was to maybe use the GitHub CI to automatically create a test run per spec file. So that instead of re-running all tests when one fails, only the spec file of the failing test needs to re-run. But this'd need to consider coverage reporting, not spamming our CI dashboard, etc.

@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 7.3.0-alpha.7

@parseplatformorg parseplatformorg added the state:released-alpha Released as alpha version label Aug 27, 2024
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 7.3.0-beta.1

@parseplatformorg parseplatformorg added the state:released-beta Released as beta version label Oct 3, 2024
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 7.3.0

@parseplatformorg parseplatformorg added the state:released Released as stable version label Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bounty:$20 Bounty applies for fixing this issue (Parse Bounty Program) state:released Released as stable version state:released-alpha Released as alpha version state:released-beta Released as beta version type:ci CI related issue
Projects
None yet
3 participants