-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SDTEST-437] Auto test retries for RSpec #213
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #213 +/- ##
==========================================
+ Coverage 98.84% 98.89% +0.04%
==========================================
Files 253 261 +8
Lines 11630 12096 +466
Branches 521 544 +23
==========================================
+ Hits 11496 11962 +466
Misses 134 134 ☔ View full report in Codecov by Sentry. |
# it is going to be reported once after retries are done | ||
return super unless @skip_reporting | ||
|
||
super(::RSpec::Core::NullReporter) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if we're delaying the reporting until after the retries, then isn't it true that we would potentially hide the fact that there were flakes? or will the final trace data include the flake information?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is hiding reporting (for now) to RSpec formatter that prints the final message and tracks tests that failed (I might fix it in later versions to format retries correctly).
As tested in instrumentation_spec every retry is traced as a separate test span, so the test will be correctly identified as flaky. Also, test suite and test session statuses will be correctly reported as pass.
What does this PR do?
Adds auto test retries functionality for RSpec test framework
How to test the change?
Tested using middleman test suite with this flaky test:
Locally test process completed normally:
Results of the test run in Datadog
New flaky test detected:
All the retries are correctly displayed in Datadog test runs:
Test session has status "pass":