Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SDTEST-437] Auto test retries for RSpec #213

Merged
merged 8 commits into from
Aug 8, 2024

Conversation

anmarchenko
Copy link
Member

@anmarchenko anmarchenko commented Aug 7, 2024

What does this PR do?
Adds auto test retries functionality for RSpec test framework

How to test the change?
Tested using middleman test suite with this flaky test:

    it "is new flaky" do
      res = rand(3)
      expect(res).to eq 1
    end

Locally test process completed normally:
image

Results of the test run in Datadog

New flaky test detected:
image

All the retries are correctly displayed in Datadog test runs:
image

Test session has status "pass":
image

@codecov-commenter
Copy link

codecov-commenter commented Aug 7, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.89%. Comparing base (f00caf6) to head (9e6f62e).
Report is 12 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #213      +/-   ##
==========================================
+ Coverage   98.84%   98.89%   +0.04%     
==========================================
  Files         253      261       +8     
  Lines       11630    12096     +466     
  Branches      521      544      +23     
==========================================
+ Hits        11496    11962     +466     
  Misses        134      134              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

# it is going to be reported once after retries are done
return super unless @skip_reporting

super(::RSpec::Core::NullReporter)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we're delaying the reporting until after the retries, then isn't it true that we would potentially hide the fact that there were flakes? or will the final trace data include the flake information?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is hiding reporting (for now) to RSpec formatter that prints the final message and tracks tests that failed (I might fix it in later versions to format retries correctly).

As tested in instrumentation_spec every retry is traced as a separate test span, so the test will be correctly identified as flaky. Also, test suite and test session statuses will be correctly reported as pass.

@anmarchenko anmarchenko marked this pull request as ready for review August 8, 2024 14:41
@anmarchenko anmarchenko requested review from a team as code owners August 8, 2024 14:41
@anmarchenko anmarchenko merged commit b891aaf into main Aug 8, 2024
28 checks passed
@anmarchenko anmarchenko deleted the anmarchenko/auto_test_retries_rspec branch August 8, 2024 14:56
@github-actions github-actions bot added this to the 1.4.0 milestone Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants