[CIVIS-7948] Test suite level visibility instrumentation for Minitest framework

[anmarchenko]

What does this PR do?
Adds test suite level visibility instrumentation for Minitest framework

Important design note
The most important decision made during the development was about the concurrent test execution in Minitest.

When tests are executed serially, they are executed suite by suite (or Runnable by Runnable in Minitest case) just like in RSpec or Cucumber. Things change when tests are executed concurrently (which is a built in feature for Minitest). In this case runners (implemented as threads or forked processes) receive jobs and execute them one by one. "Job" in this case is a single test (or Runnable's method) not the whole runnable:
https://github.com/minitest/minitest/blob/master/lib/minitest/parallel.rb#L33

The proposed implementation is a pragmatic solution to this problem:

• for serial execution we treat Runnable as a test suite that gives us the same functionality as for RSpec and Cucumber frameworks
• for parallel execution we wrap every test in a separate test suite (adding a unique prefix "(#{test_name} concurrently)")

This produces the following session traces

For serial execution:

For concurrent execution:

This approach does not allow to track suite performance in concurrent mode but it shows the execution model better:
with concurrent execution enabled the most important performance metric is time to execute individual tests, not test files.

Other Notes

• Using DatadogReporter for test suites tracking isn't possible due to the fact that Rails parallel executor's workers do not call reporter's callbacks: https://github.com/rails/rails/blob/f481353150c100dbd944d5aa84f2480264cd723c/activesupport/lib/active_support/testing/parallelization/worker.rb#L50
• During testing I hit the limitation of tracer's buffer size: concurrent threads running a lot of tests overflow the buffer, and traces are being dropped in result. This PR increases buffer size for CI mode to 10 000 traces.

How to test the change?
Tested using the following repositories:
https://github.com/anmarchenko/openstreetmap-website
https://github.com/anmarchenko/sidekiq
https://github.com/anmarchenko/feedbin
https://github.com/anmarchenko/devdocs
https://github.com/anmarchenko/middleman

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c78d26f) 99.18% compared to head (1c278b7) 99.22%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #92      +/-   ##
==========================================
+ Coverage   99.18%   99.22%   +0.04%     
==========================================
  Files         139      142       +3     
  Lines        5624     5924     +300     
  Branches      216      233      +17     
==========================================
+ Hits         5578     5878     +300     
  Misses         46       46              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nikita-tkachenko-datadog]

Does it mean a test suite can never have 'skipped' status? Not sure what the intended behaviour is, mentioning it just in case

[anmarchenko]

Yes, I haven't implemented SKIP for test suites in any of the instrumentations right now. Is it heavily used by other libraries (if we don't include ITR)?

[nikita-tkachenko-datadog]

In Java the suite status is SKIP if it contains 0 test cases, or if every test case in it is skipped. Not sure about other languages

[anmarchenko]

thank you! I will add a note for the future improvements

[nikita-tkachenko-datadog]

Looks like it's implicitly assumed here that the current dir is always the same as the repo root. Not sure if the two can be different in Ruby world. If they can, things like source code integration and ITR will break in the backend

[anmarchenko]

I haven't yet encountered cases where this wouldn't be true for Ruby as both bin/rails and rake command are always executed from the root. Note that this is actually a test suite name (which could be any arbitrary string), not the source code integration (which I will add together with ITR). For source code integration, I will check for the repository root.