[CIVIS-9332] unskippable tests for ITR #167
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds "unskippable by ITR tests" support for RSpec, Cucumber, and Minitest. It allows application developers using intelligent test runner to mark some of their test to make sure that they are always run and never skipped by ITR.
Motivation
ITR is not able to correctly track dependencies between tests and external non-code resources. Also, Ruby code coverage does not work across threads and/or processes.
Examples of tests that should be unskippable by ITR:
Additional Notes
Usage examples are below
RSpec:
Cucumber:
Mintiest:
Note that for minitest the developer's experience is not on the same level as for RSpec or Cucumber: this is due to the fact that minitest does not have test tagging capabilities out of the box. Currently it will be required to pass test names of unskippable tests to
datadog_itr_unskippable
method. We'll gather feedback on this and if the need arises, will provide better tagging capabilities (similar to what https://github.com/bernardoamc/minitag or https://github.com/jbodah/minitest-tagz can do).How to test the change?
Unit tests are provided.