Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CIVIS-9332] unskippable tests for ITR #167

Merged
merged 7 commits into from
Apr 26, 2024

Conversation

anmarchenko
Copy link
Member

@anmarchenko anmarchenko commented Apr 24, 2024

What does this PR do?
Adds "unskippable by ITR tests" support for RSpec, Cucumber, and Minitest. It allows application developers using intelligent test runner to mark some of their test to make sure that they are always run and never skipped by ITR.

Motivation
ITR is not able to correctly track dependencies between tests and external non-code resources. Also, Ruby code coverage does not work across threads and/or processes.

Examples of tests that should be unskippable by ITR:

  • tests that read files from disk
  • tests that make network requests
  • tests that call external processes (with backticks for example)
  • tests that use forking or threading

Additional Notes
Usage examples are below

RSpec:

# mark the whole suite unskippable
RSpec.describe MyClass, datadog_itr_unskippable: true do
  # tests here
end

# mark one block unskippable
RSpec.describe MyClass do
  describe "#read_csv", datadog_itr_unskippable: true do
    it "reads CSV file" do
    end
  end
end

# mark one test unskippable
RSpec.describe MyClass do
  it "works", datadog_itr_unskippable: true do
  end
end

Cucumber:

# mark the whole feature unskippable
@datadog_itr_unskippable
Feature: My feature
  Scenario: Main page opens
    Given open main page
    Then see Hello world text

# mark one scenario unskippable
Feature: My feature
  @datadog_itr_unskippable
  Scenario: Main page opens
    Given open main page
    Then see Hello world text

Mintiest:

# mark test suite unskippable
class UnskippableTest < Minitest::Test
  datadog_itr_unskippable

  def test_1
  end

  def test_2
  end
end

# mark one test unskippable
class PartiallyUnskippableTest < Minitest::Test
  datadog_itr_unskippable "test_1"

  def test_1
  end

  def test_2
  end
end

Note that for minitest the developer's experience is not on the same level as for RSpec or Cucumber: this is due to the fact that minitest does not have test tagging capabilities out of the box. Currently it will be required to pass test names of unskippable tests to datadog_itr_unskippable method. We'll gather feedback on this and if the need arises, will provide better tagging capabilities (similar to what https://github.com/bernardoamc/minitag or https://github.com/jbodah/minitest-tagz can do).

How to test the change?
Unit tests are provided.

@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.06%. Comparing base (f0592d1) to head (67c63db).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #167   +/-   ##
=======================================
  Coverage   99.06%   99.06%           
=======================================
  Files         196      196           
  Lines        9377     9404   +27     
  Branches      407      409    +2     
=======================================
+ Hits         9289     9316   +27     
  Misses         88       88           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@anmarchenko anmarchenko changed the title add #itr_unskippable! method to a Datadog::CI::Test to mark test_span as unskippable by ITR [CIVIS-9332] unskippable tests for ITR Apr 24, 2024
@anmarchenko anmarchenko marked this pull request as ready for review April 26, 2024 09:26
@anmarchenko anmarchenko requested review from a team as code owners April 26, 2024 09:26
@anmarchenko anmarchenko merged commit 4534437 into main Apr 26, 2024
26 checks passed
@anmarchenko anmarchenko deleted the anmarchenko/itr_unskippable_tests branch April 26, 2024 13:12
@github-actions github-actions bot added this to the 1.0 milestone Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants