Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for --export-skipped-tests-file #302

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

andheiberg
Copy link

@andheiberg andheiberg commented Jan 16, 2025

I've observed failing tests being skipped in a codebase even though the test file wasn't employing the clear and self documenting "skip" method to achieve this. At scale this makes it hard to reason about which tests are actually run and which are skipped.

I wish to create a central file test_todo.yml containing all tests that are allowed to be skipped like so:

skipped:
  - test/unit/a_test.rb

Then I can write a CI job that complains if tests not explicitly marked as skipped centrally is skipped.

SFR has ended up with tests being skipped that shouldn't be skipped. I want to add a CI check confirming that no test wasn't skipped that wasn't explicitly in a test_todo.yml file. To acheive this I need the ci-queue to report which tests were skipped.
@andheiberg andheiberg self-assigned this Jan 16, 2025
@ChrisBr
Copy link
Contributor

ChrisBr commented Jan 16, 2025

I would like to avoid storing more data into Redis and adding more options / complexity unless there is a very good reason.

For skipped tests I'm not sure we need to store it in a distributed way. Maybe we could just do it in a reporter.

class SkipAllowListReporter < Minitest::Reporters::BaseReporter
  def initialize(options = {})
    super
    @allow_list = Set.new(File.read_lines(options.delete(:allow_list)))
  end


  def record(test)
    super
    raise "boom" if test.skipped? && !allow_list.include?(test.name)
  end
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants