run_all_when_everything_filtered with filter_run_excluding doesn't run test file

[alexgenco]

If I configure RSpec with both of these options e.g.

RSpec.configure do |config|
  config.filter_run_excluding :integration, :other_tag
  config.run_all_when_everything_filtered = true
end

And then I have a spec file with :integration metadata:

RSpec.describe "integration tests", :integration do
  # ...
end

When I run the file using rspec path/to/file_spec.rb, it outputs the following:

Run options: exclude {:other_tag=>true, :integration=>true}

All examples were filtered out; ignoring {}

All examples were filtered out

Randomized with seed 8717


Finished in 0.00027 seconds (files took 0.63033 seconds to load)
0 examples, 0 failures

The ignoring {} is suspect to me, as I would expect it to say ignoring {:other_tag=>true, :integration=>true}, and then run the entire file.

We are using rspec-core 3.6.0, and I see that this use case is deprecated, however I can't figure out how to make this scenario work with filter_run_when_matching. We essentially want to ignore these tests when running the full test suite, but support running the file alone without specifying the filters explicitly with -t integration.

[myronmarston]

run_all_when_everything_filtered was only designed to ignore inclusion filters (such as :focus), and doesn't do anything with exclusion filters, which is why it's not working and is producing the confusing output. We consider it effectively deprecated (although it doesn't warn yet) as filter_run_when_matching is superior in every way. I doubt any of the RSpec maintainers will make time to improve it given it's deprecated status and that you're the first to report this problem, but if you wanted to submit a PR to improve it, feel free!

In the meantime, there's a better/simpler way to do this using a combination of filter_run_when_matching and define_derived_metadata:

RSpec.configure do |config|
  config.define_derived_metadata do |meta|
    meta[:integration] = false unless meta.key?(:integration)
  end

  config.filter_run_when_matching integration: false
end

Here's what this does:

• The define_derived_metadata block automatically tags every example with integration: false, but allows individual groups or example to explicitly override that with :integration or integration: true.
• config.filter_run_when_matching integration: false then excludes tests tagged with integration: true as long as there are some tests tagged with integration: false. If you run only a file with specs tagged with integration: true, it'll run them.

I think that'll do what you want.

[alexgenco]

Thanks for the explanation. Your suggestion works for a single tag, but I think when you pass multiple tags to filter_run_when_matching it runs everything again.

I made a gist to reproduce: https://gist.github.com/alexgenco/494cbbfb1fcea4154150a5b8b325c9aa

If you just run bundle exec rspec it runs the whole suite, even though it says Run options: include {:integration=>false, :other_tag=>false}

[myronmarston]

It's not clear to me what behavior you're trying to achieve (e.g. are you trying to AND the tags together or OR them?) but regardless, you can achieve it by doing a bit more in your define_derived_metadata block:

RSpec.configure do |config|
  RSpec.configure do |config|
    config.define_derived_metadata do |meta|
      meta[:integration] = false unless meta.key?(:integration)
      meta[:other_tag] = false unless meta.key?(:other_tag)
      meta[:integration_other_tag] = meta[:integration] && meta[:other_tag]
      # or meta[:integration_other_tag] = meta[:integration] || meta[:other_tag]
    end

    config.filter_run_when_matching integration_other_tag: false
  end
end

[alexgenco]

Thanks, this works!

To be clear, this was the final fix:

config.define_derived_metadata do |meta|
  meta[:filtered] = FILTERED_TAGS.any? { |tag| meta[tag] }
end

config.filter_run_when_matching :filtered => false

where FILTERED_TAGS is a list of tags we don't want to run by default.