Question: What's the best way to run groups of tests in the same process?

[floehopper]

This is a question rather than a problem per se.

I'd like to be able to run all my unit tests first (albeit in a random order) before any of my integration tests and I want all the tests to run in the same process.

I've come up with a solution of sorts in this repo, but I'm not very happy with the code. Can anyone suggest a better solution?

Thanks.

[zenspider]

What part of it makes you unhappy and why?

[floehopper]

Good question. I'm not sure any of them are deal-breakers, but I just thought there might be a more "official" way.

One small thing is that I seem to need to dup ARGV, because Minitest.run` seems to modify the args object passed in. That doesn't seem ideal.

Another thing is having to call Minitest::Runnable.reset. Is that part of Minitest's published API? i.e. is it safe for me to rely on it?

One last thing which isn't an issue in my example project is when doing this kind of thing in a project which makes use of Minitest.autorun, you have to be careful to reset the runnables in the right places to avoid tests running twice.

[zenspider]

Some simple questions...

If a "fast" test fails, what should happen? Should it stop? Should it continue on to the slow tests?

[floehopper]

If any "fast" test fails then none of the "slow" tests should be run.

[zenspider]

OK. So I'm working on your problem and I think I've addressed (most of?) the undesirable side-effects you mentioned. Specifically, there needs to be a hack in place to bypass minitest/autorun. Here's the output of my runs:

10006 % BAD=1 ruby -Ilib phases.rb; echo $?
Run options: --seed 38304

# Running:

.F...................................................................................................

Finished in 0.004959s, 20367.0095 runs/s, 20367.0095 assertions/s.

  1) Failure:
42#test_0101_should fail [/Users/ryan/Work/p4/zss/src/minitest/dev/fast.rb:11]:
Failed assertion, no message given.

101 runs, 101 assertions, 1 failures, 0 errors, 0 skips
1
10007 % ruby -Ilib phases.rb; echo $?
Run options: --seed 16157

# Running:

....................................................................................................

Finished in 0.003672s, 27233.1155 runs/s, 27233.1155 assertions/s.

100 runs, 100 assertions, 0 failures, 0 errors, 0 skips
Run options: --seed 54829

# Running:

....................................................................................................

Finished in 1.203475s, 83.0927 runs/s, 83.0927 assertions/s.

100 runs, 100 assertions, 0 failures, 0 errors, 0 skips
0

As you can see, when BAD=1, one of the fast tests will fail and it stops w/ exit 1 and in the second run BAD is not set so all tests run and it exits clean.

The code isn't much different than yours:

require "minitest"

module Minitest
  @@installed_at_exit = true
end

def minitest_phase
  yield

  if Minitest.run ARGV.dup then
    Minitest::Runnable.reset
  else
    exit 1
  end
end

minitest_phase do
  require "./fast.rb"
end

minitest_phase do
  require "./slow.rb"
end

As far as "official" goes? There is no "official" way to do what you want because there is no usecase for it. I would say that your code is the right way to do it tho.

[floehopper]

Thanks for looking into it. FWIW I think I do have a valid use case.

Historically projects (including Rails apps) used to run groups of their tests in a series of Rake::TestTasks, e.g. test:units, test:functionals & test:integration. Because of the way Rake::TestTask works, each of these groups of tests ran in its own process and if any failed, the subsequent groups wouldn't be run.

However, since this commit all Rails tests are run in a single group and in a single process. While this saves laoding the Rails environment multiple times, it also means that typically some/many integration/functional tests run before unit tests.

Personally I think this is undesirable, because unit tests are intended to give the fastest and most specific feedback and consequently they should run first. So I was trying to work out whether I could find a way to retain the idea of running all unit tests first, while not re-introducing the performance overhead of multiple processes.

However, if there's no official way of doing this with Minitest, I guess I'm not going to get a PR accepted by Rails.

Closing.