knapsack_pro Queue Mode for minitest

[ArturT]

Basic info

Queue Mode allocates tests dynamically across CI nodes. Tests are fetched from Knapsack Pro API work queue. Thanks to that the test suite split across CI nodes is optimal and you save time execution of your CI build. We would like to add Queue Mode for minitest in knapsack_pro gem.

How Queue Mode works

If you would like to see how Queue Mode works for RSpec see video at https://knapsackpro.com

Example of Queue Mode for RSpec

In case of RSpec we use RSpec::Core::Runner.run that allows providing new test files that will be executed without reloading the whole app environment.
https://relishapp.com/rspec-staging/rspec-core/docs/running-specs-multiple-times-with-different-runner-options-in-the-same-process

Here is an example how Queue Mode is done in knapsack_pro:
https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/runners/queue/rspec_runner.rb

Example of Regular Mode for minitest

Here is example how minitest works in Regular Mode (deterministic split):
https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/runners/minitest_runner.rb

Ideas how to add Queue Mode to minitest

We could do minitest Queue Mode in the simplest way by defining a new rake task for minitest after each test files batch fetched from work queue but this will be slow due to boot time for each rake task run.

The perfect solution would be to provide just a new list of test files to minitest and run them without the need to boot test runner again. As it is done this way in RSpec.

Minitest tips

The overall minitest structure of a run looks like this:
https://github.com/seattlerb/minitest/blob/master/lib/minitest.rb#L108-L118

How to reset state minitest/minitest#549

Possible issues to solve

• can other gems related to minitest affect the queue mode?
• which version of minitest will support queue mode?
• if minitest supports the running new set of tests without reloading the whole environment, will this have any downsides? Like problems with not cleaned some configs from the previous test files batch run? Similar problem happened for RSpec https://github.com/KnapsackPro/knapsack_pro-ruby#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail (there is link to issue in RSpec project)
• will reports collected by minitest work in queue mode?

[ArturT]

Hey @zenspider maybe you will have an idea how to run tests in minitest and then provide another set of test files to minitest to run it without reloading the whole environment?

I'm not sure what would be the good path. Maybe define my own runner as in this example minitest/minitest#192 (comment) or monkey patch minitest/minitest#688 (comment)
Maybe there is even simpler option.

Thanks!

[zenspider]

On Nov 16, 2017, at 03:58, Artur Trzop ***@***.***> wrote: Hey @zenspider maybe you will have an idea how to run tests in minitest and then provide another set of test files to minitest to run it without reloading the whole environment? I'm not sure what would be the good path. Maybe define my own runner as in this example minitest/minitest#192 (comment) or monkey patch minitest/minitest#688 (comment) Maybe there is even simpler option.

I have definite ideas about how to do this (and more) (and I want to) but not enough free time. Emphasis on free… If you want to talk contracting, I can make more time.

[mrsimo]

@ArturT perhaps there's something in https://github.com/tmm1/test-queue that might be useful

[ArturT]

Thanks for the info. I will try to play more with the code once I will find some time.

[ArturT]

I've just released a new version of knapsack_pro gem with support for Queue Mode in Minitest.

Now minitest tests can be allocated across CI nodes in dynamic way to get optimal CI build time.

To run Queue Mode for Minitest:

export RAILS_ENV=test # this is important!
bundle exec rake knapsack_pro:queue:minitest

Here is more detailed instruction:
http://docs.knapsackpro.com/knapsack_pro-ruby/guide/