Do not allow to use the RSpec tag option together with the RSpec split by test examples feature

[ArturT]

Do you pass RSpec option --tag in command like bundle exec rake "knapsack_pro:queue:rspec[--tag mytag]"?
If you do it and you use at the same time KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true then --tag option might not be respected by RSpec due to a bug in the RSpec. RSpec has a higher priority to run test examples like spec/a_spec.rb[1:1] than respect the --tag option. This can result in running test cases that you didn't intend to run.

You should not use the --tag option when you use RSpec split by test examples feature KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true.

See more:
https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it#warning-dont-use-rspec-tag-option

[shadre]

I suggest we improve the error messages gramatically

[ArturT]

@shadre Thanks for the review. I've applied suggestions. Please let me know if we are good to merge this PR.

[shadre]

looks great to me 👍

[ArturT]

tech note

About the following issue:

RSpec has a higher priority to run test examples like spec/a_spec.rb[1:1] than respect the --tag option. This can result in running test cases that you didn't intend to run.

To reproduce the issue make changes:

In the knapsack_pro gem:

# lib/knapsack_pro/queue_allocator.rb
    def test_file_paths(can_initialize_queue, executed_test_files)
      @@index ||= 0
      batches = [
        ['spec/features/calculator_spec.rb[1:2:1]', 'spec/controllers/articles_controller_spec.rb'],
        ['spec/collection_spec.rb'],
      ]
      tests = batches[@@index] || []
      puts 'INSPECT:'
      puts tests.inspect
      @@index += 1
      return tests
      
      # the rest of the code that is ignored thanks to the above `return tests`
      return [] if @fallback_activated
      # ...

In the rails-app-with-knapsack_pro repo:

Run the bin script: bin/knapsack_pro_queue_rspec_record_first_run with modified content as following to run tagged tests.

# bin/knapsack_pro_queue_rspec_record_first_run

RANDOM_CI_BUILD_ID=$(openssl rand -base64 32)
CI_BUILD_ID=${4:-$RANDOM_CI_BUILD_ID}
COMMIT_HASH=$(ruby -e "require 'securerandom'; puts SecureRandom.hex")
BRANCH_NAME=fake-branch

# you can set a const fake data if you need to test running tests on 2 CI nodes for the same commit/branch
#COMMIT_HASH=57dccc53cb2ee921692ad1d3df971674

#export KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true

#export EXTRA_TEST_FILES_DELAY=10 # seconds
# uncomment the following line to verify if Knapsack Pro raises an exception preventing the RSpec --tag usage
#export KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true

KNAPSACK_PRO_ENDPOINT=http://api.knapsackpro.test:3000 \
  KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=fec3c641a3c4d2e720fe1b6d9dd780bc \
  KNAPSACK_PRO_CI_NODE_BUILD_ID=$CI_BUILD_ID \
  KNAPSACK_PRO_COMMIT_HASH=${3:-$COMMIT_HASH} \
  KNAPSACK_PRO_BRANCH=$BRANCH_NAME \
  KNAPSACK_PRO_CI_NODE_TOTAL=${2:-2} \
  KNAPSACK_PRO_CI_NODE_INDEX=${1:-0} \
  bundle exec rake "knapsack_pro:queue:rspec[--format d --tag xfocus]"

Expected:
I expect the spec/controllers/articles_controller_spec.rb file to be executed because it has xfocus tag. Other test files should not be executed.

Actual:
The spec/features/calculator_spec.rb[1:2:1] test case is executed. This is the issue. It should not be executed.