Merge pull request #20 from KnapsackPro/queue

Add support for knapsack_pro queue mode

README.md

@@ -33,14 +33,16 @@ Next time when you will run tests you will get proper test files for each CI nod
 
 ## Details
 
-For instance when you will run tests with rake knapsack_pro:rspec then:
+For instance when you will run tests with `rake knapsack_pro:rspec` then:
 
 * information about all your existing test files are sent to API http://docs.knapsackpro.com/api/v1/#build_distributions_subset_post
 * API returns which files should be executed on particular CI node (example KNAPSACK_PRO_CI_NODE_INDEX=0)
 * when API server has info about previous tests runs then it will use it to return more accurate test split results, in other case API returns simple split based on directory names
 * knapsack_pro will run test files which got from API
 * after tests finished knapsack_pro will send information about time execution of each file to API http://docs.knapsackpro.com/api/v1/#build_subsets_post so data can be used for future test runs
 
+The knapsack_pro has also [queue mode](#queue-mode) to get most optimal test suite split.
+
 # Requirements
 
 * >= Ruby 2.0.0
@@ -66,6 +68,11 @@ For instance when you will run tests with rake knapsack_pro:rspec then:
   - [Repository adapter (How to set up 3 of 3)](#repository-adapter-how-to-set-up-3-of-3)
     - [When you NOT set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER` (default)](#when-you-not-set-global-variable-knapsack_pro_repository_adapter-default)
     - [When you set global variable `KNAPSACK_PRO_REPOSITORY_ADAPTER=git` (required when CI provider is not supported)](#when-you-set-global-variable-knapsack_pro_repository_adaptergit-required-when-ci-provider-is-not-supported)
+- [Queue Mode](#queue-mode)
+  - [How queue mode works?](#how-queue-mode-works)
+  - [How to use queue mode?](#how-to-use-queue-mode)
+  - [Additional info about queue mode](#additional-info-about-queue-mode)
+  - [Supported test runners in queue mode](#supported-test-runners-in-queue-mode)
 - [Extra configuration for CI server](#extra-configuration-for-ci-server)
   - [Info about ENV variables](#info-about-env-variables)
     - [KNAPSACK_PRO_FIXED_TEST_SUITE_SPLITE (test suite split based on seed)](#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed)
@@ -279,6 +286,41 @@ You can also use git as repository adapter to determine branch and commit hash,
 
 `KNAPSACK_PRO_PROJECT_DIR` - Path to the project on CI node for instance `/home/ubuntu/my-app-repository`. It should be main directory of your repository.
 
+## Queue Mode
+
+knapsack_pro has built in queue mode designed to solve problem with optimal test suite split in case of random time execution of test files caused by
+CI node overload and a random decrease of performance that may affect how long the test files are executed.
+The problem with random time execution of test files may be caused by many things like external requests done in tests.
+
+### How queue mode works?
+
+On the Knapsack Pro API side, there is test files queue generated for your CI build. Each of CI node dynamically asks the Knapsack Pro API for test files
+that should be executed. Thanks to that each CI node will finish tests at the same time.
+
+### How to use queue mode?
+
+Please use this command to run queue mode:
+
+    bundle exec rake knapsack_pro:queue:rspec
+
+If above command fails then you may need to explicitly pass an argument to require `rails_helper` file or `spec_helper` in case you are not doing this in some of your test files:
+
+    bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]"
+
+Note if you will run queue mode command for the first time it might be slower.
+The second build should have better optimal test suite split.
+
+### Additional info about queue mode
+
+If you are not using one of supported CI providers then please note that knapsack_pro gem doesn't know what is CI build ID in order to generated queue for particular CI build. This may result in two different CI builds taking tests from the same queue when CI builds are running at the same time against the same git commit.
+To avoid this you can specify unique `KNAPSACK_PRO_CI_NODE_BUILD_ID` environment variable for each CI build. This mean that each CI node that is part of particular CI build should have the same value for `KNAPSACK_PRO_CI_NODE_BUILD_ID`.
+
+### Supported test runners in queue mode
+
+At this moment the queue mode works for:
+
+* RSpec
+
 ## Extra configuration for CI server
 
 ### Info about ENV variables
@@ -367,6 +409,7 @@ Add arguments to knapsack_pro spinach task like this:
 You can install knapsack_pro globally and use binary. For instance:
 
     $ knapsack_pro rspec "--tag custom_tag_name --profile"
+    $ knapsack_pro queue:rspec "--tag custom_tag_name --profile"
     $ knapsack_pro cucumber "--name feature"
     $ knapsack_pro minitest "--verbose --pride"
     $ knapsack_pro spinach "--arg_name value"

bin/knapsack_pro

@@ -7,6 +7,7 @@ arguments = ARGV[1]
 
 MAP = {
   'rspec' => KnapsackPro::Runners::RSpecRunner,
+  'queue:rspec' => KnapsackPro::Runners::Queue::RSpecRunner,
   'cucumber' => KnapsackPro::Runners::CucumberRunner,
   'minitest' => KnapsackPro::Runners::MinitestRunner,
   'spinach' => KnapsackPro::Runners::SpinachRunner,

lib/knapsack_pro.rb

@@ -6,6 +6,7 @@
 require 'rake/testtask'
 require 'timecop'
 require 'digest'
+require 'securerandom'
 require_relative 'knapsack_pro/version'
 require_relative 'knapsack_pro/utils'
 require_relative 'knapsack_pro/logger_wrapper'
@@ -16,10 +17,12 @@
 require_relative 'knapsack_pro/config/ci/travis'
 require_relative 'knapsack_pro/config/ci/snap_ci'
 require_relative 'knapsack_pro/config/env'
+require_relative 'knapsack_pro/config/env_generator'
 require_relative 'knapsack_pro/client/api/action'
 require_relative 'knapsack_pro/client/api/v1/base'
 require_relative 'knapsack_pro/client/api/v1/build_distributions'
 require_relative 'knapsack_pro/client/api/v1/build_subsets'
+require_relative 'knapsack_pro/client/api/v1/queues'
 require_relative 'knapsack_pro/client/connection'
 require_relative 'knapsack_pro/repository_adapters/base_adapter'
 require_relative 'knapsack_pro/repository_adapters/env_adapter'
@@ -35,7 +38,10 @@
 require_relative 'knapsack_pro/task_loader'
 require_relative 'knapsack_pro/tracker'
 require_relative 'knapsack_pro/allocator'
+require_relative 'knapsack_pro/queue_allocator'
+require_relative 'knapsack_pro/base_allocator_builder'
 require_relative 'knapsack_pro/allocator_builder'
+require_relative 'knapsack_pro/queue_allocator_builder'
 require_relative 'knapsack_pro/adapters/base_adapter'
 require_relative 'knapsack_pro/adapters/rspec_adapter'
 require_relative 'knapsack_pro/adapters/cucumber_adapter'
@@ -46,6 +52,8 @@
 require_relative 'knapsack_pro/runners/cucumber_runner'
 require_relative 'knapsack_pro/runners/minitest_runner'
 require_relative 'knapsack_pro/runners/spinach_runner'
+require_relative 'knapsack_pro/runners/queue/base_runner'
+require_relative 'knapsack_pro/runners/queue/rspec_runner'
 require_relative 'knapsack_pro/crypto/encryptor'
 require_relative 'knapsack_pro/crypto/decryptor'
 require_relative 'knapsack_pro/crypto/digestor'

lib/knapsack_pro/adapters/base_adapter.rb

@@ -16,6 +16,12 @@ def bind
           bind_time_tracker
           bind_save_report
         end
+
+        if KnapsackPro::Config::Env.queue_recording_enabled?
+          KnapsackPro.logger.info('Test suite time execution queue recording enabled.')
+          bind_time_tracker
+          bind_save_queue_report
+        end
       end
 
       def bind_time_tracker
@@ -25,6 +31,10 @@ def bind_time_tracker
       def bind_save_report
         raise NotImplementedError
       end
+
+      def bind_save_queue_report
+        raise NotImplementedError
+      end
     end
   end
 end

lib/knapsack_pro/adapters/rspec_adapter.rb

@@ -49,6 +49,14 @@ def bind_save_report
           end
         end
       end
+
+      def bind_save_queue_report
+        ::RSpec.configure do |config|
+          config.after(:suite) do
+            KnapsackPro::Report.save_subset_queue_to_file
+          end
+        end
+      end
     end
 
     # This is added to provide backwards compatibility

lib/knapsack_pro/allocator_builder.rb

@@ -1,9 +1,5 @@
 module KnapsackPro
-  class AllocatorBuilder
-    def initialize(adapter_class)
-      @adapter_class = adapter_class
-    end
-
+  class AllocatorBuilder < BaseAllocatorBuilder
     def allocator
       KnapsackPro::Allocator.new(
         test_files: test_files,
@@ -12,29 +8,5 @@ def allocator
         repository_adapter: repository_adapter,
       )
     end
-
-    def test_dir
-      test_file_pattern.split('/').first
-    end
-
-    private
-
-    attr_reader :adapter_class
-
-    def env
-      KnapsackPro::Config::Env
-    end
-
-    def repository_adapter
-      KnapsackPro::RepositoryAdapterInitiator.call
-    end
-
-    def test_file_pattern
-      TestFilePattern.call(adapter_class)
-    end
-
-    def test_files
-      KnapsackPro::TestFileFinder.call(test_file_pattern)
-    end
   end
 end

lib/knapsack_pro/base_allocator_builder.rb

@@ -0,0 +1,35 @@
+module KnapsackPro
+  class BaseAllocatorBuilder
+    def initialize(adapter_class)
+      @adapter_class = adapter_class
+    end
+
+    def allocator
+      raise NotImplementedError
+    end
+
+    def test_dir
+      test_file_pattern.split('/').first
+    end
+
+    private
+
+    attr_reader :adapter_class
+
+    def env
+      KnapsackPro::Config::Env
+    end
+
+    def repository_adapter
+      KnapsackPro::RepositoryAdapterInitiator.call
+    end
+
+    def test_file_pattern
+      TestFilePattern.call(adapter_class)
+    end
+
+    def test_files
+      KnapsackPro::TestFileFinder.call(test_file_pattern)
+    end
+  end
+end

lib/knapsack_pro/client/api/v1/queues.rb

@@ -0,0 +1,27 @@
+module KnapsackPro
+  module Client
+    module API
+      module V1
+        class Queues < Base
+          class << self
+            def queue(args)
+              action_class.new(
+                endpoint_path: '/v1/queues/queue',
+                http_method: :post,
+                request_hash: {
+                  :can_initialize_queue => args.fetch(:can_initialize_queue),
+                  :commit_hash => args.fetch(:commit_hash),
+                  :branch => args.fetch(:branch),
+                  :node_total => args.fetch(:node_total),
+                  :node_index => args.fetch(:node_index),
+                  :node_build_id => KnapsackPro::Config::Env.ci_node_build_id,
+                  :test_files => args.fetch(:test_files)
+                }
+              )
+            end
+          end
+        end
+      end
+    end
+  end
+end

lib/knapsack_pro/config/ci/base.rb

@@ -8,6 +8,9 @@ def node_total
         def node_index
         end
 
+        def node_build_id
+        end
+
         def commit_hash
         end
 

lib/knapsack_pro/config/ci/buildkite.rb

@@ -10,6 +10,10 @@ def node_index
           ENV['BUILDKITE_PARALLEL_JOB']
         end
 
+        def node_build_id
+          ENV['BUILDKITE_BUILD_NUMBER']
+        end
+
         def commit_hash
           ENV['BUILDKITE_COMMIT']
         end

lib/knapsack_pro/config/ci/circle.rb

@@ -10,6 +10,10 @@ def node_index
           ENV['CIRCLE_NODE_INDEX']
         end
 
+        def node_build_id
+          ENV['CIRCLE_BUILD_NUM']
+        end
+
         def commit_hash
           ENV['CIRCLE_SHA1']
         end
@@ -26,4 +30,3 @@ def project_dir
     end
   end
 end
-

lib/knapsack_pro/config/ci/semaphore.rb

@@ -11,6 +11,10 @@ def node_index
           index.to_i - 1 if index
         end
 
+        def node_build_id
+          ENV['SEMAPHORE_BUILD_NUMBER']
+        end
+
         def commit_hash
           ENV['REVISION']
         end

lib/knapsack_pro/config/ci/snap_ci.rb

@@ -11,6 +11,10 @@ def node_index
           index.to_i - 1 if index
         end
 
+        def node_build_id
+          ENV['SNAP_PIPELINE_COUNTER']
+        end
+
         def commit_hash
           ENV['SNAP_COMMIT']
         end

lib/knapsack_pro/config/ci/travis.rb

@@ -2,6 +2,10 @@ module KnapsackPro
   module Config
     module CI
       class Travis < Base
+        def node_build_id
+          ENV['TRAVIS_BUILD_NUMBER']
+        end
+
         def commit_hash
           ENV['TRAVIS_COMMIT']
         end

lib/knapsack_pro/config/env.rb

@@ -14,6 +14,12 @@ def ci_node_index
             0).to_i
         end
 
+        def ci_node_build_id
+          ENV['KNAPSACK_PRO_CI_NODE_BUILD_ID'] ||
+            ci_env_for(:node_build_id) ||
+            'missing-build-id'
+        end
+
         def commit_hash
           ENV['KNAPSACK_PRO_COMMIT_HASH'] ||
             ci_env_for(:commit_hash)
@@ -45,6 +51,22 @@ def recording_enabled?
           recording_enabled == 'true'
         end
 
+        def queue_recording_enabled
+          ENV['KNAPSACK_PRO_QUEUE_RECORDING_ENABLED']
+        end
+
+        def queue_recording_enabled?
+          queue_recording_enabled == 'true'
+        end
+
+        def queue_id
+          ENV['KNAPSACK_PRO_QUEUE_ID'] || raise('Missing Queue ID')
+        end
+
+        def subset_queue_id
+          ENV['KNAPSACK_PRO_SUBSET_QUEUE_ID'] || raise('Missing Subset Queue ID')
+        end
+
         def test_files_encrypted
           ENV['KNAPSACK_PRO_TEST_FILES_ENCRYPTED']
         end

lib/knapsack_pro/config/env_generator.rb

@@ -0,0 +1,19 @@
+module KnapsackPro
+  module Config
+    class EnvGenerator
+      class << self
+        def set_queue_id
+          if ENV['KNAPSACK_PRO_QUEUE_ID']
+            raise 'Queue ID already generated.'
+          else
+            ENV['KNAPSACK_PRO_QUEUE_ID'] = "#{Time.now.to_i}_#{SecureRandom.uuid}"
+          end
+        end
+
+        def set_subset_queue_id
+          ENV['KNAPSACK_PRO_SUBSET_QUEUE_ID'] = SecureRandom.uuid
+        end
+      end
+    end
+  end
+end