Allow manual merging

CHANGELOG.md

@@ -1,3 +1,10 @@
+Master (unreleased)
+===================
+
+## Enhancements
+* Allow early running exit tasks and avoid the `at_exit` hook through the `SimpleCov.run_exit_tasks!` method. (thanks [@macumber]: https://github.com/macumber))
+* Allow manual collation of result sets through the `SimpleCov.collate` entrypoint. See the README for more details (thanks [@ticky](https://github.com/ticky))
+
 0.18.0.beta2 (2020-01-05)
 ===================
 

README.md

@@ -472,12 +472,11 @@ end
 
 You normally want to have your coverage analyzed across ALL of your test suites, right?
 
-Simplecov automatically caches coverage results in your (coverage_path)/.resultset.json. Those results will then
-be automatically merged when generating the result, so when coverage is set up properly for Cucumber and your
-unit / functional / integration tests, all of those test suites will be taken into account when building the
-coverage report.
-
-There are two things to note here though:
+Simplecov automatically caches coverage results in your
+(coverage_path)/.resultset.json, and will merge or override those with
+subsequent runs, depending on whether simplecov considers those subsequent runs
+as different test suites or as the same test suite as the cached results. To
+make this distinction, simplecov has the concept of "test suite names".
 
 ### Test suite names
 
@@ -531,14 +530,84 @@ SimpleCov.command_name "features" + (ENV['TEST_ENV_NUMBER'] || '')
 
 [simplecov-html] prints the used test suites in the footer of the generated coverage report.
 
-### Timeout for merge
 
-Of course, your cached coverage data is likely to become invalid at some point. Thus, result sets that are older than
-`SimpleCov.merge_timeout` will not be used any more. By default, the timeout is 600 seconds (10 minutes), and you can
-raise (or lower) it by specifying `SimpleCov.merge_timeout 3600` (1 hour), or, inside a configure/start block, with
-just `merge_timeout 3600`.
+### Merging test runs under the same execution environment
+
+Test results are automatically merged with previous runs in the same execution
+environment when generating the result, so when coverage is set up properly for
+Cucumber and your unit / functional / integration tests, all of those test
+suites will be taken into account when building the coverage report.
+
+#### Timeout for merge
+
+Of course, your cached coverage data is likely to become invalid at some point. Thus, when automatically merging
+subsequent test runs, result sets that are older than `SimpleCov.merge_timeout` will not be used any more. By default,
+the timeout is 600 seconds (10 minutes), and you can raise (or lower) it by specifying `SimpleCov.merge_timeout 3600`
+(1 hour), or, inside a configure/start block, with just `merge_timeout 3600`.
+
+You can deactivate this automatic merging altogether with `SimpleCov.use_merging false`.
+
+### Merging test runs under different execution environments
+
+If your tests are done in parallel across multiple build machines, you can fetch them all and merge them into a single
+result set using the `SimpleCov.collate` method. This can be added to a Rakefile or script file, having downloaded a set of
+`.resultset.json` files from each parallel test run.
+
+```ruby
+# lib/tasks/coverage_report.rake
+namespace :coverage do
+  desc "Collates all result sets generated by the different test runners"
+  task :report do
+    require 'simplecov'
+
+    SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"]
+  end
+end
+```
+
+`SimpleCov.collate` also takes an optional simplecov profile and an optional
+block for configuration, just the same as `SimpleCov.start` or
+`SimpleCov.configure`.  This means you can configure a separate formatter for
+the collated output. For instance, you can make the formatter in
+`SimpleCov.start` the `SimpleCov::Formatter::SimpleFormatter`, and only use more
+complex formatters in the final `SimpleCov.collate` run.
+
+```ruby
+# spec/spec_helper.rb
+require 'simplecov'
+
+SimpleCov.start 'rails' do
+  # Disambiguates individual test runs
+  command_name "Job #{ENV["TEST_ENV_NUMBER"]}" if ENV["TEST_ENV_NUMBER"]
+
+  if ENV['CI']
+    formatter SimpleCov::Formatter::SimpleFormatter
+  else
+    formatter SimpleCov::Formatter::MultiFormatter.new([
+      SimpleCov::Formatter::SimpleFormatter,
+      SimpleCov::Formatter::HTMLFormatter
+    ])
+  end
+
+  track_files "**/*.rb"
+end
+```
+
+```ruby
+# lib/tasks/coverage_report.rake
+namespace :coverage do
+  task :report do
+    require 'simplecov'
 
-You can deactivate merging altogether with `SimpleCov.use_merging false`.
+    SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' do
+      formatter SimpleCov::Formatter::MultiFormatter.new([
+        SimpleCov::Formatter::SimpleFormatter,
+        SimpleCov::Formatter::HTMLFormatter
+      ])
+    end
+  end
+end
+```
 
 ## Running coverage only on demand
 

features/test_unit_collate.feature

@@ -0,0 +1,43 @@
+@test_unit
+Feature:
+
+  Using SimpleCov.collate should get the user a coverage report
+
+  Scenario:
+    Given SimpleCov for Test/Unit is configured with:
+      """
+      require 'simplecov'
+      SimpleCov.start
+      """
+
+    When I successfully run `bundle exec rake part1`
+    Then a coverage report should have been generated
+    When I successfully run `mv coverage/.resultset.json coverage/resultset1.json`
+    And I successfully run `rm coverage/index.html`
+
+    When I successfully run `bundle exec rake part2`
+    Then a coverage report should have been generated
+    When I successfully run `mv coverage/.resultset.json coverage/resultset2.json`
+    And I successfully run `rm coverage/index.html`
+
+    When I open the coverage report generated with `bundle exec rake collate`
+    Then I should see the groups:
+      | name      | coverage | files |
+      | All Files | 91.38%   | 6     |
+
+    And I should see the source files:
+      | name                                    | coverage |
+      | lib/faked_project.rb                    | 100.0 %  |
+      | lib/faked_project/some_class.rb         | 80.0 %   |
+      | lib/faked_project/framework_specific.rb | 75.0 %   |
+      | lib/faked_project/meta_magic.rb         | 100.0 %  |
+      | test/meta_magic_test.rb                 | 100.0 %  |
+      | test/some_class_test.rb                 | 100.0 %  |
+
+      # Note: faked_test.rb is not appearing here since that's the first unit test file
+      # loaded by Rake, and only there test_helper is required, which then loads simplecov
+      # and triggers tracking of all other loaded files! Solution for this would be to
+      # configure simplecov in this first test instead of test_helper.
+
+    And the report should be based upon:
+      | Unit Tests |

lib/simplecov.rb

@@ -46,15 +46,50 @@ class << self
     #
     def start(profile = nil, &block)
       require "coverage"
-      load_profile(profile) if profile
-      configure(&block) if block_given?
+      initial_setup(profile, &block)
       @result = nil
-      self.running = true
       self.pid = Process.pid
 
       start_coverage_measurement
     end
 
+    #
+    # Collate a series of SimpleCov result files into a single SimpleCov output.
+    # You can optionally specify configuration with a block:
+    #   SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"]
+    #    OR
+    #   SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' # using rails profile
+    #    OR
+    #   SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"] do
+    #     add_filter 'test'
+    #   end
+    #    OR
+    #   SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' do
+    #     add_filter 'test'
+    #   end
+    #
+    # Please check out the RDoc for SimpleCov::Configuration to find about
+    # available config options, or checkout the README for more in-depth
+    # information about coverage collation
+    #
+    def collate(result_filenames, profile = nil, &block)
+      raise "There's no reports to be merged" if result_filenames.empty?
+
+      initial_setup(profile, &block)
+
+      results = result_filenames.flat_map do |filename|
+        # Re-create each included instance of SimpleCov::Result from the stored run data.
+        (JSON.parse(File.read(filename), symbolize_names: true) || {}).map do |command_name, coverage|
+          SimpleCov::Result.from_hash(command_name => coverage)
+        end
+      end
+
+      # Use the ResultMerger to produce a single, merged result, ready to use.
+      @result = SimpleCov::ResultMerger.merge_and_store(*results)
+
+      run_exit_tasks!
+    end
+
     #
     # Returns the result for the current coverage run, merging it across test suites
     # from cache using SimpleCov::ResultMerger if use_merging is activated (default)
@@ -159,6 +194,8 @@ def exit_status_from_exception
     # Called from at_exit block
     #
     def run_exit_tasks!
+      set_exit_exception
+
       exit_status = SimpleCov.exit_status_from_exception
 
       SimpleCov.at_exit.call
@@ -210,7 +247,7 @@ def result_exit_status(result, covered_percent)
         )
         SimpleCov::ExitCodes::MINIMUM_COVERAGE
       elsif (last_run = SimpleCov::LastRun.read)
-        coverage_diff = last_run["result"]["covered_percent"] - covered_percent
+        coverage_diff = last_run[:result][:covered_percent] - covered_percent
         if coverage_diff > SimpleCov.maximum_coverage_drop
           $stderr.printf(
             "Coverage has dropped by %<drop_percent>.2f%% since the last time (maximum allowed: %<max_drop>.2f%%).\n",
@@ -253,6 +290,12 @@ def write_last_run(covered_percent)
 
   private
 
+    def initial_setup(profile, &block)
+      load_profile(profile) if profile
+      configure(&block) if block_given?
+      self.running = true
+    end
+
     #
     # Trigger Coverage.start depends on given config coverage_criterion
     #
@@ -319,7 +362,7 @@ def add_not_loaded_files(result)
         result = result.dup
         Dir[tracked_files].each do |file|
           absolute_path = File.expand_path(file)
-          result[absolute_path] ||= SimulateCoverage.call(absolute_path)
+          result[absolute_path.to_sym] ||= SimulateCoverage.call(absolute_path)
         end
       end
 

lib/simplecov/combine/files_combiner.rb

@@ -15,10 +15,9 @@ module FilesCombiner
       # @return [Hash]
       #
       def combine(coverage_a, coverage_b)
-        {
-          lines: Combine.combine(LinesCombiner, coverage_a[:lines], coverage_b[:lines]),
-          branches: Combine.combine(BranchesCombiner, coverage_a[:branches], coverage_b[:branches])
-        }
+        combination = {lines: Combine.combine(LinesCombiner, coverage_a[:lines], coverage_b[:lines])}
+        combination[:branches] = Combine.combine(BranchesCombiner, coverage_a[:branches], coverage_b[:branches]) if SimpleCov.branch_coverage?
+        combination
       end
     end
   end

lib/simplecov/configuration.rb

@@ -313,7 +313,7 @@ def add_group(group_name, filter_argument = nil, &filter_proc)
     # * :line - coverage based on lines aka has this line been executed?
     # * :branch - coverage based on branches aka has this branch (think conditions) been executed?
     #
-    # If not set the default is is `:line`
+    # If not set the default is `:line`
     #
     # @param [Symbol] criterion
     #

lib/simplecov/defaults.rb

@@ -25,7 +25,9 @@
   # If we are in a different process than called start, don't interfere.
   next if SimpleCov.pid != Process.pid
 
-  SimpleCov.set_exit_exception
+  # If SimpleCov is no longer running then don't run exit tasks
+  next unless SimpleCov.running
+
   SimpleCov.run_exit_tasks!
 end
 

lib/simplecov/last_run.rb

@@ -15,7 +15,7 @@ def read
         json = File.read(last_run_path)
         return nil if json.strip.empty?
 
-        JSON.parse(json)
+        JSON.parse(json, symbolize_names: true)
       end
 
       def write(json)

lib/simplecov/result.rb

@@ -61,39 +61,26 @@ def command_name
 
     # Returns a hash representation of this Result that can be used for marshalling it into JSON
     def to_hash
-      {command_name => {"coverage" => coverage, "timestamp" => created_at.to_i}}
+      {command_name.to_sym => {coverage: coverage, timestamp: created_at.to_i}}
     end
 
     # Loads a SimpleCov::Result#to_hash dump
     def self.from_hash(hash)
       command_name, data = hash.first
 
       result = SimpleCov::Result.new(
-        symbolize_names_of_coverage_results(data["coverage"])
+        data[:coverage]
       )
 
-      result.command_name = command_name
-      result.created_at = Time.at(data["timestamp"])
+      result.command_name = command_name.to_s
+      result.created_at = Time.at(data[:timestamp])
       result
     end
 
-    # Manage symbolize the keys of coverage hash.
-    # JSON.parse gives coverage hash with stringified keys what breaks some logics
-    # inside the process that expects them as symboles.
-    #
-    # @return [Hash]
-    def self.symbolize_names_of_coverage_results(coverage_data)
-      coverage_data.each_with_object({}) do |(file_name, file_coverage_result), coverage_results|
-        coverage_results[file_name] = file_coverage_result.each_with_object({}) do |(k, v), cov_elem|
-          cov_elem[k.to_sym] = v
-        end
-      end
-    end
-
   private
 
     def coverage
-      keys = original_result.keys & filenames
+      keys = original_result.keys & filenames.map(&:to_sym)
       Hash[keys.zip(original_result.values_at(*keys))]
     end
 

lib/simplecov/result_adapter.rb

@@ -20,9 +20,9 @@ def adapt
 
       result.each_with_object({}) do |(file_name, cover_statistic), adapted_result|
         if cover_statistic.is_a?(Array)
-          adapted_result.merge!(file_name => {lines: cover_statistic})
+          adapted_result.merge!(file_name.to_sym => {lines: cover_statistic})
         else
-          adapted_result.merge!(file_name => cover_statistic)
+          adapted_result.merge!(file_name.to_sym => cover_statistic)
         end
       end
     end

lib/simplecov/result_merger.rb

@@ -26,7 +26,7 @@ def resultset
           data = stored_data
           if data
             begin
-              JSON.parse(data) || {}
+              JSON.parse(data, symbolize_names: true) || {}
             rescue StandardError
               {}
             end
@@ -62,6 +62,12 @@ def results
         results
       end
 
+      def merge_and_store(*results)
+        result = merge_results(*results)
+        store_result(result) if result
+        result
+      end
+
       # Merge two or more SimpleCov::Results into a new one with merged
       # coverage data and the command_name for the result consisting of a join
       # on all source result's names

spec/combine/results_combiner_spec.rb

@@ -4,7 +4,13 @@
 
 describe SimpleCov::Combine::ResultsCombiner do
   describe "with two faked coverage resultsets" do
+    after do
+      SimpleCov.clear_coverage_criteria
+    end
+
     before do
+      SimpleCov.enable_coverage :branch
+
       @resultset1 = {
         source_fixture("sample.rb") => {
           lines: [nil, 1, 1, 1, nil, nil, 1, 1, nil, nil],