diff --git a/CHANGELOG.md b/CHANGELOG.md index 69017557..5392a647 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +Master (unreleased) +=================== + +## Enhancements +* Allow early running exit tasks and avoid the `at_exit` hook through the `SimpleCov.run_exit_tasks!` method. (thanks [@macumber]: https://github.com/macumber)) +* Allow manual collation of result sets through the `SimpleCov.collate` entrypoint. See the README for more details (thanks [@ticky](https://github.com/ticky)) + 0.18.0.beta2 (2020-01-05) =================== diff --git a/README.md b/README.md index c422010a..3d982dc6 100644 --- a/README.md +++ b/README.md @@ -472,12 +472,11 @@ end You normally want to have your coverage analyzed across ALL of your test suites, right? -Simplecov automatically caches coverage results in your (coverage_path)/.resultset.json. Those results will then -be automatically merged when generating the result, so when coverage is set up properly for Cucumber and your -unit / functional / integration tests, all of those test suites will be taken into account when building the -coverage report. - -There are two things to note here though: +Simplecov automatically caches coverage results in your +(coverage_path)/.resultset.json, and will merge or override those with +subsequent runs, depending on whether simplecov considers those subsequent runs +as different test suites or as the same test suite as the cached results. To +make this distinction, simplecov has the concept of "test suite names". ### Test suite names @@ -531,14 +530,84 @@ SimpleCov.command_name "features" + (ENV['TEST_ENV_NUMBER'] || '') [simplecov-html] prints the used test suites in the footer of the generated coverage report. -### Timeout for merge -Of course, your cached coverage data is likely to become invalid at some point. Thus, result sets that are older than -`SimpleCov.merge_timeout` will not be used any more. By default, the timeout is 600 seconds (10 minutes), and you can -raise (or lower) it by specifying `SimpleCov.merge_timeout 3600` (1 hour), or, inside a configure/start block, with -just `merge_timeout 3600`. +### Merging test runs under the same execution environment + +Test results are automatically merged with previous runs in the same execution +environment when generating the result, so when coverage is set up properly for +Cucumber and your unit / functional / integration tests, all of those test +suites will be taken into account when building the coverage report. + +#### Timeout for merge + +Of course, your cached coverage data is likely to become invalid at some point. Thus, when automatically merging +subsequent test runs, result sets that are older than `SimpleCov.merge_timeout` will not be used any more. By default, +the timeout is 600 seconds (10 minutes), and you can raise (or lower) it by specifying `SimpleCov.merge_timeout 3600` +(1 hour), or, inside a configure/start block, with just `merge_timeout 3600`. + +You can deactivate this automatic merging altogether with `SimpleCov.use_merging false`. + +### Merging test runs under different execution environments + +If your tests are done in parallel across multiple build machines, you can fetch them all and merge them into a single +result set using the `SimpleCov.collate` method. This can be added to a Rakefile or script file, having downloaded a set of +`.resultset.json` files from each parallel test run. + +```ruby +# lib/tasks/coverage_report.rake +namespace :coverage do + desc "Collates all result sets generated by the different test runners" + task :report do + require 'simplecov' + + SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"] + end +end +``` + +`SimpleCov.collate` also takes an optional simplecov profile and an optional +block for configuration, just the same as `SimpleCov.start` or +`SimpleCov.configure`. This means you can configure a separate formatter for +the collated output. For instance, you can make the formatter in +`SimpleCov.start` the `SimpleCov::Formatter::SimpleFormatter`, and only use more +complex formatters in the final `SimpleCov.collate` run. + +```ruby +# spec/spec_helper.rb +require 'simplecov' + +SimpleCov.start 'rails' do + # Disambiguates individual test runs + command_name "Job #{ENV["TEST_ENV_NUMBER"]}" if ENV["TEST_ENV_NUMBER"] + + if ENV['CI'] + formatter SimpleCov::Formatter::SimpleFormatter + else + formatter SimpleCov::Formatter::MultiFormatter.new([ + SimpleCov::Formatter::SimpleFormatter, + SimpleCov::Formatter::HTMLFormatter + ]) + end + + track_files "**/*.rb" +end +``` + +```ruby +# lib/tasks/coverage_report.rake +namespace :coverage do + task :report do + require 'simplecov' -You can deactivate merging altogether with `SimpleCov.use_merging false`. + SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' do + formatter SimpleCov::Formatter::MultiFormatter.new([ + SimpleCov::Formatter::SimpleFormatter, + SimpleCov::Formatter::HTMLFormatter + ]) + end + end +end +``` ## Running coverage only on demand diff --git a/features/test_unit_collate.feature b/features/test_unit_collate.feature new file mode 100644 index 00000000..c0c95c56 --- /dev/null +++ b/features/test_unit_collate.feature @@ -0,0 +1,43 @@ +@test_unit +Feature: + + Using SimpleCov.collate should get the user a coverage report + + Scenario: + Given SimpleCov for Test/Unit is configured with: + """ + require 'simplecov' + SimpleCov.start + """ + + When I successfully run `bundle exec rake part1` + Then a coverage report should have been generated + When I successfully run `mv coverage/.resultset.json coverage/resultset1.json` + And I successfully run `rm coverage/index.html` + + When I successfully run `bundle exec rake part2` + Then a coverage report should have been generated + When I successfully run `mv coverage/.resultset.json coverage/resultset2.json` + And I successfully run `rm coverage/index.html` + + When I open the coverage report generated with `bundle exec rake collate` + Then I should see the groups: + | name | coverage | files | + | All Files | 91.38% | 6 | + + And I should see the source files: + | name | coverage | + | lib/faked_project.rb | 100.0 % | + | lib/faked_project/some_class.rb | 80.0 % | + | lib/faked_project/framework_specific.rb | 75.0 % | + | lib/faked_project/meta_magic.rb | 100.0 % | + | test/meta_magic_test.rb | 100.0 % | + | test/some_class_test.rb | 100.0 % | + + # Note: faked_test.rb is not appearing here since that's the first unit test file + # loaded by Rake, and only there test_helper is required, which then loads simplecov + # and triggers tracking of all other loaded files! Solution for this would be to + # configure simplecov in this first test instead of test_helper. + + And the report should be based upon: + | Unit Tests | diff --git a/lib/simplecov.rb b/lib/simplecov.rb index e847f23f..0f830f60 100644 --- a/lib/simplecov.rb +++ b/lib/simplecov.rb @@ -46,15 +46,50 @@ class << self # def start(profile = nil, &block) require "coverage" - load_profile(profile) if profile - configure(&block) if block_given? + initial_setup(profile, &block) @result = nil - self.running = true self.pid = Process.pid start_coverage_measurement end + # + # Collate a series of SimpleCov result files into a single SimpleCov output. + # You can optionally specify configuration with a block: + # SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"] + # OR + # SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' # using rails profile + # OR + # SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"] do + # add_filter 'test' + # end + # OR + # SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"], 'rails' do + # add_filter 'test' + # end + # + # Please check out the RDoc for SimpleCov::Configuration to find about + # available config options, or checkout the README for more in-depth + # information about coverage collation + # + def collate(result_filenames, profile = nil, &block) + raise "There's no reports to be merged" if result_filenames.empty? + + initial_setup(profile, &block) + + results = result_filenames.flat_map do |filename| + # Re-create each included instance of SimpleCov::Result from the stored run data. + (JSON.parse(File.read(filename), symbolize_names: true) || {}).map do |command_name, coverage| + SimpleCov::Result.from_hash(command_name => coverage) + end + end + + # Use the ResultMerger to produce a single, merged result, ready to use. + @result = SimpleCov::ResultMerger.merge_and_store(*results) + + run_exit_tasks! + end + # # Returns the result for the current coverage run, merging it across test suites # from cache using SimpleCov::ResultMerger if use_merging is activated (default) @@ -159,6 +194,8 @@ def exit_status_from_exception # Called from at_exit block # def run_exit_tasks! + set_exit_exception + exit_status = SimpleCov.exit_status_from_exception SimpleCov.at_exit.call @@ -210,7 +247,7 @@ def result_exit_status(result, covered_percent) ) SimpleCov::ExitCodes::MINIMUM_COVERAGE elsif (last_run = SimpleCov::LastRun.read) - coverage_diff = last_run["result"]["covered_percent"] - covered_percent + coverage_diff = last_run[:result][:covered_percent] - covered_percent if coverage_diff > SimpleCov.maximum_coverage_drop $stderr.printf( "Coverage has dropped by %.2f%% since the last time (maximum allowed: %.2f%%).\n", @@ -253,6 +290,12 @@ def write_last_run(covered_percent) private + def initial_setup(profile, &block) + load_profile(profile) if profile + configure(&block) if block_given? + self.running = true + end + # # Trigger Coverage.start depends on given config coverage_criterion # @@ -319,7 +362,7 @@ def add_not_loaded_files(result) result = result.dup Dir[tracked_files].each do |file| absolute_path = File.expand_path(file) - result[absolute_path] ||= SimulateCoverage.call(absolute_path) + result[absolute_path.to_sym] ||= SimulateCoverage.call(absolute_path) end end diff --git a/lib/simplecov/combine/files_combiner.rb b/lib/simplecov/combine/files_combiner.rb index 4732761a..e5c4ef39 100644 --- a/lib/simplecov/combine/files_combiner.rb +++ b/lib/simplecov/combine/files_combiner.rb @@ -15,10 +15,9 @@ module FilesCombiner # @return [Hash] # def combine(coverage_a, coverage_b) - { - lines: Combine.combine(LinesCombiner, coverage_a[:lines], coverage_b[:lines]), - branches: Combine.combine(BranchesCombiner, coverage_a[:branches], coverage_b[:branches]) - } + combination = {lines: Combine.combine(LinesCombiner, coverage_a[:lines], coverage_b[:lines])} + combination[:branches] = Combine.combine(BranchesCombiner, coverage_a[:branches], coverage_b[:branches]) if SimpleCov.branch_coverage? + combination end end end diff --git a/lib/simplecov/configuration.rb b/lib/simplecov/configuration.rb index 7455eed7..8791c558 100644 --- a/lib/simplecov/configuration.rb +++ b/lib/simplecov/configuration.rb @@ -313,7 +313,7 @@ def add_group(group_name, filter_argument = nil, &filter_proc) # * :line - coverage based on lines aka has this line been executed? # * :branch - coverage based on branches aka has this branch (think conditions) been executed? # - # If not set the default is is `:line` + # If not set the default is `:line` # # @param [Symbol] criterion # diff --git a/lib/simplecov/defaults.rb b/lib/simplecov/defaults.rb index 468568ab..60e4fb0f 100644 --- a/lib/simplecov/defaults.rb +++ b/lib/simplecov/defaults.rb @@ -25,7 +25,9 @@ # If we are in a different process than called start, don't interfere. next if SimpleCov.pid != Process.pid - SimpleCov.set_exit_exception + # If SimpleCov is no longer running then don't run exit tasks + next unless SimpleCov.running + SimpleCov.run_exit_tasks! end diff --git a/lib/simplecov/last_run.rb b/lib/simplecov/last_run.rb index e6c7f438..5a4bd1d7 100644 --- a/lib/simplecov/last_run.rb +++ b/lib/simplecov/last_run.rb @@ -15,7 +15,7 @@ def read json = File.read(last_run_path) return nil if json.strip.empty? - JSON.parse(json) + JSON.parse(json, symbolize_names: true) end def write(json) diff --git a/lib/simplecov/result.rb b/lib/simplecov/result.rb index 3a3e5c2c..dc9e386e 100644 --- a/lib/simplecov/result.rb +++ b/lib/simplecov/result.rb @@ -61,7 +61,7 @@ def command_name # Returns a hash representation of this Result that can be used for marshalling it into JSON def to_hash - {command_name => {"coverage" => coverage, "timestamp" => created_at.to_i}} + {command_name.to_sym => {coverage: coverage, timestamp: created_at.to_i}} end # Loads a SimpleCov::Result#to_hash dump @@ -69,31 +69,18 @@ def self.from_hash(hash) command_name, data = hash.first result = SimpleCov::Result.new( - symbolize_names_of_coverage_results(data["coverage"]) + data[:coverage] ) - result.command_name = command_name - result.created_at = Time.at(data["timestamp"]) + result.command_name = command_name.to_s + result.created_at = Time.at(data[:timestamp]) result end - # Manage symbolize the keys of coverage hash. - # JSON.parse gives coverage hash with stringified keys what breaks some logics - # inside the process that expects them as symboles. - # - # @return [Hash] - def self.symbolize_names_of_coverage_results(coverage_data) - coverage_data.each_with_object({}) do |(file_name, file_coverage_result), coverage_results| - coverage_results[file_name] = file_coverage_result.each_with_object({}) do |(k, v), cov_elem| - cov_elem[k.to_sym] = v - end - end - end - private def coverage - keys = original_result.keys & filenames + keys = original_result.keys & filenames.map(&:to_sym) Hash[keys.zip(original_result.values_at(*keys))] end diff --git a/lib/simplecov/result_adapter.rb b/lib/simplecov/result_adapter.rb index 79cf8e6f..996c8d31 100644 --- a/lib/simplecov/result_adapter.rb +++ b/lib/simplecov/result_adapter.rb @@ -20,9 +20,9 @@ def adapt result.each_with_object({}) do |(file_name, cover_statistic), adapted_result| if cover_statistic.is_a?(Array) - adapted_result.merge!(file_name => {lines: cover_statistic}) + adapted_result.merge!(file_name.to_sym => {lines: cover_statistic}) else - adapted_result.merge!(file_name => cover_statistic) + adapted_result.merge!(file_name.to_sym => cover_statistic) end end end diff --git a/lib/simplecov/result_merger.rb b/lib/simplecov/result_merger.rb index a92fd1cc..9a1e1635 100644 --- a/lib/simplecov/result_merger.rb +++ b/lib/simplecov/result_merger.rb @@ -26,7 +26,7 @@ def resultset data = stored_data if data begin - JSON.parse(data) || {} + JSON.parse(data, symbolize_names: true) || {} rescue StandardError {} end @@ -62,6 +62,12 @@ def results results end + def merge_and_store(*results) + result = merge_results(*results) + store_result(result) if result + result + end + # Merge two or more SimpleCov::Results into a new one with merged # coverage data and the command_name for the result consisting of a join # on all source result's names diff --git a/spec/combine/results_combiner_spec.rb b/spec/combine/results_combiner_spec.rb index 27cb5cf5..f5d27eeb 100644 --- a/spec/combine/results_combiner_spec.rb +++ b/spec/combine/results_combiner_spec.rb @@ -4,7 +4,13 @@ describe SimpleCov::Combine::ResultsCombiner do describe "with two faked coverage resultsets" do + after do + SimpleCov.clear_coverage_criteria + end + before do + SimpleCov.enable_coverage :branch + @resultset1 = { source_fixture("sample.rb") => { lines: [nil, 1, 1, 1, nil, nil, 1, 1, nil, nil], diff --git a/spec/helper.rb b/spec/helper.rb index 44fbf0f4..e747a5eb 100644 --- a/spec/helper.rb +++ b/spec/helper.rb @@ -10,7 +10,7 @@ SimpleCov.coverage_dir("tmp/coverage") def source_fixture(filename) - File.expand_path(File.join(File.dirname(__FILE__), "fixtures", filename)) + File.expand_path(File.join(File.dirname(__FILE__), "fixtures", filename)).to_sym end # Taken from http://stackoverflow.com/questions/4459330/how-do-i-temporarily-redirect-stderr-in-ruby diff --git a/spec/result_merger_spec.rb b/spec/result_merger_spec.rb index ac94213c..6d2c80b0 100644 --- a/spec/result_merger_spec.rb +++ b/spec/result_merger_spec.rb @@ -5,7 +5,7 @@ require "timeout" describe SimpleCov::ResultMerger do - before do + after do SimpleCov::ResultMerger.clear_resultset File.delete(SimpleCov::ResultMerger.resultset_path) if File.exist?(SimpleCov::ResultMerger.resultset_path) end @@ -63,7 +63,7 @@ end it "returns a hash containing keys ['result1' and 'result2'] for resultset" do - expect(SimpleCov::ResultMerger.resultset.keys.sort).to eq %w[result1 result2] + expect(SimpleCov::ResultMerger.resultset.keys.sort).to eq %i[result1 result2] end it "returns proper values for merged_result" do @@ -96,7 +96,7 @@ SimpleCov::ResultMerger.store_result("a" => [1]) SimpleCov::ResultMerger.clear_resultset new_set = SimpleCov::ResultMerger.resultset - expect(new_set).to eq("a" => [1]) + expect(new_set).to eq(a: [1]) end it "synchronizes writes" do diff --git a/spec/simplecov_spec.rb b/spec/simplecov_spec.rb index b1753826..88266fb1 100644 --- a/spec/simplecov_spec.rb +++ b/spec/simplecov_spec.rb @@ -230,10 +230,95 @@ end end + describe ".collate" do + let(:resultset1) do + {source_fixture("sample.rb") => {lines: [nil, 1, 1, 1, nil, nil, 1, 1, nil, nil]}} + end + + let(:resultset2) do + {source_fixture("sample.rb") => {lines: [1, nil, 1, 1, nil, nil, 1, 1, nil, nil]}} + end + + let(:resultset_path) { SimpleCov::ResultMerger.resultset_path } + + let(:resultset_folder) { File.dirname(resultset_path) } + + context "when no files to be merged" do + it "shows an error message" do + expect do + glob = Dir.glob("#{resultset_folder}/*.final", File::FNM_DOTMATCH) + SimpleCov.collate glob + end.to raise_error("There's no reports to be merged") + end + end + + context "when files to be merged" do + before do + expect(SimpleCov).to receive(:run_exit_tasks!) + end + + context "and a single report to be merged" do + before do + create_mergeable_report("result1", resultset1) + end + + after do + clear_mergeable_reports("result1") + end + + it "creates a merged report identical to the original" do + glob = Dir.glob("#{resultset_folder}/*.final", File::FNM_DOTMATCH) + SimpleCov.collate glob + + expected = {"result1": {coverage: {source_fixture("sample.rb") => {lines: [nil, 1, 1, 1, nil, nil, 1, 1, nil, nil]}}}} + collated = JSON.parse(File.read(resultset_path), symbolize_names: true).transform_values { |v| v.reject { |k| k == :timestamp } } + expect(collated).to eq(expected) + end + end + + context "and multiple reports to be merged" do + before do + create_mergeable_report("result1", resultset1) + create_mergeable_report("result2", resultset2) + end + + after do + clear_mergeable_reports("result1", "result2") + end + + it "creates a merged report" do + glob = Dir.glob("#{resultset_folder}/*.final", File::FNM_DOTMATCH) + SimpleCov.collate glob + + expected = {"result1, result2": {coverage: {source_fixture("sample.rb") => {lines: [1, 1, 2, 2, nil, nil, 2, 2, nil, nil]}}}} + collated = JSON.parse(File.read(resultset_path), symbolize_names: true).transform_values { |v| v.reject { |k| k == :timestamp } } + expect(collated).to eq(expected) + end + end + + private + + def create_mergeable_report(name, resultset) + result = SimpleCov::Result.new(resultset) + result.command_name = name + SimpleCov::ResultMerger.store_result(result) + FileUtils.mv resultset_path, "#{resultset_path}#{name}.final" + end + + def clear_mergeable_reports(*names) + SimpleCov.clear_result + SimpleCov::ResultMerger.clear_resultset + FileUtils.rm resultset_path + FileUtils.rm "#{resultset_path}.lock" + names.each { |name| FileUtils.rm "#{resultset_path}#{name}.final" } + end + end + end + # Normally wouldn't test private methods but just start has side effects that # cause errors so for time this is pragmatic (tm) describe ".start_coverage_measurement", if: SimpleCov.coverage_start_arguments_supported? do - before :each do + after :each do # SimpleCov is a Singleton/global object so once any test enables # any kind of coverage data it stays there. # Hence, we use clear_coverage_data to create a "clean slate" for these tests diff --git a/test_projects/faked_project/Rakefile b/test_projects/faked_project/Rakefile index 0d05bfa8..1509781a 100644 --- a/test_projects/faked_project/Rakefile +++ b/test_projects/faked_project/Rakefile @@ -8,3 +8,20 @@ Rake::TestTask.new(:test) do |test| test.test_files = FileList["test/**/*_test.rb"].sort test.verbose = true end + +Rake::TestTask.new(:part1) do |test| + test.libs << "lib" + test.test_files = FileList["test/**/*_test.rb"].sort + test.verbose = true +end + +Rake::TestTask.new(:part2) do |test| + test.libs << "test" + test.test_files = FileList["test/**/*_test.rb"].sort + test.verbose = true +end + +task :collate do + require "simplecov" + SimpleCov.collate Dir["coverage/resultset*.json"] +end diff --git a/test_projects/faked_project/test/faked_test.rb b/test_projects/faked_project/test/faked_test.rb index 2bf8497e..d8f35fdb 100644 --- a/test_projects/faked_project/test/faked_test.rb +++ b/test_projects/faked_project/test/faked_test.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require "test_helper" +require_relative "test_helper" class FakedTest < Test::Unit::TestCase def test_something diff --git a/test_projects/faked_project/test/meta_magic_test.rb b/test_projects/faked_project/test/meta_magic_test.rb index a03d3a3f..5e362153 100644 --- a/test_projects/faked_project/test/meta_magic_test.rb +++ b/test_projects/faked_project/test/meta_magic_test.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require "test_helper" +require_relative "test_helper" class MetaMagicTest < Test::Unit::TestCase def test_class_methods diff --git a/test_projects/faked_project/test/some_class_test.rb b/test_projects/faked_project/test/some_class_test.rb index 4da4dd4c..9f45b949 100644 --- a/test_projects/faked_project/test/some_class_test.rb +++ b/test_projects/faked_project/test/some_class_test.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require "test_helper" +require_relative "test_helper" class SomeClassTest < Test::Unit::TestCase def setup