Create a STANDARD_RESPONSES file for common issues

[bf4]

From a comment I wrote in #235

This PR is for you:

If anyone wants to help put together a 'standard response' doc, that would be great. See e.g. the one nokogiri uses https://github.com/sparklemotion/nokogiri/blob/master/STANDARD_RESPONSES.md

Where coverage comes from

So, first, background on how simplecov gets coverage.

SimpleCov uses the stdlib Coverage module.

The Coverage module, when active, tracks when code is evaluated for the first time (i.e. when loaded or run )

SimpleCov starts and stops tracking coverage (via Coverage) when you run SimpleCov.start and SimpleCov.result, respectively.

Thus, when you run SimpleCov.start, load/require a file/files, then run SimpleCov.result (which by default runs via an exit hook when your code finishes running), you will see a certain amount of 'code coverage'. That is, simply loading or requiring a Ruby file will result in some 'code coverage'. If you were to load all files in your app, then run SimpleCov.start, then run your tests, you would only capture the additional coverage for code that is executed for the first time. Likewise, if you ran SimpleCov.start, then loaded all files in your app, then ran the tests, even non-tested files would show 'coverage' as they will have code executed simple by being loaded/required.

Where SimpleCov.start should happen

As described above, as early as possible; it must be before any app code is run.

When you require 'simplecov', the .simplecov file, if present, will be executed.

Thus, if you, perhaps through Bundle.require require simplecov, but do not have a .simplecov file, coverage will not start until you explicitly run SimpleCov.start. In other words, by using a .simplecov file, you ensure that SimpleCov will start as soon as it is required.

The .simplecov

https://github.com/colszowka/simplecov/blob/master/lib/simplecov/defaults.rb#L85

# Autoload config from ~/.simplecov if present
# Autoload config from .simplecov if present

Common Problems

• Running Rails tests using rake often gives erroneous coverage because the app is loaded before SimpleCov.start is run in a test helper.
• Coverage lost when running in parallel

Rails and Railties

Vanilla Rails will evaluate all Railties before initializing the Rails app and its initializers. Thus, a Railtie is a reasonable way to hook into Rails initialization lifecycle to start SimpleCov before app code is run.

Right now, require 'simplecov/railtie' loads the file

module SimpleCov
  class Railtie < ::Rails::Railtie
    rake_tasks do
      load 'simplecov/railties/tasks.rake'
    end
  end
end

What should now be obvious is that this railtie does not require 'simplecov' nor does it call SimpleCov.start. @envygeeks is correct that missing the require is really a bug. (but checking the Rails env isn't necessary, because presumably you are only requiring simplecov in test or whenever you want to run it.)

require 'simplecov'
module SimpleCov
  class Railtie < ::Rails::Railtie
    rake_tasks do
      load 'simplecov/railties/tasks.rake'
    end
  end
end

and to get around where you put the SimpleCov.start in your code is why I consider using a .simplecov file a best practice.

Parallel testing

Using .simplecov rather than separately requiring SimpleCov multiple times is recommended if you are merging multiple test frameworks like Cucumber and RSpec that rely on each other, as invoking SimpleCov multiple times can cause coverage information to be lost.

You'll also want to use multiple 'command_names' to differentiate reports being merged in together.

See https://github.com/colszowka/simplecov/blob/master/lib/simplecov/command_guesser.rb#L19 that sets an unset command_name from the env if ENV['PARALLEL_TEST_GROUPS'] && ENV['TEST_ENV_NUMBER'], or from the command that ran the tests, or from any defined constants. to be sure, you may want to set it yourself, e.g. command_name "rails_app_#{$$}" # $$ is the processid

And then add something like merge_timeout 3600 # 1 hour should cover how long it takes the tests to run

Example .simplecov

# https://github.com/colszowka/simplecov#using-simplecov-for-centralized-config
# Maybe put some conditional here not to execute the code below unless ENV['COVERAGE'] == 'true'
SimpleCov.start do
  # see https://github.com/colszowka/simplecov/blob/master/lib/simplecov/defaults.rb
  load_profile 'rails' # load_adapter < 0.8
  coverage_dir 'tmp/coverage'
  # Use multiple 'command_names' to differentiate reports being merged in together
  command_name "rails_app_#{$$}" # $$ is the processid
  merge_timeout 3600 # 1 hour
  add_group "Jobs",        "app/jobs"
  add_group "Middleware",  "app/middleware"
  add_group "Serializers", "app/serializers"
  add_group "Engines",     "engines"
  add_group "Views",       "app/views"
  add_group "Long files" do |src_file|
    src_file.lines.count > 100
  end
  class MaxLinesFilter < SimpleCov::Filter
    def matches?(source_file)
      source_file.lines.count < filter_argument
    end
  end
  add_group "Short files", MaxLinesFilter.new(5)

  # Exclude these paths from analysis
  add_filter 'lib/plugins'
  add_filter 'vendor'
  add_filter 'bundle'
end

or even define your own profile that is backwards compatible

if SimpleCov.respond_to?(:profiles)
  SimpleCov.profiles
else
  SimpleCov.adapters
end.define 'my_app' do
  if defined?(load_profile)
    load_profile  'test_frameworks'
  else
    load_adapter 'test_frameworks'
  end
  coverage_dir 'tmp/coverage'
  # Use multiple 'command_names' to differentiate reports being merged in together
  command_name "rails_app_#{$$}" # $$ is the processid
  merge_timeout 3600 # 1 hour
end
if ENV['COVERAGE'] == 'true'
  SimpleCov.start 'my_app'
   if ENV['CODECLIMATE_REPO_TOKEN']
    begin
      require "codeclimate-test-reporter"
      # We run in parallel, so output results to disk via TO_FILE
      # and send at end, like tddium config
      # https://github.com/codeclimate/ruby-test-reporter/blob/master/lib/code_climate/test_reporter/formatter.rb#L18
      ENV['TO_FILE'] = 'true'
      SimpleCov.formatters = [
          SimpleCov::Formatter::HTMLFormatter,
          CodeClimate::TestReporter::Formatter
      ]
    rescue LoadError
      STDERR.puts "Could not loade CodeClimate SimpleCov Reporter"
    end
  end
end
SimpleCov.at_exit do
  File.open(File.join(SimpleCov.coverage_path, 'coverage_percent.txt'), 'w') do |f|
    f.write SimpleCov.result.covered_percent
  end
  SimpleCov.result.format!
end

[onebree]

Where do I add the .simplecov file? Right now, I define some groups/filters within my ./spec/spec_helper.rb file. Should I put .simplecov in my spec folder?

[envygeeks]

@onebree I do believe it goes in the root of your project where your .rspec is usually placed.

[bf4]

@onebree see above The .simplecov. @envygeeks is correct

Autoload config from ~/.simplecov if present
Autoload config from .simplecov if present

[envygeeks]

@bf4 I think the latter explanation "Autoload config from .simplecov is present" is pretty broad, that leaves open his interpretation of "should it go in spec/.simplecov, <project-root>/.simplecov or where?" Maybe that can be adjusted to be less broad?

[bf4]

@envygeeks duly noted, and thanks so much for your involvement in the comments here. Would you like me to ask @colszowka to add you as a commiter?

[envygeeks]

@bf4 I'd love to help out if you guys are looking for people to help out. We use Simplecov so much maybe I can finally give back by helping you guys.

[bf4]

@envygeeks My policy is to (ask to) invite anyone who seems interested in the library and makes quality contributions (code, comments, whatever). @colszowka would you add @envygeeks as a collab?

[onebree]

I just got back to working on simplecov for our app. After creating ./simplecov, do I still create a block in ./spec/spec_helper.rb? Where do I enter the code block under "Rails and Railities" section?

BTW, we run our tests using bundle exec rspec, rails 3.1, rspec 2.13, simplecov 0.7

Thank you for all the help, guys.

[bf4]

@onebree Are you having a problem? The .simplecov (not ./simplecov) file goes in your project root. If you have SimpleCov.start etc in your .simplecov, then all you need to do in your spec_helper is require 'simplecov'. If you'r running bundle exec rspec then the Railties shouldn't matter.

[onebree]

@bf4 By ./simplecov I meant ./.simplecov with the leading ./ meaning root directory of app. I will work on a SimpleCov profile next year (hahaha). Thank you for all the help, everyone!

[bf4]

@colszowka would you add @envygeeks as a collab?

[pnomolos]

I don't have a chance to open a PR (and not sure where it would fit), but in order to get things all playing nice with minitest/autorun and Guard on a non-rails project, I did the following:

require "simplecov"
SimpleCov.at_exit {}

gem "minitest" # ensures you"re using the gem, and not the built in MT
require "minitest/autorun"
Minitest.after_run { SimpleCov.result.format! }

This is due to how minitest/autorun injects itself with a nested at_exit block: https://github.com/seattlerb/minitest/blob/master/lib/minitest.rb#L45-L59

[bf4]

@pnomolos you may also want to take a look at some work I did in rubygems/rubygems.org#1028

[bf4]

Notes for some common issues I want to link to

Coverage not what expected

• debugging Fails to cover file loaded multiple times #389
• forking Forked process doing cleanup at_exit can cause duplicate coverage reports #228 Default code path doesn't appear to handle fork correctly? #371 (comment)
• parallel Need to wait until all tests suite complete before failing because of minimum_coverage #232
• no branch coverage (C0 only) Branch Coverage Metrics? (Ruby 2.5 is here!) #412 Scope is considerated "not ok" by simplecov-rcov when it is divided in two lines  #411 (comment) Single line conditionals: line coverage #386 Post 'if' coverage and ternary operator #326 (comment) Support for postfix conditionals and partial line executions #105 (comment)
• unexpected coverage non-covered code shown as covered. #365
• coverage overwritten, using command_name Default code path doesn't appear to handle fork correctly? #371 (comment)

Configuration

• excluding Exclude entire files from test coverage report #402

[perlun]

Nice issue, great idea! I think we should also state, somehow/somewhere, that code being dynamically evaluated via {class|instance|module}_eval will not be properly included in the coverage statistics (#38). It is one of the more severe design limitations here, not caused in any way by SimpleCov but rather because of how the coverage gathering in MRI works. (only handles load or require: http://ruby-doc.org/stdlib-2.3.0/libdoc/coverage/rdoc/Coverage.html)

[jdickey]

Hey, all, if you're unexpectedly getting zero coverage, look at your `test/test_helper.rb' (or equivalent for RSpec, etc).

When developing a Gem (let's call it the-gem, I had the line

require 'the_gem'

near the end of my 'test/test_helper.rb' file, and I was getting zero coverage. Removing that line brought my line counts and coverage % back to correct values.

Simple mistake to make, and you'll be scratching your head for ages if this doesn't just occur to you.

[PragTob]

This should all end up in the documentation, there's a trouble shooting section there arleady so we can see what to add from here :)