Spork doesn't seem to run SimpleCov correctly with RSpec and Rails

[fj]

tl;dr:

Summary of results:

summary of issues		Spork
		not running	running
SimpleCov	top of Spork.prefork	works as expected	SimpleCov doesn't run
	top of Spork.each_run	always 0 LOC, 0% coverage	SimpleCov doesn't run

---

I'm on an existing Rails project using Mongoid, Devise, RSpec, and Spork (0.9.0.rc5). Our spec/spec_helper.rb looks like this:

require 'spork'

ENV["RAILS_ENV"] ||= 'test'

Spork.prefork do
  require 'simplecov'
  SimpleCov.start 'rails'

  # trap mongoid
  require 'rails/mongoid'
  Spork.trap_class_method(Rails::Mongoid, :load_models)

  # trap devise
  require 'rails/application'
  Spork.trap_method(Rails::Application, :reload_routes!)

  require File.expand_path("../../config/environment", __FILE__)
  require 'rspec/rails'
  Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

  DatabaseCleaner.strategy = :truncation

  RSpec.configure do |config|
    config.mock_with :rspec

    config.before(:each) do
      DatabaseCleaner.clean
    end
  end
end

Spork.each_run do
  # reload factories
  Factory.definition_file_paths = [
    File.join(Rails.root, 'spec', 'factories')
  ]
  Factory.find_definitions
end

This configuration will run SimpleCov when Spork is not running, and it produces the expected result. When Spork is running, no coverage message is shown.

If I move the require 'simplecov'; SimpleCov.start 'rails' lines to the beginning of the Spork.each_run block, as SimpleCov's readme suggests, then when Spork is not running, I get this:

Coverage report generated for RSpec to /home/johnf/projects/<elided>/coverage. 0 / 0 LOC (0.0%) covered.

If Spork is running, then instead I get no coverage message, as before.

[fj]

FWIW, I'm running spork with bundle exec spork rspec, rake with bundle exec rake, and rspec with bundle exec rspec --drb spec.

[colszowka]

I'm not using Spork myself, but from what I understand is that it always spins up a new process when the test suite should be run.

The Ruby Coverage library SimpleCov uses works by tracking every required file AFTER it has been loaded. Therefore, if SimpleCov is loaded AFTER the Rails app and all of it's source files, it won't be able to see them (I think this is what you're getting with the each_run solution).

So I imagine the solution would be to make Spork require 'simplecov'; SimpleCov.start before loading the app's environment every time, so the actual code files become visible to SimpleCov. But as I said, I'm not using Spork and I'm not sure how exactly it does it's magic. Maybe the authors of Spork could shed some light onto this?

[colszowka]

Quick update: I pinged @timcharper about this, hopefully he can enlighten us a bit :)

[JeanMertz]

I too am having this issue. Seems not many people run into this problem, as I can't find anything anywhere else.

[turadg]

@fj, I bet your problem is your test environment caches classes. See @Estheth's comment in #30.

fwiw, I had been using the hack below in each_run and setting cache_classes to true but that doesn't work:

  # HACK so we can cache-classes in test.rb
  Dir["#{Rails.root}/app/**/*.rb"].each { |f| load f }

  require 'simplecov'
  SimpleCov.start 'rails'

[Godisemo]

I have this issue, i put the two lines for simplecov at the top of the Spork.prefork block. When I use guard no coverage report is generated but when I run rake specit works. Would be nice if it worked with guard though.

[denmarkin]

+1. Same applies to configuration with cucumber. I bet it's spork related

[aantix]

+1. Same config as @Godisemo along with the same issue.

[sevenseacat]

+1 Same here. I swear, it used to work....

[christophermanning]

+1

simplecov (0.5.4)
guard-rspec (0.5.10)
guard-spork (0.4.1)
spork (0.9.0)

[colszowka]

Guys, this obviously is related to #60 - the offender being Spork's requirement to set config.cache_classes = false in config/environments/test.rb. Please try to set it to config.cache_classes = true. As I do not use Spork I don't know what the implications are for spork when you do this, but without it SimpleCov will fail to operate correctly :(

[christophermanning]

@colszowka I had config.cache_classes set to true and false and it still wouldn't generate the coverage directory/report after guard finishes running tests. Additionally setting config.cache_classes = false in config/environments/test.rb causes ActiveRecord::AssociationTypeMismatch exceptions when using FactoryGirl: http://stackoverflow.com/questions/5954690/strange-activerecordassociationtypemismatch

[sunaku]

Something must be wrong in your setups. I successfully use both config.cache_classes=false and FactoryGirl to get code coverage results under Tork (not Spork).

[sunaku]

You guys should try using the Coverage library directly in your setups. If it works, then SimpleCov needs fixing. Else, you know what to do. :)

[timcharper]

I'm afraid I don't have any experience with SimpleCov to be able to shed much light on this. But, I can say this:

You shouldn't need config.cache_classes = false, although that's the easiest way to get your models and such to reload automatically. Read these wiki pages for more info:

https://github.com/sporkrb/spork/wiki/Troubleshooting (specifically: Some changes to files don’t take effect until I restart Spork.)

https://github.com/sporkrb/spork/wiki/Spork.trap_method-Jujutsu

All you should need to do is delay the initialization of SimpleCov until after the process forks, and SimpleCov should be none the wiser.

If a SimpleCov devel wants to do some remove pair programming with me to crack this together, I'd be happy to schedule some time.

[sunaku]

On Thu 09 Feb 2012 10:44:56 AM PST, Tim Harper wrote:

You shouldn't need config.cache_classes = false, although that's
the easiest way to get your models and such to reload automatically.

All you should need to do is delay the initialization of SimpleCov
until after the process forks, and SimpleCov should be none the wiser.

That is correct. You should call Coverage.start() after forking. The
coverage results should be collected and processed by an at_exit hook
that was registered once before forking. Here is a working example:

https://github.com/sunaku/tork/blob/coverage/lib/tork/config/coverage.rb#L6-10

[chetan]

FWIW, this is working for me now. I initially had some issues and after a bunch of mucking about it now works (even though my config is right where it was when I started).

The one minor change I made to my code was to skip my custom initializer which bootstrapped some of my custom libs. I now require the initializer manually in the each_run block. This is the complete contents of my test_helper.rb (I'm using minitest):

https://gist.github.com/1820849

I also disabled cache_classes as noted above, though I'm not sure that's absolutely necessary.

[ches]

So I'm not familiar with Tork, but @sunaku is the ordering of that Coverage.start() hook before app files are (re)loaded dependent simply on internal load order within Tork? It seems like we'd need something akin to prepend_each_run in Spork to get things like SimpleCov to run immediately after forking and before (for instance) spork-rails' each_run blocks kick off any reloading, etc., does that sound right @timcharper or am I oversimplifying?

@chetan It's not clear what you're doing in the files you require from your Gist -- if you're loading your app env in each_run every time, you aren't getting much out of using Spork, no?

[chetan]

@ches oops, I forgot to attach test_prefork. Basically I bootstrap the whole rails and test environment w/ the exception of all app/ lib/ and test/ classes. My test runtime has gone from about 10sec (mostly in Bundle.setup) to 1sec. Updated the gist:

https://gist.github.com/1820849

[bashcoder]

I'm having the same issue, following the same logic grid as in the first report. I tried toggling class caching in the test environment to no avail. The coverage report just doesn't run unless I manually run rake spec. Everything works fine without spork.

Versions:

simplecov 0.6.1
guard 1.0.0
spork 0.9.0
rails 3.2.0

spec_helper.rb https://gist.github.com/1934043

[bjnord]

rails (3.2.2)
rspec-rails (2.8.1)
simplecov (0.6.1)
spork (0.9.0)

For me, SimpleCov works when Spork is shut down, but not when Spork is running. I tried all the suggestions above (and from other Google searches on the same problem) to no avail; in particular, setting caching to true or false makes no difference. In the end, I learned two things:

1. With Spork running, "rake simplecov" will cause the Spork.each_run section to be run twice. Whether that's the cause, I don't know, but in any case, all LOC in app/*/.rb show up in "not covered" red.
2. However (also with Spork running), "rspec --no-drb spec" gives me a correct SimpleCov report, and Spork.each_run is only executed once.

Don't know if this will help anyone, but at least I have a way to get a coverage report without shutting Spork down every time.

[fredwu]

This works:

Spork.prefork do
  unless ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
  end

  # other code ...
end

Spork.each_run do
  if ENV['DRB']
    require 'simplecov'
    SimpleCov.start 'rails'
  end

  # other code ...
end

[fro]

@fredwu nice, it's working for me now.

[fivetwentysix]

@fredwu Works for me too!

ruby - 1.9.3
rails - 3.2.2
spork - 1.0.0rc

[bjnord]

@fredwu Excellent! That was the piece I needed to unlock the puzzle. There were two other things I had to do:

1. I had to add the ":require => false" option to "gem 'simplecov'" in my Gemfile (this means "rake simplecov" no longer works but I don't care).
2. As noted by @chetan above, I had "SimpleCov.coverage_dir(...)" in a config/initializers/*.rb file, and I had to move that into spec/spec_helper.rb next to "SimpleCov.start" -- I think that was causing SimpleCov to get loaded too soon.

Now I get identical results for "rake spec" and "rspec spec", with and without Spork running. Thank you! (I still have "config.cache_classes = false" by the way.)

[prusswan]

I am getting irregular coverage results with rspec tests while running against spork, despite applying @fredwu 's fix and playing with stuff like:

1. set config.cache_classes = false
2. add the ":require => false" option to "gem 'simplecov'"

For now I am simply going to leave it alone since the trouble proved to be too much for any potential gain (for those interested, my setup: https://github.com/prusswan/hw3_rottenpotatoes/tree/hw4)

[fredwu]

@prusswan Your repo's Gemfile doesn't have :require => false. Also, I just checked my test.rb environment, it's config.cache_classes = true.

[prusswan]

@fredwu as I said I have already tried changing those parameters but they did not improve matters, the result with rspec is pretty erratic.

[rickyrobinson]

@prusswan and others, the solutions suggested here got me part of the way. I think the problem is when some of the code you want coverage reports for is run for the first time during Rails initialization (for example, if an initializer in config/initializers does some config on models or custom code in your lib directory or whatever). SimpleCov won't pick that stuff up because it's loaded before SimpleCov is. A solution is to split loading and initialization between prefork and each_run by requiring application.rb instead of environment.rb, and then explicitly initializing your app in each_run. This solution works with config.cache_classes = true and if threadsafe mode has not been turned on. A detailed blog post is here: http://rickyrobinson.id.au/2012/07/20/when-spork-puts-a-fork-in-your-cucumber-and-a-spanner-in-your-specs.

[colszowka]

I still can't tell much about spork, but on the Ruby Toolbox I've been using spin(+guard-spin) with great success recently. It basically only preloads your app's gems (so probably there's a little time delay compared to spork), but works without any config, and it works smoothly with simplecov so far (of course, only mentioning coverage for the latetst spec that's been run - you can avoid this by expanding merge_timeout and setting a pseudo-random command_name)

[rickyrobinson]

@colszowka Hey, spin looks pretty cool. Thanks for the pointer. It runs my specs just fine after I take out all the spork-related stuff, but not sure if it'll run my cukes. I note that spin takes the same approach to splitting loading and initialization that my spork hack takes:

We require config/application because that file (typically) loads Rails and any Bundler deps, as well as loading the initialization code for the app, but it doesn't actually perform the initialization. That happens in config/environment.

In my experience that's the best we can do in terms of preloading. Rails and the gem dependencies rarely change and so don't need to be reloaded. But you can't initialize the application because any non-trivial app will involve it's models/controllers, etc. in its initialization, which you definitely don't want to preload.

http://jstorimer.github.com/spin/#section-9

Update: got spin working with cucumber and made a few other changes: http://rickyrobinson.id.au/2012/07/24/spork-to-spin

[leckylao]

Having the simplecov runs fine with spork. But the report generated is missing the helper files. But once I stopped spork and run all the specs, then the report includes helper files. Any idea? Anyone have the same issue or know how to fix it?

[phstc]

@fredwu's workaround works for me only if I don't add an add_filter in simplecov.

SimpleCov.start "rails" do
  add_filter "vendor"
end

When I do it I need to configure config.cache_classes = false in test.rb, otherwise I get /coverage. 0 / 0 LOC (0.0%) covered..

I don't feel comfortable disabling cache_classes, is that the correct way to fix it?

[nfm]

An additional observation: If you're using ActiveRecord observers, and activating them from your config/application.rb, the models they observe will still be skipped, even with @fredwu's fix.

The observed models being skipped seems to only be an issue when I run my specs through Spork.

[aminariana]

Works for me:

I followed the "this works" suggestion by fredwu. It didn't work with Spork at first. Then I additionally followed bjnord's suggestion to add :required => false to the simplecov line in the Gemfile. That combination seems to have solved the problem for me. Now coverage metrics are being generated BOTH with and without Spork.

I should note though, that in my test.rb, the following is still unchanged, and that seems to not cause a difference. It still works without modification:
config.cache_classes = true

Further update:
Some of my models weren't showing up when running Rspec under Spork with Guard iteratively, but their dependencies were showing up. After trying a few things, it turns out both of the following are needed for simplecov to deterministically figure out the dependencies on every partial run:

In test.rb do:
config.cache_classes = false

In your spec, do:
require 'spec_helper'
require '<model>'

[Lionizers]

I had the same issue with some application_helper lines that were not evaluated as covered code when running guard+spork. When running rspec without guard, coverage was 100% for that helper.

I use a gem that is located under vendor/gems at the moment which is not watched by guard, I excluded it with a filter in the .simplecov file.

I removed the vendor-filter and added :spec_paths => ["spec", "vendor/gems/.../spec"] to the Guardfile, so the gem-tests are now run together with my rails tests. With this configuration, the coverage is correct with AND without guard/spork.

[buccolo]

SimpleCov will only pickup if we pass the --no-drb flag, as in:
bundle exec rspec spec/ --no-drb

Though I'm not sure why...

[yvanross]

Here is how I solved the problem

in environment/test.rb
config.cache_classes = false

in spec_hepler.rb

Spork.each_run do
require 'simplecov'
SimpleCov.start do
add_filter '/spec/'
add_filter '/config/'
add_filter '/lib/'
add_filter '/vendor/'

add_group 'Controllers', 'app/controllers'
add_group 'Models', 'app/models'
add_group 'Helpers', 'app/helpers'
add_group 'Mailers', 'app/mailers'
add_group 'Views', 'app/views'

end

[kyletolle]

I am using Sinatra, RSpec, Guard, Spork, and SimpleCov, and I had the same issue where code metrics weren't being generated properly while running with Guard.

Posting here, with a complete spec_helper.rb file, in case other people try to do something similar with Sinatra.

This was my initial spec_helper, and the tip above where you checked for the DRB env var didn't work for me. This is what my spec_helper.rb started with. You can see the requiring files in the prefork block, which is what was the issue.

# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'

ENV['RACK_ENV'] = 'test'

def require_files
  ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
  Dir[ruby_files].each do |file|
    require file
  end
  require File.join(File.dirname(__FILE__), '..', 'app.rb')
end

Spork.prefork do
  require_files

  require 'rubygems'
  require 'sinatra'
  require 'rspec'
  require 'rack/test'

  # Set up the test env
  set :environment, :test
  set :run, false
  set :raise_errors, true
  set :logging, false

  def app
    @app ||= Sinatra::Application
  end

  RSpec.configure do |config|
    config.include Rack::Test::Methods
  end
end

Spork.each_run do
end

I used some of the ideas above to change it to what follows, which generates the correct code coverage metrics:

# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'

ENV['RACK_ENV'] = 'test'

def require_files
  ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
  Dir[ruby_files].each do |file|
    require file
  end
  require File.join(File.dirname(__FILE__), '..', 'app.rb')
end

Spork.prefork do
  require 'rubygems'
  require 'sinatra'
  require 'rspec'
  require 'rack/test'

  # Set up the test env
  set :environment, :test
  set :run, false
  set :raise_errors, true
  set :logging, false

  def app
    @app ||= Sinatra::Application
  end
end

Spork.each_run do
  require 'simplecov'
  SimpleCov.start do
    add_filter '/spec/'
  end

  require_files

  RSpec.configure do |config|
    config.include Rack::Test::Methods
  end
end

[bf4]

ref: #340

[bf4]

Closing as stale