OS facts clutter with recent ruby versions in spec tests

[benningm]

I have problems running the unit tests on recent ruby versions.

I have an environment with a stable Debian that is able to successfully execute the tests in api_spec.rb:

$ cat /etc/debian_version 
9.6
$ ruby -v
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]

And an environment running testing Debian/buster:

$ cat /etc/debian_version 
buster/sid
$ ruby -v
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux-gnu]

On this environment most tests in api_spec.rb (and in other test cases) fail.
A lot of errors are about wrong file paths. Some errors report paths with 'C:...' for centos and other linux variants.

Steps to reproduce

On both environments execute:

git clone https://github.com/Icinga/puppet-icinga2.git
cd puppet-icinga2
bundle install
rake spec_prep
bundle exec rspec spec/classes/api_spec.rb

Findings

In manifests/param.pp add:

  notify { 'test-os':                                                                     
    message => "running on ${operatingsystem}"                                            
  }                                                                                       

Add minimal new test case in spec/classes/bug_spec.rb:

require 'spec_helper'

describe('icinga2::feature::api', :type => :class) do
  let(:pre_condition) { [
    "class { 'icinga2': features => [], constants => {'NodeName' => 'host.example.org'} }"
  ] }

  on_supported_os.each do |os, facts|
    let(:facts) do
      facts.merge({
                      :icinga2_puppet_hostcert => '/var/lib/puppet/ssl/certs/host.example.org.pem',
                      :icinga2_puppet_hostprivkey => '/var/lib/puppet/ssl/private_keys/host.example.org.pem',
                      :icinga2_puppet_localcacert => '/var/lib/puppet/ssl/certs/ca.pem',
                  })
    end

    context "test" do
      it 'test notify' do
        is_expected.to contain_notify('test-os').with(
          'message' => "running on #{facts[:operatingsystem]}"
        )
      end
    end
  end
end

Execute it with:

$ bundle exec rspec spec/classes/bug_spec.rb
--cut--
       expected that the catalogue would contain Notify[test-os] with message set to "running on Ubuntu" but it is set to "running on FreeBSD"
--cut--
       expected that the catalogue would contain Notify[test-os] with message set to "running on SLES" but it is set to "running on FreeBSD"
--cut--
        expected that the catalogue would contain Notify[test-os] with message set to "running on Debian" but it is set to "running on FreeBSD"
--cut--
17 examples, 15 failures

Seems like the pre_conditions is run with the same facts for all on_supported_os.each loops.

Fix/Workaround

It works when i wrap everything inside the on_supported_os.each block in an additional context "on #{os}" do block. That's the way the examples in the rspec-puppet-facts do it.

With an additional context block the tests succeed:

    context "on #{os}" do # additional context block
      let(:facts) do
      ...
      end  
   
      context "test" do
      ...
      end
    end # end additional context block
  end

Conclusion

I did not find an difference except the ruby version yet and could not find the root cause for this change of behavior.

[lbetz]

Yes, I did it in v2.x as I rework the test. We don't will spent time on that for v1.x I think.

[lbetz]

Hm,

~/puppetcode/modules/icinga2:master±$ rake spec_standalone SPEC=spec/classes/api_spec.rb
/Users/lbetz/.rbenv/versions/2.5.3/bin/ruby -I/Users/lbetz/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rspec-support-3.8.0/lib:/Users/lbetz/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rspec-core-3.8.0/lib /Users/lbetz/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rspec-core-3.8.0/exe/rspec spec/classes/api_spec.rb --color
WARN: Unresolved specs during Gem::Specification.reset:
rake (>= 0)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Failed to retrieve Augeas version: cannot load such file -- augeas
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Deprecation Warnings:

puppetlabs_spec_helper: defaults mock_with to :mocha. See https://github.com/puppetlabs/puppetlabs_spec_helper#mock_with to choose a sensible value for you

If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
config.raise_errors_for_deprecations!, and it will turn the
deprecation warnings into errors, giving you the full backtrace.

1 deprecation warning total

Finished in 2 minutes 3.2 seconds (files took 3.51 seconds to load)
876 examples, 0 failures

Works fine on my ruby 2.5.3...