(GH-884) Update acceptance testing #887

treydock · 2018-04-02T23:22:28Z

Add travis-ci tests for all supported versions of Linux that use docker nodesets
Default PUPPET_GEM_VERSION to be Puppet 5 to avoid errors when not defined in environment

Pull Request Checklist

Description

Add automated acceptance tests.

Related Issue

Fixes #884

Motivation and Context

Improved module testing by adding automated system acceptance tests.

How Has This Been Tested?

Tested all supported Linux systems via travis-ci acceptance tests

General

Update README.md with any necessary configuration snippets
New parameters are documented
New parameters have tests
Tests pass - bundle exec rake validate lint spec

ghoneycutt · 2018-04-05T16:22:10Z

.travis.yml

@@ -24,6 +24,54 @@ matrix:
 env: PUPPET_GEM_VERSION="~> 4"
 - rvm: 2.4.1
 env: PUPPET_GEM_VERSION="~> 5"
+ - rvm: 2.4.2


Why this version of rvm?

Overlaps version use in latest Puppet 5 release, need newer Rubies to work with beaker.

Could we use 2.4.3 so it is the same as the latest puppet release? That will make testing easier for people.

We could, but the version used in travis for acceptance tests isn't actually what runs inside the container so no real impact on puppet compatibility. The only ruby version restriction for the acceptance tests is being supported by beaker and the various beaker support libraries.

I think the default Puppet version used in the tests inside the various containers is puppet-agent 1.10.0 which is Puppet 4.x. Should we force latest Puppet or stick with default from beaker-puppet_install_helper?

Further analysis seems the various helpers to install Puppet with beaker don't yet support Puppet 5, at least not obvious. puppetlabs/beaker-puppet_install_helper#33

Ugh. We need Puppet 5 as I'm looking to drop Puppet 4 in the next version of this module.

I'll see what's involved getting Puppet 5 used. May be we just can't use the helper methods and have to explicitly bootstrap puppet.

Got Puppet 5 as the version installed for acceptance tests. PUPPET_INSTALL_VERSION sets the default, leaving undefined results in latest puppet-agent present in puppet5 repo.

ghoneycutt · 2018-04-05T16:22:24Z

.travis.yml

+ services: docker
+ env: BEAKER_set="centos-6-docker"
+ bundler_args:
+ script: sudo service docker restart ; sleep 10 ; bundle exec rake beaker


why do we restart docker?

Honestly never thought about it, just something I have been doing since while ago when I started doing acceptance tests. May be unnecessary.

Please test and see if we need this.

I removed the service restarts and tests still work so was unnecessary.

ghoneycutt · 2018-04-05T16:22:58Z

Gemfile

@@ -1,6 +1,6 @@
 source ENV['GEM_SOURCE'] || "https://rubygems.org"

-if puppetversion = ENV['PUPPET_GEM_VERSION']
+if puppetversion = ENV['PUPPET_GEM_VERSION'] || "~> 5.x"


This should already choose the latest version. Why this change?

The Gemfile errors out if PUPPET_GEM_VERSION is not set because puppetversion ends up being nil. Without this change PUPPET_GEM_VERSION would have to be defined to the acceptance tests even though it has not bearing on the version of Puppet used by the tests.

$ bundle install The latest bundler is 1.16.1, but you are currently running 1.15.4. To update, run `gem install bundler` [!] There was an error parsing `Gemfile`: undefined method `<' for nil:NilClass. Bundler cannot continue. # from /Users/tdockendorf/git/sensu-puppet/Gemfile:37 # ------------------------------------------- # gem 'puppet-lint-version_comparison-check', :require => false > if puppetversion < '5.0' # gem 'semantic_puppet', :require => false # -------------------------------------------

ghoneycutt · 2018-04-05T16:24:34Z

spec/acceptance/nodesets/amazonlinux-201703-docker.yml

+ docker_cmd:
+ - '/sbin/init'
+ docker_image_commands:
+ - 'yum install -y tar wget cronie git upstart'


How are these dependencies determined?

They are things you may need to bootstrap test environment. I always do git incase git cloning modules for testing. For something like upstart is needed to support docker_cmd of /sbin/init which is necessary to have functioning container for tests. Others may be unnecessary but always added as sometimes useful. cronie likely not needed.

Further investigation, I believe cronie may have been added to support potentially adding cron jobs via Puppet in other modules and being able to test cron with serverspec: http://serverspec.org/resource_types.html#cron. Not 100% certain if serverspec uses crontab or looks in /var/spool/cron type situation.

ghoneycutt · 2018-04-05T16:24:47Z

spec/acceptance/nodesets/amazonlinux-201703-docker.yml

+ platform: el-6-x86_64
+ hypervisor: docker
+ image: amazonlinux:2017.03
+ docker_preserve_image: true


what does this do?

Allows container to be re-used between tests, useful for testing changes on local systems.

https://github.com/puppetlabs/beaker-docker/blob/master/docker.md#preserve-docker-image

These YAML may need to be updated to also allow BEAKER_provision=no which has options likely needed with beaker 3.x to fully support re-using containers.

https://github.com/puppetlabs/beaker-docker/blob/master/docker.md#reuse-docker-image

I'll update all images as really handy when doing local development to be able to re-run tests. There is a bug though where executions after first will put the module in /etc/puppet/modules for some reason which is not used by AIO Puppet. Never got around to opening an issue with Puppetlabs.

FYI - reusable docker images works as documented but annoying bug of module being installed to wrong place is still present:

centos-7 20:07:02$ echo /etc/puppet/modules /etc/puppet/modules centos-7 executed in 0.12 seconds Using scp to transfer /home/treydock/git/sensu-puppet to /etc/puppet/modules/sensu

Could you please report this bug to Puppet.

Issue has changed. I'm pretty sure I was mislead by fact my initial local environment had beaker pinned to ~> 2.x which created the container and then upgraded to latest beaker without re-creating container. Now I can't reuse containers at all which doesn't affect travis-ci, just means all tests have to run in fresh container.

https://tickets.puppetlabs.com/browse/BKR-1457

ghoneycutt · 2018-04-05T16:25:30Z

spec/acceptance/nodesets/centos-6.yml

@@ -0,0 +1,10 @@
+HOSTS:


Why a vagrant node if we have docker?

Probably not needed, would simplify the naming of docker containers. I have run into edge cases where some things don't work in docker containers, like modifying iptables, due to kernel modules not being able to load since kernel is of host and not container.

If we are not testing it with TravisCI, this seems like cruft that could be removed. What do you think?

This is done, all nodesets are docker now.

ghoneycutt · 2018-04-05T16:26:23Z

spec/acceptance/nodesets/default.yml

 platform: el-7-x86_64
- box : puppetlabs/centos-7.0-64-nocm
- box_url : https://vagrantcloud.com/puppetlabs/boxes/centos-7.0-64-nocm
+ box : centos/7


How is the default invoked? Why is it vagrant?

I believe default is invoked if no BEAKER_set environment variable is set. Left vagrant for compatibility but could just as easily make it docker.

Is a default really needed? If not, let's remove it, else let's make it docker.

ghoneycutt · 2018-04-05T16:27:06Z

spec/acceptance/nodesets/ubuntu-1404-docker.yml

+ - 'apt-get install -y -q net-tools wget'
+ - 'locale-gen en_US.UTF-8'
+CONFIG:
+ type: aio


Why are some types aio and some are foss? What is the difference?

Should probably be foss. I pulled the ubuntu files from one of my other puppet modules and the files were actually a contribution. I know the determination of what is installed is actually not in beaker and not from nodeset file, not sure if type even used but it is here: https://github.com/puppetlabs/beaker/blob/master/acceptance/fixtures/module/spec/acceptance/nodesets/default.yml

Real place Puppet install type is defined: https://github.com/puppetlabs/beaker-puppet_install_helper#run_puppet_install_helper

Probably just need to make them all foss.

Please make them all foss if that's what they should be.

ghoneycutt · 2018-04-05T16:27:49Z

spec/acceptance/nodesets/ubuntu-server-1204.yml

@@ -1,7 +1,7 @@
 HOSTS:
- ubuntu-server-1204-x64:


We don't support ubuntu 12. I don't think this should be here.

Please remove any platforms we do not support.

ghoneycutt · 2018-04-05T16:28:15Z

spec/acceptance/nodesets/ubuntu-server-1404.yml

@@ -1,7 +1,7 @@
 HOSTS:
- ubuntu-server-1404-x64:


Why do we have Ubuntu and Ubuntu Server? I don't get the distinction.

I think the server was left overs from previous vagrant boxes. I don't know of a difference, the containers are all just ubuntu, no server distinction.

Please remove all the cruft we don't need.

treydock · 2018-04-07T00:19:30Z

@ghoneycutt I can squash commits once everything looks good if you'd prefer a single commit.

ghoneycutt · 2018-04-12T16:46:07Z

.travis.yml

@@ -24,6 +24,54 @@ matrix:
 env: PUPPET_GEM_VERSION="~> 4"
 - rvm: 2.4.1


Seems we should be using ruby 2.4.3

https://puppet.com/docs/puppet/5.3/about_agent.html

I've updated PR to use 2.4.3.

Add travis-ci tests for all supported versions of Linux that use docker nodesets Default PUPPET_GEM_VERSION to be Puppet 5 to avoid errors when not defined in environment

Remove docker nodeset docker suffix Make all nodesets type "foss" Get all nodesets capable of being rerun with BEAKER_destroy=no and BEAKER_provision=no combinations

Version set by environment variable PUPPET_INSTALL_VERSION

ghoneycutt · 2018-04-25T02:33:40Z

spec/spec_helper_acceptance.rb

 end
- on host, puppet('module', 'install', 'puppetlabs-stdlib'), { :acceptable_exit_codes => [0,1] }


why does stdlib go away here?

Pulled in from metadata.json by install_module_dependencies_on(hosts) at the top of file. Removed to avoid having to update some dependencies in two places.

ghoneycutt · 2018-05-21T21:58:28Z

Released in v2.53.0

treydock force-pushed the acceptance-tests branch from d5784ed to 626f937 Compare April 2, 2018 23:24

ghoneycutt reviewed Apr 5, 2018

View reviewed changes

ghoneycutt reviewed Apr 12, 2018

View reviewed changes

treydock force-pushed the acceptance-tests branch from 58fc522 to 19d775e Compare April 12, 2018 21:49

treydock added 3 commits April 12, 2018 18:05

(sensuGH-884) Update acceptance testing

05f69c4

Add travis-ci tests for all supported versions of Linux that use docker nodesets Default PUPPET_GEM_VERSION to be Puppet 5 to avoid errors when not defined in environment

Remove vagrant nodesets

819698f

Remove docker nodeset docker suffix Make all nodesets type "foss" Get all nodesets capable of being rerun with BEAKER_destroy=no and BEAKER_provision=no combinations

Do not restart docker during acceptance tests in travis-ci

96cf580

treydock force-pushed the acceptance-tests branch from 19d775e to 2fef933 Compare April 12, 2018 22:05

treydock added 2 commits April 14, 2018 16:42

Bump puppetlabs_spec_helper requirement to work with mocha 1.5.0

2eadc72

Bump Ruby to 2.4.3 to match latest Puppet AIO version for Puppet 5.x

e7d1f47

treydock force-pushed the acceptance-tests branch from 2fef933 to e7d1f47 Compare April 14, 2018 20:43

Force acceptance tests to use Puppet 5

69e6a13

Version set by environment variable PUPPET_INSTALL_VERSION

treydock force-pushed the acceptance-tests branch from 72bf8e2 to 69e6a13 Compare April 18, 2018 21:58

ghoneycutt reviewed Apr 25, 2018

View reviewed changes

ghoneycutt merged commit 33f6c91 into sensu:master Apr 26, 2018

		@@ -24,6 +24,54 @@ matrix:
		env: PUPPET_GEM_VERSION="~> 4"
		- rvm: 2.4.1

		end
		on host, puppet('module', 'install', 'puppetlabs-stdlib'), { :acceptable_exit_codes => [0,1] }

(GH-884) Update acceptance testing #887

(GH-884) Update acceptance testing #887

Conversation

treydock commented Apr 2, 2018 • edited Loading

Pull Request Checklist

Description

Related Issue

Motivation and Context

How Has This Been Tested?

General

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghoneycutt Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

treydock commented Apr 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghoneycutt commented May 21, 2018

treydock commented Apr 2, 2018 •

edited

Loading

ghoneycutt Apr 16, 2018 •

edited

Loading