Jenkinsfile and FOSS setuo works

* Add bundle execution in Jenkinsfile setup

* Add full path to Jenkinsfile bundle

* starting works on foss setup

* run acceptance tests also for newer puppet versions (#220)

* run acceptance tests also for newer puppet versions

reusing the PUPPET_GEM_VERSION environment variable which is also taken
for rspec-puppet.

* switch from PUPPET_GEM_VERSION to installed puppet version

* remove pry

* smarter handling of installing specific puppet versions

* run acceptance tests also for newer puppet versions (#220) (#221)

* run acceptance tests also for newer puppet versions

reusing the PUPPET_GEM_VERSION environment variable which is also taken
for rspec-puppet.

* switch from PUPPET_GEM_VERSION to installed puppet version

* remove pry

* smarter handling of installing specific puppet versions

Add full path to Jenkinsfile bundle

starting works on foss setup

Works on foss vagrant env + docs

Fixed wrong hiera files chnagesdetection in Dangerfile

* Create CODE_OF_CONDUCT.md

* Tests and vagrant works (Should fix #146)

* Create CODE_OF_CONDUCT.md

Foss environment works

Force bundle run before rspec tests in gitlab

Puppetfile cleanup

Syntax fix

Puppet-lint gems from rubygems

Foss vagrant env based on Ubuntu1604

Add full path to Jenkinsfile bundle

* starting works on foss setup

* switch from PUPPET_GEM_VERSION to installed puppet version

* remove pry

* smarter handling of installing specific puppet versions

* run acceptance tests also for newer puppet versions

* reusing the PUPPET_GEM_VERSION environment variable which is also taken
for rspec-puppet.

* switch from PUPPET_GEM_VERSION to installed puppet version

* remove pry

* smarter handling of installing specific puppet versions

* Add full path to Jenkinsfile bundle

* starting works on foss setup

* Works on foss vagrant env + docs

* Fixed wrong hiera files chnagesdetection in Dangerfile

* Fix Typo

* Temporarily gemove trouble Gemfile.local

Syntax fix for bin/puppet_job_run.sh

gitlab ci integration tests

Removed spec/acceptance/psick_spec.rb

Test puppetmaster role with beaker

Foss environment works

Added domain fact to acceptance test spec

Run puppet jobs in CI only for local domain

Saner message in Dangerfile

Add docker to ostest-ubuntu1604.lab.psick.io

Lints for Codacy

Testing pdk on travis #230

Tunong pdk tests in .travis.yml

Deploy modules in travis

Travis fixes

try pdk on travis without error suppression

.gitignore

@@ -224,11 +224,14 @@ bin/metadata-json-lint
 bin/puppet
 bin/rubocop
 bin/rake
+bin/puppet-lint
 modules/**/bin/metadata-json-lint
 modules/**/bin/puppet
 modules/**/bin/rubocop
 modules/**/bin/rake
+modules/**/bin/puppet-lint
 site/**/bin/metadata-json-lint
 site/**/bin/puppet
 site/**/bin/rubocop
 site/**/bin/rake
+site/**/bin/puppet-lint

.gitlab-ci.yml

@@ -1,6 +1,6 @@
 # GitLab CI configuration - https://docs.gitlab.com/ce/ci/yaml/README.html
 # Example for a Puppet deployment workflow with integration and production
-# braches. Requires a fully configured Puppet-GitLab-CI runners
+# branches. Requires a fully configured Puppet-GitLab-CI runners
 # environment to work.
 
 stages:
@@ -86,7 +86,7 @@ rake_controlrepo:
   before_script:
     - "bin/gitlab_before.sh ${CI_BUILD_REF_NAME}"
   script:
-    - "bin/puppet_check_rake.sh site"
+    - "bin/puppet_check_rake.sh site bundle"
     - "bin/puppet_check_rake.sh controlrepo"
   cache:
     untracked: true

.travis.yml

@@ -1,32 +1,20 @@
 ---
-sudo: false
 language: ruby
-cache: bundler
-script:
-  - "bin/travis_check.sh"
-  - "bundle exec danger"
-before_install:
-  - gem update bundler
+install:
+  - wget 'https://pm.puppetlabs.com/cgi-bin/pdk_download.cgi?dist=ubuntu&rel=16.04&arch=amd64&ver=latest' -q -O pdk.deb && sudo dpkg -i pdk.deb && rm pdk.deb
   - gem install r10k
   - r10k puppetfile install -v
-matrix:
-  fast_finish: true
+stages:
+  - validate
+  - unit
+jobs:
   include:
-  - rvm: 2.1.9
-    env: PUPPET_GEM_VERSION="~> 4.6.0"
-  - rvm: 2.1.9
-    env: PUPPET_GEM_VERSION="~> 4.10.0"
-  - rvm: 2.4.1
-    env: PUPPET_GEM_VERSION="~> 5.0.0" STDLIB_LOG_DEPRECATIONS="false"
-  - rvm: 2.4.1
-    env: PUPPET_GEM_VERSION="~> 5.1.0" STDLIB_LOG_DEPRECATIONS="false"
-  - rvm: 2.1.9
-    env: PUPPET_GEM_VERSION="~> 4"
-  - rvm: 2.4.1
-    env: PUPPET_GEM_VERSION="~> 5" STDLIB_LOG_DEPRECATIONS="false"
-  allow_failures:
-  - rvm: 2.1.9
-    env: PUPPET_GEM_VERSION="~> 4.6.0"
+    - stage: validate
+      script:
+        - pdk validate metadata
+        - pdk validate puppet site/ manifests/
 
+    - stage: unit
+      script: pdk test unit
 notifications:
   email: false

.travis.yml.nopdk

@@ -0,0 +1,32 @@
+---
+sudo: false
+language: ruby
+cache: bundler
+script:
+  - "bin/travis_check.sh"
+  - "bundle exec danger"
+before_install:
+  - gem update bundler
+  - gem install r10k
+  - r10k puppetfile install -v
+matrix:
+  fast_finish: true
+  include:
+  - rvm: 2.1.9
+    env: PUPPET_GEM_VERSION="~> 4.6.0"
+  - rvm: 2.1.9
+    env: PUPPET_GEM_VERSION="~> 4.10.0"
+  - rvm: 2.4.1
+    env: PUPPET_GEM_VERSION="~> 5.0.0" STDLIB_LOG_DEPRECATIONS="false"
+  - rvm: 2.4.1
+    env: PUPPET_GEM_VERSION="~> 5.1.0" STDLIB_LOG_DEPRECATIONS="false"
+  - rvm: 2.1.9
+    env: PUPPET_GEM_VERSION="~> 4"
+  - rvm: 2.4.1
+    env: PUPPET_GEM_VERSION="~> 5" STDLIB_LOG_DEPRECATIONS="false"
+  allow_failures:
+  - rvm: 2.1.9
+    env: PUPPET_GEM_VERSION="~> 4.6.0"
+
+notifications:
+  email: false

CODE_OF_CONDUCT.md

@@ -0,0 +1,46 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at info@example42.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

Dangerfile

@@ -15,7 +15,7 @@ end
 
 # Tests changes without code changes
 if !has_puppet_changes && has_spec_changes
-  message('Changes in tests but not in manifests. If the affect tests resulat, in theory this should not happen.', sticky: false)
+  message('Changes in tests but not in manifests. Unless you are improving your tests, this should not happen.', sticky: false)
 end
 
 # Hiera changes

Puppetfile

@@ -8,13 +8,8 @@
 # From Forge
 mod 'example42/tp', :latest
 mod 'example42/tinydata', :latest
-# mod 'example42/psick', :latest
-mod 'example42/psick',
-  :git => 'https://github.com/example42/puppet-psick'
+mod 'example42/psick', :latest
 mod 'example42/network', :latest
-mod 'example42/puppet',
-  :git => 'https://github.com/example42/puppet-puppet',
-  :branch => 'master'
 
 # Third Party modules
 mod 'puppetlabs/concat', '3.0.0' # postgresql requires concat < 3.0.0
@@ -31,9 +26,10 @@ mod 'puppetlabs/inifile', :latest
 # Used by psick::puppet::foss_server
 mod 'puppetlabs/postgresql', :latest
 mod 'puppetlabs/puppetdb', :latest
-mod 'puppet/puppetserver',
-  :git => 'https://github.com/voxpupuli/puppet-puppetserver.git',
-  :tag => '2.1.0'
+mod 'camptocamp/puppetserver', '2.1.0'
+#mod 'puppet/puppetserver',
+#  :git => 'https://github.com/voxpupuli/puppet-puppetserver.git',
+#  :tag => '2.1.0'
 mod 'puppetlabs/puppetserver_gem', :latest
 mod 'puppet/r10k', :latest
 mod 'puppetlabs/hocon', :latest
@@ -44,10 +40,10 @@ mod 'puppet/puppetboard', :latest
 mod 'pltraining/rbac', '0.0.5'
 
 # Used by grafanadash server (puppet metrics)
-mod 'grafanadash',
-  :git => 'https://github.com/tuxmea/puppet-grafanadash.git',
-  :branch => 'master'
-mod 'dwerder/graphite', :latest
+#mod 'grafanadash',
+#  :git => 'https://github.com/tuxmea/puppet-grafanadash.git',
+#  :branch => 'master'
+#mod 'dwerder/graphite', :latest
 
 # Docker and Containers
 mod 'puppetlabs/dummy_service', :latest

bin/gitlab_before.sh

@@ -3,6 +3,7 @@ repo_dir="$(dirname $0)/.."
 script_dir="$(dirname $0)"
 # repo_dir=$(git rev-parse --show-toplevel)
 . "${script_dir}/functions"
+PATH=/opt/puppetlabs/puppet/bin:$PATH
 git_branch=${1:-integration}
 default_branch="production"
 ci=$(echo $0 | sed 's/_before\.sh//g' | sed 's/^bin\///g')
@@ -26,10 +27,11 @@ git config --add remote.origin.fetch +refs/heads/$default_branch:refs/remotes/or
 git fetch --no-tags
 diff_commits_number=$(git log origin/$default_branch..HEAD --pretty=oneline | wc -l)
 echo "Deploying modules via r10k if Puppetfile has changed in the last ${diff_commits_number} commits"
-if [ $(git diff HEAD~$diff_commits_number --name-only | wc -l) > 0 ]; then
+commits=$(git diff HEAD~$diff_commits_number --name-only | wc -l)
+if [ $commits -gt 0 ]; then
   changedfiles=$(git diff HEAD~$diff_commits_number --name-only | grep Puppetfile);
   if [ "x$changedfiles" == "xPuppetfile" ] || [ ! -d "${repo_dir}/modules/stdlib" ] ; then
     echo_title "Installing modules defined in Puppetfile via r10k"
-    /opt/puppetlabs/puppet/bin/r10k puppetfile install -v ${config}
+    r10k puppetfile install -v ${config}
   fi
 fi

bin/puppet_ci.sh

@@ -53,7 +53,7 @@ run_action () {
     db_query)
       $ssh_command
       certname=$(puppet config print certname)
-      puppet-query  --cacert=/etc/puppetlabs/puppet/ssl/certs/ca.pem --cert=/etc/puppetlabs/puppet/ssl/certs/$certname.pem --key=/etc/puppetlabs/puppet/ssl/private_keys/$certname.pem "nodes[certname] { latest_report_status = 'failed' and expired is null and catalog_environment = '${env}' }" | grep $n
+      $sudo_command puppet-query  --cacert=/etc/puppetlabs/puppet/ssl/certs/ca.pem --cert=/etc/puppetlabs/puppet/ssl/certs/$certname.pem --key=/etc/puppetlabs/puppet/ssl/private_keys/$certname.pem "nodes[certname] { latest_report_status = 'failed' and expired is null and catalog_environment = '${env}' }" | grep $n
       if [ "x$?" == "x0" ]; then
         echo "Node ${n} last run has failed!"
         exit 1
@@ -65,19 +65,17 @@ run_action () {
     ;;
     task_run)
       $ssh_command
-      bolt task run $task environment=$env -n $n
+      $sudo_command bolt task run $task environment=$env -n $n
       $ssh_command_post
     ;;
     git_deploy)
       $ssh_command
-      cd /etc/puppetlabs/code/environments/$env/
-      git pull origin $env
-      r10k puppetfile install
+      $sudo_command "cd /etc/puppetlabs/code/environments/${env}/ ; git pull origin ${env} ; r10k puppetfile install"
       $ssh_command_post
     ;;
     r10k_deploy)
       $ssh_command
-      r10k deploy environment $env
+      $sudo_command /opt/puppetlabs/puppet/bin/r10k deploy environment ${env} -v
       $ssh_command_post
     ;;
     *)

bin/puppet_deploy_controlrepo.sh

@@ -1,13 +1,10 @@
 #!/bin/bash
 script_dir="$(dirname $0)"
-. "${script_dir}/functions"
-backup_dir="/etc/puppetlabs/code_backup"
-backup_date=$(date +'%Y%m%d-%H%M%S')
-environment=${1:-production}
+# . "${script_dir}/functions"
 PATH=$PATH:/usr/local/bin
+control_repo=${1:-"https://github.com/example42/psick.git"}
 
-echo_title "Installing prerequisite packages and gems"
-
+echo "Installing prerequisite packages and gems"
 puppet resource package git ensure=present
 
 if [ $(facter osfamily) == 'Debian' ]; then
@@ -21,22 +18,17 @@ puppet resource package hiera-eyaml ensure=present provider=gem
 puppet resource package r10k ensure=present provider=gem
 
 mkdir -p /etc/puppetlabs/code/environments/
+mkdir -p /etc/puppetlabs/r10k/
+cat > /etc/puppetlabs/r10k/r10k.yaml << EOF
+---
+:postrun: []
+:cachedir: /opt/puppetlabs/puppet/cache/r10k
+:sources:
+  puppet:
+    basedir: /etc/puppetlabs/code/environments
+    remote: $control_repo
+EOF
+
+echo "Running r10k deploy environment -v"
+r10k deploy environment -v
 
-if [ -d /etc/puppetlabs/code/environments/$environment ]; then
-  ask_interactive "Directory /etc/puppetlabs/code/environments/$environment exists. We have to move it."
-  if [ "$?" = 0 ]; then
-    mkdir -p $backup_dir
-    mv /etc/puppetlabs/code/environments/$environment "${backup_dir}/${environment}-${backup_date}"
-    echo_subtitle "/etc/puppetlabs/code/environments/$environment moved to ${backup_dir}/${environment}-${backup_date}"
-  else
-    echo "Can't proceed. Remove /etc/puppetlabs/code/environments/$environment or pass as argument a different environment"
-    exit 1
-  fi
-fi
-echo title "Cloning git://github.com/example42/psick.git to /etc/puppetlabs/code/environments/$environment"
-git clone git://github.com/example42/psick.git /etc/puppetlabs/code/environments/$environment
-cd /etc/puppetlabs/code/environments/$environment
-echo_title "Running r10k puppetfile install -v"
-r10k puppetfile install -v
-echo_title "Moving /etc/puppetlabs/puppet/hiera.yaml to /etc/puppetlabs/puppet/hiera.yaml.orig"
-mv /etc/puppetlabs/puppet/hiera.yaml /etc/puppetlabs/puppet/hiera.yaml.orig

bin/puppet_deploy_environment.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+script_dir="$(dirname $0)"
+. "${script_dir}/functions"
+environment=${1:-production}
+PATH=/opt/puppetlabs/puppet/bin:$PATH
+
+if [ -d "/etc/puppetlabs/code/environments/${environment}/.git" ]; then
+  cd /etc/puppetlabs/code/environments/$environment
+  git pull origin $environment
+  r10k puppetfile install -v
+else
+  cd /etc/puppetlabs/code/environments/
+  git clone $origin $environment
+  r10k puppetfile install -v
+fi

bin/puppet_job_run.sh

@@ -10,13 +10,14 @@ default_nodes=${!default}
 always_nodes=${!always}
 nodes=0
 global_exit=0
+domain=$(facter domain)
 
 diff_commits_number=$(git log production..$1 --pretty=oneline | wc -l)
 
 echo "Checking for files in the last $diff_commits_number commits"
 for changedfile in $(git diff HEAD~$diff_commits_number --name-only); do
   node=''
-  if [[ $(echo "$changedfile" | grep -q 'hieradata/nodes'; echo $?) -eq 0 ]]; then
+  if [[ $(echo "$changedfile" | grep -q "^hieradata/nodes" | grep $domain ; echo $?) -eq 0 ]]; then
     node=$(echo $changedfile | sed -e "s/^hieradata\/nodes\///" -e "s/\.yaml//")
     nodes=$nodes+1
   fi
@@ -36,6 +37,8 @@ if [[ $nodes == 0 ]]; then
   echo_title "Running Puppet on nodes ${default_nodes} - Default nodes"
   puppet job run --nodes $default_nodes
   if [ $? != 0 ]; then
+    global_exit=1
+  fi
 fi
 
 echo_title "Running Puppet on nodes ${always_nodes} - Node always checked"

bin/puppet_set_env_facts.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --role)
+      export FACTER_role=$2
+      shift 2
+    ;;
+    --zone)
+      export FACTER_zone=$2
+      shift 2
+    ;;
+    --env)
+      export FACTER_env=$2
+      shift 2
+    ;;
+    --datacenter)
+      export FACTER_datacenter=$2
+      shift 2
+    ;;
+    --application)
+      export FACTER_application=$2
+      shift 2
+    ;;
+    *)
+      exit
+    ;;
+  esac
+done
+echo "### Setting facts in bash env: role=${role} env=${env} zone=${zone} datacenter=${datacenter} application=${application}"
+