puppetserver_gem failing on FOSS master

[antgel]

Hi, I get the following errors. Note the weird two spaces between the words "file" and "does". If I run bin/papply.sh again, everything works.

==> default: Debug: Puppet::Type::Package::ProviderPuppetserver_gem: file  does not exist
==> default: Debug: Puppet::Type::Package::ProviderPuppetserver_gem: file  does not exist
==> default: Debug: Puppet::Type::Package::ProviderPuppetserver_gem: file  does not exist
==> default: Error: /Package[puppetserver_r10k]: Provider puppetserver_gem is not functional on this host
==> default: Error: /Package[puppetserver_deep_merge]: Provider puppetserver_gem is not functional on this host
==> default: Error: /Package[puppetserver_hiera-eyaml]: Provider puppetserver_gem is not functional on this host

I have noticed that if I move puppetserver/lib/puppet/provider/package/puppetserver_gem.rb out of the way, it seems to work first time, but I'm not sure if I'm breaking something else by doing that. Any clues?

[tuxmea]

Just to ensure: the error occurs on the first run of bin/papply.sh and succeeds on the second run?

[antgel]

That's correct.

[tuxmea]

Are you bootstrapping a puppet server? Usually these gems are installed by the bin/puppet_setup.sh script. Have you adopted the hiera settings for your puppet server?

[antgel]

Indeed I am booting a puppet server (hence the title ;). After the first run, the three gems have been installed, but by /usr/bin/gem and /opt/puppetlabs/puppet/bin/gem, not by /opt/puppetlabs/bin/puppetserver gem. In addition, the puppetserver process does come up (not sure how sane it is at that point), but puppet finishes with errors which never looks good...

Check this out:

ubuntu@puppet:~$ /usr/bin/gem list

*** LOCAL GEMS ***

bigdecimal (1.2.8)
colored (1.2)
cri (2.6.1)
deep_merge (1.1.1)
did_you_mean (1.0.0)
faraday (0.9.2)
faraday_middleware (0.10.1)
fast_gettext (1.1.0)
gettext (3.2.4)
gettext-setup (0.28)
hiera-eyaml (2.1.0)
highline (1.6.21)
io-console (0.4.5)
json (1.8.3)
locale (2.1.2)
log4r (1.1.10)
minitar (0.6.1)
minitest (5.8.4)
multi_json (1.12.2)
multipart-post (2.0.0)
net-telnet (0.1.1)
power_assert (0.2.7)
psych (2.0.17)
puppet_forge (2.2.6)
r10k (2.5.5)
rake (10.5.0)
rdoc (4.2.1)
semantic_puppet (0.1.4)
test-unit (3.1.7)
text (1.3.1)
trollop (2.1.2)
ubuntu@puppet:~$ /opt/puppetlabs/puppet/bin/gem list

*** LOCAL GEMS ***

bigdecimal (1.2.4)
colored (1.2)
cri (2.6.1)
deep_merge (1.1.1, 1.0.1)
facter (3.6.6)
faraday (0.9.2)
faraday_middleware (0.10.1)
fast_gettext (1.1.0)
gettext (3.2.2)
gettext-setup (0.20)
hiera (3.3.2)
hiera-eyaml (2.1.0)
highline (1.6.21)
hocon (1.2.5)
io-console (0.4.3)
json (1.8.1)
locale (2.1.2)
log4r (1.1.10)
minitar (0.6.1)
minitest (4.7.5)
multi_json (1.12.2)
multipart-post (2.0.0)
net-ssh (4.1.0)
psych (2.0.5)
puppet (4.10.6)
puppet_forge (2.2.6)
r10k (2.5.5)
rake (10.1.0)
rdoc (4.1.0)
semantic_puppet (0.1.2)
stomp (1.3.3)
test-unit (2.1.9.0)
text (1.3.1)
trollop (2.1.2)
ubuntu@puppet:~$ sudo /opt/puppetlabs/bin/puppetserver gem list

*** LOCAL GEMS ***

fast_gettext (1.1.0)
gettext (3.2.2)
gettext-setup (0.26)
hocon (1.1.3)
jar-dependencies (0.2.6)
jruby-openssl (0.9.16 java)
json (1.8.0 java)
locale (2.1.2)
rake (10.1.0)
rdoc (4.1.2)
semantic_puppet (0.1.3)
text (1.3.1)
ubuntu@puppet:~$

After running bin/papply.sh a second time:

ubuntu@puppet:~$ sudo /opt/puppetlabs/bin/puppetserver gem list

*** LOCAL GEMS ***

colored (1.2)
cri (2.6.1)
deep_merge (1.1.1)
faraday (0.9.2)
faraday_middleware (0.10.1)
fast_gettext (1.1.0)
gettext (3.2.2)
gettext-setup (0.26)
hiera-eyaml (2.1.0)
highline (1.6.21)
hocon (1.1.3)
jar-dependencies (0.2.6)
jruby-openssl (0.9.16 java)
json (1.8.0 java)
locale (2.1.2)
log4r (1.1.10)
minitar (0.6.1)
multi_json (1.12.2)
multipart-post (2.0.0)
puppet_forge (2.2.6)
r10k (2.5.5)
rake (10.1.0)
rdoc (4.1.2)
semantic_puppet (0.1.3)
text (1.3.1)
trollop (2.1.2)
ubuntu@puppet:~$

[tuxmea]

Please try the following:

1. create a new role: hieradata/role/puppetserver.yaml

---
profiles:
  - profile::puppet::gems
  - profile::puppet::foss_master
profile::puppet::gems::install_puppetserver_gems: true

2. run FACTER_role=puppetserver bin/papply.sh

I suppose that at the first puppet apply run the trusted.certname is not yet set and therefor the node yaml data are not pulled in.

[antgel]

No dice. I do see something different in the log, where before Role was blank, but the same behaviour occurs. I also tried FACTER_roles=puppetserver (note the extra 's' in roles), which had no effect.

Notice: /Stage[main]/Profile::Motd/File[/etc/issue]/content:
--- /etc/issue  2017-07-31 13:36:56.000000000 +0000
+++ /tmp/puppet-file20170905-4250-9yez2z        2017-09-05 10:53:53.167985855 +0000
@@ -1,2 +1,6 @@
-Ubuntu 16.04.3 LTS \n \l
-
+System Managed by Puppet
+Role       : puppetserver
+Environment:
+Datacenter :
+Zone       :
+Application:

[tuxmea]

OK. Thanks for helping with debugging.

One more try (I suppose it has to do with orders):

hieradata/role/puppetserver.yaml

profiles:
  - profile::puppet::foss_master
  - profile::puppet::gems
profile::puppet::gems::install_puppetserver_gems: true

Then run again: FACTER_role=puppetserver bin/papply.sh

Switch order of profile::puppet::foss_master and profile::puppet::gems
If that is not working, we must place the order inside the profiles.

[antgel]

Sorry to say, that didn't help either. For the record, I'm running:

bin/puppet_install.sh
bin/puppet_setup.sh auto
bin/puppet_install_puppetfile.sh                                  
export FACTER_role=puppetserver
bin/papply.sh --test --debug

I also tried without bin/puppet_install_puppetfile.sh, no change. Please let me know if there's anything else I can try to help debug this.

[tuxmea]

You don't need to run bin/puppet_install_puppetfile.sh this is already done during bin/puppet_setup.sh auto

Please change bin/papply.sh:

  7 manifest="${repo_dir}/manifests/site.pp"
  8 extra_options=$*
  9
 10 PATH=$PATH:/opt/puppetlabs/puppet/bin:/opt/puppetlabs/bin
 11
 12 echo_title "Running Puppet version $(puppet --version) apply on ${manifest}"
 13 echo_subtitle "Role: ${FACTER_role} - $(facter -p role)"

See line 10, add /opt/puppetlabs/bin to PATH.
The puppetserver_gem uses which puppetserver.

[antgel]

Nice idea - I'd seen that last night when I was mv'ing puppetserver_gem.rb out of the way, I didn't put two and two together. However, adding /opt/puppetlabs/bin to the path didn't help. I tried it with both profile orders, and I also tried exporting PATH. :(

NB I don't see that puppet_setup.sh auto runs puppet_install_puppetfile.sh anywhere:

16:59 $ git grep puppet_install_puppetfile\.sh
docs/FOSS_puppet_server.md:    ./bin/puppet_install_puppetfile.sh
hieradata/nodes/cirunner.demo.yaml:    bootstrap_script: 'bin/puppet_install_puppetfile.sh'
hieradata/nodes/cirunner.lab.psick.io.yaml:    bootstrap_script: 'bin/puppet_install_puppetfile.sh'
site/profile/manifests/ci/octocatalog.pp:    'bootstrap_script' => 'bin/puppet_install_puppetfile.sh',

[tuxmea]

puppet_setup.sh auto has a shell function inside which runs r10k puppetfile install

Can we try one more thing (I doubt that this is working):

In site/profile/manifests/puppet/gems.pp add line 34:

 28     if $install_puppetserver_gems {
 29       package { "puppetserver_${gem}":
 30         ensure          => $ensure,
 31         name            => $gem,
 32         install_options => $install_options,
 33         provider        => 'puppetserver_gem',
 34         require         => Package['puppetserver'],
 35       }
 36     }

[antgel]

As you suggested, it didn't work. It does work when I use Terraform to create a puppet master in AWS, as opposed to doing it on Vagrant. So strange. Here's my Vagrantfile:

Vagrant.configure(2) do |config|
  config.vm.box = 'ubuntu/xenial64'
  config.vm.provider 'virtualbox' do |v|
    # Puppet master needs that much
    v.memory = 3072
  end
  config.vm.provision "shell",
    path: 'bootstrap-puppet-master.sh'
end

The executed parts of bootstrap-puppet-master.sh:

#!/bin/bash
set -e

. /etc/lsb-release

export DEBIAN_FRONTEND=noninteractive

install_system_packages() {
    apt-get install -yq silversearcher-ag vim
}

install_puppet() {
    git_root=~ubuntu/src/git/psick/environments
    control_repo_dir="${git_root}"/production
    code_dir="/etc/puppetlabs/code/environments/production"
    control_repo=https://github.com/graduway/psick
    if [ ! -d "${control_repo_dir}"/.git ];
    then
        sudo -u ubuntu mkdir -p "${git_root}"
        export GIT_DIR=~ubuntu/.git
        sudo -E -u ubuntu git clone "${control_repo}" "${control_repo_dir}"
        mkdir -p "${code_dir}"
        ln -s "${control_repo_dir}" "${code_dir}"
        pushd "${control_repo_dir}"
        bash -x bin/puppet_install.sh
        bash -x bin/puppet_setup.sh auto
        export FACTER_role=puppetserver
        bash -x bin/papply.sh --test --debug
        popd
    fi
}

install_system_packages
install_puppet

[tuxmea]

There is one last thing we can test: install puppetserver package prior running papply.sh.
I assume that the provider checks for the puppetserver binary at puppet apply startup only.

[tuxmea]

Can you provide your terraform snippet?

[antgel]

Indeed, installing puppetserver first solved the issue! Do you regard this as a "proper" fix, or is there something else to do?

Terraform is pretty boring, like this:

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region     = "${var.region}"
}

resource "aws_instance" "puppet" {
  ami            = "ami-1d4e7a66"
  key_name = "antony-us-east-1"
  instance_type = "t2.medium"
  tags {
    Name = "puppet"
  }
	vpc_security_group_ids = ["sg-0e7d297e"]

  provisioner "file" {
    source = "bootstrap-puppet-master.sh"
    destination = "/tmp/bootstrap-puppet-master.sh"

    connection {
      user = "ubuntu"
    }
  }

  provisioner "remote-exec" {
    inline = [
      "sudo bash -x /tmp/bootstrap-puppet-master.sh"
    ]

    connection {
      user = "ubuntu"
    }
  }
}

[tuxmea]

This has to do with provider pre-fetching. On initial puppet apply run the provider checks its functionality and marks itself as non-functional. The provider remains in this state during the whole run, even when the required dependency is installed during the run.
On second run the provider is functional as puppetserver binary is now available.
Maybe we should mention that bootstrapping puppetserver requires papply.sh to run twice.

[antgel]

Mention it where? Also, it's not great that the first run leaves errors. For example, it makes Terraform mark the resource as tainted unless a flag is sent to ignore that, which may hide other failures.

[antgel]

@tuxmea In our use case, it's not just that "bootstrapping puppetserver requires papply.sh to run twice.", it requires:

apt-get -qy install puppetserver
bin/puppet_setup.sh auto  # Make sure relevant gems are installed
export FACTER_role=puppet_foss_master
bin/papply.sh  # Firstrun
bin/papply.sh  # Everything else, including starting puppetserver

Hope that is useful, should we leave this open until it's documented (or fixed if you consider this a bug)?

[tuxmea]

I would like to keep this open. There are two ways how to deal with this issue:

1. have a special shell script for the puppet server (which installs the puppetserver package
2. update existing shell scripts and catch the error on the first puppet run in the shell script.