Puppet cannot load the jerakia-client gem

[den-is]

Need help,
Can't find what am I missing.

CenotOS 7.4
puppet --version 5.3.3 - PE 2017.3.2
jerakia version 2.4.0 - installed from repo

This is completely development fresh setup for experimentation purposes.
After installing following all instructions from official jerakia site I'm still getting this error:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Puppet cannot load the jerakia-client gem on node pup.dev

I wasn't able to find obvious instruction about installing jerakia-client

# puppetserver gem install jerakia-client
Fetching: jerakia-client-1.0.0.gem (100%)
Successfully installed jerakia-client-1.0.0
1 gem installed

systemctl restart pe-puppetserver

next error

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, <h1>Internal Server Error</h1> at /etc/puppetlabs/code/environments/production/manifests/site.pp:32:3 on node pup.dev

and puppetserver.log
https://pastebin.com/gnG9DjAD

trying this:

# puppetserver gem install lookup_http
Fetching: lookup_http-1.0.3.gem (100%)
Successfully installed lookup_http-1.0.3
1 gem installed

But getting same error from above.

Dummy module for tests:

class testmod (
        $param,
) {
        file {'/root/test.txt':
                ensure => present,
                content => "${param}"
        }
}

site.pp

node default {
  #require testmod
  class {'testmod': }
}

# ss -tuln | grep 9843
tcp    LISTEN     0      100    127.0.0.1:9843                  *:*

systemctl status jerakia
● jerakia.service - Jerakia Server
   Loaded: loaded (/etc/systemd/system/jerakia.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-12-06 15:25:13 +04; 1h 43min ago
 Main PID: 1780 (ruby)
   CGroup: /system.slice/jerakia.service
           └─1780 ruby /opt/jerakia/bin/jerakia server
...

All above is just working without jerakia.
Jerakia is working fine by itself

[crayfishx]

Hi @den-is

Firstly, you are right, the Jerakia puppet module should include better docs on the pre-requisites for puppetserver like installing the jerakia-client gem, I'll get this into a new release.

Your stack trace indicates that Puppet is failing because jerakia-client is not getting a successful response form the server (https://github.com/crayfishx/jerakia-client/blob/master/lib/jerakia/client.rb#L98) - note that you don't need to install lookup_http into Puppet's ruby - the client does Net::HTTP natively.

I'm missing quite a lot of info here - can you also send your hiera.yaml and your jerakia.yaml configuration files?

I suspect that maybe hiera.yaml is misconfigured... but it's hard to say without seeing the contents....

Craig

[den-is]

hiera.yaml

---
version: 5
defaults:
hierarchy:

  - name: jerakia
    lookup_key: jerakia
    options:
        token: puppet:tokeeeen...0c08ea9cad3266df091d4c38a0c
        scope:
            env: "%{environment}"
  - name: "Common"
    paths:
      - "common.yaml"

jerakia.yaml

---

policydir: /etc/jerakia/policy.d
plugindir: /var/lib/jerakia/plugins
loglevel: debug
logfile: /var/log/jerakia/jerakia.log
databasedir: /var/db/jerakia

policy

policy :default do

        lookup :couchdb do
                datasource :http, {
                        :host   => "10.0.2.2",
                        :port   => 5984,
                        :output => "json",
                        :paths  => [
                                "/config/#{scope[:env]}",
                                "/config/global",
                                "/config/app",
                        ]
                }
        end

end

By itself jerakia is working fine from CLI.
Nothing hits jerakia.log when puppet trying to do lookup and fails.

[crayfishx]

It's not not sending back a known failure, probably a 500 judging by the line that raises the exception... can you test this from the command line against the API.....

curl -X GET -H 'X-Authentication: puppet:tokeeeen...0c08ea9cad3266df091d4c38a0c' \
'http://localhost:9843/v1/lookup/param?namespace=testmod&metadata_env=production'

(assuming that you're puppet environment is production)

Can you see what that gives you?

[den-is]

First of all: good to know such troubleshooting URL.

I found issue, but actually it's not completely my issue.
The thing is that yea, default puppet environment where I do classification is production.
But my couchdb has only prod - actually it was done intentionally.
So why jerakia has failed with internal server error:

curl -X GET -H 'X-Authentication: puppet:62db180a7ff8740fc419690d62be8e7037aa7e981e0d5ac69556a0c08ea9cad3266df091d4c38a0c' \
> 'http://localhost:9843/v1/lookup/param?namespace=testmod&metadata_env=production'
<h1>Internal Server Error</h1>

As by policy it had to failover to /config/global where I actually added k/v param.
hiera-http does it normally.

After I've added production document (or just altered metadata to have prod), everything worked:

curl -X GET -H 'X-Authentication: puppet:62db180a7ff8740fc419690d62be8e7037aa7e981e0d5ac69556a0c08ea9cad3266df091d4c38a0c' 'http://localhost:9843/v1/lookup/param?namespace=testmod&metadata_env=production'
{"status":"ok","payload":"from couchdb/config/production"}

[dlouks]

For others that may be looking into Jerakia, but hitting this error. See https://tickets.puppetlabs.com/browse/SERVER-571 for more background as to why the jerakia-client currently needs to be installed twice.

Install jerakia-client gem for ruby (puppet apply)
/opt/puppetlabs/puppet/bin/gem install jerakia-client

Install jerakia-client gem for jRuby (puppetserver)
puppetserver gem install jerakia-client

I was following https://www.craigdunn.org/2017/04/managing-puppet-secrets-with-jerakia-and-vault/ and http://jerakia.io/integration/puppet

System details

Centos 7.4
Puppet 5.4.0
Jerakia 2.5.0
jerakia-client 1.0.0