upgrading to newer version of sensu with newer embeded ruby doesn't reinstall plugins

[bchaja]

We are using embedded ruby when installing newer version of sensu with newer embedded ruby the plugins not being reinstalled making them unavailable under upgraded version (also we are using provider sensu_gem). 2 puppet run fixes the issue. All plugins are defined in hiera as:

sensu_gems:
sensu-plugins-cpu-checks: {}

[dmizelle]

We have also run into this issue, and have to solve it by running puppet again. Is there some way to perhaps create a subscribe parameter on each plugin to 'refresh'/reinstall after the version of sensu changes?

If we update sensu-client right now in our environment, all checks go critical/unknown, which makes us get paged.

[jaxxstorm]

I don't use this myself, can you provide me some examples of how you're using this?

[rowleyaj]

  # When updating sensu we found that the sensu_gem installed plugins would take
  # a second puppet run to install. This ensures that when the Sensu package
  # changes, all sensu_gem installed packages will be notified, and then notifies
  # the sensu services (if installed)
  Package['sensu'] ~>
    Package<| provider == 'sensu_gem' |> ~>
      Service['sensu-client', 'sensu-server', 'sensu-api']

We added the above to our profile to work around this. There is still a brief period of time where a check may run and error, but it is better than waiting for 30 minutes for the next puppet run.

These kind of dependencies could be captured in the module using the above, there may be others that can be resolved this way too, for example #463

[glarizza]

Hopping on this ticket to replicate and see what I can update - is this still a problem for all involved with the recent module changes (presuming so but asking just to be sure). Does anyone have any code they're using to replicate this issue (I'm presuming install a specific version with all plugins, specify a new version, and then observe)?

[ghoneycutt]

@agoddard Any thoughts on how we can reproduce? Should the sensu package itself handle this so gems with the old version are attempted to be reinstalled?

[glarizza]

As I understand it, plugins to Sensu's embedded ruby installation are installed via package resources with the sensu_gem provider included in this module:

package { 'redphone':
  ensure   => 'installed',
  provider => sensu_gem,
}

package { 'sensu-plugins-nginx':
  ensure   => 'installed',
  provider => sensu_gem,
}

package { 'sensu-plugins-disk-checks':
  ensure   => 'installed',
  provider => sensu_gem,
}

The sensu_gem provider installs into /opt/sensu/embedded, and the version that I pulled up has gems installed within /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems because the current version of Sensu uses Ruby MRI 2.4 . Problems arise when you upgrade Sensu from an older version with an older Ruby MRI installed - the package drops down a new Ruby MRI version folder and the embedded gem binary uses the new folder for plugins/gems.

As stated above, there needs to be a mechanism for when the 'sensu' package changes that the package resources installing plugins into the embedded rubygems path are notified so they can do their installation checks AFTER Sensu is updated.

We can't set an "autonotify" from the plugin resources for the Sensu services because that's implemented in the core Package Type (and while we don't want EVERY package resource to notify the Sensu services Puppet doesn't give us a mechanism to confine it to only those package resources using the sensu_gem provider).

Like @rowleyaj said above, you could implement the following within your organization's Sensu Enterprise profile using collectors and the resource chain syntax:

Package['sensu'] ~>
Package<| provider == 'sensu_gem' |> ~>
Service['sensu-client', 'sensu-server', 'sensu-api']

That code would setup notify dependencies on the 'sensu' package to notify all Package resources using the sensu_gem provider, and then setup notify dependencies on all those package resources to notify the Sensu services if changes were made. This works, but is that something we want to drop right into the main Sensu module? The biggest concern would be whether this bit of magic would cause unexpected circular dependencies, but I feel like it's confined pretty well (limited to 4 specific resources that are declared within this module and any plugins declared using the provider within this module).

I'm going to replicate this and play around with it a bit more, but I'm leaning toward just accepting that code into the module since this is fundamentally a dependency issue and that's the "least magical" solution that actually works.

[ghoneycutt]

+1 to this approach. Excited to hear how it works out with your functional testing.

[glarizza]

Okay, found a way to test this out. First spin up a Sensu server with vagrant up sensu-server and then SSH into the box. The sensu module is symlinked into Puppet's module directory, so any changes made to /vagrant will get picked up by Puppet. Next, I run puppet apply on this manifest:

$deregistration = { 'handler' => 'deregister_client' }

package { 'redphone':
  ensure   => 'installed',
  provider => sensu_gem,
}

package { 'sensu-plugins-nginx':
  ensure   => 'installed',
  provider => sensu_gem,
}

package { 'sensu-plugins-disk-checks':
  ensure   => 'installed',
  provider => sensu_gem,
}

class { '::sensu':
  install_repo          => true,
  #version               => '0.23.0',
  version               => 'latest',
  server                => true,
  manage_services       => true,
  manage_user           => true,
  rabbitmq_password     => 'correct-horse-battery-staple',
  rabbitmq_vhost        => '/sensu',
  spawn_limit           => 16,
  api                   => true,
  api_user              => 'admin',
  api_password          => 'secret',
  client_address        => $::ipaddress_eth1,
  subscriptions         => ['all', 'roundrobin:poller'],
  client_deregister     => true,
  client_deregistration => $deregistration,
}

This will install 3 plugins into Sensu's embedded ruby (which was MRI 2.4 as of this writing). Next, uncomment the line that reads version => '0.23.0', and comment out the version => 'latest' line. Sensu core 0.23.0 uses Ruby MRI 2.3, so that will guarantee us a new Ruby folder. Finally, run puppet apply on the manifest again. What you'll notice is that because there aren't any dependencies on the 3 package declarations for the plugins, they get evaluated in manifest order (i.e. before the sensu class) and nothing is changed with them. Sensu itself gets downgraded, and then there are service errors about it starting (most likely because of the significant downgrade, but that's irrelevant to what we're testing). Running Puppet a second time will install the plugins into the new Ruby folder and expose the issue we're having.

Next, I destroyed the Vagrant environment, added the three lines from my previous comment into the main init.pp file for the Sensu class, and then repeated the process. This time the plugins were evaluated after the package was downgraded, and a second run of Puppet was unnecessary.

I'm going to file a PR against this and then move it along.

[ghoneycutt]

Great work!

Could we have a simple shell script that does what you are doing and tests that output of the last puppet apply? That would give a quick way to functionally test this in the future to ensure we don't have regressions.

[agoddard]

Sorry, I thought I’d commented the other day but I guess I’d left it w/out submitting it - the way we’ve seen this done before is to have the plugin installation tasks run after the package installer, wherein it checks the embedded gem’s installed gems, and any that are missing are installed. This works on the first install and if gems are uninstalled manually on the box, as well as if the embedded ruby is upgraded, completely separate from any interaction with the package itself. Would this approach work with Puppet? Basically it would just check /opt/sensu/embedded/bin/gem to see if a gem is installed, and if it’s missing, it will resintall it.

[glarizza]

@ghoneycutt The problem is that because those service resources fail to start on the downgrade the exit code reflects that. I can write up the script so a human can review it for regressions, but automating that check will be hard because of the exit code. A Beaker test might be better if you want to automate that check, and I'll look at that next

[ghoneycutt]

@glarizza the existing beaker tests are very outdated and should be removed, don't use those.

[glarizza]

@ghoneycutt Good to know, I'll focus my efforts on the script

[glarizza]

@agoddard What you wrote is the same angle I ended up taking, so that works out. Can you tell me if the sensu-enterprise service (or anything on the Enterprise side) needs to be refreshed on plugin changes or if the client/server/api are the only services that would need refreshed? Basically, do we have to do anything extra on the enterprise side that I haven't already done?

[glarizza]

@ghoneycutt Validation script was added in ebeeeef and is on the PR

[ghoneycutt]

Released in v2.46.0