A Chef resource for mutual exclusion of blocks of recipe code. Useful for cross-cluster rolling restarts.
The easiest way to install this is to use knife-github-cookbooks:
gem install knife-github-cookbooks
knife cookbook github install websterclay/chef-dominodes
This cookbook implements a LWRP that exposes a new dominodes resource with
two actions: execute and release.
The dominodes resources expects a name and a recipe attribute, which is
expected to be a block of recipe code. The provided recipe block will never
be applied on more than one node in your cluster simultaneously.
dominodes 'rolling_apache_restarts'
recipe do
execute 'service apache2 restart'
end
action :nothing
end
template "/etc/www/configures-apache.conf" do
notifies :execute, "dominodes[rolling_apache_restarts]"
end
You can also notify the release action to release the lock used for mutual
exclusion if you need to for some reason:
notifies :release, "dominodes[rolling_apache_restarts]"
To facilitate the mutual execution of the embedded recipe, a data bag item is
created and used as a lock. The execution of the resource loops while
attempting to obtain the lock when it is in a free state. By default, the
recipe will give up on obtaining the lock after 1 hour. You can adjust this
behavior by setting the timeout directive to the desired number of seconds.
The recipe block inside a dominodes resources is essentially an in-line
Chef run. This means that the resources within this block cannot interact with
resources outside, e.g., manipulating notifies.
Here's my test recipe:
dominodes 'test_with_uptime' do
recipe do
execute 'hostname'
execute 'date'
execute 'sleep 10'
execute 'hostname'
execute 'date'
end
end
And here's how it looks while executing
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:34:59 +0000] INFO: execute[hostname] sh(hostname)
ec2-174-129-167-180.compute-1.amazonaws.com ip-10-117-33-108
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:34:59 +0000] INFO: execute[hostname] ran successfully
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:34:59 +0000] INFO: execute[date] sh(date)
ec2-174-129-167-180.compute-1.amazonaws.com Tue May 24 14:34:59 UTC 2011
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:34:59 +0000] INFO: execute[date] ran successfully
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:34:59 +0000] INFO: execute[sleep 10] sh(sleep 10)
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:09 +0000] INFO: execute[sleep 10] ran successfully
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:09 +0000] INFO: execute[hostname] sh(hostname)
ec2-174-129-167-180.compute-1.amazonaws.com ip-10-117-33-108
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:09 +0000] INFO: execute[hostname] ran successfully
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:09 +0000] INFO: execute[date] sh(date)
ec2-174-129-167-180.compute-1.amazonaws.com Tue May 24 14:35:09 UTC 2011
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:09 +0000] INFO: execute[date] ran successfully
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:10 +0000] INFO: Chef Run complete in 15.108392 seconds
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:10 +0000] INFO: Running report handlers
ec2-174-129-167-180.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:10 +0000] INFO: Report handlers complete
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:11 +0000] INFO: execute[hostname] sh(hostname)
ec2-50-19-47-63.compute-1.amazonaws.com domU-12-31-39-02-6E-F2
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:11 +0000] INFO: execute[hostname] ran successfully
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:11 +0000] INFO: execute[date] sh(date)
ec2-50-19-47-63.compute-1.amazonaws.com Tue May 24 14:35:11 UTC 2011
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:11 +0000] INFO: execute[date] ran successfully
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:11 +0000] INFO: execute[sleep 10] sh(sleep 10)
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:21 +0000] INFO: execute[sleep 10] ran successfully
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:21 +0000] INFO: execute[hostname] sh(hostname)
ec2-50-19-47-63.compute-1.amazonaws.com domU-12-31-39-02-6E-F2
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:21 +0000] INFO: execute[hostname] ran successfully
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:21 +0000] INFO: execute[date] sh(date)
ec2-50-19-47-63.compute-1.amazonaws.com Tue May 24 14:35:21 UTC 2011
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:21 +0000] INFO: execute[date] ran successfully
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:23 +0000] INFO: Chef Run complete in 28.427305 seconds
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:23 +0000] INFO: Running report handlers
ec2-50-19-47-63.compute-1.amazonaws.com [Tue, 24 May 2011 14:35:23 +0000] INFO: Report handlers complete
Locks should be cleaned up even if exceptions are raised in the critical section. However, if you somehow work yourself into a situation that requires clearing a lock manually, you can do so with knife:
knife data bag edit dominodes test_with_uptime
Jesse Newland
jesse@websterclay.com
@jnewland
jnewland on freenode
Author:: Jesse Newland (<jesse@websterclay.com>)
Copyright:: Copyright (c) 2011 Webster Clay, LLC
License:: Apache License, Version 2.0
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.