Puppetfiles are a simple Ruby based DSL that specifies a list of modules to install, what version to install, and where to fetch them from. r10k can use a Puppetfile to install a set of Puppet modules for local development, or they can be used with r10k environment deployments to install additional modules into a given environment.
Unlike librarian-puppet, the r10k implementation of Puppetfiles does not include dependency resolution, but it is on the roadmap.
When directly working with Puppetfiles, you can use the r10k puppetfile
subcommand to interact with a Puppetfile.
When using r10k's deploy functionality, interacting with Puppetfiles is handled on a case by case basis.
Because the Puppetfile format is actually implemented using a Ruby DSL any valid Ruby expression can be used. That being said, being a bit too creative in the DSL can lead to surprising (read: bad) things happening, so consider keeping it simple.
Puppetfile subcommands assume that the Puppetfile to operate on is in the current working directory and modules should be installed in the 'modules' directory relative to the current working directory.
Install or update all modules in a given Puppetfile into ./modules)
r10k puppetfile install
Verify the Puppetfile syntax
r10k puppetfile check
Remove any modules in the 'modules' directory that are not specified in the Puppetfile:
r10k puppetfile purge
The following settings can be used to control how the Puppetfile installs and handles modules.
The forge setting specifies which server that Forge based modules are fetched
from. This is currently a noop and is provided for compatibility with
librarian-puppet.
The moduledir setting specifies where modules from the Puppetfile will be
installed. This defaults to the modules directory relative to the Puppetfile.
If the path is absolute then the modules will be installed to that absolute
path, otherwise it's assumed that the moduledir setting should be relative and
the modules will be installed in that directory relative to the Puppetfile.
The moduledir setting should be placed before any modules are declared.
Install modules to an absolute path:
moduledir '/etc/puppet/modules'
mod 'branan/eight_hundred' # will be installed into '/etc/puppet/modules/eight_hundred'Install modules to a relative path:
moduledir 'thirdparty'
mod 'branan/eight_hundred' # will be installed into `dirname /path/to/Puppetfile`/thirdparty/eight_hundredNote: support for a relative moduledir was added in r10k 1.4.0; the behavior of a relative moduledir path is undefined on earlier versions of r10k.
r10k can install Puppet modules from a number of different sources. Right now modules can be installed from the Puppet Forge, Git, or SVN.
Modules can be installed from the Puppet Forge.
If no version is specified the latest version available at the time will be installed, and will be kept at that version.
mod 'puppetlabs/apache'
If a version is specified then that version will be installed.
mod 'puppetlabs/apache', '0.10.0'
If the version is set to :latest then the module will be always updated to the latest version available.
mod 'puppetlabs/apache', :latest
Git repositories that contain a Puppet module can be cloned and used as modules.
When Git is used, the module version can be specified by using :ref, :tag,
:commit, and :branch.
When a module is installed using :ref, r10k uses some simple heuristics to
determine the type of Git object that should be checked out. This can be used
with a git commit, branch reference, or a tag.
When a module is installed using :tag or :commit, r10k assumes that the
given object is a tag or commit and can do some optimizations around fetching
the object. If the tag or commit is already available r10k will skip network
operations when updating the repo, which can speed up install times.
Module versions can also be specified using :branch to track a specific
branch reference.
# Install puppetlabs/apache and keep it up to date with 'master'
mod 'apache',
:git => 'https://github.com/puppetlabs/puppetlabs-apache'
# Install puppetlabs/apache and track the 'docs_experiment' branch
mod 'apache',
:git => 'https://github.com/puppetlabs/puppetlabs-apache',
:ref => 'docs_experiment'
# Install puppetlabs/apache and pin to the '0.9.0' tag
mod 'apache',
:git => 'https://github.com/puppetlabs/puppetlabs-apache',
:tag => '0.9.0'
# Install puppetlabs/apache and pin to the '83401079' commit
mod 'apache',
:git => 'https://github.com/puppetlabs/puppetlabs-apache',
:commit => '83401079053dca11d61945bd9beef9ecf7576cbf'
# Install puppetlabs/apache and track the 'docs_experiment' branch
mod 'apache',
:git => 'https://github.com/puppetlabs/puppetlabs-apache',
:branch => 'docs_experiment'Since r10k 2.4.0, the :branch option can be set to the special value
:control_branch to indicate that the content should track a branch
reference matching the containing control repo branch. For example, if a
Puppetfile containing a Git content declaration is in the "testing" branch
of a control repo, a setting of :control_branch will attempt to deploy that
content from a "testing" branch of the content repo.
Additionally, you can specify a :default_branch option which is the branch
reference that content will be deployed from if the the given :ref, :tag,
:commit, or :branch option cannot be resolved and deployed. If the desired
content cannot be resolved and no default branch is given, or if the default
branch can also not be resolved, an error will be logged and the content will
not be deployed or updated.
# Deploy content from branch matching control repo branch.
mod 'hieradata',
:git => 'git@git.example.com:organization/hieradata.git',
:branch => :control_branch
# Track control branch and fall-back to master if no matching branch.
mod 'hieradata',
:git => 'git@git.example.com:organization/hieradata.git',
:branch => :control_branch,
:default_branch => 'master'Modules can be installed via SVN. If no version is given, the module will track the latest version available in the main SVN repository.
mod 'apache',
:svn => 'https://github.com/puppetlabs/puppetlabs-apache/trunk'
If an SVN revision number is specified with :rev (or :revision), that
SVN revision will be kept checked out.
mod 'apache',
:svn => 'https://github.com/puppetlabs/puppetlabs-apache/trunk',
:rev => '154'
mod 'apache',
:svn => 'https://github.com/puppetlabs/puppetlabs-apache/trunk',
:revision => '154'
If the SVN repository requires credentials, you can supply the :username and
:password options.
mod 'apache',
:svn => 'https://github.com/puppetlabs/puppetlabs-apache/trunk',
:username => 'azurediamond',
:password => 'hunter2'
Note: SVN credentials are passed as command line options, so the SVN credentials may be visible in the process table when r10k is running. If you choose to supply SVN credentials make sure that the system running r10k is appropriately secured.
In the event you want to store locally written modules in your r10k-managed
repository in the Puppetfile managed path, you can use the :local type.
For instance, if you have a Git repository with the following structure:
# tree -L 2
.
├── environment.conf
├── modules
│ └── local_module
└── Puppetfile
4 directories, 2 files
And you want to prevent local_module from being removed, you can add a 'local'
module in your Puppetfile:
mod 'local_module', :local => true
# Include your other modules as normal
mod 'branan/eight_hundred'
mod 'puppetlabs/apache'
If you run r10k against this Git branch, you'll get the following:
# tree -L 2
.
├── environment.conf
├── modules
│ ├── apache
│ ├── eight_hundred
│ └── local_module
└── Puppetfile
4 directories, 2 files
This is a workaround for r10k not being able to determine that modules created via VCS should not be purged, but is not meant to be a long term solution. The general practice around having local and remote modules in the same Git repository is to move modules versioned into a separate directory, like so:
# tree -L 2
.
├── environment.conf
├── site-modules
│ └── local_module
├── modules
│ ├── apache
│ └── eight_hundred
└── Puppetfile
4 directories, 2 files
Moving modules stored in the Git repository into a separate directory will remove the need to have Puppetfile entries for every locally versioned Puppet module.
For more information see the FAQ entry on managing internal and external modules in the same directory.
Git and SVN content types support installing into an alternate path without changing the value of moduledir by specifying an 'install_path' option:
# This will install the 'apache' module into 'external/apache'.
mod 'apache',
:git => 'git@github.com:puppetlabs/puppetlabs-apache.git',
:install_path => 'external'
The given 'install_path' can be an absolute path or a path relative to the base of the environment. Note that r10k will exit with an error if you attempt to set the 'path' option to a directory outside of the environment.
When using r10k, Git branches correspond to Puppet environments. For small teams using trunk-based development, with one or maybe two deployment environments, it's reasonable to incorporate and deploy changes using standard git merge workflows.
In large enterprise organizations with more complex deployment requirements, however, it may become untenable to try and manage change deployments across more than two environments using only git merge. For example, consider the following ordered tasks:
- Deploy change A to
devint - Deploy change B to
devint - Deploy change A to
staging - Deploy change B to
staging - Deploy change B to
production - Deploy change C to
devint - Deploy change C to
staging - Deploy change C to
production - Deploy change A to
production
This example uses only three deployment environments, and three changes. While it's not impossible to manage this kind of change decoulping in Git, it is prohibitively difficult. Furthermore, when this is managed entirely with Git merging, it can become difficult to look at any individual environment and understand which changes have been deployed to it.
A trunk-based deployment workflow is considered a better practice and results in a simpler r10k workflow. Sometimes it's not possible to achieve that though, or getting to trunk-based deployment may take some time.
R10k's Puppetfile.<BRANCH> feature exists to help address complicated deployment use cases that are prohibitively difficult to manage with Git merging.
Consider a use case with three deployment environments:
- devint
- staging
- production
Choose a trunk branch. The Puppetfile.<BRANCH> pattern extends the normal Git/r10k use case. In this example, we'll use the traditional git "master" branch as our trunk.
Development of features occurs as per normal in the master branch. Its contents look something like this:
(master)
├── Puppetfile
├── environment.conf
├── hiera.yaml
└── site-modules
├── profile
└── role
Perform normal development against this branch. It does not necessarily represent an important Puppet environment, though of course it is possible to perform canary agent runs against it using puppet agent -t --environment=master.
The Puppetfile in this branch should list all of the modules deployed which do not need decoupled change deployment.
Suppose that Team C owns a Puppet module, team_c, which needs changes deployed at an odd cadence across the three deployment environments. The Puppetfile should list modules such as ntp, stdlib, and so forth, but it should not list Team C's module.
For example, the Puppetfile may look something like this:
# Puppetfile
mod 'puppetlabs-ntp', '6.0.0'
In order to enable decoupled change deployment, Team C's module will be defined independently per-environment in the Puppetfile.<BRANCH>.
Because Git branches correspond to Puppet environments, there is a Git branch for devint, staging, and production. However, these branches are special. No merging is ever performed on these branches, and they can only contain a single file. Here's what these branches look like:
(devint)
└── Puppetfile.DEVINT
(staging)
└── Puppetfile.STAGING
(production)
└── Puppetfile.PRODUCTION
The Puppetfile.<BRANCH> is metadata that tells r10k how to deploy the environment. It configures two things:
- Which version of Puppet code to deploy from the trunk branch
- Which decoupled modules to deploy at which versions
The Puppetfile.DEVINT file may look something like this:
# Puppetfile.DEVINT
environment 'devint',
:branch => 'master'
mod 'team_c',
:git => 'git:///team_c.git',
:ref => 'v22'
The Puppetfile.PRODUCTION may look something like this:
# Puppetfile.PRODUCTION
environment 'production',
:tag => 'release-302'
mod 'team_c',
:git => 'git:///team_c.git',
:ref => 'v21'
When r10k deploys one of these environments, this is what the Puppet environment will look like on the master (using devint as the example):
/etc/puppetlabs/code/environments/devint
├── Puppetfile
├── Puppetfile.DEVINT
├── environment.conf
├── hiera.yaml
├── modules
│ ├── ntp
│ └── team_c
└── site-modules
├── profile
└── role
Note that modules from the Puppetfile AND modules from the Puppetfile.<BRANCH> are deployed.
To deploy changes to an environment defined using Puppetfile.<BRANCH>, edit the Puppetfile.<BRANCH> file (in its corresponding Git branch).
To deploy a module release, edit the appropriate mod directive with the version to deploy.
To deploy an environment code release, edit the environment directive with the version to deploy.
A manual deployment of the named environment may be performed at any time in the normal way (e.g. r10k deploy environment devint). If any references in the Puppetfile point to a branch this will result in the environment being updated to reflect any changes in referenced branch(es) since the last deploy.
To use the Puppetfile.<BRANCH> feature, a Git branch must:
- Contain one, and only one, committed file: Puppetfile.<BRANCH>
- The <BRANCH> component of the file name MUST be the upper-case name of the branch it is committed to
- The name given in the environment directive of Puppetfile.<BRANCH> MUST match the name of the branch it is committed to
The Puppetfile.<BRANCH> files should contain a special "environment" directive. This tells r10k which version of code to deploy to the named environment. The environment directive behaves similarly to how the mod directive behaves when using a Git source.
# Puppetfile.PRODUCTION
environment 'production',
:ref => '13.0.2'
The environment directive accepts two arguments.
As in a module, :ref may refer to a git tag, commit, or branch to deploy.
By default, environment assumes you want to check out a different ref in the control repo. Using the :git parameter, it is possible to link to and deploy an environment code version from a separate repository entirely. For example:
# Puppetfile.PRODUCTION
environment 'production',
:git => 'git://github.com/reidmv/puppet-control-repo.git',
:ref => 'b05b956'[root@master:~] # r10k deploy environment production -pv
INFO -> Using Puppetfile '/Users/reidmv/tmp/environments/production/Puppetfile'
INFO -> Deploying environment /etc/puppetlabs/code-staging/environments/production
INFO -> Environment production is now at b8fabbeac16374cfdeb5596c87edcedd144fafb9
INFO -> Puppetfile at b8fabbeac16374cfdeb5596c87edcedd144fafb9 redirects to git://github.com/reidmv/puppet-control-repo.git "b05b956"
INFO -> Environment production is now at b05b956a972358a8f649f66f0993c0f0bdce9c5a
INFO -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/site
INFO -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/stdlib
INFO -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/remote_file
INFO -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/concat
...
Users of this feature might implement a clean separation of development and stable deployment using separate r10k sources configured as follows.
---
cachedir: '/var/cache/r10k'
sources:
stable:
remote: 'git://github.com/reidmv/control-repo.git'
prefix: false
development:
remote: 'git://github.com/reidmv/puppet-environment.git'
prefix: 'git'The "stable" source above is a typical control repo, where branches correspond to named environments. Every branch in this repo though is a "link" to either a branch or a code version from the puppet-environment.git repo.
the "development" source, puppet-environment.git, has a typical git master branch, and feature branches for work in progress. Every branch in puppet-environment.git is available as a Puppet environment, but has the "git_*" prefix. For example, "git_master".
In the control-repo, an early environment such as development might be pinned to the master branch of puppet-environment.git.
#! Puppetfile
#=============================================================================
# ENVIRONMENT: development
#=============================================================================
environment 'development',
:git => 'git://github.com/reidmv/puppet-environment.git',
:ref => 'master'Higher tier environments are defined the same way, but likely pinned to commits or tags. E.g.
#! Puppetfile
#=============================================================================
# ENVIRONMENT: staging
#=============================================================================
environment 'staging',
:git => 'git://github.com/reidmv/puppet-environment.git',
:ref => '2.1'This pattern results in named environments without prefixes being defined and controlled in a deployment-centric workflow, while developers working on new features retain the same benefits they're used to; the main difference when compared to traditional development-only workflows is that Puppet environments that are deployed for development use are clearly identifyable by their "git_*" prefixes.