(CODEMGMT-1177) Add environment definition feature

doc/puppetfile.mkd

@@ -305,3 +305,142 @@ mod 'apache',
 The given 'install\_path' can be an absolute path or a path relative to the base of
 the environment. Note that r10k will exit with an error if you attempt to set the
 'path' option to a directory outside of the environment.
+
+## Environment
+
+When using a Git control repo, branches, and a Puppetfile, it is possible to "link" the branch to an environment version, rather than comitting code to the branch. This advanced feature may be used to aid in deployment workflows.
+
+Example: the "production" branch of a control repository should have version 3.2.0 of the puppet environment code deployed. Development of puppet code is happening constantly in the master branch, but when releases are ready they are tagged so that the code may be deployed to upper-tier environments. Ensuring that the version of code checked out to the production branch matches a given tag or commit is a process that requires a high level of Git knowledge. A simpler approach to controlling that deployment may be to make the production branch link to a code version instead.
+
+To link the production environment branch to a code version:
+
+1. The production branch should have one, and only one, file committed. The Puppetfile.
+2. The environment declaration in the Puppetfile must match the git branch name
+
+For example:
+
+```
+[reidmv@halcyon:~/src/control-repo/] % git status
+On branch production
+nothing to commit, working tree clean
+[reidmv@halcyon:~/src/control-repo/] % tree
+.
+└── Puppetfile
+
+0 directories, 1 file
+```
+
+```ruby
+#! Puppetfile
+#=============================================================================
+# ENVIRONMENT: production
+#=============================================================================
+
+environment 'production',
+  :ref => '3.2.0'
+```
+
+When r10k deploys this environment, the output will look something like this:
+
+```
+[root@master:~] # r10k deploy environment production -pv
+INFO     -> Using Puppetfile '/etc/puppetlabs/code-staging/environments/production/Puppetfile'
+INFO     -> Deploying environment /etc/puppetlabs/code-staging/environments/production
+INFO     -> Environment production is now at b8fabbeac16374cfdeb5596c87edcedd144fafb9
+INFO     -> Puppetfile at b8fabbeac16374cfdeb5596c87edcedd144fafb9 redirects to "3.2.0"
+INFO     -> Environment production is now at b05b956a972358a8f649f66f0993c0f0bdce9c5a
+INFO     -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/stdlib
+INFO     -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/remote_file
+INFO     -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/concat
+...
+```
+
+The `environment` directive accepts two arguments.
+
+#### `:ref`
+
+As in a module, `:ref` may refer to a git tag, commit, or branch to deploy.
+
+#### `:git`
+
+By default, `environment` assumes you want to check out a different ref in the control repo. Using the `:git` parameter, it is possible to link to and deploy an environment code version from a separate repository entirely. For example:
+
+```ruby
+#! Puppetfile
+#=============================================================================
+# ENVIRONMENT: production
+#=============================================================================
+
+environment 'production',
+  :git => 'git://github.com/reidmv/puppet-control-repo.git',
+  :ref => 'b05b956'
+```
+
+```
+[root@master:~] # r10k deploy environment production -pv
+INFO     -> Using Puppetfile '/Users/reidmv/tmp/environments/production/Puppetfile'
+INFO     -> Deploying environment /etc/puppetlabs/code-staging/environments/production
+INFO     -> Environment production is now at b8fabbeac16374cfdeb5596c87edcedd144fafb9
+INFO     -> Puppetfile at b8fabbeac16374cfdeb5596c87edcedd144fafb9 redirects to git://github.com/reidmv/puppet-control-repo.git "b05b956"
+INFO     -> Environment production is now at b05b956a972358a8f649f66f0993c0f0bdce9c5a
+INFO     -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/site
+INFO     -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/stdlib
+INFO     -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/remote_file
+INFO     -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/concat
+...
+```
+
+### Combining environment with mod
+
+A Puppetfile containing an environment declaration can also declare modules. The modules declared will be deployed, as well as any moduled declared in the new environment's Puppetfile. It will be as if the two Puppetfiles were concatenated together. Note that this means a module declared in both Puppetfiles will cause an error on deployment (conflicting/duplicate name).
+
+The use case for allowing a Puppetfile that redirects to another environment to also declare modules is to allow independently managed modules to be released and deployed to environments without requiring a shared Puppetfile to be edited.
+
+### Example uses
+
+Users of this feature might implement a clean separation of development and stable deployment using separate r10k sources configured as follows.
+
+```yaml
+---
+cachedir: '/var/cache/r10k'
+
+sources:
+  stable:
+    remote: 'git://github.com/reidmv/control-repo.git'
+    prefix: false
+  development:
+    remote: 'git://github.com/reidmv/puppet-environment.git'
+    prefix: 'git'
+```
+
+The "stable" source above is a typical control repo, where branches correspond to named environments. Every branch in this repo though is a "link" to either a branch or a code version from the puppet-environment.git repo.
+
+the "development" source, puppet-environment.git, has a typical git master branch, and feature branches for work in progress. Every branch in puppet-environment.git is available as a Puppet environment, but has the "git\_\*" prefix. For example, "git\_master".
+
+In the control-repo, an early environment such as development might be pinned to the master branch of puppet-environment.git.
+
+```ruby
+#! Puppetfile
+#=============================================================================
+# ENVIRONMENT: development
+#=============================================================================
+
+environment 'development',
+  :git => 'git://github.com/reidmv/puppet-environment.git',
+  :ref => 'master'
+```
+
+Higher tier environments are defined the same way, but likely pinned to commits or tags. E.g.
+
+```ruby
+#! Puppetfile
+#=============================================================================
+# ENVIRONMENT: staging
+#=============================================================================
+
+environment 'staging',
+  :git => 'git://github.com/reidmv/puppet-environment.git',
+  :ref => '2.1'
+```
+
+This pattern results in named environments without prefixes being defined and controlled in a deployment-centric workflow, while developers working on new features retain the same benefits they're used to; the main difference when compared to traditional development-only workflows is that Puppet environments that are deployed for development use are clearly identifyable by their "git\_\*" prefixes.

lib/r10k/action/deploy/environment.rb

@@ -82,18 +82,29 @@ def visit_environment(environment)
 
           started_at = Time.new
 
-          status = environment.status
           logger.info _("Deploying environment %{env_path}") % {env_path: environment.path}
 
-          environment.sync
-          logger.info _("Environment %{env_dir} is now at %{env_signature}") % {env_dir: environment.dirname, env_signature: environment.signature}
+          # Puppetfiles may redirect an environment to another repository
+          # reference. Perform the sync in a loop so that redirects may be
+          # followed.
+          i = 0
+          loop do
+            raise R10K::Error.new('Too many Puppetfile redirects! Aborting') if (i >= 3)
 
-          if status == :absent || @puppetfile
-            if status == :absent
-              logger.debug(_("Environment %{env_dir} is new, updating all modules") % {env_dir: environment.dirname})
-            end
+            status = environment.status
+            environment.sync
+
+            logger.info _("Environment %{env_dir} is now at %{env_signature}") % {env_dir: environment.dirname, env_signature: environment.signature}
+            logger.debug(_("Environment %{env_dir} is new, updating all modules") % {env_dir: environment.dirname}) if status == :absent
 
-            yield
+            if @puppetfile && yield == :redirect
+              redirect = environment.puppetfile.environment_redirect
+              environment.ref = redirect[:ref]
+              environment.remote = redirect[:git] if redirect[:git]
+              i += 1
+            else
+              break
+            end
           end
 
           if @purge_levels.include?(:environment)
@@ -111,6 +122,19 @@ def visit_environment(environment)
         def visit_puppetfile(puppetfile)
           puppetfile.load
 
+          if puppetfile.environment_redirect?
+            signature = puppetfile.environment.signature
+            newremote = puppetfile.environment_redirect[:git]
+            newref    = puppetfile.environment_redirect[:ref]
+            remotestr = newremote ? "#{newremote} " : ''
+            logger.info(_('Puppetfile at %{signature} redirects to %{remote}"%{ref}"') % {signature: signature, remote: remotestr, ref: newref})
+            unless puppetfile.modules.empty?
+              names = puppetfile.modules.map{ |m| m.name }.join(',')
+              logger.info(_('Loaded modules from Puppetfile at %{signature}: %{names}') % {signature: signature, names: names})
+            end
+            return :redirect
+          end
+
           yield
 
           if @purge_levels.include?(:puppetfile)

lib/r10k/environment/git.rb

@@ -16,7 +16,7 @@ class R10K::Environment::Git < R10K::Environment::Base
 
   # @!attribute [r] ref
   #   @return [String] The git reference to use for this environment
-  attr_reader :ref
+  attr_accessor :ref
 
   # @!attribute [r] repo
   #   @api private
@@ -40,6 +40,11 @@ def initialize(name, basedir, dirname, options = {})
     @repo = R10K::Git::StatefulRepository.new(@remote, @basedir, @dirname)
   end
 
+  def remote=(addr)
+    @remote = addr
+    @repo = R10K::Git::StatefulRepository.new(@remote, @basedir, @dirname)
+  end
+
   # Clone or update the given Git environment.
   #
   # If the environment is being created for the first time, it will

lib/r10k/git/rugged/thin_repository.rb

@@ -3,10 +3,10 @@
 require 'r10k/git/rugged/cache'
 
 class R10K::Git::Rugged::ThinRepository < R10K::Git::Rugged::WorkingRepository
-  def initialize(basedir, dirname, cache_repo)
+  def initialize(basedir, dirname, gitdirname, cache_repo)
     @cache_repo = cache_repo
 
-    super(basedir, dirname)
+    super(basedir, dirname, gitdirname)
   end
 
   # Clone this git repository
@@ -29,8 +29,9 @@ def clone(remote, opts = {})
     # update 'objects/info/alternates' with the path. We don't actually
     # fetch any objects because we don't need them, and we don't actually
     # use any refs in this repository so we skip all those steps.
-    ::Rugged::Repository.init_at(@path.to_s, false)
-    @_rugged_repo = ::Rugged::Repository.new(@path.to_s, :alternates => [cache_objects_dir])
+    ::Rugged::Repository.init_at(@git_dir.to_s, true)
+    @_rugged_repo = ::Rugged::Repository.new(@git_dir.to_s, :alternates => [cache_objects_dir])
+    @_rugged_repo.workdir = @path.to_s
     alternates << cache_objects_dir
 
     with_repo do |repo|

lib/r10k/git/rugged/working_repository.rb

@@ -4,15 +4,16 @@
 
 class R10K::Git::Rugged::WorkingRepository < R10K::Git::Rugged::BaseRepository
 
-  # @return [Pathname] The path to the Git repository inside of this directory
-  def git_dir
-    @path + '.git'
-  end
+  # @attribute [r] git_dir
+  #   @return [Pathname]
+  attr_reader :git_dir
 
   # @param basedir [String] The base directory of the Git repository
   # @param dirname [String] The directory name of the Git repository
-  def initialize(basedir, dirname)
-    @path = Pathname.new(File.join(basedir, dirname))
+  # @param gitdirname [String] The relative name of the gitdir
+  def initialize(basedir, dirname, gitdirname = '.git')
+    @path = Pathname.new(File.join(basedir, dirname)).cleanpath
+    @git_dir = Pathname.new(File.join(basedir, dirname, gitdirname)).cleanpath
   end
 
   # Clone this git repository
@@ -34,13 +35,15 @@ def clone(remote, opts = {})
     # repository. However alternate databases can be handled when an existing
     # repository is loaded, so loading a cloned repo will correctly use
     # alternate object database.
-    options = {:credentials => credentials}
+    options = {:credentials => credentials, :bare => true}
     options.merge!(:alternates => [File.join(opts[:reference], 'objects')]) if opts[:reference]
 
     proxy = R10K::Git.get_proxy_for_remote(remote)
 
     R10K::Git.with_proxy(proxy) do
-      @_rugged_repo = ::Rugged::Repository.clone_at(remote, @path.to_s, options)
+      @_rugged_repo = ::Rugged::Repository.clone_at(remote, @git_dir.to_s, options)
+      @_rugged_repo.workdir = @path.to_s
+      @_rugged_repo.checkout(@_rugged_repo.head.canonical_name)
     end
 
     if opts[:reference]
@@ -53,7 +56,7 @@ def clone(remote, opts = {})
       checkout(opts[:ref])
     end
   rescue Rugged::SshError, Rugged::NetworkError => e
-    raise R10K::Git::GitError.new(e.message, :git_dir => git_dir, :backtrace => e.backtrace)
+    raise R10K::Git::GitError.new(e.message, :git_dir => @git_dir, :backtrace => e.backtrace)
   end
 
   # Check out the given Git ref
@@ -66,7 +69,7 @@ def checkout(ref, opts = {})
     if sha
       logger.debug2 { _("Checking out ref '%{ref}' (resolved to SHA '%{sha}') in repository %{path}") % {ref: ref, sha: sha, path: @path} }
     else
-      raise R10K::Git::GitError.new("Unable to check out unresolvable ref '#{ref}'", git_dir: git_dir)
+      raise R10K::Git::GitError.new("Unable to check out unresolvable ref '#{ref}'", git_dir: @git_dir)
     end
 
     # :force defaults to true
@@ -98,7 +101,7 @@ def fetch(remote_name = 'origin')
 
     report_transfer(results, remote)
   rescue Rugged::SshError, Rugged::NetworkError => e
-    raise R10K::Git::GitError.new(e.message, :git_dir => git_dir, :backtrace => e.backtrace)
+    raise R10K::Git::GitError.new(e.message, :git_dir => @git_dir, :backtrace => e.backtrace)
   end
 
   def exist?
@@ -110,7 +113,7 @@ def head
   end
 
   def alternates
-    R10K::Git::Alternates.new(git_dir)
+    R10K::Git::Alternates.new(@git_dir)
   end
 
   def origin
@@ -133,13 +136,15 @@ def dirty?
   private
 
   def with_repo
-    if @_rugged_repo.nil? && git_dir.exist?
+    if @_rugged_repo.nil? && @git_dir.exist?
       setup_rugged_repo
     end
     super
   end
 
   def setup_rugged_repo
-    @_rugged_repo = ::Rugged::Repository.new(@path.to_s, :alternates => alternates.to_a)
+    @_rugged_repo = ::Rugged::Repository.new(@git_dir.to_s, :alternates => alternates.to_a)
+    @_rugged_repo.workdir = @path.to_s
+    @_rugged_repo
   end
 end

lib/r10k/git/shellgit/thin_repository.rb

@@ -8,9 +8,9 @@
 # making new clones very lightweight.
 class R10K::Git::ShellGit::ThinRepository < R10K::Git::ShellGit::WorkingRepository
 
-  def initialize(basedir, dirname, cache_repo)
+  def initialize(basedir, dirname, gitdirname, cache_repo)
     @cache_repo = cache_repo
-    super(basedir, dirname)
+    super(basedir, dirname, gitdirname)
   end
 
   # Clone this git repository
@@ -36,11 +36,11 @@ def fetch(remote = 'cache')
 
   # @return [String] The origin remote URL
   def cache
-    git(['config', '--get', 'remote.cache.url'], :path => @path.to_s, :raise_on_fail => false).stdout
+    git(['config', '--get', 'remote.cache.url'], :raise_on_fail => false).stdout
   end
 
   def tracked_paths(ref="HEAD")
-    git(['ls-tree', '-t', '-r', '--name-only', ref], :path => @path.to_s).stdout.split("\n")
+    git(['ls-tree', '-t', '-r', '--name-only', ref]).stdout.split("\n")
   end
 
   private