Add Yaml environment source #983

reidmv · 2019-09-27T01:39:58Z

Feature demonstration: https://youtu.be/P0lljPd_9L8

Two major ideas here:

Make it so that a YAML object enumerating environments and their versions/sources can be consumed, rather than consulting Git branches in a repository for this information
Make it so that for environments defined this way, a list of modules can also be supplied directly. No need to involve a Puppetfile.

Example environments.yaml file:

env_with_only_modules:
  type: bare
  modules:
    puppetlabs-stdlib: 6.0.0
    puppetlabs-concat: 6.1.0
    puppetlabs-exec: 0.5.0
    puppetlabs-ntp: 8.1.0
    reidmv-xampl:
      git: https://github.com/reidmv/reidmv-xampl.git
      ref: 62d07f2
    puppetlabs-pe_xl:
      git: https://github.com/puppetlabs/puppetlabs-pe_xl.git
      ref: bcbd7b9

env_with_control_repo_base:
  type: git
  remote: file:///Users/reidmv/Workspace/repos/control-repo/.git
  ref: dddbd8d
  modules:
    puppetlabs-pe_xl:
      git: file:///Users/reidmv/Workspace/repos/puppetlabs-pe_xl/.git
      ref: bcbd7b9
    reidmv-xampl:
      git: file:///Users/reidmv/Workspace/repos/reidmv-xampl/.git
      ref: 62d07f2

reidmv · 2019-09-27T01:42:31Z

lib/r10k/source/hash.rb

+# can be used to designate module content for the environment, independent of
+# the base source parameters.
+#
+# Example:


Start reading here ☝️ for a summary of how it would work

This commit implements a new capability for R10K environment objects: environments can now contain modules directly, when built from an appropriate source type and environment type. Previously, modules could only be attached to an environment by loading an environment, then reading the Puppetfile it contained. This constrained the system because in order for modules to be added, removed, or updated, either the core environment content needed to be changed (the Puppetfile edited), or some kind of magic pattern needed to be specified in the Puppetfile which delegated part of the work of defining a module to an external source (the :control_branch pattern). Now, the framework exists that enables an environment source to add, remove, or update deployed module versions without ever touching or modifying a Puppetfile.

The Hash environment source type is an abstract base class that makes it easy to define environments by simply enumerating them. For example, a REST API could provide a JSON object enumerating each environment that should exist, along with details such as modules to deploy to them. The Hash source baseclass can consume this data and produce the appropriate environments. Note that such a source would no longer need be tightly coupled with version control branches, which is the only way today's r10k sources work.

This commit changes the R10K::Environment::Git class's parent from R10K::Environment::Base to R10K::Environment::WithModules. The practical effect of this is that Git environments are now compatible with environment sources that directly define modules to deploy (in addition to or instead of modules defined in a Puppetfile).

Prior to this fix, the .r10k-deploy.json file wouldn't have a module version in it on first install. It would be present after an upgrade or other modification of an already-present module, just not the first time the module was added to an environment. After this commit, the .r10k-deploy.json file is correctly populated with new-install Forge modules' version numbers.

This allows r10k to read environment definitions from a YAML file, located by default at /etc/puppetlabs/r10k/environments.yaml.

This type has no content besides environment-attached modules

binford2k · 2019-11-20T21:15:31Z

@reidmv Where would said environments.yaml file live on disk? How would it be version controlled, or is that an exercise left to the reader?

Also, I'm a fan of the general idea.

dhollinger

LGTM

reidmv · 2019-11-20T21:53:53Z

Where would said environments.yaml file live on disk? How would it be version controlled, or is that an exercise left to the reader?

@binford2k the environments.yaml file would live on disk on the master. Management and version control of the file is left as an exercise to the reader.

For shops that like keeping the content in Git, it could be as simple as a one-file Git repo with a hook to read the latest file on commit.

For shops that prefer to be API-driven, either a script could query the API for a hash and write the file as part of a deploy action, or r10k itself could read a hash from the API directly; see the R10k::Source::Hash abstract class for suggestions on how that would be implemented in a later iteration.

This is a plumbing feature that user interfaces (such as PE's Code Manager) would build on.

ekohl · 2019-11-21T21:29:30Z

Makes me wonder if you want environments.d/*.yaml so people can share an environment. Branching would be as simple as a cp. Also less chance of merge conflicts.

Yaml source Environment modules Bare environment type

binford2k · 2019-12-13T00:49:39Z

@ekohl Reid's working on an environments.d/ source now and will submit a separate PR

justinstoller · 2020-01-09T23:05:17Z

Why doesn't this work for SVN?

reidmv · 2020-01-13T17:30:54Z

Why doesn't this work for SVN?

As this is currently an experimental feature, SVN support is untested. Focus has been on testing with Git sources.

Technically, I think it should partly work for SVN sources. What will likely work is listing an SVN source as an environment in a YAML file. What won't work is specifying individual modules to deploy to that SVN source, as the SVN source type has not been converted from using the Base environment class to using the WithModules environment class. See here.

reidmv commented Sep 27, 2019

View reviewed changes

reidmv force-pushed the add-environment-modules branch from 4ff1038 to 1104d5e Compare September 27, 2019 15:54

reidmv and others added 4 commits September 27, 2019 09:51

reidmv force-pushed the add-environment-modules branch from 1104d5e to 779a8fb Compare September 27, 2019 16:52

reidmv mentioned this pull request Sep 27, 2019

(CODEMGMT-1177) Add environment definition feature #802

Closed

reidmv added 3 commits October 3, 2019 08:18

Implement yaml source type

7500495

This allows r10k to read environment definitions from a YAML file, located by default at /etc/puppetlabs/r10k/environments.yaml.

Fix error relating to purging environments w/ mods

f68bfda

Implement "bare" environment type

4e57af5

This type has no content besides environment-attached modules

reidmv force-pushed the add-environment-modules branch from 9b64dd5 to 4e57af5 Compare October 17, 2019 22:50

reidmv changed the title ~~Add Hash environment source~~ Add Yaml environment source Oct 17, 2019

reidmv marked this pull request as ready for review October 18, 2019 20:41

reidmv requested review from Sharpie and binford2k November 20, 2019 17:23

dhollinger approved these changes Nov 20, 2019

View reviewed changes

binford2k approved these changes Nov 21, 2019

View reviewed changes

Add experimental documentation for new features

1311dbe

Yaml source Environment modules Bare environment type

binford2k merged commit 14039f1 into puppetlabs:master Dec 13, 2019

reidmv deleted the add-environment-modules branch January 13, 2020 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Yaml environment source #983

Add Yaml environment source #983

reidmv commented Sep 27, 2019 •

edited

Loading

reidmv Sep 27, 2019

binford2k commented Nov 20, 2019

dhollinger left a comment

reidmv commented Nov 20, 2019

ekohl commented Nov 21, 2019

binford2k commented Dec 13, 2019

justinstoller commented Jan 9, 2020

reidmv commented Jan 13, 2020

Add Yaml environment source #983

Add Yaml environment source #983

Conversation

reidmv commented Sep 27, 2019 • edited Loading

reidmv Sep 27, 2019

Choose a reason for hiding this comment

binford2k commented Nov 20, 2019

dhollinger left a comment

Choose a reason for hiding this comment

reidmv commented Nov 20, 2019

ekohl commented Nov 21, 2019

binford2k commented Dec 13, 2019

justinstoller commented Jan 9, 2020

reidmv commented Jan 13, 2020

reidmv commented Sep 27, 2019 •

edited

Loading