Feature request: Passing environment

[tobias-urdin]

Feature request:
Seems like the 'production' environment is hardcoded into octocatalog-diff.
I would like to be able to give the environment as an parameter.

[kpaulisse]

Hey @tobias-urdin ! Thanks for the note.

By way of background, octocatalog-diff runs Puppet against a single directory of code (and of course, it does this twice, separately, to compare the results). That directory might be a checkout of a specific branch of a git repo (configured by the -f / -t options), or a directory you provide that you've already prepared (configured with --bootstrapped-from-dir / bootstrapped-to-dir). For each catalog it compiles, octocatalog-diff constructs a temporary directory to hold the Puppet configuration, and in this temporary directory it constructs one environment which points at the code you've supplied. The naming of this environment (which is production) within the temporary directory is not meaningful. The hard-coded production here shouldn't prevent you from using different branches of your code from a git repository via the -f and -t command line options.

So that I can better understand your request, could you please let me know what you're unable to do with the current implementation? Thanks in advance!

[tobias-urdin]

Hello @kpaulisse,
Thanks for answering!

We have all our puppet code in the master branch which contains our three environments and then use tags for deploying the actual changes.

I run octocatalog-diff like this:
octocatalog-diff -d -n $NODE --puppet-binary=which puppet --display-detail-add --bootstrapped-from-dir=before --bootstrapped-to-dir=after --hiera-config=$OLD_DIR/hiera.yaml --hiera-path-strip=$HIERA_STRIP_PATH --fact-file yaml/facts/$NODE.yaml

Some more information:

• before is the folder containg our checkout of the master branch in it's current state
• after is the folder containing our checkout of the master branch after a change

So the structure is like this:
├── after
│   ├── environments
│   │   ├── production
│   │   ├── shared
│   │   └── testing
│   └── modules
│   └── apache
├── before
│   ├── environments
│ │ ├── production
│ │ ├── shared
│ │ └── testing
│   └── modules
│ └── apache
└── yaml
├── facts
└── node

The yaml directory is synced from the puppetmaster node.

Now let's say I would like to test the 'testing' environment changes between before and after. How would I do that?
I'm quite certain I must have missed something here, but this line only allows me to compile our production environment.

Hope this gave you an idea what I'm trying to achieve, this would be a great addition to our CI process.
Best regards

[tobias-urdin]

So to give you an example, I could specify the bootstrapped-from-dir and bootstrapped-to-dir to the environment code, like --bootstrapped-from-dir=before/environments/testing but then the modules in before/modules (modules that is shared between all environments) won't be loaded and cause the catalog to fail and I can't see that there is any option to add a module path or similar.

Best regards

[kpaulisse]

Could you post your environment.conf file? I'm assuming that the configuration of modulepath is how you're making this all work? I think I understand what would be needed to support this but I want to make sure before I write any code.

You can wrap your text in ```(your text here)``` to format it as code/plain text. Thanks in advance!

[tobias-urdin]

I assume you mean the environment.conf file in one of the environments. I'm using the default i.e the environment.conf file only contains commented lines and no active changes.

What would be great, and what I'm trying to achieve, is if I could simply use the actual structure of my puppet code when doing the compiles+diffs and not have to script any changes instead just clone the code and check.

I already have puppet compiles and diffs working however it's a long bash script which octocatalog-diff could easily replace with a single command, which would be, to say the least, awesome!

# Each environment can have an environment.conf file. Its settings will only
# affect its own environment. See docs for more info:
# https://docs.puppetlabs.com/puppet/latest/reference/config_file_environment.html

# Any unspecified settings use default values; some of those defaults are based
# on puppet.conf settings.

# If these settings include relative file paths, they'll be resolved relative to
# this environment's directory.

# Allowed settings and default values:

# modulepath = ./modules:$basemodulepath
# manifest = (default_manifest from puppet.conf, which defaults to ./manifests)
# config_version = (no script; Puppet will use the time the catalog was compiled)
# environment_timeout = (environment_timeout from puppet.conf, which defaults to 0)
    # Note: unless you have a specific reason, we recommend only setting
    # environment_timeout in puppet.conf.

[tobias-urdin]

Hello @kpaulisse,
Great work on the PR! The environment stuff is working flawlessly.

I added the --preserve-environments --create-symlinks modules,environments --environment=testing options when running the command it it properly does the symlinks for the environment, modules folder outside of the environment.

However it seems the modules folder outside of the environments isn't included in the compile for me.
When we do the compilations manually using our scripts we pass the modulepath manually and set the paths like this

--modulepath=$CURRENT_DIR/after/environments/$ENVIRONMENT/modules:$CURRENT_DIR/after/modules

Which would translate to something like this during runtime. Having priority by loading the environment modules first (first found = first used) and then loading the modules shared between the environments.

--modulepath=/home/xx/differ/after/environments/testing/modules:/home/xx/differ/after/modules

Should I be doing any changes to the environments.conf file? See the default one we have right now above, I don't want to change something in there that could break the actual puppet usage of the code so I assume if edits would be required one would need a similar command to the hiera-strip-path.

Looking forward to your input. When running the command enabling debug mode there is no modulepath being appended.

D, [2016-12-28T11:17:51.499472 #23343] DEBUG -- : Symlinked /tmp/d20161228-23343-66isgd/environments -> /home/xx/differ/before/environments
D, [2016-12-28T11:17:51.499473 #23340] DEBUG -- : Symlinked /tmp/d20161228-23340-dbi0ff/environments -> /home/xx/differ/after/environments
D, [2016-12-28T11:17:51.499507 #23343] DEBUG -- : Symlinked /tmp/d20161228-23343-66isgd/modules -> /home/xx/differ/before/modules
D, [2016-12-28T11:17:51.499507 #23340] DEBUG -- : Symlinked /tmp/d20161228-23340-dbi0ff/modules -> /home/xx/differ/after/modules
D, [2016-12-28T11:17:51.499588 #23343] DEBUG -- : Symlinked /tmp/d20161228-23343-66isgd/environments -> /home/xx/differ/before/environments
D, [2016-12-28T11:17:51.499575 #23340] DEBUG -- : Symlinked /tmp/d20161228-23340-dbi0ff/environments -> /home/xx/differ/after/environments
D, [2016-12-28T11:17:51.500690 #23343] DEBUG -- : Installed hiera.yaml from /home/xx/differ/hiera.yaml to /tmp/d20161228-23343-66isgd/hiera.yaml
D, [2016-12-28T11:17:51.500690 #23340] DEBUG -- : Installed hiera.yaml from /home/xx/differ/hiera.yaml to /tmp/d20161228-23340-dbi0ff/hiera.yaml
D, [2016-12-28T11:17:51.509606 #23340] DEBUG -- : Installed fact file at /tmp/d20161228-23340-dbi0ff/var/yaml/facts/my.node.com.yaml
D, [2016-12-28T11:17:51.509602 #23343] DEBUG -- : Installed fact file at /tmp/d20161228-23343-66isgd/var/yaml/facts/my.node.com.yaml
D, [2016-12-28T11:17:52.018817 #23343] DEBUG -- : (from) Try 1 executing Puppet 4.8.1: /usr/local/bin/puppet master --compile my.node.com --no-storeconfigs --factpath=/tmp/d20161228-23343-66isgd/var/yaml/facts --facts_terminus=yaml --no-daemonize --no-ca --color=false --config_version="/bin/echo catalogscript" --environment=testing --hiera_config=/tmp/d20161228-23343-66isgd/hiera.yaml --environmentpath=/tmp/d20161228-23343-66isgd/environments --vardir=/tmp/d20161228-23343-66isgd/var --logdir=/tmp/d20161228-23343-66isgd/var --ssldir=/tmp/d20161228-23343-66isgd/var/ssl --confdir=/tmp/d20161228-23343-66isgd

I would then get this compilation error

[Puppet Error] Evaluation Error: Error while evaluating a Function Call, Could not find class ::network

Since it would not load the network module from the modules folder shared between the environments.
I assume this might be an issue on my side however overriding the modulepath might be a good option to have.

Best regards and a huge thank you for the work, it's very much appreciated!

[kpaulisse]

#39 adds the ability to add additional command line arguments to the Puppet catalog compile. If you stay with the approach of specifying modulepath via the command line, this new option should allow you to accomplish that.

You might want to look at doing your module path via the environment.conf file though, since I think that's the more standard approach. The environment.conf file I built for integration tests (https://github.com/github/octocatalog-diff/blob/kpaulisse-arbitrary-command-line-pass/spec/octocatalog-diff/fixtures/repos/preserve-environments/environments/one/environment.conf) is probably pretty close to what you need. Specifically modulepath = ../../modules:./modules might be what you need.

[tobias-urdin]

I think I found a issue with the environments change.
When passing my hiera config to octocatalog-diff like this with the environment set to testing it doesn't strip the path correctly and instead uses the production environment.

--environment=testing --hiera-config=$CURRENT_DIR/hiera.yaml --hiera-path-strip=/etc/puppetlabs/code

My hiera.yaml config file looks like this:

---
:backends:
  - yaml
:hierarchy:
  - "nodes/%{::trusted.certname}"
  - common

:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix
# - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows
# When specifying a datadir, make sure the directory exists.
  :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"

The one generated in the temporary file is like this:

cat /tmp/d20170103-5712-y0aear/hiera.yaml
---
:backends:
- yaml
:hierarchy:
- nodes/%{::trusted.certname}
- common
:yaml:
  :datadir: "/tmp/d20170103-5712-y0aear/environments/production/hieradata"

So it doesn't use the correct hiera data, the "production" in the generated/stripped hiera.yaml should be testing as specified by ---environment to octocatalog-diff.

Best regards

[tobias-urdin]

@kpaulisse Also another question, is there any documentation about return codes/exit codes? I would like to know so I can halt in my script properly depending on the execution result of octocatalog-diff.

Thanks for the hiera fix! 👍
Best regards