Merge pull request #155 from github/kpaulisse-mega-merge

Release 1.5.0
github · Oct 18, 2017 · 93df209 · 93df209
2 parents 4069cb3 + 61cd024
commit 93df209
Show file tree

Hide file tree

Showing 56 changed files with 1,373 additions and 414 deletions.
diff --git a/.version b/.version
@@ -1 +1 @@
-1.4.1
+1.5.0
diff --git a/README.md b/README.md
@@ -53,6 +53,7 @@ The example above reflects the changes in the Puppet catalog from switching an u
 - [Requirements](/doc/requirements.md)
 - [Limitations](/doc/limitations.md)
 - [List of all command line options](/doc/optionsref.md)
+- [Environment variables](/doc/advanced-environment-variables.md)
 
 ### Project
 

diff --git a/doc/CHANGELOG.md b/doc/CHANGELOG.md
@@ -8,6 +8,15 @@
 </tr>
 </thead><tbody>
 <tr valign=top>
+<td>1.5.0</td>
+<td>2017-10-18</td>
+<td>
+<li><a href="https://github.com/github/octocatalog-diff/pull/151">#151</a>: (Enhancement) Support text differences in files where `source` is an array</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/153">#153</a>: (Enhancement) Support for hiera 5</li>
+<li><a href="https://github.com/github/octocatalog-diff/pull/152">#152</a>: (Internal) Better temporary directory handling</li>
+</td>
+</tr>
+<tr valign=top>
 <td>1.4.1</td>
 <td>2017-10-02</td>
 <td>

diff --git a/doc/advanced-environment-variables.md b/doc/advanced-environment-variables.md
@@ -0,0 +1,70 @@
+# Environment variables
+
+The following environment variables have special meaning to octocatalog-diff:
+
+### `OCTOCATALOG_DIFF_CONFIG_FILE`
+
+This environment variable is used to locate the configuration file for the CLI. The use of configuration files is described generally in:
+
+- [Configuration](/doc/configuration.md)
+
+### `OCTOCATALOG_DIFF_CUSTOM_VERSION`
+
+When set, the `octocatalog-diff` CLI will display this as the version number within debugging, instead of the version number in the package. This is most useful if you want to use or include a git SHA in the version number.
+
+```
+$ export OCTOCATALOG_DIFF_CUSTOM_VERSION="@$(git rev-parse HEAD)"
+$ octocatalog-diff -d ...
+D, [2017-10-12T08:57:46.454738 #35205] DEBUG -- : Running octocatalog-diff @504d7f3c91267e5193beb103caae5d4d8cebfee3 with ruby 2.3.1
+...
+```
+
+### `OCTOCATALOG_DIFF_DEVELOPER_PATH`
+
+When set, instead of loading libraries from the system or bundler location, libraries will be loaded from the specified value of this environment variable. This is used internally for development as we point users to unreleased code for debugging or testing.
+
+```
+$ export OCTOCATALOG_DIFF_DEVELOPER_PATH=$HOME/git-checkouts/octocatalog-diff
+$ octocatalog-diff ...
+```
+
+### `OCTOCATALOG_DIFF_TEMPDIR`
+
+When set:
+
+- `octocatalog-diff` will create all of its temporary directories within the specified directory.
+- `octocatalog-diff` will not attempt to remove any temporary directories it creates.
+
+This is useful in the following situations:
+
+- You are calling `octocatalog-diff` from within another program which is highly parallelized, and `at_exit` handlers are difficult to implement. Instead of figuring that all out (if it can even be figured out), you create a temporary directory before the parallelized logic, and remove it afterwards.
+
+- You wish to debug intermediate output. For example, you may be instructed to set this variable and send some of the output to the project maintainers if you request assistance.
+
+This variable is used internally for the parallelized logic for catalog compilation, but the value set from the environment will override any internal usage.
+
+### `OCTOCATALOG_DIFF_VERSION`
+
+This variable is used when building the gem, to override the default version. This is used for internal testing of `octocatalog-diff` before public releases. This variable is not useful outside the build context.
+
+### `PUPPETDB_HOST`
+
+This variable specifies the fully qualified domain name or IP address of the PuppetDB server.
+
+Note: If `PUPPETDB_URL` is specified, then `PUPPETDB_HOST` is not consulted.
+
+### `PUPPETDB_PORT`
+
+This variable specifies the port number of the PuppetDB server.
+
+Note: If `PUPPETDB_URL` is specified, then `PUPPETDB_PORT` is not consulted.
+
+### `PUPPETDB_URL`
+
+This variable specifies the URL to the PuppetDB server.
+
+Example: `https://puppetdb.example.net:8081`
+
+### `PUPPET_FACT_DIR`
+
+This variable specifies the directory path where puppet fact files are stored. (Fact files must be named `<fqdn>.yaml` where `<fqdn>` is specified when running `octocatalog-diff`.)
diff --git a/doc/configuration-hiera.md b/doc/configuration-hiera.md
@@ -1,6 +1,28 @@
 # Configuring octocatalog-diff to use Hiera
 
-If you are using Hiera with Puppet, then you must already have a [`hiera.yaml`](https://docs.puppet.com/puppet/latest/reference/config_file_hiera.html) file to configure it. These instructions will guide you through pointing octocatalog-diff at that configuration file.
+## Hiera 5
+
+Hiera 5 is included with Puppet 4.9 and higher.
+
+If there is a `hiera.yaml` file in the base directory of the environment that is in hiera 5 format, and you are running Puppet 4.9 or higher, then that file will be recognized by Puppet (and therefore, by octocatalog-diff). There is no special configuration for octocatalog-diff needed to make this work. Similarly, there is no command line option or setting to changed this behavior, because there is no corresponding option to change Puppet's behavior.
+
+If you are running Puppet 4.8 or lower, then the `hiera.yaml` file in the base directory of the environment will be ignored (unless you use `--hiera-config` to specify it as your global configuration file).
+
+If you have no global hiera configuration and you wish to rely on a `hiera.yaml` file in the base directory of your environment, make sure that you are *not* using any of the following command line options or [configuration settings](/doc/configuration.md):
+
+- `--hiera-path` or `settings[:hiera_path]`
+- `--hiera-path-strip` or `settings[:hiera_path_strip]`
+- `--hiera-config` or `settings[:hiera_config]`
+
+There is more information about Hiera 5 in Puppet's documentation:
+
+- [Enable the environment layer for existing Hiera data](https://puppet.com/docs/puppet/5.3/hiera_migrate_environments.html)
+
+## Hiera global configuration
+
+If you are using Hiera 5 with a global configuration, or you are using Hiera 3 or before, then you must already have a [`hiera.yaml`](https://docs.puppet.com/puppet/latest/reference/config_file_hiera.html) file to configure it. These instructions will guide you through pointing octocatalog-diff at that configuration file.
+
+octocatalog-diff will automatically determine the version of your Hiera configuration file and treat it accordingly. (Hiera 5 configuration files are identified as such by a `version: 5` line in the file itself.)
 
 Before you start, please understand how octocatalog-diff compiles a catalog:
 
@@ -10,9 +32,9 @@ Before you start, please understand how octocatalog-diff compiles a catalog:
 - It compiles the catalog, based on the temporary directory, for environment=production
 - It removes the temporary directory
 
-## Configuring the location of hiera.yaml
+### Configuring the location of global hiera.yaml
 
-The command line option `--hiera-config PATH` allows you to set the path to hiera.yaml.
+The command line option `--hiera-config PATH` allows you to set the path to the global hiera.yaml.
 
 You may specify this as either an absolute or a relative path.
 
@@ -24,12 +46,9 @@ You may specify this as either an absolute or a relative path.
     bin/octocatalog-diff --hiera-config hiera.yaml ...
     ```
 
-  The path is relative to a checkout of your Puppet repository. With the setting above, it will use the file named `hiera.yaml` that is at the top level
-  of your Puppet checkout.
+  The path is relative to a checkout of your Puppet repository. With the setting above, it will use the file named `hiera.yaml` that is at the top level of your Puppet checkout.
 
-  Perhaps your hiera.yaml file is in a subdirectory of your Puppet checkout. In that case, just use the relative directory path. Be sure not to add a leading `/` though,
-  because you don't want octocatalog-diff to treat it as an absolute path. In the following example, suppose you have a top level directory called `config` and your
-  `hiera.yaml` file is contained within it. You could then use:
+  Perhaps your hiera.yaml file is in a subdirectory of your Puppet checkout. In that case, just use the relative directory path. Be sure not to add a leading `/` though, because you don't want octocatalog-diff to treat it as an absolute path. In the following example, suppose you have a top level directory called `config` and your `hiera.yaml` file is contained within it. You could then use:
 
     ```
     bin/octocatalog-diff --hiera-config config/hiera.yaml ...
@@ -65,7 +84,7 @@ You may specify this as either an absolute or a relative path.
 
   We strongly recommend that you version-control your hiera.yaml file within your Puppet repository, and use the relative path option described above.
 
-## Configuring the directory in your repository in which hiera data files are found
+### Configuring the directory in your repository in which hiera data files are found
 
 The command line option `--hiera-path PATH` allows you to set the directory path, relative to the checkout of your Puppet repository, of your Hiera YAML/JSON data files.
 
@@ -88,7 +107,7 @@ If you are specifying the Hiera data path in the [configuration file](/doc/confi
 
 octocatalog-diff will fail if you specify a path that is not a directory.
 
-## Configuring the prefix path to strip
+### Configuring the prefix path to strip
 
 This is a different, and potentially more complex, alternative to `--hiera-path` / `settings[:hiera_path]` described in the prior section. Unless you have a very good reason, you should prefer to use the instructions above.
 

diff --git a/doc/dev/releasing.md b/doc/dev/releasing.md
@@ -6,24 +6,13 @@ The project maintainers are responsible for bumping the version number, regenera
 
 *This procedure is performed by a GitHubber.*
 
-To test the new version of `octocatalog-diff` in the GitHub Puppet repository:
+To test the new version of `octocatalog-diff` in the GitHub Puppet repository, check out `github/puppet` and:
 
-0. In a checkout of the GitHub Puppet repository, start a new branch based off master.
-0. In the `octocatalog-diff` checkout:
-  - Ensure that the desired branch is checked out.
-  - Choose a unique internal version number which has never been used in CI. A good guideline is that if you're planning to release a version `0.6.0` then for these tests, use `0.6.0a`, `0.6.0b`, ...
-  - Build the gem using your internal version number:
-
-    ```
-    OCTOCATALOG_DIFF_VERSION=0.6.0a rake gem:force-build
-    ```
-  - Run the task to install the gem into your Puppet checkout:
-
-    ```
-    OCTOCATALOG_DIFF_VERSION=0.6.0a rake gem:localinstall
-    ```
-
-0. Back in the Puppet checkout, ensure that the changes are as expected (updates to Gemfile / Gemfile.lock, addition of new gem). Push the change and build appropriate CI job(s) to validate the changes.
+- Start a new branch based off master
+- Run `script/update-octocatalog-diff -r <ocd_branch_name>`
+- Confirm and commit the result
+- Make sure all CI jobs pass
+- Run the `puppet-catalog-diff` CI job and make sure it passes and shows expected results
 
 ## Merging one PR
 

diff --git a/doc/optionsref.md b/doc/optionsref.md
@@ -43,11 +43,18 @@ Usage: octocatalog-diff [command line options]
         --master-cache-branch BRANCH Branch to cache
         --safe-to-delete-cached-master-dir PATH
                                      OK to delete cached master directory at this path
-        --hiera-config PATH          Relative path to hiera YAML file
+        --hiera-config STRING        Full or relative path to global Hiera configuration file globally
+        --to-hiera-config STRING     Full or relative path to global Hiera configuration file for the to branch
+        --from-hiera-config STRING   Full or relative path to global Hiera configuration file for the from branch
         --no-hiera-config            Disable hiera config file installation
-        --hiera-path PATH            Path to hiera data directory, relative to top directory of repository
+        --hiera-path STRING          Path to hiera data directory, relative to top directory of repository globally
+        --to-hiera-path STRING       Path to hiera data directory, relative to top directory of repository for the to branch
+        --from-hiera-path STRING     Path to hiera data directory, relative to top directory of repository for the from branch
         --no-hiera-path              Do not use any default hiera path settings
-        --hiera-path-strip PATH      Path prefix to strip when munging hiera.yaml
+        --hiera-path-strip STRING    Path prefix to strip when munging hiera.yaml globally
+        --to-hiera-path-strip STRING Path prefix to strip when munging hiera.yaml for the to branch
+        --from-hiera-path-strip STRING
+                                     Path prefix to strip when munging hiera.yaml for the from branch
         --no-hiera-path-strip        Do not use any default hiera path strip settings
         --ignore-attr "attr1,attr2,..."
                                      Attributes to ignore
@@ -81,10 +88,10 @@ Usage: octocatalog-diff [command line options]
         --from-puppet-binary STRING  Full path to puppet binary for the from branch
         --facts-terminus STRING      Facts terminus: one of yaml, facter
         --puppetdb-ssl-ca FILENAME   CA certificate that signed the PuppetDB certificate
-        --puppetdb-ssl-client-password PASSWORD
-                                     Password for SSL client key to connect to PuppetDB
         --puppetdb-ssl-client-cert FILENAME
                                      SSL client certificate to connect to PuppetDB
+        --puppetdb-ssl-client-password PASSWORD
+                                     Password for SSL client key to connect to PuppetDB
         --puppetdb-ssl-client-key FILENAME
                                      SSL client key to connect to PuppetDB
         --puppetdb-ssl-client-password-file FILENAME
@@ -643,6 +650,43 @@ by permitting a data type specification as well. (<a href="../lib/octocatalog-di
     </td>
   </tr>
 
+  <tr>
+    <td valign=top>
+      <pre><code>--from-hiera-config STRING</code></pre>
+    </td>
+    <td valign=top>
+      Full or relative path to global Hiera configuration file for the from branch
+    </td>
+    <td valign=top>
+      Specify a relative path to the Hiera yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_config.rb">hiera_config.rb</a>)
+    </td>
+  </tr>
+
+  <tr>
+    <td valign=top>
+      <pre><code>--from-hiera-path STRING</code></pre>
+    </td>
+    <td valign=top>
+      Path to hiera data directory, relative to top directory of repository for the from branch
+    </td>
+    <td valign=top>
+      Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the
+Puppet control repo template, the value of this should be 'hieradata', which is the default. (<a href="../lib/octocatalog-diff/cli/options/hiera_path.rb">hiera_path.rb</a>)
+    </td>
+  </tr>
+
+  <tr>
+    <td valign=top>
+      <pre><code>--from-hiera-path-strip STRING</code></pre>
+    </td>
+    <td valign=top>
+      Path prefix to strip when munging hiera.yaml for the from branch
+    </td>
+    <td valign=top>
+      Specify the path to strip off the datadir to munge hiera.yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_path_strip.rb">hiera_path_strip.rb</a>)
+    </td>
+  </tr>
+
   <tr>
     <td valign=top>
       <pre><code>--from-puppet-binary STRING</code></pre>
@@ -773,10 +817,10 @@ This timeout is specified in seconds. (<a href="../lib/octocatalog-diff/cli/opti
 
   <tr>
     <td valign=top>
-      <pre><code>--hiera-config PATH</code></pre>
+      <pre><code>--hiera-config STRING</code></pre>
     </td>
     <td valign=top>
-      Relative path to hiera YAML file
+      Full or relative path to global Hiera configuration file globally
     </td>
     <td valign=top>
       Specify a relative path to the Hiera yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_config.rb">hiera_config.rb</a>)
@@ -785,10 +829,10 @@ This timeout is specified in seconds. (<a href="../lib/octocatalog-diff/cli/opti
 
   <tr>
     <td valign=top>
-      <pre><code>--hiera-path PATH</code></pre>
+      <pre><code>--hiera-path STRING</code></pre>
     </td>
     <td valign=top>
-      Path to hiera data directory, relative to top directory of repository
+      Path to hiera data directory, relative to top directory of repository globally
     </td>
     <td valign=top>
       Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the
@@ -798,10 +842,10 @@ Puppet control repo template, the value of this should be 'hieradata', which is
 
   <tr>
     <td valign=top>
-      <pre><code>--hiera-path-strip PATH</code></pre>
+      <pre><code>--hiera-path-strip STRING</code></pre>
     </td>
     <td valign=top>
-      Path prefix to strip when munging hiera.yaml
+      Path prefix to strip when munging hiera.yaml globally
     </td>
     <td valign=top>
       Specify the path to strip off the datadir to munge hiera.yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_path_strip.rb">hiera_path_strip.rb</a>)
@@ -1541,6 +1585,43 @@ by permitting a data type specification as well. (<a href="../lib/octocatalog-di
     </td>
   </tr>
 
+  <tr>
+    <td valign=top>
+      <pre><code>--to-hiera-config STRING</code></pre>
+    </td>
+    <td valign=top>
+      Full or relative path to global Hiera configuration file for the to branch
+    </td>
+    <td valign=top>
+      Specify a relative path to the Hiera yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_config.rb">hiera_config.rb</a>)
+    </td>
+  </tr>
+
+  <tr>
+    <td valign=top>
+      <pre><code>--to-hiera-path STRING</code></pre>
+    </td>
+    <td valign=top>
+      Path to hiera data directory, relative to top directory of repository for the to branch
+    </td>
+    <td valign=top>
+      Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the
+Puppet control repo template, the value of this should be 'hieradata', which is the default. (<a href="../lib/octocatalog-diff/cli/options/hiera_path.rb">hiera_path.rb</a>)
+    </td>
+  </tr>
+
+  <tr>
+    <td valign=top>
+      <pre><code>--to-hiera-path-strip STRING</code></pre>
+    </td>
+    <td valign=top>
+      Path prefix to strip when munging hiera.yaml for the to branch
+    </td>
+    <td valign=top>
+      Specify the path to strip off the datadir to munge hiera.yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_path_strip.rb">hiera_path_strip.rb</a>)
+    </td>
+  </tr>
+
   <tr>
     <td valign=top>
       <pre><code>--to-puppet-binary STRING</code></pre>