Skip to content

Commit

Permalink
docs(map): document the load of .yaml.jinja after each .yaml
Browse files Browse the repository at this point in the history
  • Loading branch information
@baby-gnu
baby-gnu committed Feb 8, 2021
1 parent 536d7a5 commit 77c60f2
Showing 1 changed file with 126 additions and 95 deletions.
221 changes: 126 additions & 95 deletions docs/map.jinja.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ For formula users
Quick start: configure per role and per DNS domain name values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We will see a quick setup to configure the ``TEMPLATE`` formula for different DNS domain name and several roles.
We will see a quick setup to configure the ``TEMPLATE`` formula for different DNS domain names and several roles.

For this example, I'll define 2 kinds of `fileserver`_ sources:

Expand Down Expand Up @@ -93,7 +93,7 @@ We create a configuration for the DNS domain ``example.net`` in ``/srv/salt/TEMP
config: /etc/template-formula-example-net.conf
...
We create another configuration for the DNS domain ``example.com`` in ``/srv/salt/TEMPLATE/parameters/dns:domain/example.com.yaml``:
We create another configuration for the DNS domain ``example.com`` in the Jinja YAML template ``/srv/salt/TEMPLATE/parameters/dns:domain/example.com.yaml.jinja``:

.. code-block:: yaml
Expand Down Expand Up @@ -149,29 +149,28 @@ We create the global parameters file ``/srv/salt/parameters/map_jinja.yaml``:
---
values:
map_jinja:
sources:
# default values
- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
# Roles activate/deactivate things
# then thing are configured depending on environment
# So roles comes before `dns:domain`, `domain` and `id`
- "Y:C@roles"
# DNS domain configured (DHCP or resolv.conf)
- "Y:G@dns:domain"
# Based on minion ID
- "Y:G@domain"
# default values
- "Y:G@id"
sources:
# default values
- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
# Roles activate/deactivate things
# then thing are configured depending on environment
# So roles comes before `dns:domain`, `domain` and `id`
- "Y:C@roles"
# DNS domain configured (DHCP or resolv.conf)
- "Y:G@dns:domain"
# Based on minion ID
- "Y:G@domain"
# default values
- "Y:G@id"
...
The syntax is explained later at `Sources of configuration values`_.
Expand All @@ -196,7 +195,7 @@ For the clients:
.. note::

Since we used ``Y:C@roles``, ``map.jinja`` will do a ``salt['config.get']('roles')`` to retrieve the roles so you could use any other method to bind roles to minions (`pillars`_ or `SDB`_) but `grains`_ seems to be the prefered method.
Since we used ``Y:C@roles``, ``map.jinja`` will do a ``salt['config.get']('roles')`` to retrieve the roles so you could use any other method to bind roles to minions (`pillars`_ or `SDB`_) but `grains`_ seems to be the preferred method.

Note for Microsoft Windows systems
``````````````````````````````````
Expand All @@ -209,29 +208,28 @@ Modify ``/srv/salt/parameters/map_jinja.yaml`` to change the query for ``dns:dom
---
values:
map_jinja:
sources:
# default values
- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
# Roles activate/deactivate things
# then thing are configured depending on environment
# So roles comes before `dns:domain`, `domain` and `id`
- "Y:C@roles"
# DNS domain configured (DHCP or resolv.conf)
- "Y:G:!@dns!domain"
# Based on minion ID
- "Y:G@domain"
# default values
- "Y:G@id"
sources:
# default values
- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
# Roles activate/deactivate things
# then thing are configured depending on environment
# So roles comes before `dns:domain`, `domain` and `id`
- "Y:C@roles"
# DNS domain configured (DHCP or resolv.conf)
- "Y:G:!@dns!domain"
# Based on minion ID
- "Y:G@domain"
# default values
- "Y:G@id"
...
And then, rename the directory:
Expand Down Expand Up @@ -265,29 +263,56 @@ Here is a valid example:
config: '/path/to/a/configuration/file'
...
You can use `Jinja`_ as with any SLS files:
.. code-block:: yaml
Using Jinja2 YAML template
``````````````````````````

You can provide a Jinja2 YAML template file with a name suffixed with ``.yaml.jinja``, it must produce a YAML file conform to the `Format of configuration YAML files`_, for example:

.. code-block:: jinja2
---
strategy: 'overwrite'
merge_lists: 'true'
values:
output_dir: /tmp/{{ grains['id'] }}
{%- if grains["os"] == "Debian" %}
output_dir: /tmp/{{ grains["id"] }}
{%- endif %}
...
Sources of configuration values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The ``map.jinja`` file aggregates configuration values from several sources:

- YAML files stored in the `fileserver`_
- `pillars`_
- `grains`_
- configuration gathered with `salt['config.get']`_

For the values loaded from YAML files, ``map.jinja`` will automatically try to load a Jinja2 template with the same name as the YAML file with the addition of the ``.jinja`` extension, for example ``foo/bar/quux.yaml.jinja``.

After loading values from all sources, it will try to include the ``salt://parameters/post-map.jinja`` Jinja file if it exists which can post-process the ``mapdata`` variable.

Configuring ``map.jinja`` sources
`````````````````````````````````

The ``map.jinja`` file uses several sources where to lookup parameter values. The list of sources can be modified by two files:
The ``map.jinja`` file uses several sources where to lookup parameter values. The list of sources can be configured in two places:

1. a global ``salt://parameters/map_jinja.yaml``
2. a per formula ``salt://{{ tplroot }}/parameters/map_jinja.yaml``, it overrides the global configuration
1. globally

1. with a plain YAML file ``salt://parameters/map_jinja.yaml``
2. with a Jinja2 YAML template file ``salt://parameters/map_jinja.yaml.jinja``

2. per formula

1. with a plain YAML file ``salt://{{ tplroot }}/parameters/map_jinja.yaml``
2. with a Jinja2 YAML template file ``salt://{{ tplroot }}/parameters/map_jinja.yaml.jinja``

.. note::

The ``map.jinja`` configuration files must conform to the `format of configuration YAML files`_.

Each source definition has the form ``[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<KEY>`` where ``<TYPE>`` can be one of:

Expand All @@ -310,84 +335,90 @@ Finally, the ``<KEY>`` describes what to lookup to either build the YAML filenam

.. note::

For the YAML type, if the ``<KEY>`` can't be looked up, then it's used a literal string path to a YAML file, for example: ``any/path/can/be/used/here.yaml`` will result in the loading of ``salt://{{ tplroot }}/parameters/any/path/can/be/used/here.yaml`` if it exists.
For the YAML type:

- if the ``<KEY>`` can't be looked up, then it's used a literal string path to a YAML file, for example: ``any/path/can/be/used/here.yaml`` will result in the loading of ``salt://{{ tplroot }}/parameters/any/path/can/be/used/here.yaml`` if it exists
- ``map.jinja`` will automatically try to load a Jinja2 template, after the corresponding YAML file, with the same name as the YAML file extended with the ``.jinja`` extension, for example ``any/path/can/be/used/here.yaml.jinja``

The built-in ``map_jinja:sources`` is:
The built-in ``map.jinja`` sources are:

.. code-block:: yaml
values:
map_jinja:
sources:
- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
- "Y:G@id"
This is strictly equivalent to the following ``map_jinja.yaml`` using `Jinja`_:
- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
- "Y:G@id"
This is strictly equivalent to the following ``map_jinja.yaml.jinja``:

.. code-block:: sls
values:
map_jinja:
sources:
- "parameters/osarch/{{ salt['grains.get']('osarch') }}.yaml"
- "parameters/os_family/{{ salt['grains.get']('os_family') }}.yaml"
- "parameters/os/{{ salt['grains.get']('os') }}.yaml"
- "parameters/osfinger/{{ salt['grains.get']('osfinger') }}.yaml"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
- "parameters/id/{{ salt['grains.get']('id') }}.yaml"
sources:
- "parameters/osarch/{{ salt['grains.get']('osarch') }}.yaml"
- "parameters/osarch/{{ salt['grains.get']('osarch') }}.yaml.jinja"
- "parameters/os_family/{{ salt['grains.get']('os_family') }}.yaml"
- "parameters/os_family/{{ salt['grains.get']('os_family') }}.yaml.jinja"
- "parameters/os/{{ salt['grains.get']('os') }}.yaml"
- "parameters/os/{{ salt['grains.get']('os') }}.yaml.jinja"
- "parameters/osfinger/{{ salt['grains.get']('osfinger') }}.yaml"
- "parameters/osfinger/{{ salt['grains.get']('osfinger') }}.yaml.jinja"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
- "parameters/id/{{ salt['grains.get']('id') }}.yaml"
- "parameters/id/{{ salt['grains.get']('id') }}.yaml.jinja"
Loading values from the configuration sources
`````````````````````````````````````````````

For each configuration source defined in ``map_jinja:sources``, ``map.jinja`` will:
For each configuration source defined, ``map.jinja`` will:

#. load values depending on the source type:

- for YAML file sources

- if the ``<KEY>`` can be looked up, load values from the YAML file named ``salt://{{ tplroot }}/paramaters/<KEY>/{{ salt['<QUERY_METHOD>']('<KEY>') }}.yaml`` if it exists
- otherwise, load the YAML file named ``salt://{{ tplroot }}/parameters/<KEY>.yaml`` if it exists

- for ``C``, ``G`` or ``I`` source type, lookup the value of ``salt['<QUERY_METHOD>']('<KEY>')``

#. merge the loaded values with the previous ones using `salt.slsutil.merge`_
- if the ``<KEY>`` can be looked up:

There will be no error if a YAML file does not exists, they are all optional.
- load values from the YAML file named ``salt://{{ tplroot }}/paramaters/<KEY>/{{ salt['<QUERY_METHOD>']('<KEY>') }}.yaml`` if it exists
- load values from the Jinja2 YAML template file named ``salt://{{ tplroot }}/paramaters/<KEY>/{{ salt['<QUERY_METHOD>']('<KEY>') }}.yaml.jinja`` if it exists

- otherwise:

Configuration values from ``salt['config.get']``
````````````````````````````````````````````````
- load the YAML file named ``salt://{{ tplroot }}/parameters/<KEY>.yaml`` if it exists
- load the Jinja2 YAML template file named ``salt://{{ tplroot }}/parameters/<KEY>.yaml.jinja`` if it exists

For sources with of type ``C`` declared in ``map_jinja:sources``, you can configure the ``merge`` option of `salt['config.get']`_ by defining per formula ``strategy`` configuration key (retrieved with ``salt['config.get'](tplroot ~ ':strategy')`` with one of the following values:
- for ``C``, ``G`` or ``I`` source type, lookup the value of ``salt['<QUERY_METHOD>']('<KEY>')``

- ``recurse`` merge recursively dictionaries. Non dictionary values replace already defined values
- ``overwrite`` new value completely replace old ones
#. merge the loaded values with the previous ones using `salt.slsutil.merge`_

By default, no merging is done, the first value found is returned.
There will be no error if a YAML or Jinja2 file does not exists, they are all optional.


Global view of the order of preferences
```````````````````````````````````````

To summarize, here is a complete example of the load order of formula configuration values for an ``AMD64`` ``Ubuntu 18.04`` minion named ``minion1.example.net`` for the ``libvirt`` formula:
To summarise, here is a complete example of the load order of formula configuration values for an ``AMD64`` ``Ubuntu 18.04`` minion named ``minion1.example.net`` for the ``libvirt`` formula:

#. ``parameters/defaults.yaml``
#. ``parameters/defaults.yaml.jinja``
#. ``parameters/osarch/amd64.yaml``
#. ``parameters/osarch/amd64.yaml.jinja``
#. ``parameters/os_family/Debian.yaml``
#. ``parameters/os_family/Debian.yaml.jinja``
#. ``parameters/os/Ubuntu.yaml``
#. ``parameters/os/Ubuntu.yaml.jinja``
#. ``parameters/osfinger/Ubuntu-18.04.yaml``
#. ``parameters/osfinger/Ubuntu-18.04.yaml.jinja``
#. ``salt['config.get']('libvirt:lookup')``
#. ``salt['config.get']('libvirt')``
#. ``parameters/id/minion1.example.net``
#. ``parameters/id/minion1.example.net.yaml``
#. ``parameters/id/minion1.example.net.yaml.jinja``

Remember that the order is important, for example, the value of ``key1:subkey1`` loaded from ``parameters/os_family/Debian.yaml`` is overridden by a value loaded from ``parameters/id/minion1.example.net``.
Remember that the order is important, for example, the value of ``key1:subkey1`` loaded from ``parameters/os_family/Debian.yaml`` is overridden by a value loaded from ``parameters/id/minion1.example.net.yaml``.


For formula authors and contributors
Expand Down

0 comments on commit 77c60f2

Please sign in to comment.