From 746c58906b291a1082a582116f3ae4601595676d Mon Sep 17 00:00:00 2001 From: Daniel Dehennin Date: Tue, 12 Jan 2021 10:31:29 +0100 Subject: [PATCH] feat(map): update to v5 `map.jinja` BREAKING CHANGE: `map.jinja` now export a generic `mapdata` variable BREAKING CHANGE: The parameters per grains are now under `openvpn/parameters/` --- docs/README.rst | 6 + docs/map.jinja.rst | 498 ++++++++++++++++++ openvpn/_mapdata/init.sls | 4 +- openvpn/config.sls | 2 +- openvpn/dhparams.sls | 2 +- openvpn/general_config.sls | 2 +- openvpn/hardening.sls | 2 +- openvpn/ifconfig_pool_persist.sls | 2 +- openvpn/install.sls | 2 +- openvpn/libmapstack.jinja | 300 +++++++++++ openvpn/libmatchers.jinja | 222 ++++++++ openvpn/libsaltcli.jinja | 16 + openvpn/map.jinja | 85 ++- .../files/connection.jinja | 2 +- openvpn/network_manager_networks/init.sls | 2 +- openvpn/osfamilymap.yaml | 32 -- openvpn/osfingermap.yaml | 5 - openvpn/osmap.yaml | 20 - openvpn/{ => parameters}/defaults.yaml | 6 +- openvpn/parameters/map_jinja.yaml | 17 + openvpn/parameters/os/Debian.yaml | 17 + openvpn/parameters/os/Fedora.yaml | 19 + openvpn/parameters/os/Ubuntu.yaml | 17 + openvpn/parameters/os_family/Arch.yaml | 16 + openvpn/parameters/os_family/Debian.yaml | 21 + openvpn/parameters/os_family/FreeBSD.yaml | 19 + openvpn/parameters/os_family/RedHat.yaml | 16 + openvpn/parameters/os_family/Windows.yaml | 17 + openvpn/parameters/osfinger/CentOS-6.yaml | 14 + openvpn/repo.sls | 2 +- openvpn/service.sls | 2 +- .../default/files/_mapdata/amazonlinux-1.yaml | 8 + .../default/files/_mapdata/amazonlinux-2.yaml | 8 + .../files/_mapdata/arch-base-latest.yaml | 8 + .../default/files/_mapdata/centos-6.yaml | 8 + .../default/files/_mapdata/centos-7.yaml | 8 + .../default/files/_mapdata/centos-8.yaml | 8 + .../default/files/_mapdata/debian-10.yaml | 8 + .../default/files/_mapdata/debian-9.yaml | 8 + .../default/files/_mapdata/fedora-31.yaml | 8 + .../default/files/_mapdata/fedora-32.yaml | 8 + .../default/files/_mapdata/fedora-33.yaml | 8 + .../default/files/_mapdata/gentoo-2-sysd.yaml | 8 + .../default/files/_mapdata/gentoo-2-sysv.yaml | 8 + .../default/files/_mapdata/opensuse-15.yaml | 8 + .../default/files/_mapdata/oraclelinux-7.yaml | 8 + .../default/files/_mapdata/oraclelinux-8.yaml | 8 + .../default/files/_mapdata/ubuntu-16.yaml | 8 + .../default/files/_mapdata/ubuntu-18.yaml | 8 + .../default/files/_mapdata/ubuntu-20.yaml | 8 + .../files/_mapdata/windows-2019-server.yaml | 8 + .../default/files/_mapdata/windows-8.yaml | 8 + 52 files changed, 1460 insertions(+), 95 deletions(-) create mode 100644 docs/map.jinja.rst create mode 100644 openvpn/libmapstack.jinja create mode 100644 openvpn/libmatchers.jinja create mode 100644 openvpn/libsaltcli.jinja delete mode 100644 openvpn/osfamilymap.yaml delete mode 100644 openvpn/osfingermap.yaml delete mode 100644 openvpn/osmap.yaml rename openvpn/{ => parameters}/defaults.yaml (83%) create mode 100644 openvpn/parameters/map_jinja.yaml create mode 100644 openvpn/parameters/os/Debian.yaml create mode 100644 openvpn/parameters/os/Fedora.yaml create mode 100644 openvpn/parameters/os/Ubuntu.yaml create mode 100644 openvpn/parameters/os_family/Arch.yaml create mode 100644 openvpn/parameters/os_family/Debian.yaml create mode 100644 openvpn/parameters/os_family/FreeBSD.yaml create mode 100644 openvpn/parameters/os_family/RedHat.yaml create mode 100644 openvpn/parameters/os_family/Windows.yaml create mode 100644 openvpn/parameters/osfinger/CentOS-6.yaml diff --git a/docs/README.rst b/docs/README.rst index eb0d55f..a5cc91f 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -33,6 +33,12 @@ which contains the currently released version. This formula is versioned accordi See `Formula Versioning Section `_ for more details. +If you need (non-default) configuration, please refer to: + +- `how to configure the formula with map.jinja `_ +- the ``pillar.example`` file + + Contributing to this repo ------------------------- diff --git a/docs/map.jinja.rst b/docs/map.jinja.rst new file mode 100644 index 0000000..3a75e57 --- /dev/null +++ b/docs/map.jinja.rst @@ -0,0 +1,498 @@ +.. _map.jinja: + +``map.jinja``: gather formula configuration values +================================================== + +The `documentation`_ explains the use of a ``map.jinja`` to gather parameters values for a formula. + +As `pillars`_ are rendered on the Salt master for every minion, this increases the load on the master as the pillar values and the number of minions grows. + +As a good practice, you should: + +- store non-secret data in YAML files distributed by the `fileserver`_ +- store secret data in: + + - `pillars`_ (and look for the use of something like `pillar.vault`_) + - `SDB`_ (and look for the use of something like `sdb.vault`_) + +Current best practice is to let ``map.jinja`` handle parameters from all sources, to minimise the use of pillars, grains or configuration from ``sls`` files and templates directly. + + +.. contents:: **Table of Contents** + + +For formula users +----------------- + + +Quick start: configure per role and per DNS domain name values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We will see a quick setup to configure the ``TEMPLATE`` formula for different DNS domain name and several roles. + +For this example, I'll define 2 kinds of `fileserver`_ sources: + +1. formulas git repositories with hard-coded version reference to avoid breaking my setup randomly at upstream update. they are the last sources where files are looked up +2. parameters of the formulas in the file backend `roots`_ + + +Configure the fileserver backends +````````````````````````````````` + +I configure the `fileserver`_ backends to serve: + +1. files from `roots`_ first +2. `gitfs`_ repositories last + +Create the file ``/etc/salt/master.d/fileserver.conf`` and restart the ``master``: + +.. code-block:: yaml + + --- + ## + ## file server + ## + fileserver_backend: + # parameters values and override + - roots + # formulas + - gitfs + + # The files in this directory will take precedence over git repositories + file_roots: + base: + - /srv/salt + + # List of formulas I'm using + gitfs_remotes: + - https://github.com/saltstack-formulas/template-formula.git: + - base: v4.1.1 + - https://github.com/saltstack-formulas/openssh-formula.git: + - base: v2.0.1 + ... + + +Create per DNS configuration for ``TEMPLATE`` formula +````````````````````````````````````````````````````` + +Now, we can provides the per DNS domain name configuration files for the ``TEMPLATE`` formulas under ``/srv/salt/TEMPLATE/parameters/``. + +We create the directory for ``dns:domain`` grain and we add a symlink for the ``domain`` grain which is extracted from the minion ``id``: + +.. code-block:: console + + mkdir -p /srv/salt/TEMPLATE/parameters/dns:domain/ + ln -s dns:domain /srv/salt/TEMPLATE/parameters/domain + +We create a configuration for the DNS domain ``example.net`` in ``/srv/salt/TEMPLATE/parameters/dns:domain/example.net.yaml``: + +.. code-block:: yaml + + --- + values: + config: /etc/template-formula-example-net.conf + ... + +We create another configuration for the DNS domain ``example.com`` in ``/srv/salt/TEMPLATE/parameters/dns:domain/example.com.yaml``: + +.. code-block:: yaml + + --- + values: + config: /etc/template-formula-{{ grains['os_family'] }}.conf + ... + + +Create per role configuration for ``TEMPLATE`` formula +`````````````````````````````````````````````````````` + +Now, we can provides the per role configuration files for the ``TEMPLATE`` formulas under ``/srv/salt/TEMPLATE/parameters/``. + +We create the directory for roles: + +.. code-block:: console + + mkdir -p /srv/salt/TEMPLATE/parameters/roles + +We will define 2 roles: + +- ``TEMPLATE/server`` +- ``TEMPLATE/client`` + +We create a configuration for the role ``TEMPLATE/server`` in ``/srv/salt/TEMPLATE/parameters/roles/TEMPLATE/server.yaml``: + +.. code-block:: yaml + + --- + values: + config: /etc/template-formula-server.conf + ... + +We create another configuration for the role ``TEMPLATE/client`` in ``/srv/salt/TEMPLATE/parameters/roles/TEMPLATE/client.yaml``: + +.. code-block:: yaml + + --- + values: + config: /etc/template-formula-client.conf + ... + + +Enable roles and the ``dns:domain`` and ``domain`` grains for ``map.jinja`` +``````````````````````````````````````````````````````````````````````````` + +We need to redefine the sources for ``map.jinja`` to load values from our new configuration files, we provide a global configuration for all our minions. + +We create the global parameters file ``/srv/salt/parameters/map_jinja.yaml``: + +.. code-block:: yaml + + --- + values: + map_jinja: + sources: + # default values + - "Y:G@osarch" + - "Y:G@os_family" + - "Y:G@os" + - "Y:G@osfinger" + - "C@{{ tplroot ~ ':lookup' }}" + - "C@{{ tplroot }}" + + # Roles activate/deactivate things + # then thing are configured depending on environment + # So roles comes before `dns:domain`, `domain` and `id` + - "Y:C@roles" + + # DNS domain configured (DHCP or resolv.conf) + - "Y:G@dns:domain" + + # Based on minion ID + - "Y:G@domain" + + # default values + - "Y:G@id" + ... + +The syntax is explained later at `Sources of configuration values`_. + + +Bind roles to minions +````````````````````` + +We associate roles `grains`_ to minion using `grains.append`_. + +For the servers: + +.. code-block:: console + + salt 'server-*' grains.append roles TEMPLATE/server + +For the clients: + +.. code-block:: console + + salt 'client-*' grains.append roles TEMPLATE/client + +.. note:: + + Since we used ``Y:C@roles``, ``map.jinja`` will do a ``salt['config.get']('roles')`` to retrieve the roles so you could use any other method to bind roles to minions (`pillars`_ or `SDB`_) but `grains`_ seems to be the prefered method. + +Note for Microsoft Windows systems +`````````````````````````````````` + +If you have a minion running under windows, you can't use colon ``:`` as a delimiter for grain path query (see `bug 58726`_) in which case you should use an alternate delimiter: + +Modify ``/srv/salt/parameters/map_jinja.yaml`` to change the query for ``dns:domain`` to define the `alternate delimiter`_: + +.. code-block:: yaml + + --- + values: + map_jinja: + sources: + # default values + - "Y:G@osarch" + - "Y:G@os_family" + - "Y:G@os" + - "Y:G@osfinger" + - "C@{{ tplroot ~ ':lookup' }}" + - "C@{{ tplroot }}" + + # Roles activate/deactivate things + # then thing are configured depending on environment + # So roles comes before `dns:domain`, `domain` and `id` + - "Y:C@roles" + + # DNS domain configured (DHCP or resolv.conf) + - "Y:G:!@dns!domain" + + # Based on minion ID + - "Y:G@domain" + + # default values + - "Y:G@id" + ... + +And then, rename the directory: + +.. code-block:: console + + mv /srv/salt/TEMPLATE/parameters/dns:domain/ '/srv/salt/TEMPLATE/parameters/dns!domain/' + + +Format of configuration YAML files +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When you write a new YAML file, note that it must conform to the following layout: + +- a mandatory ``values`` key to store the configuration values +- two optional keys to configure the use of `salt.slsutil.merge`_ + + - an optional ``strategy`` key to configure the merging strategy, for example ``strategy: 'recurse'``, the default is ``smart`` + - an optional ``merge_lists`` key to configure if lists should be merged or overridden for the ``recurse`` and ``overwrite`` strategy, for example ``merge_lists: 'true'`` + +Here is a valid example: + +.. code-block:: yaml + + --- + strategy: 'recurse' + merge_lists: 'false' + values: + pkg: + name: 'some-package' + config: '/path/to/a/configuration/file' + ... + +You can use `Jinja`_ as with any SLS files: + +.. code-block:: yaml + + --- + strategy: 'overwrite' + merge_lists: 'true' + values: + output_dir: /tmp/{{ grains['id'] }} + ... + + +Sources of configuration values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +Configuring ``map.jinja`` sources +````````````````````````````````` + +The ``map.jinja`` file uses several sources where to lookup parameter values. The list of sources can be modified by two files: + +1. a global ``salt://parameters/map_jinja.yaml`` +2. a per formula ``salt://{{ tplroot }}/parameters/map_jinja.yaml``, it overrides the global configuration + +Each source definition has the form ``[[: