-
Notifications
You must be signed in to change notification settings - Fork 75
/
stack.py
516 lines (432 loc) · 21.3 KB
/
stack.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
# -*- coding: utf-8 -*-
'''
Simple and flexible YAML ext_pillar which can read pillar from within pillar.
.. versionadded:: 2016.3.0
This custom saltstack ``ext_pillar`` is inspired by
`varstack <https://github.com/conversis/varstack>`_ but is heavily based on
Jinja2 for maximum flexibility.
It supports the following features:
- multiple config files that are jinja2 templates with support for ``pillar``,
``__grains__``, ``__salt__``, ``__opts__`` objects
- a config file renders as an ordered list of files (paths of these files are
relative to the current config file)
- this list of files are read in ordered as jinja2 templates with support for
``stack``, ``pillar``, ``__grains__``, ``__salt__``, ``__opts__`` objects
- all these rendered files are then parsed as ``yaml``
- then all yaml dicts are merged in order with support for the following
merging strategies: ``merge-first``, ``merge-last``, ``remove``, and
``overwrite``
- stack config files can be matched based on ``pillar``, ``grains``, or
``opts`` values, which make it possible to support kind of self-contained
environments
Installation
------------
PillarStack is already bundled with Salt since 2016.3.0 version so there is
nothing to install from version 2016.3.0.
If you use an older Salt version or you want to override PillarStack with a
more recent one, follow the installation procedure below.
Installing the PillarStack ``ext_pillar`` is as simple as dropping the
``stack.py`` file in the ``<extension_modules>/pillar`` directory (no external
python module required), given that ``extension_modules`` is set in your
salt-master configuration, see:
http://docs.saltstack.com/en/latest/ref/configuration/master.html#extension-modules
Configuration in Salt
---------------------
Like any other external pillar, its configuration takes place through the
``ext_pillar`` key in the master config file.
However, you can configure PillarStack in 3 different ways:
Single config file
~~~~~~~~~~~~~~~~~~
This is the simplest option, you just need to set the path to your single
PillarStack config file like below:
.. code:: yaml
ext_pillar:
- stack: /path/to/stack.cfg
List of config files
~~~~~~~~~~~~~~~~~~~~
You can also provide a list of config files:
.. code:: yaml
ext_pillar:
- stack:
- /path/to/stack1.cfg
- /path/to/stack2.cfg
Select config files through grains|pillar|opts matching
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can also opt for a much more flexible configuration: PillarStack allows to
select the config files for the current minion based on matching values from
either grains, or pillar, or opts objects.
Here is an example of such a configuration, which should speak by itself:
.. code:: yaml
ext_pillar:
- stack:
pillar:environment:
dev: /path/to/dev/stack.cfg
prod: /path/to/prod/stack.cfg
grains:custom:grain:
value:
- /path/to/stack1.cfg
- /path/to/stack2.cfg
opts:custom:opt:
value: /path/to/stack0.cfg
PillarStack configuration files
-------------------------------
The config files that are referenced in the above ``ext_pillar`` configuration
are jinja2 templates which must render as a simple ordered list of ``yaml``
files that will then be merged to build pillar data.
The path of these ``yaml`` files must be relative to the directory of the
PillarStack config file.
The following variables are available in jinja2 templating of PillarStack
configuration files:
- ``pillar``: the pillar data (as passed by Salt to our ``ext_pillar``
function)
- ``minion_id``: the minion id ;-)
- ``__opts__``: a dictionary of mostly Salt configuration options
- ``__grains__``: a dictionary of the grains of the minion making this pillar
call
- ``__salt__``: a dictionary of Salt module functions, useful so you don't have
to duplicate functions that already exist (note: runs on the master)
So you can use all the power of jinja2 to build your list of ``yaml`` files
that will be merged in pillar data.
For example, you could have a PillarStack config file which looks like:
.. code:: jinja
$ cat /path/to/stack/config.cfg
core.yml
osarchs/{{ __grains__['osarch'] }}.yml
oscodenames/{{ __grains__['oscodename'] }}.yml
{%- for role in pillar.get('roles', []) %}
roles/{{ role }}.yml
{%- endfor %}
minions/{{ minion_id }}.yml
And the whole directory structure could look like:
.. code::
$ tree /path/to/stack/
/path/to/stack/
├── config.cfg
├── core.yml
├── osarchs/
│ ├── amd64.yml
│ └── armhf.yml
├── oscodenames/
│ ├── wheezy.yml
│ └── jessie.yml
├── roles/
│ ├── web.yml
│ └── db.yml
└── minions/
├── test-1-dev.yml
└── test-2-dev.yml
Overall process
---------------
In the above PillarStack configuration, given that test-1-dev minion is an
amd64 platform running Debian Jessie, and which pillar ``roles`` is ``["db"]``,
the following ``yaml`` files would be merged in order:
- ``core.yml``
- ``osarchs/amd64.yml``
- ``oscodenames/jessie.yml``
- ``roles/db.yml``
- ``minions/test-1-dev.yml``
Before merging, every files above will be preprocessed as Jinja2 templates.
The following variables are available in Jinja2 templating of ``yaml`` files:
- ``stack``: the PillarStack pillar data object that has currently been merged
(data from previous ``yaml`` files in PillarStack configuration)
- ``pillar``: the pillar data (as passed by Salt to our ``ext_pillar``
function)
- ``minion_id``: the minion id ;-)
- ``__opts__``: a dictionary of mostly Salt configuration options
- ``__grains__``: a dictionary of the grains of the minion making this pillar
call
- ``__salt__``: a dictionary of Salt module functions, useful so you don't have
to duplicate functions that already exist (note: runs on the master)
So you can use all the power of jinja2 to build your pillar data, and even use
other pillar values that has already been merged by PillarStack (from previous
``yaml`` files in PillarStack configuration) through the ``stack`` variable.
Once a ``yaml`` file has been preprocessed by Jinja2, we obtain a Python dict -
let's call it ``yml_data`` - then, PillarStack will merge this ``yml_data``
dict in the main ``stack`` dict (which contains already merged PillarStack
pillar data).
By default, PillarStack will deeply merge ``yml_data`` in ``stack`` (similarly
to the ``recurse`` salt ``pillar_source_merging_strategy``), but 3 merging
strategies are currently available for you to choose (see next section).
Once every ``yaml`` files have been processed, the ``stack`` dict will contain
your whole own pillar data, merged in order by PillarStack.
So PillarStack ``ext_pillar`` returns the ``stack`` dict, the contents of which
Salt takes care to merge in with all of the other pillars and finally return
the whole pillar to the minion.
Merging strategies
------------------
The way the data from a new ``yaml_data`` dict is merged with the existing
``stack`` data can be controlled by specifying a merging strategy. Right now
this strategy can either be ``merge-last`` (the default), ``merge-first``,
``remove``, or ``overwrite``.
Note that scalar values like strings, integers, booleans, etc. are always
evaluated using the ``overwrite`` strategy (other strategies don't make sense
in that case).
The merging strategy can be set by including a dict in the form of:
.. code:: yaml
__: <merging strategy>
as the first item of the dict or list.
This allows fine grained control over the merging process.
``merge-last`` (default) strategy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the ``merge-last`` strategy is selected (the default), then content of dict
or list variables is merged recursively with previous definitions of this
variable (similarly to the ``recurse`` salt
``pillar_source_merging_strategy``).
This allows for extending previously defined data.
``merge-first`` strategy
~~~~~~~~~~~~~~~~~~~~~~~~
If the ``merge-first`` strategy is selected, then the content of dict or list
variables are swapped between the ``yaml_data`` and ``stack`` objects before
being merged recursively with the ``merge-last`` previous strategy.
``remove`` strategy
~~~~~~~~~~~~~~~~~~~
If the ``remove`` strategy is selected, then content of dict or list variables
in ``stack`` are removed only if the corresponding item is present in the
``yaml_data`` dict.
This allows for removing items from previously defined data.
``overwrite`` strategy
~~~~~~~~~~~~~~~~~~~~~~
If the ``overwrite`` strategy is selected, then the content of dict or list
variables in ``stack`` is overwritten by the content of ``yaml_data`` dict.
So this allows one to overwrite variables from previous definitions.
Merging examples
----------------
Let's go through small examples that should clarify what's going on when a
``yaml_data`` dict is merged in the ``stack`` dict.
When you don't specify any strategy, the default ``merge-last`` strategy is
selected:
+----------------------+-----------------------+-------------------------+
| ``stack`` | ``yaml_data`` | ``stack`` (after merge) |
+======================+=======================+=========================+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | tom: | tom: |
| uid: 500 | uid: 1000 | uid: 1000 |
| roles: | roles: | roles: |
| - sysadmin | - developer | - sysadmin |
| root: | mat: | - developer |
| uid: 0 | uid: 1001 | mat: |
| | | uid: 1001 |
| | | root: |
| | | uid: 0 |
+----------------------+-----------------------+-------------------------+
Then you can select a custom merging strategy using the ``__`` key in a dict:
+----------------------+-----------------------+-------------------------+
| ``stack`` | ``yaml_data`` | ``stack`` (after merge) |
+======================+=======================+=========================+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | __: merge-last | tom: |
| uid: 500 | tom: | uid: 1000 |
| roles: | uid: 1000 | roles: |
| - sysadmin | roles: | - sysadmin |
| root: | - developer | - developer |
| uid: 0 | mat: | mat: |
| | uid: 1001 | uid: 1001 |
| | | root: |
| | | uid: 0 |
+----------------------+-----------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | __: merge-first | tom: |
| uid: 500 | tom: | uid: 500 |
| roles: | uid: 1000 | roles: |
| - sysadmin | roles: | - developer |
| root: | - developer | - sysadmin |
| uid: 0 | mat: | mat: |
| | uid: 1001 | uid: 1001 |
| | | root: |
| | | uid: 0 |
+----------------------+-----------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | __: remove | root: |
| uid: 500 | tom: | uid: 0 |
| roles: | mat: | |
| - sysadmin | | |
| root: | | |
| uid: 0 | | |
+----------------------+-----------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | __: overwrite | tom: |
| uid: 500 | tom: | uid: 1000 |
| roles: | uid: 1000 | roles: |
| - sysadmin | roles: | - developer |
| root: | - developer | mat: |
| uid: 0 | mat: | uid: 1001 |
| | uid: 1001 | |
+----------------------+-----------------------+-------------------------+
You can also select a custom merging strategy using a ``__`` object in a list:
+----------------+-------------------------+-------------------------+
| ``stack`` | ``yaml_data`` | ``stack`` (after merge) |
+================+=========================+=========================+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| - tom | - __: merge-last | - tom |
| - root | - mat | - root |
| | | - mat |
+----------------+-------------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| - tom | - __: merge-first | - mat |
| - root | - mat | - tom |
| | | - root |
+----------------+-------------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| - tom | - __: remove | - root |
| - root | - mat | |
| | - tom | |
+----------------+-------------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| - tom | - __: overwrite | - mat |
| - root | - mat | |
+----------------+-------------------------+-------------------------+
'''
from __future__ import absolute_import
import os
import logging
from functools import partial
import yaml
from jinja2 import FileSystemLoader, Environment, TemplateNotFound
import six
log = logging.getLogger(__name__)
strategies = ('overwrite', 'merge-first', 'merge-last', 'remove')
def ext_pillar(minion_id, pillar, *args, **kwargs):
import salt.utils
stack = {}
stack_config_files = list(args)
traverse = {
'pillar': partial(salt.utils.traverse_dict_and_list, pillar),
'grains': partial(salt.utils.traverse_dict_and_list, __grains__),
'opts': partial(salt.utils.traverse_dict_and_list, __opts__),
}
for matcher, matchs in six.iteritems(kwargs):
t, matcher = matcher.split(':', 1)
if t not in traverse:
raise Exception('Unknown traverse option "{0}", '
'should be one of {1}'.format(t, list(traverse.keys())))
cfgs = matchs.get(traverse[t](matcher, None), [])
if not isinstance(cfgs, list):
cfgs = [cfgs]
stack_config_files += cfgs
for cfg in stack_config_files:
if not os.path.isfile(cfg):
log.warning('Ignoring pillar stack cfg "{0}": '
'file does not exist'.format(cfg))
continue
stack = _process_stack_cfg(cfg, stack, minion_id, pillar)
return stack
def _process_stack_cfg(cfg, stack, minion_id, pillar):
log.debug('Config: {0}'.format(cfg))
basedir, filename = os.path.split(cfg)
jenv = Environment(loader=FileSystemLoader(basedir))
jenv.globals.update({
"__opts__": __opts__,
"__salt__": __salt__,
"__grains__": __grains__,
"minion_id": minion_id,
"pillar": pillar,
})
for path in _parse_stack_cfg(jenv.get_template(filename).render(stack=stack)):
try:
log.debug('YAML: basedir={0}, path={1}'.format(basedir, path))
obj = yaml.safe_load(jenv.get_template(path).render(stack=stack))
log.debug('obj: {0}'.format(obj))
if not isinstance(obj, dict):
log.info('Ignoring pillar stack template "{0}": Can\'t parse '
'as a valid yaml dictionary'.format(path))
continue
stack = _merge_dict(stack, obj)
log.debug('stack: {0}'.format(stack))
except TemplateNotFound as e:
if hasattr(e, 'name') and e.name != path:
log.info('Jinja include file "{0}" not found '
'from root dir "{1}", which was included '
'by stack template "{2}"'.format(
e.name, basedir, path))
else:
log.info('Ignoring pillar stack template "{0}": can\'t find from '
'root dir "{1}"'.format(path, basedir))
continue
return stack
def _cleanup(obj):
if obj:
if isinstance(obj, dict):
obj.pop('__', None)
for k, v in six.iteritems(obj):
obj[k] = _cleanup(v)
elif isinstance(obj, list) and isinstance(obj[0], dict) \
and '__' in obj[0]:
del obj[0]
return obj
def _merge_dict(stack, obj):
strategy = obj.pop('__', 'merge-last')
if strategy not in strategies:
raise Exception('Unknown strategy "{0}", should be one of {1}'.format(
strategy, strategies))
if strategy == 'overwrite':
return _cleanup(obj)
else:
for k, v in six.iteritems(obj):
if strategy == 'remove':
stack.pop(k, None)
continue
if k in stack:
if strategy == 'merge-first':
# merge-first is same as merge-last but the other way round
# so let's switch stack[k] and v
stack_k = stack[k]
stack[k] = _cleanup(v)
v = stack_k
if type(stack[k]) != type(v):
log.debug('Force overwrite, types differ: '
'\'{0}\' != \'{1}\''.format(stack[k], v))
stack[k] = _cleanup(v)
elif isinstance(v, dict):
stack[k] = _merge_dict(stack[k], v)
elif isinstance(v, list):
stack[k] = _merge_list(stack[k], v)
else:
stack[k] = v
else:
stack[k] = _cleanup(v)
return stack
def _merge_list(stack, obj):
strategy = 'merge-last'
if obj and isinstance(obj[0], dict) and '__' in obj[0]:
strategy = obj[0]['__']
del obj[0]
if strategy not in strategies:
raise Exception('Unknown strategy "{0}", should be one of {1}'.format(
strategy, strategies))
if strategy == 'overwrite':
return obj
elif strategy == 'remove':
return [item for item in stack if item not in obj]
elif strategy == 'merge-first':
return obj + stack
else:
return stack + obj
def _parse_stack_cfg(content):
'''
Allow top level cfg to be YAML
'''
try:
obj = yaml.safe_load(content)
if isinstance(obj, list):
return obj
except Exception as e:
pass
return content.splitlines()