[Resolves #1187] Add configuration for inheritance strategies for lis…

…t and dict configs (#1445)

Add configuration for inheritance strategies for list and dict configs.

This resolves #1187 by allowing selecting a dict_merge strategy at a stack group/stack level, while remaining backwards compatible with existing stacks by retaining the default of child_wins.

This does not however address the example from the original issue of including tags of dependencies (or dependents?), as I don't believe this fits with the existing model (or makes sense as a use case).

This PR additionally adds the same support to all list and dict based configs.

The config for inheritance strategy itself is also inheritable, but not configurable.

From the original issue example:

Example:
Assume Sceptre project A, B and C with dependency C -> B

root StackConfig config/config.yaml

```
stack_tags_inheritance: dict_merge
stack_tags:
  - country: US
```

config/prod/A.yaml

```
stack_tags:
  - city: anaheim    # stack A tags are 'city:anaheim', 'country:US'
```

config/prod/C.yaml

```
dependencies:
  - prod/B.yaml
stack_tags:
  - county: collin   # stack C tags are 'county:collin', 'country:US'
```

config/prod/B.yaml

```
stack_tags:         # In contrast to original issue example, dependencies do not impact inherited values
  - city: boston    # stack B tags are 'city:boston', 'country:US'
```

docs/_source/docs/stack_config.rst

@@ -15,13 +15,16 @@ particular Stack. The available keys are listed below.
 
 - `template_path`_ or `template`_ *(required)*
 - `dependencies`_ *(optional)*
+- `dependencies_inheritance`_ *(optional)*
 - `hooks`_ *(optional)*
+- `hooks_inheritance`_ *(optional)*
 - `ignore`_ *(optional)*
 - `notifications`_ *(optional)*
 - `obsolete`_ *(optional)*
 - `on_failure`_ *(optional)*
 - `disable_rollback`_ *(optional)*
 - `parameters`_ *(optional)*
+- `parameters_inheritance`_ *(optional)*
 - `protected`_ *(optional)*
 - `role_arn`_ *(optional)*
 - `cloudformation_service_role`_ *(optional)*
@@ -30,8 +33,10 @@ particular Stack. The available keys are listed below.
 - `iam_role_session_duration`_ *(optional)*
 - `sceptre_role_session_duration`_ *(optional)*
 - `sceptre_user_data`_ *(optional)*
+- `sceptre_user_data_inheritance`_ *(optional)*
 - `stack_name`_ *(optional)*
 - `stack_tags`_ *(optional)*
+- `stack_tags_inheritance`_ *(optional)*
 - `stack_timeout`_ *(optional)*
 
 It is not possible to define both `template_path`_ and `template`_. If you do so,
@@ -80,7 +85,7 @@ dependencies
 ~~~~~~~~~~~~
 * Resolvable: No
 * Can be inherited from StackGroup: Yes
-* Inheritance strategy: Appended to parent's dependencies
+* Inheritance strategy: Appended to parent's dependencies. Configurable with ``dependencies_inheritance`` parameter.
 
 A list of other Stacks in the environment that this Stack depends on. Note that
 if a Stack fetches an output value from another Stack using the
@@ -97,15 +102,39 @@ and that Stack need not be added as an explicit dependency.
  situation by either (a) setting those ``dependencies`` on individual Stack Configs rather than the
  the StackGroup Config, or (b) moving those dependency stacks outside of the StackGroup.
 
+dependencies_inheritance
+~~~~~~~~~~~~~~~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
+
+This configuration will override the default inheritance strategy of `dependencies`.
+
+The default value for this is ``merge``.
+
+Valid values for this config are: ``merge``, or ``override``.
+
 hooks
 ~~~~~
 * Resolvable: No (but you can use resolvers _in_ hook arguments!)
 * Can be inherited from StackGroup: Yes
-* Inheritance strategy: Overrides parent if set
+* Inheritance strategy: Overrides parent if set. Configurable with ``hooks_inheritance`` parameter.
 
 A list of arbitrary shell or Python commands or scripts to run. Find out more
 in the :doc:`hooks` section.
 
+hooks_inheritance
+~~~~~~~~~~~~~~~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
+
+This configuration will override the default inheritance strategy of `hooks`.
+
+The default value for this is ``override``.
+
+Valid values for this config are: ``merge``, or ``override``.
+
 ignore
 ~~~~~~
 * Resolvable: No
@@ -155,7 +184,7 @@ notifications
 
 List of SNS topic ARNs to publish Stack related events to. A maximum of 5 ARNs
 can be specified per Stack. This configuration will be used by the ``create``,
-``update``, and ``delete`` commands. More information about Stack notifications
+``update``, or ``delete`` commands. More information about Stack notifications
 can found under the relevant section in the `AWS CloudFormation API
 documentation`_.
 
@@ -231,7 +260,7 @@ parameters
 ~~~~~~~~~~
 * Resolvable: Yes
 * Can be inherited from StackGroup: Yes
-* Inheritance strategy: Overrides parent if set
+* Inheritance strategy: Overrides parent if set. Configurable with ``parameters_inheritance`` parameter.
 
 .. warning::
 
@@ -241,7 +270,7 @@ parameters
  environment variable resolver.
 
 A dictionary of key-value pairs to be supplied to a template as parameters. The
-keys must match up with the name of the parameter, and the value must be of the
+keys must match up with the name of the parameter, or the value must be of the
 type as defined in the template.
 
 .. note::
@@ -292,6 +321,18 @@ Example:
  - !stack_output security-groups.yaml::BaseSecurityGroupId
  - !file_contents /file/with/security_group_id.txt
 
+parameters_inheritance
+~~~~~~~~~~~~~~~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
+
+This configuration will override the default inheritance strategy of `parameters`.
+
+The default value for this is ``override``.
+
+Valid values for this config are: ``merge``, or ``override``.
+
 protected
 ~~~~~~~~~
 * Resolvable: No
@@ -391,12 +432,24 @@ sceptre_user_data
 ~~~~~~~~~~~~~~~~~
 * Resolvable: Yes
 * Can be inherited from StackGroup: Yes
-* Inheritance strategy: Overrides parent if set
+* Inheritance strategy: Overrides parent if set. Configurable with ``sceptre_user_data_inheritance`` parameter.
 
 Represents data to be passed to the ``sceptre_handler(sceptre_user_data)``
 function in Python templates or accessible under ``sceptre_user_data`` variable
 key within Jinja2 templates.
 
+sceptre_user_data_inheritance
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
+
+This configuration will override the default inheritance strategy of `sceptre_user_data`.
+
+The default value for this is ``override``.
+
+Valid values for this config are: ``merge``, or ``override``.
+
 stack_name
 ~~~~~~~~~~
 * Resolvable: No
@@ -436,10 +489,22 @@ stack_tags
 ~~~~~~~~~~
 * Resolvable: Yes
 * Can be inherited from StackGroup: Yes
-* Inheritance strategy: Overrides parent if set
+* Inheritance strategy: Overrides parent if set. Configurable with ``stack_tags_inheritance`` parameter.
 
 A dictionary of `CloudFormation Tags`_ to be applied to the Stack.
 
+stack_tags_inheritance
+~~~~~~~~~~~~~~~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
+
+This configuration will override the default inheritance strategy of `stack_tags`.
+
+The default value for this is ``override``.
+
+Valid values for this config are: ``merge``, or ``override``.
+
 stack_timeout
 ~~~~~~~~~~~~~
 * Resolvable: No

docs/_source/docs/stack_group_config.rst

@@ -175,7 +175,7 @@ configurations should be defined at a lower directory level.
 
 YAML files that define configuration settings with conflicting keys, the child
 configuration file will usually take precedence (see the specific config keys as documented
-for the inheritance strategy employed).
+for the inheritance strategy employed and `Inheritance Strategy Override`_).
 
 In the above directory structure, ``config/config.yaml`` will be read in first,
 followed by ``config/account-1/config.yaml``, followed by
@@ -185,6 +185,16 @@ For example, if you wanted the ``dev`` StackGroup to build to a different
 region, this setting could be specified in the ``config/dev/config.yaml`` file,
 and would only be applied to builds in the ``dev`` StackGroup.
 
+Inheritance Strategy Override
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The inheritance strategy of some properties may be overridden by the stack group config.
+
+Strategy options:
+
+* ``merge``: Child config is merged with parent configs, with child taking precedence for conflicting dictionary keys.
+* ``override``: Overrides the parent config, if set.
+
 .. _setting_dependencies_for_stack_groups:
 
 Setting Dependencies for StackGroups

sceptre/config/reader.py

@@ -39,16 +39,30 @@
 
 ConfigAttributes = collections.namedtuple("Attributes", "required optional")
 
+
+CONFIG_MERGE_STRATEGY_OVERRIDES = {
+ "dependencies": strategies.LIST_STRATEGIES,
+ "hooks": strategies.LIST_STRATEGIES,
+ "notifications": strategies.LIST_STRATEGIES,
+ "parameters": strategies.DICT_STRATEGIES,
+ "sceptre_user_data": strategies.DICT_STRATEGIES,
+ "stack_tags": strategies.DICT_STRATEGIES,
+}
+
 CONFIG_MERGE_STRATEGIES = {
  "dependencies": strategies.list_join,
+ "dependencies_inheritance": strategies.child_or_parent,
  "hooks": strategies.child_wins,
+ "hooks_inheritance": strategies.child_or_parent,
  "iam_role": strategies.child_wins,
  "sceptre_role": strategies.child_wins,
  "iam_role_session_duration": strategies.child_wins,
  "sceptre_role_session_duration": strategies.child_wins,
  "notifications": strategies.child_wins,
+ "notifications_inheritance": strategies.child_or_parent,
  "on_failure": strategies.child_wins,
  "parameters": strategies.child_wins,
+ "parameters_inheritance": strategies.child_or_parent,
  "profile": strategies.child_wins,
  "project_code": strategies.child_wins,
  "protect": strategies.child_wins,
@@ -57,8 +71,10 @@
  "role_arn": strategies.child_wins,
  "cloudformation_service_role": strategies.child_wins,
  "sceptre_user_data": strategies.child_wins,
+ "sceptre_user_data_inheritance": strategies.child_or_parent,
  "stack_name": strategies.child_wins,
  "stack_tags": strategies.child_wins,
+ "stack_tags_inheritance": strategies.child_or_parent,
  "stack_timeout": strategies.child_wins,
  "template_bucket_name": strategies.child_wins,
  "template_key_value": strategies.child_wins,
@@ -68,6 +84,7 @@
  "obsolete": strategies.child_wins,
 }
 
+
 STACK_GROUP_CONFIG_ATTRIBUTES = ConfigAttributes(
  {"project_code", "region"},
  {
@@ -84,21 +101,26 @@
  "template_path",
  "template",
  "dependencies",
+ "dependencies_inheritance",
  "hooks",
+ "hooks_inheritance",
  "iam_role",
  "sceptre_role",
  "iam_role_session_duration",
  "sceptre_role_session_duration",
  "notifications",
  "on_failure",
  "parameters",
+ "parameters_inheritance",
  "profile",
  "protect",
  "role_arn",
  "cloudformation_service_role",
  "sceptre_user_data",
+ "sceptre_user_data_inheritance",
  "stack_name",
  "stack_tags",
+ "stack_tags_inheritance",
  "stack_timeout",
  },
 )
@@ -352,11 +374,8 @@ def _read(self, rel_path, base_config=None):
 
  # Parse and read in the config files.
  this_config = self._recursive_read(directory_path, filename, config)
-
- if "dependencies" in config or "dependencies" in this_config:
- this_config["dependencies"] = CONFIG_MERGE_STRATEGIES["dependencies"](
- this_config.get("dependencies"), config.get("dependencies")
- )
+ # Apply merge strategies with the config that includes base_config values.
+ this_config.update(self._get_merge_with_stratgies(config, this_config))
  config.update(this_config)
 
  self._check_version(config)
@@ -395,16 +414,39 @@ def _recursive_read(
 
  # Read config file and overwrite inherited properties
  child_config = self._render(directory_path, filename, config_group) or {}
+ child_config.update(self._get_merge_with_stratgies(config, child_config))
+ config.update(child_config)
+ return config
 
- for config_key, strategy in CONFIG_MERGE_STRATEGIES.items():
- value = strategy(config.get(config_key), child_config.get(config_key))
+ def _get_merge_with_stratgies(self, left: dict, right: dict) -> dict:
+ """
+ Returns a new dict with only the merge values of the two inputs, using the
+ merge strategies defined for each key.
+ """
+ merge = {}
+
+ # Then apply the merge strategies to each item
+ for config_key, default_strategy in CONFIG_MERGE_STRATEGIES.items():
+ strategy = default_strategy
+ override_key = f"{config_key}_inheritance"
+ if override_key in CONFIG_MERGE_STRATEGIES:
+ name = CONFIG_MERGE_STRATEGIES[override_key](
+ left.get(override_key), right.get(override_key)
+ )
+ if not name:
+ pass
+ elif name not in CONFIG_MERGE_STRATEGY_OVERRIDES[config_key]:
+ raise SceptreException(
+ f"{name} is not a valid inheritance strategy for {config_key}"
+ )
+ else:
+ strategy = CONFIG_MERGE_STRATEGY_OVERRIDES[config_key][name]
 
+ value = strategy(left.get(config_key), right.get(config_key))
  if value:
- child_config[config_key] = value
+ merge[config_key] = value
 
- config.update(child_config)
-
- return config
+ return merge
 
  def _render(self, directory_path, basename, stack_group_config):
  """

sceptre/config/strategies.py

@@ -71,3 +71,26 @@ def child_wins(a, b):
  :returns: b
  """
  return b
+
+
+def child_or_parent(a, b):
+ """
+ Returns the second arg if it is not empty, else the first.
+
+ :param a: An object.
+ :type a: object
+ :param b: An object.
+ :type b: object
+ :returns: b
+ """
+ return b or a
+
+
+LIST_STRATEGIES = {
+ "merge": list_join,
+ "override": child_wins,
+}
+DICT_STRATEGIES = {
+ "merge": dict_merge,
+ "override": child_wins,
+}