Add new lib and filter for merging datas in form of list or dict. #4799

indelog · 2022-06-07T13:36:23Z

This follow my first PR #4686 with attempt to add new module for manage content in JSON, YAML and TOML file.

The PR was closed because it was to heavy a complicated to review and test and need to be split in several parts.

This is the firs part of new functionality that i attempt to add with data_merge a lib that can be used to merge tow dataset in form of dict or list by ensuring that expected datas will be present, absent or indentic to the result in function to the wanted state.

It also add a new filter merge_data, that permit to use the functionality directly in playbook (see examples in filter documentation).

NOTE

With this you can already do things similar to my final goal like this :

- name: Get the current datas
  ansible.builtin.command: cat /path/to/my/file.json
  register: cat_json
- name: Parse JSON
  ansible.builtin.set_fact:
    current: "{{ cat_json.stdout|from_json }}"
    expected:
      'A': 1
      'B': 2
      'C': 3
- name: Merge with expected
  ansible.builtin.set_fact:
    result: "{{ current|community.general.merge_data(expected, state='present') }}"
  vars:
- name: Put merged json in the file
  ansible.builtin.copy:
    dest: cat /path/to/my/file.json
    content: "{{ result|to_nice_json(indent=2) }}"

This is a bit rough, but this is working. I'm going anyway to try to do a complete module for doing that in better way.

NOTE 2

I've found by accident a module with a similar goal to mine (in fact, all ideas that you can have, another people get it before you) :

https://docs.ansible.com/ansible/latest/collections/ansible/utils/update_fact_module.html

I think both has different approach and be complementary by permitting to provide the most appropriate answer according with the encountered problem.

felixfontein · 2022-06-08T05:35:54Z

plugins/filter/merge_data.py

+name: merge_data
+short_description: Merge dict or lists by ensuring the comparing datas are
+                   presents, absents or identics.
+version_added: 5.0.2


Suggested change

version_added: 5.0.2

version_added: 5.2.0

(5.0.2 would be a bugfix release anyway, that cannot have new features.)

felixfontein · 2022-06-08T05:36:33Z

plugins/filter/merge_data.py

+positional: expected, state, list_diff_type
+options:
+  expected:
+    type: dict|list


This is an invalid type. Use raw if you cannot decide on one.

Suggested change

type: dict|list

type: raw

plugins/filter/merge_data.py

felixfontein · 2022-06-08T05:40:02Z

plugins/module_utils/data_merge.py

+        if not isinstance(current, (dict, list)) or not isinstance(expected, (dict, list)):
+            raise ValueError('Only dict or list can be merged, {0} and {1} given.'.format(type(current), type(expected)))
+        if self.merge_type == 'identic':
+            return deepcopy(expected)


I don't think calling deepcopy is a good idea, since for a filter you can pass in arbitrary ansible-core objects that probably shoudn't simply be copied.

The filter use DataMerge that I designed to be reusable in various context, not only in this filter.

I'm afraid that if I remove the deepcopy in get_new_merged_data, get_new_merged_dict and get_new_merged_list we got unexpected behaviours later.

Firstly, we get a problem when we use merge_type='absent. get_new_merged_dict call itself recursively for each dict element that it contains. If we do not return deepcopy for the result we modifing the original object by removing some element on it, and we get problems when we try to continus to iterating over its elements (concretely, it raises RuntimeError: dictionary changed size during iteration).

Secondly, when we call get_new_merged_data, get_new_merged_dict and get_new_merged_list, we expect to get a new object. If we remove deepcopy it possible to get one of current or expected object, depending on the used params, modified by the call of the method. If I need to reuse later one of current or expected I can't because the original value has been replaced by the result. It's bit tricky...

I don't see very clearly what ansible-core objects can be passed to the filter. Please, can you give me some sample to help me that understand this. Maybe I can treat the problem in merge_data, the function called by the filter, avoiding modifying anything in DataMerge.

I'm afraid that if I remove the deepcopy in get_new_merged_data, get_new_merged_dict and get_new_merged_list we got unexpected behaviours later.

I do not mean simply removing deepcopy without replacement, but writing code which handles this correctly.

I don't see very clearly what ansible-core objects can be passed to the filter. Please, can you give me some sample to help me that understand this. Maybe I can treat the problem in merge_data, the function called by the filter, avoiding modifying anything in DataMerge.

For example hostvars is of type ansible.vars.hostvars.HostVars, and hostvars[inventory_hostname] is of type ansible.vars.hostvars.HostVarsVars. Also lookups and filters can return arbitrary Python objects.

You should probably look at how merge_hash is implemented in lib/ansible/utils/vars.py in https://github.com/ansible/ansible/

felixfontein · 2022-06-08T05:41:22Z

tests/unit/plugins/module_utils/test_data_merge.py

+        'data_expected': ['A', 'C'],
+    },
+]
+DATA_MERGE_TEST_CASE_LIST_IDS = (item['id']


Create a list instead of a generator.

Suggested change

DATA_MERGE_TEST_CASE_LIST_IDS = (item['id']

DATA_MERGE_TEST_CASE_LIST_IDS = [item['id']

(Ending needs to be changed as well.)

felixfontein · 2022-06-10T05:33:15Z

plugins/filter/merge_data.py

+version_added: 5.0.2
+author: DEMAREST Maxime (@indelog)
+description:
+  - Merge two datas, current and expected in form of dict or list.


Suggested change

- Merge two datas, current and expected in form of dict or list.

- Merge two data, current and expected in form of dict or list.

data is already plural, but I don't think you can say "two data" (https://www.merriam-webster.com/dictionary/data#usage-1). How about

Suggested change

- Merge two datas, current and expected in form of dict or list.

- Merge two data structures, current and expected in form of dict or list.

?

felixfontein · 2022-06-10T05:35:15Z

plugins/filter/merge_data.py

+description:
+  - Merge two datas, current and expected in form of dict or list.
+  - You can have mixed data like dict->list or list->dict. If the two type
+    are different, the expected datas are always returned.


I don't think I understand this. Do you mean that one argument is a dictionary, while the other a list? And with "expected", do you mean the input named "expected" (should we written as I(expected)), or the english word "expected"?

felixfontein · 2022-06-10T05:36:43Z

plugins/filter/merge_data.py

+  - For the list, you have two type of diff. You can check if the expected
+    datas are present in a list on the current datas or you can
+    check a value at the specific index of the list.
+positional: expected, state, list_diff_type


Suggested change

positional: expected, state, list_diff_type

positional: expected

I would keep state and list_diff_type as keyword arguments.

felixfontein · 2022-06-10T05:38:05Z

plugins/module_utils/data_merge.py

+        Getting the merge of two element if it can't be sure that they have the
+        same type.
+        """
+        if not isinstance(current, (dict, list)) or not isinstance(expected, (dict, list)):


Instead of list and dict you should use Sequence and Mapping (from ansible.module_utils.common._collections_compat import Mapping, Sequence).

If we do not ensure that the type of element is exactly a list or a dict, we can get some unexpected behaviour. For example, for get_new_merged_list, by replacing isinstance(current, list) or not isinstance(expected, list) by isinstance(current, Sequence) or not isinstance(expected, Sequence) we can get a str as current and expected which will result to an error.

I think I can integrate Mapping or Sequence as parameter rather than list or dict but DataMerge will be more complicated and heavy because we will have more case to treat. Is really useful to do that ?

If we do not ensure that the type of element is exactly a list or a dict, we can get some unexpected behaviour. For example, for get_new_merged_list, by replacing isinstance(current, list) or not isinstance(expected, list) by isinstance(current, Sequence) or not isinstance(expected, Sequence) we can get a str as current and expected which will result to an error.

Well, then make sure it is not isinstance of string_types (from ansible.module_utils.six import string_types).

I think I can integrate Mapping or Sequence as parameter rather than list or dict but DataMerge will be more complicated and heavy because we will have more case to treat. Is really useful to do that ?

Since this is used by a plugin, you can easily get data structures in here that are lists or dictionaries, but not of type list or dict. So yes, it is very useful to handle these correctly.

indelog · 2022-06-13T13:42:03Z

I've done all thinks indicated on comments except two on which I encounter problems. I provided details on comments responses.

My English is weak. I've difficulty to express clearly complicated things, sorry for that. I fixed indicated grammar problems and tried to rewrite some parts of the documentation. I expect that is more understandable like this.

I've also splited some elements of filter documentation in docs fragments which can be reusable by modules that may use DataMerge.

ansibullbot · 2022-06-13T14:02:40Z

The test ansible-test sanity --test import --python 2.7 [explain] failed with 1 error:

plugins/filter/merge_data.py:279:36: traceback: SyntaxError: invalid syntax

The test ansible-test sanity --test compile --python 2.7 [explain] failed with 1 error:

plugins/filter/merge_data.py:279:36: SyntaxError: def merge_data(current, expected, *, state, list_diff_type='value'):

The test ansible-test sanity --test compile --python 2.6 [explain] failed with 1 error:

plugins/filter/merge_data.py:279:36: SyntaxError: def merge_data(current, expected, *, state, list_diff_type='value'):

click here for bot help

felixfontein · 2022-06-17T19:38:02Z

plugins/filter/merge_data.py

+version_added: 5.2.0
+author: DEMAREST Maxime (@indelog)
+description:
+  - Compare data structure provided in I(_input) with data structure provided


Hmm, I don't think "compare" is the correct verb here. The filter returns a modified copy of the original data. To compute the result it does compare something, but more importantly is that it returns a modified copy.

felixfontein · 2022-06-17T19:38:41Z

plugins/filter/merge_data.py

+
+DOCUMENTATION = r'''
+name: merge_data
+short_description: Compare one data structure provided in I(_input) with an


The short description should be something really short. It is what is printed next to the plugin name in https://docs.ansible.com/ansible/devel/collections/community/general/. (It also should not end with a period.)

felixfontein · 2022-06-17T19:40:45Z

plugins/filter/merge_data.py

+EXAMPLES = r'''
+- name: Ensure that the expected data will be present with lists merged by values.
+  ansible.builtin.debug:
+    msg: "{{ current|community.general.merge_data(expected, state='present') }}"


The state parameter feels really strange for a filter. I would rather expect an operation here, as a filter doesn't ensure idempotency, but modifies something. Especially the value identic seems wrong, doesn't it simply mean "return expected"?

felixfontein · 2022-06-17T19:46:40Z

plugins/module_utils/data_merge.py

+        Getting the merge of two element if it can't be sure that they have the
+        same type.
+        """
+        if not isinstance(current, (dict, list)) or not isinstance(expected, (dict, list)):


If we do not ensure that the type of element is exactly a list or a dict, we can get some unexpected behaviour. For example, for get_new_merged_list, by replacing isinstance(current, list) or not isinstance(expected, list) by isinstance(current, Sequence) or not isinstance(expected, Sequence) we can get a str as current and expected which will result to an error.

Well, then make sure it is not isinstance of string_types (from ansible.module_utils.six import string_types).

I think I can integrate Mapping or Sequence as parameter rather than list or dict but DataMerge will be more complicated and heavy because we will have more case to treat. Is really useful to do that ?

Since this is used by a plugin, you can easily get data structures in here that are lists or dictionaries, but not of type list or dict. So yes, it is very useful to handle these correctly.

felixfontein · 2022-06-17T20:00:03Z

plugins/module_utils/data_merge.py

+        if not isinstance(current, (dict, list)) or not isinstance(expected, (dict, list)):
+            raise ValueError('Only dict or list can be merged, {0} and {1} given.'.format(type(current), type(expected)))
+        if self.merge_type == 'identic':
+            return deepcopy(expected)


I'm afraid that if I remove the deepcopy in get_new_merged_data, get_new_merged_dict and get_new_merged_list we got unexpected behaviours later.

I do not mean simply removing deepcopy without replacement, but writing code which handles this correctly.

I don't see very clearly what ansible-core objects can be passed to the filter. Please, can you give me some sample to help me that understand this. Maybe I can treat the problem in merge_data, the function called by the filter, avoiding modifying anything in DataMerge.

For example hostvars is of type ansible.vars.hostvars.HostVars, and hostvars[inventory_hostname] is of type ansible.vars.hostvars.HostVarsVars. Also lookups and filters can return arbitrary Python objects.

You should probably look at how merge_hash is implemented in lib/ansible/utils/vars.py in https://github.com/ansible/ansible/

indelog · 2022-07-01T09:12:44Z

I've been a bit long...

I've done some changes but, the background remain the same.

Firstly, i have solved the deepcopy problem but not with merge_hash way. In fact, we should not directly making copy of the original data structure as it, even with copy, because in some case it can be not mutable (like in the example that you give me with hostvars) and we will not be able to update the copy to make the new updated data data structure.

I considered a very simple solution (that i hope not be too naive). To get a new Mutable or Sequence object that contains items given by the original data structure, instead to use deepcopy or copy, just create an empty dict or list that be filled with the items found in original or modifications data, depending on that what you expect to find in the result, by directly iterating over them (rather iterating directly on the copy before to try to modifying its items). I by this way, i think that we be able to create a new data structure in all cases not depending on the nature of source data.

To to this i needed to refactor the DataMerge (i think it now more cleaner) and on the way i renamed it to IntersectData and i have redefines its parameters (i have also renamed the filter in intersect_data_with). I have also totally rewrited the documentation by trying to not express myself like a stroke victim. I hope with this that the purpose the this feature will be better defined and more understandable.

The IntersectData class was moved in module_utils/vars.py to attempt to have similar logic that done in main ansible code with lib/ansible/utils/vars.py but, I am not sure that it is correct.

As a 5.2.0 release was published, i change the version_added added number to 5.3.0.

The doc fragment data_intersection is intended to be used by modules that implement this feature.

indelog · 2022-07-15T08:36:01Z

@felixfontein
I don't understand why Build Ansible Docs fail...

ansibullbot · 2022-08-05T18:28:12Z

@indelog This PR was evaluated as a potentially problematic PR for the following reasons:

More than 50 commits.

Such PR can only be merged by human. Contact a Core team member to review this PR on IRC: #ansible-devel on Libera.chat IRC

click here for bot help

indelog · 2022-08-05T18:52:40Z

@russoz

I fixed all that is pointed in your comments.

I have renamed the filter into structure_merging and the class that it use into StructureMerging.
I wanted to prevent that this functionality to be assimilated to functions like merge_hash or dict_merge to keep it own purpose separate (which is basically to update keys/items in configuration structure like JSON or YAML). Maybe that in fact, it does not mean anything...

I'll let you tell me if other things are wrong.

russoz

Please see further comments.

russoz · 2022-08-07T10:01:39Z

.github/workflows/docs-pr.yml

@@ -21,6 +21,7 @@ jobs:
      provide-link-targets: |
        ansible_collections.ansible.builtin.dict2items_filter
        ansible_collections.ansible.builtin.path_join_filter
+        ansible_collections.ansible.utils.update_fact_module


Why is this PR touching GH workflows?

To avoid documentation build errors. Only references in this list are allowed to be used in module docs. As long as

seealso: - module: ansible.utils.update_fact description: Do something similar in another way.

is in the plugins/filter/structure_merging.py docs, this modification is needed.

russoz · 2022-08-07T21:13:31Z

plugins/module_utils/structure_merging.py

+        `present` parameter is a boolean that set if items in `changes` be used
+        to adding/updating or to removing items from `base`.
+        - When `True`, it acts to add or update items.
+        - When `False`, it acts to remove items.


The name and semantics of this parameter/feature seems off: when we say it is not present, we actually mean that it will be removed. I think this would make more sense as action having the choices ADD_OR_UPDATE and REMOVE.

And you may hate me for picking on the name again, but I think merging is defeated by present=False - we are actually subtracting rather than merging.

In fact, my general feeling is that this class is more complicated than it needs to be and, to a great length, that derives from the fact that it is trying to do many different things in one single class. I think this should be broken down into something that handles dict and something that handles list, and maybe we should break it further down into something that merges stuff and something that removes stuff (maybe this last one is a bit overkill).

russoz · 2022-08-07T21:26:37Z

plugins/module_utils/structure_merging.py

+        * get this result :
+          `{'A': {'AB': '2'}, 'B': ['4'], 'C': '5'}`
+
+        `merge_seq_by_index` is a boolean that set if `Sequence` items are


maybe name it keep_seq_index or keep_list_index? Or maybe match_seq_index - probably better.

russoz · 2022-08-07T21:28:02Z

plugins/module_utils/structure_merging.py

+        * or if using `present=False`, get :
+          `['A', {'C': '1'}]`
+
+        `keep_empty` parameter defines if emptied `Sequence` or `Mapping` items


maybe keep_empty_collections or keep_empty_containers ?

russoz · 2022-08-07T21:30:29Z

plugins/module_utils/structure_merging.py

+          when is needed to ensure that a key will be removed.
+
+        Example that ignoring items at a specific position with
+        `remove_none=False` and `merge_seq_by_index=True` :


There is so much documentation in this file that maintaining it has become an issue: the cost of change is too high because you have to go back to all the docs and perform the change in every single occurrence. Instead of over-documenting, it is better to give more meaningful names to the variables/parameters/functions so that their purpose become self-evident.

remove_none is probably a previous incarnation of keep_null?

russoz · 2022-08-07T21:58:08Z

plugins/module_utils/structure_merging.py

+            # Ignoring value of changes[k] when it None permit to avoid to
+            # update this item when it explicitly skipped with
+            # `merge_seq_by_index=True`.
+            if k not in keys_changes or changes[k] is None:


Simpler as:

Suggested change

if k not in keys_changes or changes[k] is None:

if changes.get(k) is None:

russoz · 2022-08-07T22:07:35Z

plugins/module_utils/structure_merging.py

+            base_dict = self._convert_generic_sequence_to_dict(base)
+            changes_dict = self._convert_generic_sequence_to_dict(changes)
+            res_dict = self._get_new_dict_with_changes_present(base_dict, changes_dict)
+            new = self._convert_generic_mapping_to_list(res_dict)


Converting back and forth has a couple of problems there:

increased memory usage (to hold the same data), which might become a problem if someone decides to use this for larger datasets

lists, by definition, maintain order, and the conversion method back to list does not do that

It could be solved by something (untested) like:

Suggested change

base_dict = self._convert_generic_sequence_to_dict(base)

changes_dict = self._convert_generic_sequence_to_dict(changes)

res_dict = self._get_new_dict_with_changes_present(base_dict, changes_dict)

new = self._convert_generic_mapping_to_list(res_dict)

for base_elem, chg_elem in zip(base, changes):

res_elem = self._choose_merge_method(base_elem, chg_elem, present=True)(base_elem, chg_elem)

res.append(res_elem)

russoz · 2022-08-07T22:12:16Z

plugins/module_utils/structure_merging.py

+          `{'A': '1', 'C': {'CA': '3'}}`
+        """
+
+        if self._both_are_assimilable_to_dict(base, changes):


If I may suggest, instead of repeating this if block multiple times in the code, make a small static method that performs that logic and returns the right method to you, then here that method would be assigned to _get_method (or a better name as suggested before), then lines 181-187 would actually run whatever comes out, same for 194-204, 210-215, and 221-226.

For the sake of argument, will call that function _choose_merge_method(base, change, present).

russoz · 2022-08-07T22:24:49Z

plugins/module_utils/structure_merging.py

+        return new
+
+    @staticmethod
+    def _both_are_assimilable_to_dict(base, changes):


too verbose a name... maybe _both_dict, so that we can code it like:

if _both_dict(base, changes): # blablabla

russoz · 2022-08-07T22:27:56Z

plugins/module_utils/structure_merging.py

+        if (isinstance(base, string_types) or isinstance(changes, string_types)):
+            return False
+        return (isinstance(base, Sequence) and isinstance(changes, Sequence))


I suggest you use ansible.module_utils.common.collections.is_sequence() for simplification.

felixfontein · 2022-08-08T06:25:01Z

plugins/doc_fragments/structure_merging.py

+# Copyright (c) 2022     , DEMAREST Maxime <maxime@indelog.fr>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)


Suggested change

# Copyright (c) 2022 , DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

# Copyright (c) 2022, DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)

# SPDX-License-Identifier: GPL-3.0-or-later

felixfontein · 2022-08-08T06:25:23Z

plugins/filter/structure_merging.py

+# Copyright (c) 2022     , DEMAREST Maxime <maxime@indelog.fr>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)


Suggested change

# Copyright (c) 2022 , DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

# Copyright (c) 2022, DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)

# SPDX-License-Identifier: GPL-3.0-or-later

felixfontein · 2022-08-08T06:25:49Z

plugins/module_utils/structure_merging.py

+# Copyright: (c) 2022, DEMAREST Maxime <maxime@indelog.fr>
+# GNU General Public License v3.0+
+# (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)


Suggested change

# Copyright: (c) 2022, DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+

# (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

# Copyright (c) 2022, DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)

# SPDX-License-Identifier: GPL-3.0-or-later

felixfontein · 2022-08-08T06:26:12Z

tests/integration/targets/filter_structure_merging/aliases

@@ -0,0 +1,2 @@
+shippable/posix/group3


Suggested change

shippable/posix/group3

# Copyright (c) 2022, DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)

# SPDX-License-Identifier: GPL-3.0-or-later

shippable/posix/group3

felixfontein · 2022-08-08T06:26:30Z

tests/integration/targets/filter_structure_merging/tasks/main.yml

+# (c) 2022 DEMAREST Maxime <maxime@indelog.fr>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)


Suggested change

# (c) 2022 DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

# Copyright (c) 2022, DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)

# SPDX-License-Identifier: GPL-3.0-or-later

felixfontein · 2022-08-08T06:26:44Z

tests/integration/targets/filter_structure_merging/tasks/run.yml

+# (c) 2022 DEMAREST Maxime <maxime@indelog.fr>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)


Suggested change

# (c) 2022 DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

# Copyright (c) 2022, DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)

# SPDX-License-Identifier: GPL-3.0-or-later

felixfontein · 2022-08-08T06:26:54Z

tests/integration/targets/filter_structure_merging/vars/main.yml

@@ -0,0 +1,99 @@
+---
+


Suggested change

# Copyright (c) 2022, DEMAREST Maxime <maxime@indelog.fr>

# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)

# SPDX-License-Identifier: GPL-3.0-or-later

felixfontein · 2022-08-08T06:27:18Z

tests/unit/plugins/module_utils/test_structure_merging.py

+# (c) 2022, Maxime DEMAREST <maxime@indelog.fr>
+# Copyright: (c) 2022, Ansible Project
+# GNU General Public License v3.0+
+# (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)


Suggested change

# (c) 2022, Maxime DEMAREST <maxime@indelog.fr>

# Copyright: (c) 2022, Ansible Project

# GNU General Public License v3.0+

# (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

# Copyright (c) 2022, DEMAREST Maxime <maxime@indelog.fr>

# Copyright (c) 2022, Ansible Project

# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)

# SPDX-License-Identifier: GPL-3.0-or-later

felixfontein · 2022-10-11T05:38:56Z

tests/integration/targets/filter_structure_merging/aliases

@@ -0,0 +1,2 @@
+shippable/posix/group3


Suggested change

shippable/posix/group3

azp/posix/3

(see #5344)

russoz · 2022-11-12T07:07:12Z

Hi @indelog are you still working on this?

needs_info

indelog · 2022-11-13T19:09:04Z

@russoz
I'm sorry, I haven't time to continue it for now, but I'm intended to continue this later (it will be better for me in January).

russoz · 2022-11-16T01:04:10Z

Hi @indelog please be aware that there is another PR with a similar proposition being worked on: #5533

I recommend you and this other PR author to touch base and see whether your efforts can be combined. Both PRs are about merging data structures, and it would be nice if we were not to write duplicated pieces of code to do the same stuff.

felixfontein · 2023-05-10T05:43:22Z

needs_info

ansibullbot · 2023-06-11T06:12:25Z

@indelog This pullrequest is waiting for your response. Please respond or the pullrequest will be closed.

click here for bot help

ansibullbot · 2023-07-13T06:35:39Z

@indelog You have not responded to information requests in this pullrequest so we will assume it no longer affects you. If you are still interested in this, please create a new pullrequest with the requested information.

click here for bot help

indelog changed the title ~~Add new lib and filter for merging datas in form of list or dict.~~ Add new lib and filter for merging datas in form of list or dict. 🏷️module_utils Jun 7, 2022

indelog changed the title ~~Add new lib and filter for merging datas in form of list or dict. 🏷️module_utils~~ Add new lib and filter for merging datas in form of list or dict. Jun 7, 2022

felixfontein added check-before-release PR will be looked at again shortly before release and merged if possible. backport-5 labels Jun 8, 2022

felixfontein reviewed Jun 10, 2022

View reviewed changes

ansibullbot added docs_fragments docs_fragments plugin (shared docs) needs_ci This PR requires CI testing to be performed. Please close and re-open this PR to trigger CI and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR labels Jun 13, 2022

ansibullbot removed the needs_ci This PR requires CI testing to be performed. Please close and re-open this PR to trigger CI label Jun 13, 2022

felixfontein reviewed Jun 17, 2022

View reviewed changes

ansibullbot added stale_ci CI is older than 7 days, rerun before merging and removed stale_ci CI is older than 7 days, rerun before merging labels Jun 25, 2022

ansibullbot added stale_ci CI is older than 7 days, rerun before merging and removed stale_ci CI is older than 7 days, rerun before merging labels Jul 9, 2022

indelog added 6 commits August 5, 2022 12:08

Replace param list_as_dict by merge_list_by_index

f182ea0

Rename class NewStructWith in DataStuctureMerging

17303eb

Rename filter new_struct_with in data_structure_merging

1ff7dec

Rename files according to the new name

bd0e316

Udpdating documentation

4d532aa

Finally, rename data_structure_merging into structure_merging

6602af5

ansibullbot added needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html ci_verified Push fixes to PR branch to re-run CI labels Aug 5, 2022

Forgot BOOTMETA.yml updating

6e3e0d5

ansibullbot removed the ci_verified Push fixes to PR branch to re-run CI label Aug 5, 2022

russoz requested changes Aug 7, 2022

View reviewed changes

felixfontein reviewed Aug 8, 2022

View reviewed changes

felixfontein reviewed Oct 11, 2022

View reviewed changes

felixfontein removed the backport-5 label Oct 25, 2022

felixfontein added the backport-6 label Nov 8, 2022

ansibullbot added the needs_info This issue requires further information. Please answer any outstanding questions label Nov 12, 2022

ansibullbot removed the needs_info This issue requires further information. Please answer any outstanding questions label Nov 13, 2022

russoz mentioned this pull request Nov 16, 2022

Add new merge_variables lookup plugin #5533

Merged

felixfontein removed the backport-6 label Apr 24, 2023

felixfontein added the backport-7 Automatically create a backport for the stable-7 branch label May 10, 2023

ansibullbot added the needs_info This issue requires further information. Please answer any outstanding questions label May 10, 2023

ansibullbot closed this Jul 13, 2023

	DATA_MERGE_TEST_CASE_LIST_IDS = (item['id']
	DATA_MERGE_TEST_CASE_LIST_IDS = [item['id']

	- Merge two datas, current and expected in form of dict or list.
	- Merge two data, current and expected in form of dict or list.

	- Merge two datas, current and expected in form of dict or list.
	- Merge two data structures, current and expected in form of dict or list.

	positional: expected, state, list_diff_type
	positional: expected

	if k not in keys_changes or changes[k] is None:
	if changes.get(k) is None:

		# Copyright (c) 2022 , DEMAREST Maxime <maxime@indelog.fr>
		# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

-# Copyright (c) 2022     , DEMAREST Maxime <maxime@indelog.fr>
-# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+# Copyright (c) 2022, DEMAREST Maxime <maxime@indelog.fr>
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later

		# (c) 2022 DEMAREST Maxime <maxime@indelog.fr>
		# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

Add new lib and filter for merging datas in form of list or dict. #4799

Add new lib and filter for merging datas in form of list or dict. #4799

Conversation

indelog commented Jun 7, 2022

NOTE

NOTE 2

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

indelog Jun 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

indelog commented Jun 13, 2022 • edited Loading

ansibullbot commented Jun 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

indelog commented Jul 1, 2022 • edited Loading

indelog commented Jul 15, 2022

ansibullbot commented Aug 5, 2022

indelog commented Aug 5, 2022

russoz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

russoz Aug 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

russoz commented Nov 12, 2022

indelog commented Nov 13, 2022 • edited Loading

russoz commented Nov 16, 2022

felixfontein commented May 10, 2023

ansibullbot commented Jun 11, 2023

ansibullbot commented Jul 13, 2023

indelog Jun 13, 2022 •

edited

Loading

indelog commented Jun 13, 2022 •

edited

Loading

indelog commented Jul 1, 2022 •

edited

Loading

russoz Aug 7, 2022 •

edited

Loading

indelog commented Nov 13, 2022 •

edited

Loading