Merge release-0.16 back into develop (#4272)

* Move credentials objects within the menu to the admin section * Remove mention of isolated mode in docs Fixes #3968 * diff lifecycle tests and bugs (#4168) * test and fix for bug when removed on main and updated on branch * diff merge integration test * more tests * confirm nodes added on branch are added * update path identifiers to match values expected by merge logic * more tests and bug fixes * fix query bug * formatting * add delete to integration test * final integration tests and bug fixes * UI improvement and cleaning on Search Anywhere (#4183) cleaning some old code reuse some component and utils functions Small UI update * Update on Diff tree UX/UI (#4180) * Pre-release 1 for Python SDK 0.13.0 (#4194) * IFC-264 - Diff refresh fix in branch details view (#4193) * fix branch props for branch view * display kind if no label provided * manually adds label for schema items * IFC-585 diff relationship cardinality (#4196) * IFC-585 add cardinality to diff relationship response in DiffTree * formatting * Update proposed change overview UI (#4190) - removed name and description on side card - Added description about comment and styled it as a comment - some cleaning of dead code - removed "comment added" toast because we already have a visual update of threads * IFC-597 - Handle error messages from context in apollo hook + update diff summary error state (#4195) * hide icon for diff summary error screen * add processErrorMessage function in context * enable context from hook options * enable className in error screen * disable alerts for diff summary if the branch is not found * Allow generators and artifacts to run on node without name attribute (#4187) * Use display label instead of name * Add changelog shard * Add support for using default_branch in git other than main * Diff updates for UI and E2E test (#4178) * update toggle button to be disabled * add new test for diff * uncomment diff trigger * update diff click * remove flaky verification * Update on Diff tree UX/UI (#4180) * update text locator for filters * update empty and loading states * update node titles to remove branches and give more space * update reload icon * udpate branche display * ui updtaes for branch display * add loading button for loading state to prevent UI clipping * update test * add disbaled filter buttons * fix disabled cursor * fix e2e * update reset filter condition * update locators * refresh also summary * improve lint and error message --------- Co-authored-by: Bilal ABBAD <bilal@opsmill.com> * IFC-604 - Update repositories fields (#4208) * add fields for read only and regular repo * update schema label typo * Set attribute source when assigning from number pools * IFC-601 set contains_conflict correctly on diff relationship element (#4216) * Add support for using partial_match queries with array variables (#4201) * update states filters (#4191) * add namespaces in number pool form object dropdown (#4212) * add unique for field in number pool form (#4213) * Add newsfragment for importing non main default branch Together with other PRs fixes #3435 * Use correct attribute names for update errors The model and model_attribute attributes on the number pools were renamed to node and node_attribute but this error message never changed. This PR fixes this mistake. * Remove unique_for from CoreNumberPool It was decided that we'd go with a different approach instead. * Fix number pool from_pool inconsistency * Update diff tree construction by using "parent" value of DiffTree query (#4199) * Add colors to generic repo dropdown choices * update from pool value in mutation (#4222) * IFC-587 track conflicts with an attribute instead of by ID (#4223) * IFC-587 track conflicts with an attribute instead of by ID * formatting * mode updates * update protocols again * Allow seaching by HFID using GraphQL (#4205) * [FIX] Fixed a bug prevent thread on proposed change overview (#4226) * [FIX] Edit form contains only relationship field of cardinality many if kind is Attribute or Parent (#4229) * Added test on getRelationshipsForForm func when create form * hide relationship field of cardinality many on edit form * IFC-598 - Rebase button on diff view (#4215) * add rebase button to rebase the branch + update the diff * disable filter button if no qsp * remove buttons hover effect if disabled * update reset filter UI * fix locators * add small close icons on badges for filters * fix padding * update filter locator * refactor close badges * IFC-583 improve diff labels (#4197) * IFC-583 allow getting a deleted node and its attributes * a little more unit test * IFC-583 include deleted node labels * fix unit test * add labels all over the place * finish adding more labels and testing * remove invalid conflicts during enrichment * include empty diffs in the coordinator to delete them if appropriate * fix action determination bug in cardinality one enricher * update types for leaf diff values to str | None * remove commented out code * update handling for cardinality one relationship diffs, add tests * mypy fixes * Revert "IFC-583 allow getting a deleted node and its attributes" This reverts commit 32e4bc9. * formatting * update labels without getting deleted nodes * fix un-reverted unit test * fix flappy test * npm audit: update lint-staged * Documentation for enhanced git integration (#4227) * Create placeholder for credentials * Update repository documentation following integration v2 * Update generator with new queries section * IFC-576 fields for untracked changes on DiffSummary and DiffTree (#4231) * IFC-583 allow getting a deleted node and its attributes * a little more unit test * IFC-583 include deleted node labels * fix unit test * Revert "IFC-583 allow getting a deleted node and its attributes" This reverts commit 32e4bc9. * update labels without getting deleted nodes * fix un-reverted unit test * IFC-576 add num_untracked_changes field to DiffTree * untracked fields for DiffSummary too * fix unit tests * Fix HFID when using generic relationship attributes (#4238) * make diff responses consistent with NULL vs being empty (#4242) * make diff responses consistent with NULL vs being empty * mypy fix * Fix idempotency for number pools * IFC-572 - Account permissions fix (#4230) * add is admin permission in hook + test for accounts * add permissions test * update isAdmin structure * update buttons component for account object * update adming message * update permissions messages * udpate permission message * remove log * IFC-613 - Form pool update cleanup (#4241) * fix clean up for same pool source * clean test * add override to display value in edit form * update from pool display test * 🔧 * add unit test * Apply a lock to DiffCoordinator._update_diffs() * Added counter and icon of checks conclusion on proposed change list (#4240) * Added counter on pc list tab * overflow on diff tree + tooltip * Added icon checks on pc list * adjust padding * fix typo * Apply correct relationship filter * Added help button on proposed change + update the way branches are on PC's header (#4251) * Differentiate when diff is empty vs when diff is being computed (#4247) * Rename Repository.admin_status -> internal_status Also adds missing order_weight * Fixed css issue on Time selector calendar when no datepicker is loaded (#4258) * ignore pkg_resources is deprecated as an API message in pytest config (#4256) * ignore pkg_resources is deprecated as an API message in pytest config * Add description in pyproject * Update Python packages ahead of 0.16 release (#4255) * UPdate all packages with poetry update * Pin pylint to 3.1 * Update neo4j driver and fastapi * fix message from usePermission * Move response parsing to query * IFC-618 fix diff relationship bug when conflicting changes with main changes after branch changes (#4244) * IFC-618 test that fails on root cause of the bug * IFC-618 fix bug in applying previous base values in DiffCalculator * mypy fix and remove comment * IFC-623 diff locking enhancements (#4263) * IFC-623 diff locking enhancements * add locked method to NATSLock * add unit test * add some debug logging statements * add exception handling for deleted proposed change * pylint exclusion * Added link to the related tab on each column of proposed change list (#4267) * e2e diff tests with better locking (#4264) * add cardnality in query * use node attribute for relationship one * udpate relationshiip structure for attribute * add more details in response * make relationships match attribute node diff * fix test with new UI * default open for items with conflicts * fix test with new UI * fix test after rebase * fix test * update test * avoid retry failure * increase timeout for diff update query * IFC-623 diff locking enhancements * add locked method to NATSLock * add unit test * add some debug logging statements * add exception handling for deleted proposed change * pylint exclusion --------- Co-authored-by: pa-lem <pa-lem@hotmail.fr> Co-authored-by: bilalabbad <bilal@opsmill.com> * Minor tutorial docs changes related to 'Integration with Git' * add diff conflict at the attribute level for value conflicts (#4253) * add cardnality in query * use node attribute for relationship one * udpate relationshiip structure for attribute * add more details in response * make relationships match attribute node diff * fix test with new UI * default open for items with conflicts * fix test with new UI * fix test after rebase * add diff conflict at the attribute level for value conflicts * remove HAS_VALUE conflict from the property * fix test * update test * avoid retry failure * increase timeout for diff update query * IFC-623 diff locking enhancements * add locked method to NATSLock * add unit test * add some debug logging statements * add exception handling for deleted proposed change * pylint exclusion * add conflict at top level if needed * add conflict badge in title for node attribute * update types and details * update toggle button ui * update conflict resolution ui * add diff conflict at the attribute level for value conflicts * remove HAS_VALUE conflict from the property * e2e test updates for new diff --------- Co-authored-by: pa-lem <pa-lem@hotmail.fr> Co-authored-by: bilalabbad <bilal@opsmill.com> * uncomment refetch --------- Co-authored-by: Patrick Ogenstad <patrick@ogenstad.com> Co-authored-by: Aaron McCarty <ajtmccarty@pm.me> Co-authored-by: Bilal ABBAD <bilal@opsmill.com> Co-authored-by: Guillaume Mazoyer <guillaume@opsmill.com> Co-authored-by: Baptiste <32564248+BaptisteGi@users.noreply.github.com> Co-authored-by: Damien Garros <dgarros@gmail.com> Co-authored-by: LucasG0 <lucas@opsmill.com> Co-authored-by: LucasG0 <44552904+LucasG0@users.noreply.github.com>
opsmill · Sep 6, 2024 · 5243661 · 5243661
1 parent 900f2df
commit 5243661
Show file tree

Hide file tree

Showing 186 changed files with 9,815 additions and 3,557 deletions.
diff --git a/backend/infrahub/api/menu.py b/backend/infrahub/api/menu.py
@@ -133,11 +133,6 @@ async def get_menu(branch: Branch = Depends(get_branch_dep)) -> list[InterfaceMe
     unified_storage = InterfaceMenu(
         title="Unified Storage",
         children=[
-            InterfaceMenu(
-                title="Credentials",
-                path=f"/objects/{InfrahubKind.CREDENTIAL}",
-                icon=_extract_node_icon(full_schema[InfrahubKind.CREDENTIAL]),
-            ),
             InterfaceMenu(title="Schema", path="/schema", icon="mdi:file-code"),
             InterfaceMenu(
                 title="Repository",
@@ -207,6 +202,11 @@ async def get_menu(branch: Branch = Depends(get_branch_dep)) -> list[InterfaceMe
                 path=f"/objects/{InfrahubKind.GENERICACCOUNT}",
                 icon=_extract_node_icon(full_schema[InfrahubKind.GENERICACCOUNT]),
             ),
+            InterfaceMenu(
+                title="Credentials",
+                path=f"/objects/{InfrahubKind.CREDENTIAL}",
+                icon=_extract_node_icon(full_schema[InfrahubKind.CREDENTIAL]),
+            ),
             InterfaceMenu(
                 title="Webhooks",
                 children=[

diff --git a/backend/infrahub/core/attribute.py b/backend/infrahub/core/attribute.py
@@ -88,7 +88,7 @@ def __init__(  # pylint: disable=too-many-branches
         self.at = at
         self.is_default = is_default
         self.is_from_profile = is_from_profile
-        self.from_pool: Optional[str] = None
+        self.from_pool: Optional[dict] = None
 
         self._init_node_property_mixin(kwargs)
         self._init_flag_property_mixin(kwargs)

diff --git a/backend/infrahub/core/constants/__init__.py b/backend/infrahub/core/constants/__init__.py
@@ -107,7 +107,7 @@ class CheckType(InfrahubStringEnum):
     ALL = "all"
 
 
-class RepositoryAdminStatus(InfrahubStringEnum):
+class RepositoryInternalStatus(InfrahubStringEnum):
     ACTIVE = "active"
     INACTIVE = "inactive"
     STAGING = "staging"

diff --git a/backend/infrahub/core/diff/combiner.py b/backend/infrahub/core/diff/combiner.py
@@ -3,8 +3,8 @@
 from typing import Iterable
 from uuid import uuid4
 
-from infrahub.core import registry
 from infrahub.core.constants import DiffAction, RelationshipCardinality
+from infrahub.core.constants.database import DatabaseEdgeType
 
 from .model.path import (
     EnrichedDiffAttribute,
@@ -25,7 +25,6 @@ class NodePair:
 
 class DiffCombiner:
     def __init__(self) -> None:
-        self.schema_manager = registry.schema
         # {child_uuid: (parent_uuid, parent_rel_name)}
         self._child_parent_uuid_map: dict[str, tuple[str, str]] = {}
         self._parent_node_uuids: set[str] = set()
@@ -89,20 +88,26 @@ def _should_include(self, earlier: DiffAction, later: DiffAction) -> bool:
         actions = {earlier, later}
         if actions == {DiffAction.UNCHANGED}:
             return False
-        if actions == {DiffAction.ADDED, DiffAction.REMOVED}:
+        if earlier is DiffAction.ADDED and later is DiffAction.REMOVED:
             return False
         return True
 
     def _combine_actions(self, earlier: DiffAction, later: DiffAction) -> DiffAction:
         actions = {earlier, later}
-        combined_action = DiffAction.UPDATED
-        if DiffAction.ADDED in actions:
-            combined_action = DiffAction.ADDED
-        elif DiffAction.REMOVED in actions:
-            combined_action = DiffAction.REMOVED
-        elif actions == {DiffAction.UNCHANGED}:
-            combined_action = DiffAction.UNCHANGED
-        return combined_action
+        if len(actions) == 1:
+            return actions.pop()
+        if DiffAction.UNCHANGED in actions:
+            actual_action = actions - {DiffAction.UNCHANGED}
+            return actual_action.pop()
+        actions_map = {
+            (DiffAction.ADDED, DiffAction.REMOVED): DiffAction.UPDATED,
+            (DiffAction.ADDED, DiffAction.UPDATED): DiffAction.ADDED,
+            (DiffAction.UPDATED, DiffAction.ADDED): DiffAction.UPDATED,
+            (DiffAction.UPDATED, DiffAction.REMOVED): DiffAction.REMOVED,
+            (DiffAction.REMOVED, DiffAction.ADDED): DiffAction.UPDATED,
+            (DiffAction.REMOVED, DiffAction.UPDATED): DiffAction.UPDATED,
+        }
+        return actions_map[(earlier, later)]
 
     def _combine_conflicts(
         self, earlier: EnrichedDiffConflict | None, later: EnrichedDiffConflict | None
@@ -141,16 +146,15 @@ def _combine_properties(
             combined_conflict = self._combine_conflicts(
                 earlier=earlier_property.conflict, later=later_property.conflict
             )
-            combined_property = EnrichedDiffProperty(
-                property_type=later_property.property_type,
-                changed_at=later_property.changed_at,
-                previous_value=earlier_property.previous_value,
-                new_value=later_property.new_value,
-                path_identifier=later_property.path_identifier,
-                action=self._combine_actions(earlier=earlier_property.action, later=later_property.action),
-                conflict=combined_conflict,
+            combined_properties.add(
+                replace(
+                    later_property,
+                    previous_label=earlier_property.previous_label,
+                    previous_value=earlier_property.previous_value,
+                    action=self._combine_actions(earlier=earlier_property.action, later=later_property.action),
+                    conflict=combined_conflict,
+                )
             )
-            combined_properties.add(combined_property)
         combined_properties |= {
             deepcopy(prop) for prop in later_properties if prop.property_type not in common_property_types
         }
@@ -200,11 +204,25 @@ def _combine_cardinality_one_relationship_elements(
                 earlier_properties=combined_properties, later_properties=element.properties
             )
         final_element = ordered_elements[-1]
+        peer_id = final_element.peer_id
+        peer_label = final_element.peer_label
+        # if this relationship is removed and was updated earlier, use the previous peer ID from the update
+        if combined_action is DiffAction.REMOVED:
+            for element in ordered_elements:
+                for prop in element.properties:
+                    if (
+                        prop.property_type is DatabaseEdgeType.IS_RELATED
+                        and prop.action is DiffAction.UPDATED
+                        and prop.previous_value
+                    ):
+                        peer_id = prop.previous_value
+                        peer_label = prop.previous_label
+                        break
         return EnrichedDiffSingleRelationship(
             changed_at=final_element.changed_at,
             action=combined_action,
-            peer_id=final_element.peer_id,
-            peer_label=final_element.peer_label,
+            peer_id=peer_id,
+            peer_label=peer_label,
             path_identifier=final_element.path_identifier,
             properties=combined_properties,
             conflict=self._combine_conflicts(earlier=ordered_elements[0].conflict, later=final_element.conflict),
@@ -244,9 +262,7 @@ def _combine_relationships(
         self,
         earlier_relationships: set[EnrichedDiffRelationship],
         later_relationships: set[EnrichedDiffRelationship],
-        node_kind: str,
     ) -> set[EnrichedDiffRelationship]:
-        node_schema = self.schema_manager.get_node_schema(name=node_kind, branch=self.diff_branch_name, duplicate=False)
         earlier_rels_by_name = {rel.name: rel for rel in earlier_relationships}
         later_rels_by_name = {rel.name: rel for rel in later_relationships}
         common_rel_names = set(earlier_rels_by_name.keys()) & set(later_rels_by_name.keys())
@@ -257,12 +273,10 @@ def _combine_relationships(
                 copied.nodes = set()
                 combined_relationships.add(copied)
                 continue
-            relationship_schema = node_schema.get_relationship(name=earlier_relationship.name)
-            is_cardinality_one = relationship_schema.cardinality is RelationshipCardinality.ONE
             later_relationship = later_rels_by_name[earlier_relationship.name]
             if len(earlier_relationship.relationships) == 0 and len(later_relationship.relationships) == 0:
                 combined_relationship_elements = set()
-            elif is_cardinality_one:
+            elif earlier_relationship.cardinality is RelationshipCardinality.ONE:
                 combined_relationship_elements = {
                     self._combine_cardinality_one_relationship_elements(
                         elements=(earlier_relationship.relationships | later_relationship.relationships)
@@ -275,6 +289,7 @@ def _combine_relationships(
             combined_relationship = EnrichedDiffRelationship(
                 name=later_relationship.name,
                 label=later_relationship.label,
+                cardinality=later_relationship.cardinality,
                 changed_at=later_relationship.changed_at or earlier_relationship.changed_at,
                 action=self._combine_actions(earlier=earlier_relationship.action, later=later_relationship.action),
                 path_identifier=later_relationship.path_identifier,
@@ -315,7 +330,6 @@ def _combine_nodes(self, node_pairs: list[NodePair]) -> set[EnrichedDiffNode]:
             combined_relationships = self._combine_relationships(
                 earlier_relationships=node_pair.earlier.relationships,
                 later_relationships=node_pair.later.relationships,
-                node_kind=node_pair.later.kind,
             )
             combined_action = self._combine_actions(earlier=node_pair.earlier.action, later=node_pair.later.action)
             combined_nodes.add(

diff --git a/backend/infrahub/core/diff/conflicts_enricher.py b/backend/infrahub/core/diff/conflicts_enricher.py
@@ -50,6 +50,8 @@ async def add_conflicts_to_branch_diff(
     def _add_node_conflicts(self, base_node: EnrichedDiffNode, branch_node: EnrichedDiffNode) -> None:
         if base_node.action != branch_node.action:
             self._add_node_conflict(base_node=base_node, branch_node=branch_node)
+        elif branch_node.conflict:
+            branch_node.conflict = None
         base_attribute_map = {a.name: a for a in base_node.attributes}
         branch_attribute_map = {a.name: a for a in branch_node.attributes}
         common_attribute_names = set(base_attribute_map.keys()) & set(branch_attribute_map.keys())
@@ -103,6 +105,8 @@ def _add_attribute_conflicts(
                     base_property=base_property,
                     branch_property=branch_property,
                 )
+            elif branch_property.conflict:
+                branch_property.conflict = None
 
     def _add_relationship_conflicts(
         self,
@@ -147,36 +151,49 @@ def _add_relationship_conflicts_for_one_peer(
         for property_type in common_property_types:
             base_property = base_properties_by_type[property_type]
             branch_property = branch_properties_by_type[property_type]
-            if base_property.new_value != branch_property.new_value:
-                if branch_property.conflict:
-                    conflict_uuid = branch_property.conflict.uuid
-                    selected_branch = branch_property.conflict.selected_branch
+            same_value = base_property.new_value == branch_property.new_value
+            # special handling for cardinality-one peer ID conflict
+            if branch_property.property_type is DatabaseEdgeType.IS_RELATED and is_cardinality_one:
+                if same_value:
+                    branch_element.conflict = None
+                    continue
+                if branch_element.conflict:
+                    conflict_uuid = branch_element.conflict.uuid
+                    selected_branch = branch_element.conflict.selected_branch
                 else:
                     conflict_uuid = str(uuid4())
                     selected_branch = None
-                # special handling for cardinality-one peer ID conflict
-                if branch_property.property_type is DatabaseEdgeType.IS_RELATED and is_cardinality_one:
-                    branch_element.conflict = EnrichedDiffConflict(
-                        uuid=conflict_uuid,
-                        base_branch_action=base_element.action,
-                        base_branch_value=base_property.new_value,
-                        base_branch_changed_at=base_property.changed_at,
-                        diff_branch_action=branch_element.action,
-                        diff_branch_value=branch_property.new_value,
-                        diff_branch_changed_at=branch_property.changed_at,
-                        selected_branch=selected_branch,
-                    )
-                else:
-                    branch_property.conflict = EnrichedDiffConflict(
-                        uuid=conflict_uuid,
-                        base_branch_action=base_property.action,
-                        base_branch_value=base_property.new_value,
-                        base_branch_changed_at=base_property.changed_at,
-                        diff_branch_action=branch_property.action,
-                        diff_branch_value=branch_property.new_value,
-                        diff_branch_changed_at=branch_property.changed_at,
-                        selected_branch=selected_branch,
-                    )
+                conflict = EnrichedDiffConflict(
+                    uuid=conflict_uuid,
+                    base_branch_action=base_element.action,
+                    base_branch_value=base_property.new_value,
+                    base_branch_changed_at=base_property.changed_at,
+                    diff_branch_action=branch_element.action,
+                    diff_branch_value=branch_property.new_value,
+                    diff_branch_changed_at=branch_property.changed_at,
+                    selected_branch=selected_branch,
+                )
+                branch_element.conflict = conflict
+                continue
+            if same_value:
+                branch_property.conflict = None
+                continue
+            if branch_property.conflict:
+                conflict_uuid = branch_property.conflict.uuid
+                selected_branch = branch_property.conflict.selected_branch
+            else:
+                conflict_uuid = str(uuid4())
+                selected_branch = None
+            branch_property.conflict = EnrichedDiffConflict(
+                uuid=conflict_uuid,
+                base_branch_action=base_property.action,
+                base_branch_value=base_property.new_value,
+                base_branch_changed_at=base_property.changed_at,
+                diff_branch_action=branch_property.action,
+                diff_branch_value=branch_property.new_value,
+                diff_branch_changed_at=branch_property.changed_at,
+                selected_branch=selected_branch,
+            )
 
     def _add_property_conflict(
         self,