Refactor Graph de/serialization

[yann-lty]

Description

Refactoring graph de/serialization for code-base simplification, fixing issues and better testability.

This PR introduces a major refactoring of the graph serialization/deserialization logic (aka graph IO).
With those changes, we address some of the known issues in this area, most notably when copy/pasting nodes.

Overall, the main concept is that any operation that needs to operate on serialized Graph data uses deserialization first, to work with Graph API, instead of manipulating a raw dictionary of serialized data.
It also creates a clear distinction in the API between the loading of a Meshroom file, the initialization of a Graph from a template and importing the content of another Graph.

Features

• Fix copy pasting issues that was failling to re-connect nodes correctly on paste.
• Deal with advanced copy/pasting scenarios, eg. handling compatibility nodes.
• Improved graph loading performance.

Implementation details

The goal of this PR is to move away from a monolothic Graph class that mixes a lot of responsibilities, to make the code more manageable and testable.

NodeFactory

The nodeFactory function has been refactored to rely on an underlying class that manages the compatibility checks. While preserving the existing behavior, this class helps understanding the different steps that occurs during that that process.

Graph

Deserialization

The current implementation exposes a unique entry point, as a single load function with several booleans to manage the different cases, to deal with:

• loading a Meshroom file
• initializing a Graph instance from a template file
• importing another graph's content

classDiagram
class Graph {
  load(filepath, setupProjectFile=True, importProject=False, publishOutputs=False)
}

Loading

This makes it complex to manage the possible combinations of options, as all of them are not necessarily meaningful.

In the new approach, those are decoupled into as many entry points as required to handle the distinct logic those operations have:

classDiagram

class Graph {
 + load(filepath: PathLike)
 + initFromTemplate(filepath: PathLike, publishOutputs: bool)
 + importGraphContentFromFile(filepath: PathLike)
 + importGraphContent(graph: Graph)
}

Loading

UID conflict check

The current implementation creates all nodes twice to deal with UID compatibility issues, which increases load times and may keep false-positives depending on the order in which nodes are handled.

In the new approach, we only consider nodes that actually have a UID conflict, and solve them by depth ordering.
Indeed, UID conflicts are contagious: an upstream node can impact the downstream ones, as UID flow through connections.
By replacing upstream nodes first with CompatibilityNodes, we restore their serialized UID, which may solve the "false-positive" conflicts downstream, that were only due to the contagion by this node.

Importing another graph content

This area benefits the most from the refactoring, which solves errors we may currently encounter in the copy-pasting of nodes.
Instead of manipulating raw dictionaries to solve conflicting node names w.r.t the current content of the Graph, it now relies on:

• the Graph API to make changes to node names in a separate Graph instance
• node serialization, to get the correctly updated link expressions (serialized Edges)
• node deserialization, to import nodes in the target Graph instance

Serialization

Introducing new GraphSerializer classes, following a Template Method pattern.
This makes it easier to define several node serialization logic, while inheriting from the standard serialization structure.

[codecov]

Codecov Report

Attention: Patch coverage is 96.40288% with 25 lines in your changes missing coverage. Please review.

Project coverage is 73.32%. Comparing base (c2d4159) to head (dbafe84).
Report is 6 commits behind head on develop.

Files with missing lines	Patch %	Lines
meshroom/core/graph.py	92.46%	11 Missing ⚠️
meshroom/core/graphIO.py	91.50%	9 Missing ⚠️
meshroom/core/nodeFactory.py	94.94%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2612      +/-   ##
===========================================
+ Coverage    69.85%   73.32%   +3.46%     
===========================================
  Files          120      125       +5     
  Lines         7067     7449     +382     
===========================================
+ Hits          4937     5462     +525     
+ Misses        2130     1987     -143     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.