directed graph serialization

[CagtayFabry]

Changes

Here is a simple implementation to handle networkx.DiGraph more generically via ASDF, represent a basic graph in a tree like structure.

The motivation is to enable a simple way to define, describe and validate of specific graph structure/layout in ASDF schemas with minmal custom code on the python side.

Checks

• updated CHANGELOG.md
• updated tests
• updated doc/
• update example/tutorial notebooks

basic implementation

Two basic types are introduced to represent and tag generic directed graph nodes and edges in weldx files.

@dataclass
class DiEdge:
    """Generic directed edge type."""

    target_node: "DiNode"
    attributes: dict = field(default_factory=dict)
    direction: str = "fwd"

@dataclass
class DiNode:
    """Generic directed graph node type."""

    edges: List["DiEdge"]
    name: str = field(default_factory=uuid4)
    attributes: dict = field(default_factory=dict)

Each DiNode has list of DiEdges that descend further "down" into the tree.

Each edge points to a single target_node and also if the original edge direction descents into the tree "fwd" or is pointing towards root "bwd".

Here is the output of a simple directed graph A->B->C

graph: !<tag:weldx.bam.de:weldx/core/graph/graph-1.0.0>
  root_node: !<tag:weldx.bam.de:weldx/core/graph/node-1.0.0>
    name: A
    edges:
    - !<tag:weldx.bam.de:weldx/core/graph/edge-1.0.0>
      direction: fwd
      target_node: !<tag:weldx.bam.de:weldx/core/graph/node-1.0.0>
        name: B
        edges:
        - !<tag:weldx.bam.de:weldx/core/graph/edge-1.0.0>
          direction: fwd
          target_node: !<tag:weldx.bam.de:weldx/core/graph/node-1.0.0>
            name: C
            edges: []

Here the same graph in reverse order, also starting at node A A<-B<-C

graph: !<tag:weldx.bam.de:weldx/core/graph/graph-1.0.0>
  root_node: !<tag:weldx.bam.de:weldx/core/graph/node-1.0.0>
    name: A
    edges:
    - !<tag:weldx.bam.de:weldx/core/graph/edge-1.0.0>
      direction: bwd
      target_node: !<tag:weldx.bam.de:weldx/core/graph/node-1.0.0>
        name: B
        edges:
        - !<tag:weldx.bam.de:weldx/core/graph/edge-1.0.0>
          direction: bwd
          target_node: !<tag:weldx.bam.de:weldx/core/graph/node-1.0.0>
            name: C
            edges: []

Attributes that are added to nodes or edges via networkx will be serialized under the respective attributes property.

ASDF customization

To use the graph with custom types, specialized node and edge ASDF schemas can be defined.
Here is an example on how one could specify a measurement chain graph with the following layout and restrictions.

• each node represents a Signal
• each edge represents a SignalTransform
• no other node or edge types are allowed
• every SignalTransform must be defined in forward direction (no inverse transformations)
• every Signal can only have one transformation attached

This simple structure corresponds to the arborescence layout in networkx, see https://networkx.org/documentation/stable//reference/algorithms/tree.html

In essence, a MeasurementChain is simply a graph where nodes and edges have a Signal or SignalTransform in their attributes plus the above layout restrictions.

Here is the yaml schema needed measurement_chain-1.0.0:

%YAML 1.1
---
$schema: "http://stsci.edu/schemas/yaml-schema/draft-01"
id: "http://weldx.bam.de/schemas/weldx/core/graph/measurement_chain-1.0.0"
tag: "tag:weldx.bam.de:weldx/core/graph/measurement_chain-1.0.0"

title: |
  measurement_chain

definitions:
  signal_node:
    allOf:
      - tag: "tag:weldx.bam.de:weldx/core/graph/node-1.0.0"
      - type: object
        properties:
          attributes:
            type: object
            properties:
              signal:
                tag: "tag:weldx.bam.de:weldx/core/graph/signal-1.0.0"
            required: [signal]
          edges:
            type: array
            items:
              $ref: "#/definitions/signal_transform_edge"
            maxItems: 1
        required: [attributes]

  signal_transform_edge:
    allOf:
      - tag: "tag:weldx.bam.de:weldx/core/graph/edge-1.0.0"
      - type: object
        properties:
          attributes:
            type: object
            properties:
              signal_transform:
                tag: "tag:weldx.bam.de:weldx/core/graph/signal_transform-1.0.0"
            required: [signal_transform]
          target_node:
            $ref: "#/definitions/signal_node"
          direction:
            enum: [fwd]
        required: [attributes]

type: object
properties:
  root_node:
    $ref: "#/definitions/signal_node"

required: [root_node]
...

The basic measurement chain only requires the root_node.
The internal definitions

The minimal python implementation needed:

@dataclass
class MeasurementChain:
    """Example Measurement Chain implementation."""

    graph: nx.DiGraph


class MeasurementChainTypeASDF(WeldxType):
    """ASDF type for dummy Measurement Chain."""

    name = "core/graph/measurement_chain"
    version = "1.0.0"
    types = [MeasurementChain]
    requires = ["weldx"]

    @classmethod
    def to_tree(cls, node: MeasurementChain, ctx):
        """convert to python dict"""
        root_node = build_tree(node.graph, tuple(node.graph.nodes)[0])  # set root node
        return dict(root_node=root_node)

    @classmethod
    def from_tree(cls, tree, ctx):
        """Reconstruct form tree."""
        graph = nx.DiGraph()
        build_graph(graph, tree["root_node"])
        return MeasurementChain(graph=graph)

All schema requirements get validated on the ASDF level.

expanded example

If we want to introduce another node type to represent a Source and every Measurement to start with a source we could do the following:

%YAML 1.1
---
$schema: "http://stsci.edu/schemas/yaml-schema/draft-01"
id: "http://weldx.bam.de/schemas/weldx/core/graph/measurement_chain-1.0.0"
tag: "tag:weldx.bam.de:weldx/core/graph/measurement_chain-1.0.0"

title: |
  measurement_chain

definitions:
  signal_node:
    allOf:
      - tag: "tag:weldx.bam.de:weldx/core/graph/node-1.0.0"
      - type: object
        properties:
          attributes:
            type: object
            properties:
              signal:
                tag: "tag:weldx.bam.de:weldx/core/graph/signal-1.0.0"
            required: [signal]
          edges:
            type: array
            items:
              $ref: "#/definitions/signal_transform_edge"
            maxItems: 1
        required: [attributes]

  signal_transform_edge:
    allOf:
      - tag: "tag:weldx.bam.de:weldx/core/graph/edge-1.0.0"
      - type: object
        properties:
          attributes:
            type: object
            properties:
              signal_transform:
                tag: "tag:weldx.bam.de:weldx/core/graph/signal_transform-1.0.0"
            required: [signal_transform]
          target_node:
            $ref: "#/definitions/signal_node"
          direction:
            enum: [fwd]
        required: [attributes]

  source_node:
    allOf:
      - tag: "tag:weldx.bam.de:weldx/core/graph/node-1.0.0"
      - type: object
        properties:
          attributes:
            type: object
            properties:
              source:
                tag: "tag:weldx.bam.de:weldx/core/graph/source-1.0.0"
            required: [ source ]
          edges:
            type: array
            items:
              $ref: "#/definitions/source_edge"
            maxItems: 1
        required: [ attributes ]

  source_edge:
    allOf:
      - tag: "tag:weldx.bam.de:weldx/core/graph/edge-1.0.0"
      - type: object
        properties:
          target_node:
            $ref: "#/definitions/source_node"
          direction:
            enum: [fwd]
        required: [attributes]

type: object
properties:
  root_node:
    $ref: "#/definitions/source_node"

required: [root_node]
...

Possible additions:

• support cyclic graphs (should be simple)
• subgraphs (point to graph nodes somewhere else?)

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[pep8speaks]

Hello @CagtayFabry! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-04-16 05:35:43 UTC

[codecov]

Codecov Report

Merging #330 (64e572b) into master (61555e0) will increase coverage by 0.06%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #330      +/-   ##
==========================================
+ Coverage   96.89%   96.96%   +0.06%     
==========================================
  Files          82       84       +2     
  Lines        4769     4871     +102     
==========================================
+ Hits         4621     4723     +102     
  Misses        148      148              

Impacted Files	Coverage Δ
weldx/asdf/tags/weldx/core/common_types.py	100.00% <ø> (ø)
weldx/asdf/tags/weldx/__init__.py	100.00% <100.00%> (ø)
weldx/asdf/tags/weldx/base_types.py	100.00% <100.00%> (ø)
weldx/asdf/tags/weldx/core/__init__.py	100.00% <100.00%> (ø)
weldx/asdf/tags/weldx/core/graph.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 61555e0...64e572b. Read the comment docs.

[vhirtham]

So how do we proceed? Clear all the examples and merge the graph-related stuff?

[CagtayFabry]

So how do we proceed? Clear all the examples and merge the graph-related stuff?

Maybe we can dicuss the general approach here and if we are happy build on from there
And yes I can clear the examples afterwards

[CagtayFabry]

I have updated the default behavior to not serialize the node name if the node name is of type UUID

This should make it possible to use custom implementations for other classes that don't want or need to store node names

[CagtayFabry]

any thoughts on this @marscher? Otherwise I will just merge :)

[marscher]

The new directed graph implementation looks very good to me.