AIP-84 Add external dependencies to a dag's graph_data

[pierrejeambrun]

from @bbovenzi:

In 2.10, we cobbled together a UI to show dataset conditions and upstream and downstream dataset dependencies. We should update this to include a boolean param show_external_dependencies to show connected datasets, dataset conditions, triggers and sensors.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[bbovenzi]

We can rename graph_data to dag structure, with edges and nodes. The nested list of nodes should have the topological sort, which we may move to happen during DAG serialization. With a sorted list of nodes, specifying which are tasks and which aren't, we can reuse the endpoint to help render the grid and gantt views.

[bbovenzi]

Actually, the sort should be based off of the webserver.grid_view_sorting_order config.

[bbovenzi]

Updated types I am using on the frontend mockups:

export type Edge = {
  is_setup_teardown?: boolean;
  label?: string;
  source_id: string;
  target_id: string;
};

export type Node = {
  children?: Array<Node>;
  id: string;
  is_mapped?: boolean;
  label: string;
  tooltip?: string;
  setup_teardown_type?: "setup" | "teardown";
  type:
    | "asset_alias"
    | "asset_condition"
    | "asset"
    | "dag"
    | "join"
    | "sensor"
    | "task"
    | "trigger";
};

export type GraphData = {
  arrange: "BT" | "LR" | "RL" | "TB";
  edges: Array<Edge>;
  nodes: Array<Node>;
};

[pierrejeambrun]

Having fields optional in the payload, i.e ?: makes it a little bit more complicated to handle and I think it might be easier to always return all fields, defaulting to None in case they are not filled, and the front knows what to read based on the type value. (ideally we would need multiple implem of the node interface depending on each type allowed, but that might be overkill at first)