Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Column lineage graph endpoint #2114

Closed
pawel-big-lebowski opened this issue Sep 13, 2022 · 0 comments · Fixed by #2124 or #2269
Closed

Column lineage graph endpoint #2114

pawel-big-lebowski opened this issue Sep 13, 2022 · 0 comments · Fixed by #2124 or #2269
Assignees

Comments

@pawel-big-lebowski
Copy link
Collaborator

Create an endpoint to return column lineage graph according to the proposal:

GET /column-lineage?nodeId=dataset:{namespace}:{dataset}
GET /column-lineage?nodeId=datasetField:{namespace}:{dataset}:{field}
GET /column-lineage?nodeId=datasetField:db1:table1:a
...
{
  graph: [
    {
       "id": "datasetField:db1:table1:a",
       "type": "DATASET_FIELD",
       "data": {
         "namespace": "DB1",
         "name": "table2",
         "field": "a",
         "type": "integer",
         "transformationDescription": "identical",
         "transformationType": "IDENTITY",    
         "inputFields": [
            { "namespace": "DBA", "name": "tableA", "field": "columnA"},
            { "namespace": "DBB", "name": "tableB", "field": "columnB"},
            { "namespace": "DBC", "name": "tableC", "field": "columnC"}
         ]
         "inEdges": [
          {
             "origin": "datasetField:db1:table1:a",
             "destination": "datasetField:DBA:tableA:columnA"
          },
          {
             "origin": "datasetField:db1:table1:a",
             "destination": "datasetField:DBB:tableB:columnB"
          },
          {
             "origin": "datasetField:db1:table1:a",
             "destination": "datasetField:DBB:tableB:columnC"
          }
         ],
      },
      ...
      # Input fields, present within "inEdges", can be also returned within a graph due to a `depth` parameter greate than 0.
  }
  ]
}  

The purpose of the endpoint is to return column lineage graph with a configured depth and the ability to traverse further nodes with other requests.

Graph will contain only inEdges, whileoutEdges will not be included in PR resolving this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant