Capturing column level lineage in Spline

[AtindraHazra]

we need to capture a table's(can be on-prem or cloud) column level lineage like capturing changes on particular column's values and having it captured in Spine.

So, just wanted to ask if it would be doable in Spline as I couldn't get much info about capturing column level lineages in any Spline related docs.
Any help on this is appreciated

[cerveada]

Spline is able to capture column level lineage for spark jobs. You can see it in the UI on the level of individual jobs. The only thing that is not implemented is tracking column level lineage through multiple jobs. But the information is in the database, there is just no API/UI to show it now.

But of course the safest way to make sure it works as you want is to test it yourself.

[wajda]

@AtindraHazra you may look at this service to get an idea of how the end-to-end column lineage could be implemented. I promise it would be a lot of fun to implement it. And it also would be super useful :D

https://github.com/AbsaOSS/spline/blob/develop/arangodb-foxx-services/src/main/services/lineage-overview.ts