feat: Column Lineage API

[allisonsuarez]

Summary of Changes

• Created Table Lineage API
• Manually tested

Tests

What tests did you add or modify and why? If no tests were added or modified, explain why. Remove this line

Documentation

What documentation did you add or modify and why? Add any relevant links then remove this line

CheckList

Make sure you have checked all steps below to ensure a timely review.

• PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• PR includes a summary of changes.
• PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
• PR passes make test

[codecov-io]

Codecov Report

Merging #280 (101b52c) into master (2752492) will increase coverage by 3.51%.
The diff coverage is 72.83%.

@@            Coverage Diff             @@
##           master     #280      +/-   ##
==========================================
+ Coverage   74.10%   77.61%   +3.51%     
==========================================
  Files          25       27       +2     
  Lines        1255     1385     +130     
  Branches      136      163      +27     
==========================================
+ Hits          930     1075     +145     
+ Misses        297      263      -34     
- Partials       28       47      +19     

Impacted Files	Coverage Δ
metadata_service/api/popular_tables.py	100.00% <ø> (ø)
metadata_service/api/system.py	66.66% <ø> (ø)
metadata_service/api/user.py	100.00% <ø> (ø)
metadata_service/proxy/statsd_utilities.py	81.25% <ø> (ø)
metadata_service/util.py	100.00% <ø> (ø)
metadata_service/api/column.py	54.54% <28.57%> (-45.46%)	⬇️
metadata_service/proxy/shared.py	28.57% <28.57%> (ø)
metadata_service/api/badge.py	61.29% <61.29%> (ø)
metadata_service/proxy/neo4j_proxy.py	71.55% <61.53%> (-3.45%)	⬇️
metadata_service/proxy/base_proxy.py	67.07% <73.33%> (-0.07%)	⬇️
... and 17 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed93cab...101b52c. Read the comment docs.

[allisonsuarez]

cc: @youngyjd @dkunitskiy

[feng-tao]

hey @allisonsuarez, I wonder how you test it, is it by building a sub private class with CW to use the API directly?

[dorianj]

Code LGTM

[dorianj]

If this id generation is a common pattern, should it be handled by a util or wrapper?

[allisonsuarez]

good question, I don't think we have any generic method for both tables and columns, in any methods that call columns like get_column_description on neo4j proxy we get column through its association with table name node, but if we have the entire column key we can just call directly with the key rather than querying neo4j for table->column->what_you_want

[dorianj]

ah, yeah -- i guess ultimately this is tied to ColumnMetadata.COLUMN_KEY_FORMAT -- we're "getting lucky" in that column shares a prefix with TableMetadata.TABLE_KEY_FORMAT. I'm sure there are other places that rely on this behavior

[allisonsuarez]

hey @allisonsuarez, I wonder how you test it, is it by building a sub private class with CW to use the API directly?

In this case I just used a private subclass for neo4j proxy and returned some mock data from the get_lineage method. Still working on getting column lineage from CW