Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add facets to openapi marquez spec #894

Merged

Conversation

henneberger
Copy link
Contributor

@henneberger henneberger commented Jan 19, 2021

This proposal adds open lineage facets to the marquez API for jobs, datasets, and runs. Additionally, this deprecates the context parameter and convert contexts to a special context facet. This will allow cross compatibility between the existing marquez apis and open lineage.

See: #895

Signed-off-by: henneberger git@danielhenneberger.com

Signed-off-by: henneberger <git@danielhenneberger.com>
@codecov
Copy link

codecov bot commented Jan 19, 2021

Codecov Report

Merging #894 (10c0f8c) into main (07d671f) will decrease coverage by 0.71%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##               main     #894      +/-   ##
============================================
- Coverage     75.01%   74.30%   -0.72%     
+ Complexity      730      500     -230     
============================================
  Files           152      117      -35     
  Lines          3118     2370     -748     
  Branches        308      234      -74     
============================================
- Hits           2339     1761     -578     
+ Misses          369      304      -65     
+ Partials        410      305     -105     
Impacted Files Coverage Δ Complexity Δ
...main/java/marquez/service/models/LineageEvent.java 37.25% <0.00%> (-4.57%) 0.00% <0.00%> (ø%)
api/src/main/java/marquez/db/OpenLineageDao.java 89.94% <0.00%> (-1.51%) 36.00% <0.00%> (-3.00%)
...a/src/main/java/marquez/client/models/JobMeta.java
...java/src/main/java/marquez/client/MarquezHttp.java
.../java/src/main/java/marquez/client/models/Tag.java
...ava/marquez/client/MarquezWriteOnlyClientImpl.java
...in/java/marquez/client/MarquezClientException.java
...c/main/java/marquez/client/models/DbTableMeta.java
.../java/src/main/java/marquez/client/models/Job.java
...va/src/main/java/marquez/client/models/Stream.java
... and 27 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07d671f...459872e. Read the comment docs.

@wslulciuc
Copy link
Member

@henneberger : Should we also add facets to the source and namespace models? I think it'd make sense to have the core models of marquez contain the facet field for extensibility.

Copy link
Member

@julienledem julienledem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me. I like that we live all facets be generic in this spec. Marquez treats all facets the same whether they are standard or custom.
I would advise against adding facets to source and namespace in that spec.
In openlineage, the namespace is just a mechanism to make unique name, the metadata is around jobs, datasets and runs.

spec/openapi.yml Show resolved Hide resolved
spec/openapi.yml Show resolved Hide resolved
spec/openapi.yml Show resolved Hide resolved
spec/openapi.yml Show resolved Hide resolved
Copy link
Member

@wslulciuc wslulciuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really excited to begin adding support for facets to the marquez model 💯

@wslulciuc wslulciuc merged commit e561576 into MarquezProject:main Feb 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants