Ability to add custom facet in OpenLineage events #38982

anandhimurali · 2024-04-12T23:33:19Z

Motivation
Currently, users need to write custom extractor inorder to add custom facets in the lineage events. Instead, providing a way to inject custom facets by accepting a config.

Modifications
The get_custom_facets function in the OpenLineage provider is enhanced to fetch the Airflow config AIRFLOW__OPENLINEAGE__CUSTOM_RUN_FACETS, execute those functions and append their return values to the run facet of the lineage event for all the operators.

Result
Enables users to add custom facets to the lineage events by defining functions instead of custom extractor.

Configs documentation for AIRFLOW__OPENLINEAGE__CUSTOM_RUN_FACETS

User Docs

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

boring-cyborg · 2024-04-12T23:33:23Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

airflow/providers/openlineage/provider.yaml

XD-DENG · 2024-04-18T18:08:23Z

Hi @eladkal , @anandhimurali is my colleague from the same team and I'm really excited about her 1st PR to Airflow!

Do you have an idea who among the committers may be the best to help review this openlineage-related change?

Any suggestion/input would be greatly appreciated. Thanks a lot!

eladkal · 2024-04-18T19:52:25Z

Ping @mobuchowski for review :)

mobuchowski · 2024-04-25T10:03:13Z

Hey @anandhimurali @XD-DENG I think ability to add custom facets to arbitrary operators is a great idea.

I have two points I think it's worth addressing:

Whether we pass all the necessary info. I think given task_instance we can call get_template_context and get all the data, so this might not be an issue.
Whether this acts "per DAG/Task" or it's a mechanism that works for all the tasks equally - this might be defined inside the function so it still might not be an issue other than complicating the function a bit more.

Overall I like the solution, but might be good to specify the goals regarding 1) and 2) - because for sure, later, someone will come and ask "how to make this custom facet work only on some particular tasks" and "how do I pass some custom info to my function".

kacpermuda

I like the idea 🚀 Apart from what @mobuchowski mentioned above, I added some small comments about the code itself.

airflow/providers/openlineage/conf.py

airflow/providers/openlineage/utils/utils.py

airflow/providers/openlineage/provider.yaml

airflow/providers/openlineage/utils/utils.py

kacpermuda · 2024-05-14T14:23:14Z

Hey @anandhimurali @XD-DENG, is there anything i could help you with here? I really like the idea and would like to see it happen, let me know 😄

anandhimurali · 2024-05-14T16:00:34Z

Hey @anandhimurali @XD-DENG, is there anything i could help you with here? I really like the idea and would like to see it happen, let me know 😄

Hi @kacpermuda, Thanks for the interest and I'm excited to hear that. Sorry I've been busy for the last few weeks. I'll look into the comments this week.

anandhimurali · 2024-05-16T05:58:28Z

I have two points I think it's worth addressing:

Whether we pass all the necessary info. I think given task_instance we can call get_template_context and get all the data, so this might not be an issue.

Whether this acts "per DAG/Task" or it's a mechanism that works for all the tasks equally - this might be defined inside the function so it still might not be an issue other than complicating the function a bit more.

Overall I like the solution, but might be good to specify the goals regarding 1) and 2) - because for sure, later, someone will come and ask "how to make this custom facet work only on some particular tasks" and "how do I pass some custom info to my function".

Hi @mobuchowski,

Yes, I think task_instance is sufficient to fetch necessary info about the run context.
The custom_facet_functions defined works equally for all tasks. Yes, user can choose to return None for scenarios where they won't to add the custom facet.

Let me know if this needs to be documented anywhere. Thanks.

kacpermuda

As you already mentioned, i think this should be documented for the users. We should include the function signature with type hints, so that users know how to properly implement it + some examples from our side.

Probably here and here

airflow/providers/openlineage/utils/utils.py

mobuchowski · 2024-07-01T14:08:04Z

@anandhimurali any interest in getting this PR into shape and approved? I think this is good, really useful feature.

anandhimurali · 2024-07-01T19:41:04Z

@anandhimurali any interest in getting this PR into shape and approved? I think this is good, really useful feature.

@mobuchowski Yes, definitely. I just have to add some documentation and will get this done this week. Thanks and sorry for the delay.

ahidalgob · 2024-07-03T10:23:44Z

I think it would be good to rename this functionality to custom_run_facets, we might need to inject other facets too, specifically Job.facets.

This is a great feature:)

…_FACET_FUNCTIONS

Co-authored-by: Kacper Muda <mudakacper@gmail.com>

tests/system/providers/google/cloud/datapipelines/resources/__init__.py

kacpermuda

LGTM

JDarDagran

one small thing, otherwise great job! 🚀

docs/apache-airflow-providers-openlineage/guides/developer.rst

tests/providers/openlineage/utils/custom_facet_fixture.py

boring-cyborg · 2024-07-22T11:17:22Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

mobuchowski · 2024-07-22T11:17:41Z

Thanks @anandhimurali for implementing this feature 🚀

* Ability to add custom facet in OpenLineage events * Update airflow/providers/openlineage/provider.yaml Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com> * Update airflow/providers/openlineage/provider.yaml Co-authored-by: Kacper Muda <mudakacper@gmail.com> * Adding None type hint for the custom facet function * Fix a test after rebase * Removed the legacy OPENLINEAGE_ configs format for OPENLINEAGE_CUSTOM_FACET_FUNCTIONS * Duplicate facet key check * Update airflow/providers/openlineage/utils/utils.py Co-authored-by: Kacper Muda <mudakacper@gmail.com> * Update airflow/providers/openlineage/utils/utils.py Co-authored-by: Kacper Muda <mudakacper@gmail.com> * Fixes after rebase * Adding user docs for custom_facet_functions * Rename custom_facet_functions as custom_run_facets * Increment version for custom_run_facets feature * Enrich example with access to operator and return value as None. * Add try-except for custom facet function execution * Fix the typing for the custom facet fucntion return type * Documentation: funcs are executed only for START events * Fix the typing for the custom facet function return type * Fixes after pre-commit hook checks * Adding start_date to test DAGs for 2.7 compatibility tests * Removing a out of scope __init__ file added by pre-commit check --------- Co-authored-by: Anandhi <anandhi_murali@apple.com> Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com> Co-authored-by: Kacper Muda <mudakacper@gmail.com>

boring-cyborg bot added area:providers provider:openlineage AIP-53 labels Apr 12, 2024

anandhimurali force-pushed the ol-custom-facet branch from 446c159 to 435db7d Compare April 12, 2024 23:34

anandhimurali marked this pull request as ready for review April 13, 2024 00:47

anandhimurali requested a review from mobuchowski as a code owner April 13, 2024 00:47

eladkal reviewed Apr 14, 2024

View reviewed changes

airflow/providers/openlineage/provider.yaml Outdated Show resolved Hide resolved

kacpermuda reviewed Apr 25, 2024

View reviewed changes

anandhimurali force-pushed the ol-custom-facet branch from d0265d1 to b4414cb Compare May 16, 2024 04:09

anandhimurali force-pushed the ol-custom-facet branch from 6e96d73 to 03ed172 Compare May 16, 2024 06:43

kacpermuda mentioned this pull request May 21, 2024

[openlineage] fix: Prevent error when extractor can't be imported #39736

Merged

kacpermuda reviewed May 21, 2024

View reviewed changes

airflow/providers/openlineage/utils/utils.py Outdated Show resolved Hide resolved

airflow/providers/openlineage/utils/utils.py Outdated Show resolved Hide resolved

eladkal mentioned this pull request May 26, 2024

Status of testing Providers that were prepared on May 26, 2024 #39842

Closed

anandhimurali force-pushed the ol-custom-facet branch from 421e0fe to e851940 Compare June 21, 2024 04:46

eladkal force-pushed the ol-custom-facet branch from e851940 to 1eea113 Compare June 30, 2024 05:01

anandhimurali force-pushed the ol-custom-facet branch 6 times, most recently from 4f193aa to 0d63f59 Compare July 3, 2024 05:02

Anandhi and others added 17 commits July 18, 2024 10:47

Adding None type hint for the custom facet function

ca0224c

Fix a test after rebase

5597404

Removed the legacy OPENLINEAGE_ configs format for OPENLINEAGE_CUSTOM…

457584e

…_FACET_FUNCTIONS

Duplicate facet key check

d246f3a

Update airflow/providers/openlineage/utils/utils.py

5465664

Co-authored-by: Kacper Muda <mudakacper@gmail.com>

Update airflow/providers/openlineage/utils/utils.py

16fe262

Co-authored-by: Kacper Muda <mudakacper@gmail.com>

Fixes after rebase

1d9b1f5

Adding user docs for custom_facet_functions

9847bd6

Rename custom_facet_functions as custom_run_facets

f268e9d

Increment version for custom_run_facets feature

bfffb9e

Enrich example with access to operator and return value as None.

531f996

Add try-except for custom facet function execution

9b6f0c8

Fix the typing for the custom facet fucntion return type

88f8c07

Documentation: funcs are executed only for START events

ead05fd

Fix the typing for the custom facet function return type

6505f4f

Fixes after pre-commit hook checks

497bdba

Adding start_date to test DAGs for 2.7 compatibility tests

8c404b9

anandhimurali force-pushed the ol-custom-facet branch from 81a8282 to 8c404b9 Compare July 18, 2024 17:48

kacpermuda reviewed Jul 19, 2024

View reviewed changes

tests/system/providers/google/cloud/datapipelines/resources/__init__.py Outdated Show resolved Hide resolved

kacpermuda approved these changes Jul 19, 2024

View reviewed changes

Anandhi and others added 2 commits July 19, 2024 08:50

Removing a out of scope __init__ file added by pre-commit check

18526d7

Merge branch 'main' into ol-custom-facet

75c10d1

JDarDagran approved these changes Jul 22, 2024

View reviewed changes

docs/apache-airflow-providers-openlineage/guides/developer.rst Show resolved Hide resolved

tests/providers/openlineage/utils/custom_facet_fixture.py Show resolved Hide resolved

mobuchowski approved these changes Jul 22, 2024

View reviewed changes

mobuchowski merged commit e30f810 into apache:main Jul 22, 2024
52 checks passed

kacpermuda mentioned this pull request Jul 23, 2024

openlineage: extend custom_run_facets to on_failed and on_complete #40953

Merged

eladkal mentioned this pull request Aug 3, 2024

Status of testing Providers that were prepared on August 03, 2024 #41237

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ability to add custom facet in OpenLineage events #38982

Ability to add custom facet in OpenLineage events #38982

anandhimurali commented Apr 12, 2024 •

edited

Loading

boring-cyborg bot commented Apr 12, 2024

XD-DENG commented Apr 18, 2024

eladkal commented Apr 18, 2024

mobuchowski commented Apr 25, 2024

kacpermuda left a comment

kacpermuda commented May 14, 2024

anandhimurali commented May 14, 2024

anandhimurali commented May 16, 2024

kacpermuda left a comment

mobuchowski commented Jul 1, 2024

anandhimurali commented Jul 1, 2024

ahidalgob commented Jul 3, 2024 •

edited

Loading

kacpermuda left a comment

JDarDagran left a comment

boring-cyborg bot commented Jul 22, 2024

mobuchowski commented Jul 22, 2024

Ability to add custom facet in OpenLineage events #38982

Ability to add custom facet in OpenLineage events #38982

Conversation

anandhimurali commented Apr 12, 2024 • edited Loading

boring-cyborg bot commented Apr 12, 2024

XD-DENG commented Apr 18, 2024

eladkal commented Apr 18, 2024

mobuchowski commented Apr 25, 2024

kacpermuda left a comment

Choose a reason for hiding this comment

kacpermuda commented May 14, 2024

anandhimurali commented May 14, 2024

anandhimurali commented May 16, 2024

kacpermuda left a comment

Choose a reason for hiding this comment

mobuchowski commented Jul 1, 2024

anandhimurali commented Jul 1, 2024

ahidalgob commented Jul 3, 2024 • edited Loading

kacpermuda left a comment

Choose a reason for hiding this comment

JDarDagran left a comment

Choose a reason for hiding this comment

boring-cyborg bot commented Jul 22, 2024

mobuchowski commented Jul 22, 2024

anandhimurali commented Apr 12, 2024 •

edited

Loading

ahidalgob commented Jul 3, 2024 •

edited

Loading