Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add basic system tests for OpenLineage #43643

Merged
merged 1 commit into from
Nov 27, 2024

Conversation

mobuchowski
Copy link
Contributor

Added system tests to make sure our OpenLineage integration actually works as expected. Here's what's new:

  • Built a VariableTransport that dumps OpenLineage events into Airflow Variables so we can check them later
  • Created a test operator (OpenLineageTestOperator) that compares the events we got against what we expected
  • Added a simple test DAG to show how it all works together
  • Threw in some helper functions to make event validation less painful (checking UUIDs, dates, etc.)

The main idea is to have a reliable way to verify that we're sending the right OpenLineage events with the right format.
The test setup is pretty flexible - you can either define your expected events inline or keep them in JSON files, whatever works better for your use case.

@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch 2 times, most recently from 459da4f to db41441 Compare November 14, 2024 14:39
@mobuchowski mobuchowski marked this pull request as ready for review November 14, 2024 14:57
@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch from db41441 to 8b5b3c8 Compare November 14, 2024 20:27
Copy link
Contributor

@kacpermuda kacpermuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, leaving some nits - mostly naming or something similar.

@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch from 8b5b3c8 to 9ec2b3f Compare November 15, 2024 12:46
airflow/models/variable.py Outdated Show resolved Hide resolved
@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch 2 times, most recently from 9173b7a to f9a59d7 Compare November 19, 2024 14:58
@mobuchowski
Copy link
Contributor Author

@Lee-W fixed all the typing/style comments.

@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch from f9a59d7 to 3a97b27 Compare November 19, 2024 16:02
@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch from 3a97b27 to 09ac15a Compare November 19, 2024 19:12
@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch from 09ac15a to 8d4f14b Compare November 20, 2024 14:10
Copy link
Contributor

@dstandish dstandish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @mobuchowski

@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch from 8d4f14b to f81bb0f Compare November 20, 2024 14:56
@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch 2 times, most recently from d23957f to 6e5aa55 Compare November 25, 2024 16:58
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski mobuchowski force-pushed the openlineage-systemtests branch from 6e5aa55 to ab743c1 Compare November 25, 2024 17:17
@potiuk potiuk merged commit 7cc3610 into apache:main Nov 27, 2024
45 checks passed
ArshiaZr pushed a commit to ArshiaZr/airflow that referenced this pull request Nov 27, 2024
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this pull request Jan 5, 2025
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants