Airflow 3.0 not emitting OpenLineage Task Failure Events #46915
Description
Apache Airflow Provider(s)
openlineage
Versions of Apache Airflow Providers
openlineage 2.0.0 (part of 3.0.0alpha)
Apache Airflow version
3.0.0alpha
Operating System
linux
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
What happened
I am testing airflow 3.0 and open lineage.
What you think should happen instead
No response
How to reproduce
I created an Airflow 3.0 instance and deployed this DAG. When I ran the DAG and looked at the emitted OL events, I did not see any task failure events. I did see the DAG start, task starts, and DAG failure event.
from datetime import datetime, timedelta
import random
import time
import logging
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.common.compat.openlineage.facet import Dataset as OLDataset
from airflow.datasets import Dataset
ol_dataset = OLDataset(
name="oldataset",
namespace="olnamespace",
facets={}
)
af_dataset = Dataset("afdataset")
# Define default args
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email_on_failure': False,
'email_on_retry': False,
'retries': 0,
'retry_delay': timedelta(minutes=5),
}
def fail():
raise ValueError("Task failed.")
# Define DAG
dag = DAG(
'faildag',
default_args=default_args,
description='A DAG with complex task dependencies and logic.',
start_date=datetime(2023, 12, 1),
catchup=False,
tags=['example'],
)
# Define tasks
airflow_dataset_fail = PythonOperator(
task_id='airflow_dataset_fail',
python_callable=fail,
dag=dag,
inlets=[af_dataset],
outlets=[af_dataset]
)
ol_dataset_fail = PythonOperator(
task_id='ol_dataset_fail',
python_callable=fail,
dag=dag,
inlets=[ol_dataset],
outlets=[ol_dataset]
)
[airflow_dataset_fail, ol_dataset_fail]
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct