What causes these Airflow ingestion errors?

[scastria]

The particular task it is complaining about was removed just this morning. If I check the AirflowMetadata DB that powers the MWAA environment, I do see that task in the task_instance table, but it has a state of "removed" as it should. So I am wondering if the Airflow metadata ingestion process is not ignoring "removed" tasks like it should or perhaps the error message needs to be more clear??

[[34m2023-10-17T19:57:43.604+0000[0m] {[34mmetadata_rest.py:[0m356} ERROR[0m - Unexpected error writing pipeline status [pipeline_fqn='airflow.third_party-mountain-seed-ingest' pipeline_status=PipelineStatus(timestamp=Timestamp(__root__=1697553453), executionStatus=<StatusType.Successful: 'Successful'>, taskStatus=[TaskStatus(name='task_notify_qa_validate', executionStatus=<StatusType.Successful: 'Successful'>, startTime=Timestamp(__root__=1697554057935), endTime=Timestamp(__root__=1697554142340), logLink=None), TaskStatus(name='task_mountain_seed_post_validation', executionStatus=<StatusType.Successful: 'Successful'>, startTime=Timestamp(__root__=1697553971398), endTime=Timestamp(__root__=1697554054861), logLink=None), TaskStatus(name='task_mountain_seed_data_ingest', executionStatus=<StatusType.Successful: 'Successful'>, startTime=Timestamp(__root__=1697553886154), endTime=Timestamp(__root__=1697553965523), logLink=None), TaskStatus(name='task_mountain_seed_id_resolution', executionStatus=<StatusType.Successful: 'Successful'>, startTime=Timestamp(__root__=1697553787741), endTime=Timestamp(__root__=1697553882382), logLink=None), TaskStatus(name='task_mountain_seed_raw_ingest', executionStatus=<StatusType.Successful: 'Successful'>, startTime=Timestamp(__root__=1697553703752), endTime=Timestamp(__root__=1697553784035), logLink=None), TaskStatus(name='task_mountain_seed_data_qa_validate', executionStatus=<StatusType.Successful: 'Successful'>, startTime=Timestamp(__root__=1697553597671), endTime=Timestamp(__root__=1697553697428), logLink=None), TaskStatus(name='task_mountain_seed_add_unique_key', executionStatus=<StatusType.Pending: 'Pending'>, startTime=Timestamp(__root__=1697553555525), endTime=Timestamp(__root__=1697553601349), logLink=None), TaskStatus(name='task_mountain_seed_remove_unique_key', executionStatus=<StatusType.Pending: 'Pending'>, startTime=None, endTime=None, logLink=None), TaskStatus(name='task_mountain_seed_copy_raw', executionStatus=<StatusType.Successful: 'Successful'>, startTime=Timestamp(__root__=1697553465252), endTime=Timestamp(__root__=1697553548777), logLink=None)])]: Invalid task name task_mountain_seed_add_unique_key[0m

[pmbrull]

@scastria can you please give us some version info?

OM version & MWAA versions to replicate this. thanks

[pmbrull]

hi @scastria need a bit of more info here to know how to reproduce

[scastria]

Sorry for the delay.

OM Version = 1.1.7
MWAA version = 2.6.3

[pmbrull]

looks like task deletion reconciliation in the process. to review