GlueJobOperator - resume_glue_job_on_retry doesn't seem to work on MWAA 2.11.0

[wilsonhooi86]

Apache Airflow version

2.11.X

If "Other Airflow 3 version" selected, which one?

MWAA 2.11.0

What happened?

Hi,

I tried to run the GlueJobOperator with resume_glue_job_on_retry=True but somehow during retry when the task failed, it creates a new glue job ID while the existing glue job ID is still running.

Sample of the GlueJobOperator:

 drug_program = GlueJobOperator(
            task_id="drug_program",
            job_name="gtm-core-dl-dev-euc1-glue-job-glo_informa_prd",
            verbose=False,
            script_args={
                "--name": var_glue_job_name,
                "--db_name": var_database_name,
                "--tbl_name": "drug_program",
            },
            stop_job_run_on_kill=False,
            deferrable=False,
            pool=var_glue_pool,
            botocore_config=botocore_overide_config,
            resume_glue_job_on_retry=True,
        )

1st Run:
Create a new glue job ID as per screenshot

2nd Run (1st Retry):

Job failed due to internal errors, during retry, it didn't find back the existing glue job ID but created 1 new glue job ID. End results, 2 same glue jobs are running together.

What you think should happen instead?

During task retry, it should check the existing glue job ID, if found, it should not create a new glue job ID and resume run status in Airflow Task

How to reproduce

1. Install the apache-airflow-providers-amazon==9.22.0rc1 in MWAA Airflow 2.11 environment

2.. Create GlueJobOperator Dag
Sample of the GlueJobOperator:

 drug_program = GlueJobOperator(
            task_id="drug_program",
            job_name="gtm-core-dl-dev-euc1-glue-job-glo_informa_prd",
            verbose=False,
            script_args={
                "--name": var_glue_job_name,
                "--db_name": var_database_name,
                "--tbl_name": "drug_program",
            },
            stop_job_run_on_kill=False,
            deferrable=False,
            pool=var_glue_pool,
            botocore_config=botocore_overide_config,
            resume_glue_job_on_retry=True,
        )

3. Run the glue job task.
4. During 1st run, it failed (to test, produce by marking the task as failed)
5. Task went to for retry but somehow it creates a new glue job ID

Operating System

Amazon Linux 2023

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.22.0rc1

Deployment

Amazon (AWS) MWAA

Deployment details

MWAA 2.11.0 deployment with requirements to install apache-airflow-providers-amazon==9.22.0rc1

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wilsonhooi86]

Hi @henry3260 , just to share this is related to the recent feature: #59392

Thank you so much and let me know if you need more info

[henry3260]

Ｉwill take a look. Thanks for sharing!

[wilsonhooi86]

Ｉwill take a look. Thanks for sharing!

Thank you so much, @henry3260 .
This feature will be very helpful for our current use cases. Really appreciate your support.