Batch Operator: Realtime container execution logs in Airflow task execution log

[rchui]

Description

Add functionality to Airflow BatchOperator to capture the task execution logs as they happen from AWS CloudWatch to the BatchOperator's task execution logs.

Use case/motivation

When executing Batch tasks via BatchOperator the logs are not readily available in Airflow task during execution; only a link is provided to point to where the logs can be viewed in CloudWatch. As a result one should navigate the task logs in Airflow to get a handle of the container log path in AWS Cloudwatch and then access the log from AWS Cloudwatch in realtime.

The current behavior introduces the following challenge:

The ongoing execution logs are not captured in Airflow task ( BatchOperator ) log. This typically involves couple of additional steps - which can be avoided.

Related issues

This is almost identical to the ECSOperator realtime log streaming that was raised in this issue: #22512. In fact most of the description, use case, and motivation are lifted from that issue.

An EcsTaskLogFetcher class already exists in airflow.providers.amazon.aws.hooks.ecs which AWS Batch can use as a backend. It just needs to be initialized, started, and stopped at the correct time. In the event of an array job, the Batch Operator already defaults to the first task and the same could be done when choosing which log stream to fetch.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Sure. Maybe you would like to add it? Just to set the expectations - the thing with OSS project features like that is that they will get implemented if anyone picks interest int them . Just creating an issue makes it visible and when we (as @hussein-awala did ) can mark it as "good first issue" for someone to pick it up, but that's about it. If you would like to see it implemented @rchui , the fastest and most certain way is that you implement it yourself or find someone who will do it for you if you do not have enough skills. Otherwise it might just stick here until someone will find enough interest and skill together to implement ot.

Just to set the expectations.

[killua1zoldyck]

Hello, can I work on this?

[phanikumv]

Assigned it to you @killua1zoldyck