Cloudwatch remote logging is not working on Airflow 3

[sdg9670]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

No response

Apache Airflow version

3.0.2

Operating System

ebian GNU/Linux 12 (bookworm)

Deployment

Other Docker-based deployment

Deployment details

Airflow verison: 3.0.2
Provider version: 9.9.0
Executor: AWS Batch
Permissions: Handled by the IAM role attached to the EC2 instances in the compute environment.

What happened

The task running on AWS Batch is not writing any logs to Amazon CloudWatch. It completes successfully without leaving any specific error messages or logs.

Logs about amazon

[2025-06-27T04:55:46.050+0000] {base_aws.py:623} WARNING - Unable to find AWS Connection ID 'aws_default', switching to empty.
--
[2025-06-27T04:55:46.051+0000] {base_aws.py:189} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='ap-northeast-2'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
--
{
    "timestamp": "2025-06-27T04:56:20.255576",
    "level": "info",
    "event": "No connection ID provided. Fallback on boto3 credential strategy (region_name=None). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html",
    "logger": "airflow.providers.amazon.aws.hooks.base_aws.BaseSessionFactory"
}

What you think should happen instead

No response

How to reproduce

We are using Apache Airflow with the AWSBatchExecutor to schedule and submit our worker tasks to AWS Batch.

AWS batch service role

{
    "Effect": "Allow",
    "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
    ],
    "Resource": "*"
}

Worker Configuration

AIRFLOW__LOGGING__REMOTE_LOGGING='true'
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER="cloudwatch://${AWS_CLOUDWATCH_GROUP_ARN}"
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID='aws_default'

Anything else

This configuration was working correctly in Airflow 2, but logging to CloudWatch stopped after we upgraded to Airflow 3.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[djadeau]

Hello ! I have the same problem with remote logging on s3. I suspect a problem with remote logging in general unfortunately.
I use the same provider and airflow versions. I didn't find any work around yet.

[rjtshrm]

I gets the same error. Working fine in 2.10.5 but after upgrading to 3.0.2, remote logs are not working anymore

[eladkal]

cc @o-nikolas @vincbeck

[rjtshrm]

Hi, I compared the airflow info for our stable and upgrade release to 3.0.2. I can see that in 3.0.2, task_logging_handler is set as airflow.utils.log.file_task_handler.FileTaskHandler even my remote_base_log_folder is cloudwatch ARN.

Apache Airflow
version                | 2.10.5                                           
executor               | KubernetesExecutor                               
task_logging_handler   | airflow.providers.amazon.aws.log.cloudwatch_task_
                       | handler.CloudwatchTaskHandler                    
sql_alchemy_conn       | postgresql://xxx.xxx.xxx                      
dags_folder            | /opt/airflow/dags                                
plugins_folder         | /opt/airflow/plugins                             
base_log_folder        | /opt/airflow/logs                                
remote_base_log_folder | cloudwatch://arn:aws:logs:xxx  

Apache Airflow
version                | 3.0.2                                                                                                                      
executor               | KubernetesExecutor                                                                                                         
task_logging_handler   | airflow.utils.log.file_task_handler.FileTaskHandler                                                                        
sql_alchemy_conn       | postgresql://xxx.xxx.xxx                                                                                                                                          
dags_folder            | /opt/airflow/dags                                                                                                          
plugins_folder         | /opt/airflow/plugins                                                                                                       
base_log_folder        | /opt/airflow/logs                                                                                                          
remote_base_log_folder | cloudwatch://arn:aws:logs:xxx

In helm, I have enabled the logging like this and is same in both 2.10 and 3.0.2 except auth_manager addition.

config:
    core:
      lazy_load_plugins: 'False'
      max_active_tasks_per_dag: 64
      max_active_runs_per_dag: 10
      parallelism: 64
      auth_manager: 'airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager'
      remote_logging: 'True'
    kubernetes_executor:
      worker_pods_creation_batch_size: 30

  logging:
        remote_logging: 'True'
        remote_base_log_folder: cloudwatch://arn:aws:logs:xxx
        remote_log_conn_id: MyCloudwatchConn

Are there any specific changes in logging related to Airflow 3.x that need to be taken care of externally?

[rjtshrm]

One more thing I noticed, why didn't the DEFAULT_LOGGING_CONFIG["handlers"]["task"] is updated here with the CloudwatchTaskHandler. Is it something specific with 3.x change.

[rjtshrm]

@eladkal Any update on the issue. Can you confirm if its a bug or some configuration issue? This blocks us to upgrade the airflow 3.x.

@sdg9670 @djadeau. Were you managed to solve this?