Dag bundles from S3 do not work

[nabuskey]

Apache Airflow version

3.1.0

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

When I configure DAG bundle from S3 according to the documentation, it fails with IsADirectoryError.

This occurs because it does not ignore prefix objects in

airflow/providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py

Lines 1776 to 1785 in 44f8479

	for obj in s3_bucket.objects.filter(Prefix=s3_prefix):
	obj_path = Path(obj.key)
	local_target_path = local_dir.joinpath(obj_path.relative_to(s3_prefix))
	if not local_target_path.parent.exists():
	local_target_path.parent.mkdir(parents=True, exist_ok=True)
	self.log.debug("Created local directory: %s", local_target_path.parent)
	self._sync_to_local_dir_if_changed(
	s3_bucket=s3_bucket, s3_object=obj, local_target_path=local_target_path
	)
	local_s3_objects.append(local_target_path)

bucket.objects() returns prefixes as well.

>>> objects =  bucket.objects.filter(Prefix='dags/')
>>> for obj in objects:
...     print(obj)
...
s3.ObjectSummary(bucket_name='data-on-eks-airflow-20251023154905245200000001', key='dags/')
s3.ObjectSummary(bucket_name='data-on-eks-airflow-20251023154905245200000001', key='dags/simple.py')

What you think should happen instead?

It should be able to retrieve DAGs without errors.

How to reproduce

Setup standalone:

uv pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
uv pip install apache-airflow-providers-amazon

Update airflow.cfg to use S3 as a source for DAG bundles

[dag_processor]
dag_bundle_config_list = [
      {
        "name": "dags-s3",
        "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
        "kwargs": {
          "bucket_name": "some-bucket",
          "prefix": "dags/"
        }
      }
    ]

Put DAGs in s3:

aws s3 ls --recursive  data-on-eks-airflow-20251023154905245200000001/dags/
2025-10-23 15:49:08          0 dags/
2025-10-23 16:18:07       1000 dags/simple.py
2025-10-23 18:54:56          0 dags/test1/
2025-10-23 18:56:06       1007 dags/test1/simple-test.py

Run airflow:

airflow standalone

Fails with:

dag-processor | File "/tmp/airflow/.venv/lib/python3.13/site-packages/s3transfer/utils.py", line 299, in rename_file
dag-processor | rename_file(current_filename, new_filename)
dag-processor | ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dag-processor | IsADirectoryError: [Errno 21] Is a directory: '/tmp/airflow/dag_bundles/dags-s3.64d6feDD' -> '/tmp/airflow/dag_bundles/dags-s3'

Operating System

Ubuntu 24.04

Versions of Apache Airflow Providers

apache-airflow-providers-amazon          9.15.0
apache-airflow-providers-common-compat   1.7.4
apache-airflow-providers-common-io       1.6.3
apache-airflow-providers-common-sql      1.28.1
apache-airflow-providers-http            5.3.4
apache-airflow-providers-smtp            2.2.1
apache-airflow-providers-standard        1.8.0

it fails with:

dag-processor | IsADirectoryError: [Errno 21] Is a directory: '/tmp/airflow/dag_bundles/dags-s3.64d6feDD' -> '/tmp/airflow/dag_bundles/dags-s3'

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.