Skip to content

s3 key sensor (sync version) has bad memory footprint #55039

@dstandish

Description

@dstandish

Body

It currently loads all keys into memory. This is not necessary and, on the wrong bucket, can takedown the worker.

We saw this (and fixed) with the async version some time ago #40473.

But the sync version needs to be fixed.

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Metadata

Metadata

Assignees

Labels

good first issuekind:bugThis is a clearly a bugkind:metaHigh-level information important to the communityprovider:amazonAWS/Amazon - related issues

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions