Skip to content

Conversation

@isaiahiruoha
Copy link
Contributor

This PR adds a system test for AWS Lambda executor's Dead Letter Queue (DLQ) functionality, addressing a gap in test coverage for Lambda service-level failures.

Before, all Lambda executor system tests only covered the "happy path" where tasks execute successfully (whether the test ultimately passes or fails). However, the Lambda executor has unique handling for cases where the Lambda service itself fails to complete invocation (timeouts, memory limits, crashes, etc.). In these scenarios:

  1. Normal path: Lambda completes → sends result to results queue → executor processes
  2. DLQ path: Lambda fails at service level → AWS sends message to DLQ → executor processes DLQ message → marks task as failed

The test uses executor_config functionality (from #53994) to implement a "poison pill" pattern:

  1. Task definition: Uses @task(executor_config={"poison_pill": True}) to flag a task for intentional failure
  2. Lambda handler: Checks for poison pill flag and raises RuntimeError before any task execution
  3. DLQ trigger: RuntimeError causes Lambda service failure → message sent to DLQ
  4. Verification: Polls CloudWatch SQS metrics to confirm DLQ processing occurred

Testing: Tested locally on Lambda executor: Confirms complete DLQ flow works correctly


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Aug 1, 2025
@isaiahiruoha isaiahiruoha force-pushed the iiruoha/add-lambda-executor-dlq-test branch from 7def5dc to f2db666 Compare August 5, 2025 16:45
@isaiahiruoha isaiahiruoha force-pushed the iiruoha/add-lambda-executor-dlq-test branch from 8ca4266 to 1e93b4f Compare August 5, 2025 23:17
@vincbeck vincbeck merged commit 60ad539 into apache:main Aug 6, 2025
103 checks passed
@vincbeck vincbeck deleted the iiruoha/add-lambda-executor-dlq-test branch August 6, 2025 13:41
ferruzzi pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 7, 2025
apache#54042)

* Create lambda executor dlq system test file

* Pre-commit changes for CI tests

* Add DLQ to spellcheck

* Adjusted spellling wordlist

* Ignore system test related generated doc files

* Fix static checks

* Fix static checks

* Fix doc building

---------

Co-authored-by: vincbeck <vincbeck@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants