Skip to content

Add handle for circular exceptions in SecretsMasker.#61254

Closed
Crowiant wants to merge 1 commit intoapache:v2-11-testfrom
VladaZakharova:add-handle-of-circular-exceptions
Closed

Add handle for circular exceptions in SecretsMasker.#61254
Crowiant wants to merge 1 commit intoapache:v2-11-testfrom
VladaZakharova:add-handle-of-circular-exceptions

Conversation

@Crowiant
Copy link
Contributor

In the case of circular exceptions, Airflow 2 runs into a retry cycle without showing anything inside the task logs.

Here is the example that I used to reproduce the issue:

try:
    try:
        raise ValueError("Original crash simulation")
    except ValueError as err:
        # Create an intentional circular reference in the context
        err.__context__ = err
        raise RuntimeError("Wrapper error") from err
except Exception:
    # logger.exception triggers _redact_exception_with_context (no depth limit)
    logger.exception("This will trigger RecursionError in the SecretsMasker")

These changes will prevent this and show all exceptions to the user.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Copy link
Contributor

@molcay molcay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@potiuk potiuk force-pushed the add-handle-of-circular-exceptions branch from 385d91b to 3520fbb Compare February 11, 2026 10:40
potiuk added a commit to potiuk/airflow that referenced this pull request Feb 11, 2026
Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Initially implemented as a fix to v2-11-test in apache#61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.
potiuk added a commit to potiuk/airflow that referenced this pull request Feb 11, 2026
Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Initially implemented as a fix to v2-11-test in apache#61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>
potiuk added a commit to potiuk/airflow that referenced this pull request Feb 11, 2026
Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in apache#61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>
@potiuk
Copy link
Member

potiuk commented Feb 11, 2026

I am closing it in favour of #61776. Since it is also present in main, I recreated it there, and will backport it to both v3-1-test and v2-11-test, I also added test coverage, but also updated that change (result of testing) - rather than stopping redacting when recursion level is reached, we will SKIP those exceptions and replace the remaining stack trace with sentinel exception explaining that this happened.

This is much safer, because potentially we could leak some secrets in those unredacted exceptions remaining on stack trace.

I made you co-author of #61776 @Crowiant

@potiuk potiuk closed this Feb 11, 2026
potiuk added a commit to potiuk/airflow that referenced this pull request Feb 11, 2026
Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in apache#61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>
potiuk added a commit that referenced this pull request Feb 12, 2026
* Fix recursion depth error in _redact_exception_with_context

Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in #61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Anton Nitochkin <nitochkin@google.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
github-actions bot pushed a commit that referenced this pull request Feb 12, 2026
…xt (#61776)

* Fix recursion depth error in _redact_exception_with_context

Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in #61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------
(cherry picked from commit 1533ecf)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Anton Nitochkin <nitochkin@google.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
potiuk added a commit to potiuk/airflow that referenced this pull request Feb 12, 2026
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in apache#61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>
potiuk added a commit to potiuk/airflow that referenced this pull request Feb 12, 2026
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in apache#61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>

(cherry picked from commit (1533ecf)
@potiuk
Copy link
Member

potiuk commented Feb 12, 2026

Cherry-picked to 2-11-test in #61797

github-actions bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Feb 12, 2026
…xt (apache#61776)

* Fix recursion depth error in _redact_exception_with_context

Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in apache#61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------
(cherry picked from commit 1533ecf)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Anton Nitochkin <nitochkin@google.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ytoprakc pushed a commit to ytoprakc/airflow that referenced this pull request Feb 12, 2026
…1776)

* Fix recursion depth error in _redact_exception_with_context

Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in apache#61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Anton Nitochkin <nitochkin@google.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
potiuk added a commit that referenced this pull request Feb 12, 2026
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in #61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>

(cherry picked from commit (1533ecf)
Ratasa143 pushed a commit to Ratasa143/airflow that referenced this pull request Feb 15, 2026
…1776)

* Fix recursion depth error in _redact_exception_with_context

Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in apache#61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Anton Nitochkin <nitochkin@google.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
potiuk added a commit that referenced this pull request Feb 15, 2026
…xt (#61776)

* Fix recursion depth error in _redact_exception_with_context

Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in #61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------
(cherry picked from commit 1533ecf)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Anton Nitochkin <nitochkin@google.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
potiuk added a commit that referenced this pull request Feb 15, 2026
…xt (#61776) (#61795)

* Fix recursion depth error in _redact_exception_with_context

Fix recursion depth check in _redact_exception_with_context_or_cause.
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in #61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.



* Apply suggestions from code review



---------
(cherry picked from commit 1533ecf)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Anton Nitochkin <nitochkin@google.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
potiuk added a commit that referenced this pull request Feb 16, 2026
There are some obscure cases where exception might point to itself
in cause/context - this PR protects against it.

Changed name to include cause as well.

Initially implemented as a fix to v2-11-test in #61254 - enhanced
with the case of removal of too-deep exceptions rather than
not-redacting it (and replacing it with sentinel exception explaining
that reminder of the stack trace has been removed.

Co-authored-by: Anton Nitochkin <nitochkin@google.com>

(cherry picked from commit (1533ecf)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments