Ensure that Connection extra can get masked without causing an error #54769

ashb · 2025-08-21T10:08:35Z

This was caused by pydantic#9541 and improper testing on my part. Sorry
folks. Thsi happens because Iterable is too open-ended a type

Yes, this should absolutely have unit tests to go with the fix, but a fix is
better than nothing, and I'm about to leave on a camping holiday.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Fixes #54768 This was caused by pydantic#9541 and improper testing on my part. Sorry folks. Thsi happens because `Iterable` is too open-ended a type Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

VladaZakharova · 2025-08-21T10:12:55Z

Hi
thank you for the fix
Did you check this on different types? can you please add unit tests for that?

potiuk

Damn. Pydantic runtime type usage has side effects - that change looks lile it changes nothing....

potiuk · 2025-08-21T10:14:20Z

Hi thank you for the fix Did you check this on different types? can you please add unit tests for that?

@ashb wrote:

Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

I guess we should add unit tests as follow-up without breaking @ashb's plans.

ashb · 2025-08-21T10:14:53Z

Anyone hitting this on 3.0.5 andwho doesn't want to rollback to 3.0.4: a work around for now is to manually apply this patch to your Airflow.

If you are using docker to deploy your airflow then something like this should help hopefully:

USER root

RUN apt update && apt install -y patch patchutils

RUN set -ex; \
    cd /usr/local/lib/python3.12/site-packages/airflow; \
    curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54769.patch \
    | filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose

USER airflow

The path to site-packages and the user to switch back to at the end would need adapting to your specific Dockerfile

ashb · 2025-08-21T10:16:11Z

Did you check this on different types? can you please add unit tests for that?

I manually tested it with this:

First set the conneciton:

export AIRFLOW_CONN_TEST='{"conn_type": "google_cloud_default", "extra": {"key_path": "/files/airflow-breeze-config/keys2/keys.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "project_id", "num_retries": 6}}

Than I ran this DAG:

@task
def my_function() -> None:
    conn = Connection.get("test")

    print(f"{conn.conn_id=} {conn.password=} {conn.extra_dejson=} {conn=}")

with DAG("test_dag") as dag:
    my_function()

potiuk · 2025-08-21T10:17:55Z

I manually tested it with this:

I will take it from now on @ashb -> thanks for the repro scenarios

amoghrajesh

Damn!

amoghrajesh · 2025-08-21T10:23:02Z

@ashb mypy will need a fix:

  task-sdk/src/airflow/sdk/execution_time/secrets_masker.py:127: error: Argument "value" to "MaskSecret" has incompatible type "str | Iterable[Any]"; expected "JsonValue"  [arg-type]
              comms.send(MaskSecret(value=secret, name=name))
                                          ^~~~~~
  task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
  "str | dict[Any, Any] | Iterable[Any]"  [arg-type]
                  mask_secret(msg.value, msg.name)
                              ^~~~~~~~~
  Found 2 errors in 2 files (checked 126 source files)

potiuk · 2025-08-21T10:24:44Z

^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

I'll copy the PR and fix it -> and I will ask others to help :) .. @ashb will not be available to fix it @amoghrajesh

potiuk · 2025-08-21T10:25:12Z

Ah... It's in `apache/airflow" so we can push to it directly

VladaZakharova · 2025-08-21T10:26:16Z

Originally, there was also the problem with triggerers with the following errors:

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

do you think we can also fix the types for masks to solve this problem too?

gopidesupavan · 2025-08-21T10:26:46Z

oh geeee...

amoghrajesh · 2025-08-21T10:27:17Z

@potiuk you can push directly to the PR. No need to copy it.

potiuk · 2025-08-21T10:28:33Z

@potiuk you can push directly to the PR. No need to copy it.

Yep. Will do

potiuk · 2025-08-21T10:29:14Z

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

potiuk · 2025-08-21T10:29:46Z

Let's just push fixups without -f : to not override each-others's changes.

amoghrajesh · 2025-08-21T10:41:39Z

Dropped you a message on slack, looking into mypy failures

… error

potiuk · 2025-08-21T10:43:59Z

pushed mypy fix :)

VladaZakharova · 2025-08-21T10:46:09Z

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

potiuk · 2025-08-21T10:48:33Z

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

VladaZakharova · 2025-08-21T10:49:08Z

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

agree :)

amoghrajesh · 2025-08-21T10:52:03Z

Agreed too. Let's fix the error at hand first and we can always follow up for the non breaking / burning things...

…sing an error

potiuk · 2025-08-21T11:24:06Z

fixup with tests is here

potiuk · 2025-08-21T11:28:12Z

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

amoghrajesh · 2025-08-21T11:28:53Z

Yeah I do not think it is directly linked to this PR

kaxil · 2025-08-21T11:34:26Z

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

Yeah unrelated

…out causing an error

amoghrajesh

Looks good to me!

task-sdk/tests/task_sdk/definitions/test_secrets_masker.py

kaxil · 2025-08-21T12:19:59Z

Compat test change: #54776

potiuk · 2025-08-21T12:40:06Z

Merging it 3.0.5 fails clearly because of yanking :) and #54766 is already on the way.

…ng an error (#54769) * Ensure that Connection extra can get masked without causing an error Fixes #54768 This was caused by pydantic#9541 and improper testing on my part. Sorry folks. Thsi happens because `Iterable` is too open-ended a type Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday. * fixup! Ensure that Connection extra can get masked without causing an error * fixup! fixup! Ensure that Connection extra can get masked without causing an error * fixup! fixup! fixup! Ensure that Connection extra can get masked without causing an error --------- (cherry picked from commit 5aec867) Co-authored-by: Ash Berlin-Taylor <ash@apache.org> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

github-actions · 2025-08-21T12:41:04Z

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test

…ng an error (#54769) (#54780) * Ensure that Connection extra can get masked without causing an error Fixes #54768 This was caused by pydantic#9541 and improper testing on my part. Sorry folks. Thsi happens because `Iterable` is too open-ended a type Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday. * fixup! Ensure that Connection extra can get masked without causing an error * fixup! fixup! Ensure that Connection extra can get masked without causing an error * fixup! fixup! fixup! Ensure that Connection extra can get masked without causing an error --------- (cherry picked from commit 5aec867) Co-authored-by: Ash Berlin-Taylor <ash@apache.org> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…pache#54769) * Ensure that Connection extra can get masked without causing an error Fixes apache#54768 This was caused by pydantic#9541 and improper testing on my part. Sorry folks. Thsi happens because `Iterable` is too open-ended a type Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday. * fixup! Ensure that Connection extra can get masked without causing an error * fixup! fixup! Ensure that Connection extra can get masked without causing an error * fixup! fixup! fixup! Ensure that Connection extra can get masked without causing an error --------- Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

ashb requested review from amoghrajesh and kaxil as code owners August 21, 2025 10:08

boring-cyborg bot added the area:task-sdk label Aug 21, 2025

eladkal approved these changes Aug 21, 2025

View reviewed changes

potiuk approved these changes Aug 21, 2025

View reviewed changes

ashb mentioned this pull request Aug 21, 2025

TaskSDK: Make secrets masking work when conns are loaded from secrets backends #54574

Merged

Lee-W approved these changes Aug 21, 2025

View reviewed changes

amoghrajesh approved these changes Aug 21, 2025

View reviewed changes

gopidesupavan approved these changes Aug 21, 2025

View reviewed changes

fixup! Ensure that Connection extra can get masked without causing an…

67f7b98

… error

fixup! fixup! Ensure that Connection extra can get masked without cau…

df480d3

…sing an error

kaxil approved these changes Aug 21, 2025

View reviewed changes

fixup! fixup! fixup! Ensure that Connection extra can get masked with…

8d6c570

…out causing an error

amoghrajesh approved these changes Aug 21, 2025

View reviewed changes

potiuk added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Aug 21, 2025

potiuk mentioned this pull request Aug 21, 2025

Use of conn.extra_dejson in Airflow 3.0.5 fails with NotImplementedError error #54768

Closed

dshvedchenko reviewed Aug 21, 2025

View reviewed changes

task-sdk/tests/task_sdk/definitions/test_secrets_masker.py Show resolved Hide resolved

potiuk merged commit 5aec867 into main Aug 21, 2025
74 of 75 checks passed

potiuk deleted the fix-extra-connection-masking branch August 21, 2025 12:40

kaxil mentioned this pull request Aug 21, 2025

Improve test maintainability for supervisor request handling tests #54792

Merged

amoghrajesh mentioned this pull request Aug 21, 2025

Relocate MaskSecret comms tests into task SDK comms tests #54795

Merged

Ensure that Connection extra can get masked without causing an error #54769

Ensure that Connection extra can get masked without causing an error #54769

Uh oh!

Conversation

ashb commented Aug 21, 2025

Uh oh!

VladaZakharova commented Aug 21, 2025

Uh oh!

potiuk left a comment

Choose a reason for hiding this comment

Uh oh!

potiuk commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Aug 21, 2025

Uh oh!

ashb commented Aug 21, 2025

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

amoghrajesh left a comment

Choose a reason for hiding this comment

Uh oh!

amoghrajesh commented Aug 21, 2025

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

potiuk commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VladaZakharova commented Aug 21, 2025

Uh oh!

gopidesupavan commented Aug 21, 2025

Uh oh!

amoghrajesh commented Aug 21, 2025

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

amoghrajesh commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

VladaZakharova commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

VladaZakharova commented Aug 21, 2025

Uh oh!

amoghrajesh commented Aug 21, 2025

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

amoghrajesh commented Aug 21, 2025

Uh oh!

kaxil commented Aug 21, 2025

Uh oh!

amoghrajesh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kaxil commented Aug 21, 2025

Uh oh!

potiuk commented Aug 21, 2025

Uh oh!

Uh oh!

github-actions bot commented Aug 21, 2025

Backport successfully created: v3-0-test

Uh oh!

Reviewers

Assignees

Labels

Projects

potiuk commented Aug 21, 2025 •

edited

Loading

potiuk commented Aug 21, 2025 •

edited

Loading

amoghrajesh commented Aug 21, 2025 •

edited

Loading

VladaZakharova commented Aug 21, 2025 •

edited

Loading