Automatic JSON deserialization of variables not working with AWS Secrets Manager secrets backend

[tom-gordon]

Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

In Airflow 3, the deserialize_json=True parameter in Variable.get() is no longer automatically deserializing JSON strings when using AWS Secrets Manager as the secrets backend. The method returns a string type. Deserialization still works with variables created in Airflow and not stored in AWS Secrets Manager.

The connection I have stored as JSON in AWS Secrets Manager also no longer works in Airflow 3

What you think should happen instead?

No response

How to reproduce

1. Configure your Airflow deployment to use AWS Secrets Manager as a secrets backend:
   https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html
   https://www.astronomer.io/docs/astro/secrets-backend/aws-secretsmanager/

2. Store an Airflow variable as JSON in AWS Secrets Manager e.g. {"num1": 23, "num2": 42}

3. Retrieve the variable in a DAG using Variable.get("my_json_variable", deserialize_json=True)

4. the get() method will return the entire JSON as a string type

Operating System

Linux

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.7.0

Deployment

Astronomer

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[amoghrajesh]

Thanks for raising, i will self assign this and try to look into it soon.

[amoghrajesh]

I am going to take a stab at this in the coming week.

[amoghrajesh]

Did some investigation on this one. This issue is not applicable only to AWS secrets manager but is a general issue with any secrets backend, because of a bug in Variable.get().

Set up a airflow.secrets.local_filesystem.LocalFilesystemBackend and checked with both Airflow 2 and Airflow 3.

var.json

{
    "A": "{\"num1\": 23, \"num2\": 42}"
}

Dag used:

from __future__ import annotations

from airflow.models import Variable
from airflow.models.baseoperator import BaseOperator
from airflow import DAG
# from airflow.sdk import Variable

class CustomOperator(BaseOperator):
    def execute(self, context):
        v = Variable.get(key="A")
        print("Got value", v, type(v))


        v1 = Variable.get(key="A", deserialize_json=True)
        print("Got value from deser", v1, type(v1))


with DAG("get-var-sdk", schedule=None, catchup=False) as dag:
    CustomOperator(task_id="set_var")

Airflow 2:

# With secrets backned
# [2025-05-20, 16:38:47 IST] {logging_mixin.py:190} INFO - Got value {"num1": 23, "num2": 42} <class 'str'>
# [2025-05-20, 16:38:47 IST] {logging_mixin.py:190} INFO - Got value from deser {'num1': 23, 'num2': 42} <class 'dict'>

Airflow 3:

[2025-05-21, 15:35:41] INFO - Got value {"num1": 23, "num2": 42} <class 'str'>: chan="stdout": source="task"
[2025-05-21, 15:35:41] INFO - Got value from deser {"num1": 23, "num2": 42} <class 'str'>: chan="stdout": source="task"

As observed the second print for Airflow 3 is a string when it should really be a dictionary.