Serialization returns unserialized numpy value

[astro-anand]

Apache Airflow version

3.0.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

numpy float64 subclasses float, so the serialize function returns the object without passing it to the custom numpy serializer. This means that when the numpy value is serialized to the xcom backend, it will get coerced to float and read back in as a float in subsequent tasks.

What you think should happen instead?

Run the built-in serializer code first, then check if the object is an instance of the primitive types. This ensure that a custom serializer would be called before returning the object

How to reproduce

import random
from airflow.sdk import dag, task, Asset, AssetAlias

ALIAS = AssetAlias("s3://alias-example")

TEST_ASSET = Asset("s3://my-bucket/data/file.csv")


@dag(tags=["asset_alias_demo"])
def asset_alias_demo():

    @task
    def get_file_ids():
        import numpy as np
        import pandas as pd

        return [np.float64(142.13412), np.float64(123.12351242), pd.DataFrame({"a": [1, 2, 3]})]

    @task(outlets=[TEST_ASSET])
    def add_assets_to_alias(i, outlet_events):
        """
        adds assets to alias
        """

        import numpy as np

        random_value = random.randint(1, 100)
        print(random_value, "is the random value for file_id:", i)
        print(type(i), "is the type of i")
        print("it is", isinstance(i, np.float64), "that i is a numpy float64")
        outlet_events[TEST_ASSET].extra = {
            "file_id": i,
            "random_value": random_value,
        }
        print("the outlet events are", outlet_events)

running this DAG will show you that the deserialized value of i is not a numpy float64

Operating System

MacOS

Versions of Apache Airflow Providers

N/A

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

@sjyangkevin @bolkedebruin -> that is very realated to #52753 and it would be nice to check if #53690 also addresses this float issue

[sjyangkevin]

@sjyangkevin @bolkedebruin -> that is very realated to #52753 and it would be nice to check if #53690 also addresses this float issue

Thanks, will look into this.

[astro-anand]

@sjyangkevin and @potiuk unfortunately, logic in the numpy serializer will not fix the issue described above. The source of the issue is the order of ops in the serialization function in serde.py. Since float64 is a subclass of float, the object is returned without ever being passed thru the numpy serializer. Both fixes will probably be required

[sjyangkevin]

I think it might be something we can have a deeper look about how the ordering. We had a previous discussion #51059 (comment) about it a little bit (see change summary the second bullet point). Probably we can also look into if there are differences on data types across major numpy versions.

[astro-anand]

@sjyangkevin I agree. In general, I do think the custom serialization check should happen before the checking if the object is a primitive. With the current implementation, if an object is a subclass of a primitive, it'll be returned unmodified.