Move the traces and metrics code under a common observability package

[xBis7]

This patch is refactoring the metrics and the traces packages and moves them under a common package named observability.

This is done so that it will be easier to add common files in the future.

The need for these changes came from this discussion

#56150 (comment)

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[jason810496]

Nice! Thanks for the cleanup!

LGTM for the Airflow-Core and Providers change, but I'm not sure about TaskSDK part.
I will leave it to someone more familiar to answer.

[ferruzzi]

Approved pending Ash weighing in on the SDK part.

[xBis7]

@jason810496 @ferruzzi Thank you for the review!

[xBis7]

@jason810496 @ferruzzi @ashb It's ready for review. The CI is green.

The step for running the task sdk integration tests seems to be flaky.

https://github.com/apache/airflow/actions/runs/18505099702/job/52734949908?pr=56187

I ran it locally and it worked, so I triggered the CI again and it passed.

To help you understand the changes faster

• The new observability package has been moved under shared
• The unit tests have also been moved there
• The otel integration tests haven't been moved because I don't think there is a mechanism to run them on the CI if the change is made
• airflow-core and task-sdk both have their own version of shared directory. The files aren't copies but symlinks to the original ones
• airflow-core imports from its shared symlinks
• task-sdk imports from its own shared symlinks
• providers import from the task-sdk/shared symlinks
• in the task-sdk and the providers, the imports are wrapped with try-except for backwards compat

[jason810496]

Thanks for the update! The overall change LGMT, but there are still question about the dependencies for Airflow-Core and TaskSDK.

[jscheffl]

Wow - this is also a big one!

Also having two sanity questions but no general objections after good review by others. Would be important to clarify minor open points and then LGTM!

#protm

[amoghrajesh]

This PR is an absolute delight. Thanks for collaborating with myself and @potiuk on this one and being our first real developer who wasn't involved in designing the shared module approach for client server separation. This helped us identify gaps in the process and fix them as needed.

The changes look good to me minus the comments I sent you offline regarding the task sdk docs, preemptively approving now assuming those changes will be made prior to merge and the last few comments I have will be handled.

This PR will go down as a PR of the month candidate for me.
#protm

[amoghrajesh]

@xBis7 the failure on your PR:

================================================================================================= ERRORS =================================================================================================
__________________________________________________________ ERROR collecting airflow-core/tests/unit/api_fastapi/core_api/routes/test_routes.py ___________________________________________________________
airflow-core/tests/unit/api_fastapi/core_api/routes/test_routes.py:19: in <module>
    from airflow.api_fastapi.core_api.routes.public import authenticated_router, public_router
airflow-core/src/airflow/api_fastapi/core_api/routes/public/__init__.py:24: in <module>
    from airflow.api_fastapi.core_api.routes.public.assets import assets_router
airflow-core/src/airflow/api_fastapi/core_api/routes/public/assets.py:127: in <module>
    def get_assets(
airflow-core/src/airflow/api_fastapi/common/router.py:37: in decorator
    self.add_api_route(
/usr/python/lib/python3.10/site-packages/fastapi/routing.py:1011: in add_api_route
    route = route_class(
/usr/python/lib/python3.10/site-packages/fastapi/routing.py:636: in __init__
    get_parameterless_sub_dependant(depends=depends, path=self.path_format),
/usr/python/lib/python3.10/site-packages/fastapi/dependencies/utils.py:132: in get_parameterless_sub_dependant
    return get_dependant(
/usr/python/lib/python3.10/site-packages/fastapi/dependencies/utils.py:290: in get_dependant
    sub_dependant = get_dependant(
/usr/python/lib/python3.10/site-packages/fastapi/dependencies/utils.py:259: in get_dependant
    endpoint_signature = get_typed_signature(call)
/usr/python/lib/python3.10/site-packages/fastapi/dependencies/utils.py:196: in get_typed_signature
    signature = inspect.signature(call, eval_str=True)
/usr/python/lib/python3.10/inspect.py:3254: in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped,
/usr/python/lib/python3.10/inspect.py:3002: in from_callable
    return _signature_from_callable(obj, sigcls=cls,
/usr/python/lib/python3.10/inspect.py:2463: in _signature_from_callable
    return _signature_from_function(sigcls, obj,
/usr/python/lib/python3.10/inspect.py:2307: in _signature_from_function
    annotations = get_annotations(func, globals=globals, locals=locals, eval_str=eval_str)
/usr/python/lib/python3.10/inspect.py:175: in get_annotations
    return_value = {key:
/usr/python/lib/python3.10/inspect.py:176: in <dictcomp>
    value if not isinstance(value, str) else eval(value, globals, locals)
<string>:1: in <module>
    ???
E   NameError: name 'HTTPAuthorizationCredentials' is not defined

is unrelated and is due to a fastAPI bump it seems.

[xBis7]

is unrelated and is due to a fastAPI bump it seems.

@amoghrajesh I see, it's on main as well.

[amoghrajesh]

@xBis7 yes, the fix is here: #59048

[amoghrajesh]

The fix is merged, rebasing it.

[xBis7]

The fix is merged, rebasing it.

@amoghrajesh I was about to do it. Thanks.

[xBis7]

@amoghrajesh @potiuk Both failures seem random. I ran the redis step locally and it worked.

[amoghrajesh]

Just rebased again, hopefully its green and ready to merge!

[xBis7]

@amoghrajesh @potiuk Thank you for all the help! @jason810496 @ferruzzi @jscheffl Thank you for the reviews!

[potiuk]

Wooohooo

[jason810496]

#protm

[ferruzzi]

Well done, and thanks for sticking with it. That was a long trek, but it turned out great.

[xBis7]

@ferruzzi Thank you!